This tool helps catch common typos in code and documentation, which is
particularly useful for our many contributors who are not native
English speakers.
The config is based on the codespell that I ran in
https://github.com/zulip/zulip/pull/18535.
CircleCI expected 3434; GitHub Actions expects 1001. This is the
reason zulip-ci.yml currently needs to mess with the permissions in
/__w.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This will be useful to let users enable/disable
sharing read receipts once we add that feature.
Note: Added "I've" to IGNORED_PHRASES in
tools/lib/capitalization.py to avoid capitalization
errors for the label text of this setting.
We make zero invalid value for message_content_delete_limit_seconds and
for handling the case of "Allow to delete message any time", the API-level
value of message_content_delete_limit_seconds is "anytime" and "None"
as the DB-level value. We also use these values for message retention
setting, so it helps maintain consistency.
On a 2 GiB, 1 CPU system, webpack would hit the Node.js heap
limit (which is half of physical memory up to 4 GiB, on 64-bit
systems).
Signed-off-by: Anders Kaseorg <anders@zulip.com>
gitlint has a bunch of pinned requirements that hold back important
upgrades and conflict with other packages’ requirements. The gitlint
author has rejected proposals to unpin them because it might increase
the amount of maintenance he needs to do
(https://github.com/jorisroovers/gitlint/pull/133). That decision is
his to make, but _somebody_ needs to do the maintenance, so we
delegate it to Debian and Ubuntu. If that means using a significantly
older version of gitlint, that’s a tradeoff we need to make to keep
the rest of our requirements current.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
The semgrep --dangerously-allow-arbitrary-code-execution-from-rules
flag is deprecated and no longer used.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This commit adds a new module settings_defaults.js which calls
the functions in settings_display passing appropriate container
element and settings object as parameters.
We also add one more parameter for_realm_settings to some of the
functions in settings_dislay to differentiate between the user
and realm-level settings.
When the upstream provides a chunked response, proxying this header
causes a protocol-level miscommunication.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This environment variable is not a thing and has never been a thing,
while the path it purportedly pointed to does not exist and has never
existed. It appears to have been inexplicably both cargo-culted and
renamed from test-js-with-casper.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
We will use this modal for any narrow / hash or other UI element that
requires an actual account to use, to provide something reasonable to
occur when a user clicks on those things.
This adds the X-Smokescreen-Role header to proxy connections, to track
usage from various codepaths, and enforces a timeout. Timeouts were
kept consistent with their previous values, or set to 5s if they had
none previously.
This was originally meant to fix the emoji mapping conflict during a
Slack import. In Slack, 🎉 and ㊗️ have different
symbols, but they both map to 🎉 in Zulip prior to this commit.
㊗️ now refers to the Japanese character version, as is
observed in Matrix and Slack.
I expand the fix to include all other Japanese characters. Matrix.org
and Slack already have those characters in their symbol section, and so
this is to reach feature parity.
See the discussion thread in https://chat.zulip.org/#narrow/stream/9-issues/topic/duplicate.20emoji.20in.20data.20import
These changes are all independent of each other; I just didn’t feel
like making dozens of commits for them.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
None results in an uninitialized image (that happens to be transparent
most of the time); we want to explicitly initialize the image to
transparent.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
We rework the landing page for companies in the same way we've
recently revamped the landing pages for other use cases.
This implementation unfortunately duplicates a lot of content from
/plans; we should clean that up at some point.
We recently added a lot of new pages to our top navigation and
restructured top-navigation in general. This commit updates the
footer to reflect the recent changes to our top navigation.
Partially reverts commit cc55393671.
mode="r" cannot be combined with buffering=0.
This is not a correct thing to do, since a multibyte character might
get split between two binary reads and cause UnicodeDecodeError, but
it’s good enough for a known-broken test we don’t run.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This removes some steps which are no longer necessary to be run
in the production upgrade script. The steps were used due to
errors related to supervisor failing to restart which was resolved
in the commit 08c39a7388.
This removes a bunch of non-functional duplicate JavaScript, HTML, and
CSS that was interfering with maintenance on the functional originals,
because it was never clear how to update the duplicates or how to
check that you’d updated the duplicates correctly.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This API change removes unnecessary complexity from a client that
wants to change a user's personal settings, and also saves developers
from needing to make decisions about what sort of setting something is
at the API level.
We preserve the old settings endpoints as mapping to the same function
as the new one for backwards-compatibility. We delete the
documentation for the old endpoints, though the documentation for the
merged /settings endpoint mentions how to use the old endpoints when
needed.
We migrate all backend tests to the new endpoints, except for
individual tests for each legacy endpoint to verify they still work.
Co-authored-by: sahil839 <sahilbatra839@gmail.com>
We move templates related to listing streams in left pannel of
stream settings overlay to stream_settings folder by making
following changes:
* Rename `subscriptions.hbs` to `browse_streams_list.hbs`.
* Move `settings_stream_list.hbs` to stream_settings folder.
* Rename `subscription.hbs` to `browse_streams_list_item.hbs`.
* Move `settings_stream_list_item.hbs` to stream_settings folder.
This makes several changes:
* Fixes a bug where the help text explaining our policies was not displayed.
* No help text was defined for many organization types.
* Copy-edits the help text somewhat.
* Offers all of the organization type options.
* Removes the 100% coverage requirement because it's annoying to test
the e.currentTarget click handler.
This commit changes the bot-edit modal to use dialog_widget instead of
edit_fields_modal.
This commit also removes edit_fields_modal module as it is no longer used.
This commit adds a new dialog_widget.js file containing most
of the code of confirm_dialog.js with some minor changes and
changes confirm_dialog to be a wrapper around dialog_widget.js.
We pass 'is_confim_dialog' as true in dialog_widget for a
confirm_dialog modal. This commit also renames confirm_dialog.hbs
and confirm_dialog_heading.hbs to dialog_widget.js,
dialog_widget.hbs and dialog_widget_heading.hbs respectively.
We currently configure ‘APT::Get::Assume-Yes’ in our custom Docker
image, but this is the only place we rely on it (outside of the
Dockerfile itself), and it’s better not to.
Also ‘apt-get remove && apt-get purge’ is the same as just ‘apt-get
purge’.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This commit adds a check to avoid the use of assertTrue
for cases like: assertTrue(len(data) == 2).
We should use assert_length, assertGreater, or
assertGreaterEqual, whatever suits, in cases like these.
We use subs as a common variable name for a collection of stream
data structure used in settings, in lot of modules. So this
rename clears a bunch of related shadowed variables.
This commit migrates the `navbar.html` Django template
to handlebars by creating a new file as `navbar.hbs`
within `/static/templates` which is then rendered
using `ui_init` module.
As a part of migration, we also remove the `search_pills_enabled`
and `embedded` parameters from the context attribute as they
are no longer needed now.
Fixes part of #18792.
Now that we are starting to link this pages from the landing page's
top navigation, it makes sense to have proper backlinks to the
homepage so that there is some continuity when the user clicks on
a link that takes them to a ReadTheDocs page from the main website.
This commit first moves the compose.validate() function out
with the functions that are needed by it. Then one by one
checked for which function is now not needed in compose.js.
This moves all validation related functions out of "compose.js"
to "compose_validate.js".
Splitting compose announce variables out of compose.js.
This commit moves the "user_acknowledged_all_everyone" and
"user_acknowledged_announce" out of compose.js to reduce
cyclic dependency of compose_validate on compose.js.
Moving wildcard mentions to compose_validate.
The wildcard mention settings are mostly used while validating.
Also to reduce the cyclic dependence of compose in
compose_validate, the related wildcard mentions are moved out to
compose_vaidate.js.
This also converts reset_acknowledged functions to set values
by passing values.
Mypy can’t follow absolute imports based on directories other than the
root. This was hiding some type errors due to ignore_missing_imports.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This is consistent with how we handle JsonableError and friends; it
doesn't make sense for translators to spend time on strings only
visible in a development environment.
Even though this looks like an independently runnable script, it
should not be run independently: a SHA-256 mismatch will fail to stop
the script, unless it was sourced from another script that has ‘set
-e’.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This commits ports the `keyboard_shortcuts.html` file from
using the Django template to handlebars, essentially creating
a new file as `keyboard_shortcuts.hbs` within /static/templates
which is then rendered using info_overlays.js.
Fixes part of #18792.
We create a new widget edit_fields_modal such that this common
framework can be used in bot-edit modal, linkifier-edit modal
and user-edit modal, which have very similar implementations.
The "edit-fields-modal-status" is used only for edit-linkifier
modal and remains empty for others, so this change does not
cause problems with other modals.
We had a lot of functions and click handlers that were only
involved with user profile modal and were not related to
popovers logic in any way. So we extract these functions
into a separate module `user_profile.js`.
We turn off the eslint no-use-before-define for TypeScript files
because it does not work correctly for types. There is already
a typescript-eslint version of it that is enabled for TS.
We also update the error handler on window to use instanceof check
for ErrorEvent instead of checking the error property.
The plan for type annotating the page_params is to set it to
Record<string, unknown> for now and then annotate individual
properties on it as we use it in typescript modules.
We add a exclude pattern that makes sure we don't catch two edge
cases: a variable declaration `const style =` and setting a
variable ending in style such as `require_cmd_style =`. We don't add
and exclude pattern for let declaration because it will catch lines
that modify it later in the code.
(Removed other files from exclude list that no longer needed to be
excluded from this lint rule.)
Moved `subscription_invites_warning` modal to `confirm_dialog`
folder and renamed the modal to `confirm_subscription_invites_warning.hbs`
to follow the naming convention.
Generally, we never want to recommend sudo for an operation that can
be done as a non-root user, and it's normal to configure Docker to be
usable by normal users.
We currently have created a copy of the
`clean_unused_caches.main` function in
`provision_inner.py` to clean the unused caches. But as
we have now converted the script into a python file we
can directly call that function.
This commit replaces that function (introduced in adc0ed4206) with
`clean_unused_caches.main`.
We split recent_topics module into recent_topics_(ui + data + util).
This allows us to reduce cyclical dependencies which were
created due to large list of imports in recent topics. Also, this
refactor on its own makes sense.
It's sufficiently tiny that the shared code benefits don't justify the
cost, given that we plan to move index.html to a different templating
system soon.
The current linter disallowed the pattern
where multiple refs were present without
an additional parameter in allOf, which should
be valid.
Fixed the condition to allow the change.
It appears that some server-side change to Transifex resulted in the
"onlytranslated" mode deleting some (all?) strings from django.po files that
were not translated.
Testing determined that the "translator" mode appears to now be the
only mode that works with both our django.po and translations.json
files (We want to avoid both copying the English strings and deleting
strings), so we're switching to that.
Background is available here:
https://chat.zulip.org/#narrow/stream/3-backend/topic/4.2Ex.20branch.20translations.20sync/near/1187324
Using puppet modules from the puppet forge judiciously will allow us
to simplify the configuration somewhat; this specifically pulls in the
stdlib module, which we were already using parts of.
We now organize the pygment language codes into meaningful categories
- default, custom and aliases.
Further the `lang.json` list now contains a dataset extracted from the
"language" section of https://insights.stackoverflow.com/survey/2020
and is prioritized based on current language trends.
We failed to update this fork for the Django 3.2 upgrade. Unfork it
so that’s not something we need to remember to do.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This ensures that we exercise the fact that the Zulip installer may be
unpacked to a directory that may not be world-readable.
bc45525369 fixed a recent regression in
this behavior that would have been caught by this commit.
We record Git details about the merge-base with upstream branches in
the zulip-git-version file, if the upstream repository is available.
Note that the first Git upgrade after merging the parent commit will
not include the merge-base details, since the upstream repository will
not have been available.
Co-authored-by: Tim Abbott <tabbott@zulip.com>
Signed-off-by: Anders Kaseorg <anders@zulip.com>
New bot avatars are generated with this tool. Having the avatars generated,
we can run generate-integration-docs-screenshot to generate the doc
screenshots.
Fixes: #17792
Non-webhook integrations should have fixtures containing mock messages
in json format with fields "subject" and "body" indicating the topic
and content respectively.
Thumbor and tc-aws have been dragging their feet on Python 3 support
for years, and even the alphas and unofficial forks we’ve been running
don’t seem to be maintained anymore. Depending on these projects is
no longer viable for us.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Currently users that try to deploy Zulip through docker has errors
because LDAP group search configuration can't be automated.
Reverts a hunk of f5197518a9.
The design of the form is similar to the linkifiers page
and is styled similarly.
The introduction text for "Code playgrounds" is improved
with more details and examples.
Also, we can remove the hardcoded playground and the fix
we had previously done to prevent breaking the hardcoded
playground.
The `en_US.UTF-8` locale may not be configured or generated on all
installs; it also requires that the `locales` package be installed.
If users generate the `en_US.UTF-8` locale without adding it to the
permanent set of system locales, the generated `en_US.UTF-8` stops
working when the `locales` package is updated.
Switch to using `C.UTF-8` in all cases, which is guaranteed to be
installed.
Fixes#15819.
This is a straightforward upgrade in terms of changes needed.
Necessary changes were:
- Set `DEFAULT_AUTO_FIELD`
https://docs.djangoproject.com/en/3.2/releases/3.2/#customizing-type-of-auto-created-primary-keys
- `The default_app_config application configuration variable is deprecated, due
to the now automatic AppConfig discovery.`
https://docs.djangoproject.com/en/3.2/releases/3.2/#automatic-appconfig-discovery
To handle this one, we can remove default_app_config from
zerver/__init__.py because it satisfies what release notes describe in
https://docs.djangoproject.com/en/3.2/releases/3.2/#automatic-appconfig-discovery:
"Most pluggable applications define an AppConfig subclass in an apps.py
submodule. Many define a default_app_config variable pointing to this
class in their __init__.py. When the apps.py submodule exists and
defines a single AppConfig subclass, Django now uses that configuration
automatically, so you can remove default_app_config."
An important note is that rebuild-test-database needs to be run after
this upgrade in dev environment - if tests are run with test db that was
built on the previous version, they will fail due to a mysterious bug
(?), where changing attributes of a user and .save()ing after logging in
in the test via self.login_user, causes getting logged out - the next
requests via self.client_get etc. are unauthed for some reason,
unless self.login_user is called again. This behavior is no longer
exhibited upon rebuilding the test db - and I can't reproduce it in
production or dev db. So this can likely be reasonably dismissed as some
quirk of the test client system that won't be relevant in the future and
doesn't impact production.
Currently only enabled in development, since the exact details don't
seem right..
Co-Author-By: Signior-X <b19188@students.iitmandi.ac.in>
Co-Author-By: Aman Agrawal <amanagr@zulip.com>
Implements UI for #8005.
This is a feature of GNU readlink that isn't in the BSD readlink
found on macOS.
For using this and other GNU coreutils features in our scripts in
general, we could use a solution like mobile's tools/lib/ensure-coreutils.sh
to get GNU coreutils on the PATH -- check if it's there already,
if not then try to find a Homebrew install of it and use that, if not
then print a helpful message.
But even then there'd be a bootstrapping problem of how to find
ensure-coreutils.sh . That involves exactly the same problem as we
have for finding git-tools.sh in these lines. So in fact in mobile
for the task of finding ensure-coreutils.sh in the first place, we
do without `readlink -f` anyway.
The one consequence of this behavior-wise is that if you make a
symlink somewhere that points directly at that script (say in your
`~/bin/`), and try to run it using that symlink, it won't work.
(It'll still work just fine if there are symlinks somewhere higher
up in the paths involved -- just not for the script itself.)
An ideal CLI program really should support that, I think, but
lacking a better idea, this seems an acceptable compromise.
Currently the tools/build-docs was slow
because the clean option was rebulding everything.
But this is only required if one wants the left
sidebar to update.
So now set the default to exclude clean and
add clean option only if --clean is passed.
Also a warning is displayed if clean option is
not passed that the left sidebar won't update.
Fixes#17961.
Change the script to python. This is done
for the following reasons.
* It enables us to use the sanity_check.
* Later when we add warning to include
--clean flag we can use the pre-existing
WARNING from zulip_tools rather than using
terminal color codes.
TODO: Currently this script is slow as the
clean option is expensive so instead use only
html by default and clean only if --clean
option is passed.
Part of #17961.
Fixes#17795
In PR #17014, we added support for deactivate-own-user.
And while doing so, we first deactivated the client and
then reactivated it. But this implementation is a bit
hacky.
So, to fix this, we're now deactivating a test_user so that
we don't have to reactivate it. We did so by changing the value
of authentication_line.
As we want to keep endpoint code out of the
“test_curl_examples”, we changed the value of
authentication_line in `curl_param_value_generators.py`.
To work this out, we create a new global variable named
AUTHENTICATION_LINE in “curl_param_value_generators.py”
and change its value in function “deactivate_own_user” and
to use this change in “test_curl_examples,” we import
AUTHENTICATION_LINE.
AUTHENTICATION_LINE is of list data type because we want a
pointer to original mutable object so that changes made during
run time show across the module. Another way to do this is to change
the way we import variable, but that will be inconsistent to
the way we had in all other files.
To remove confusion between AUTHENTICATION_LINE and
authentication_line we renamed authentication_line
to default_authentication_line.
3f4d0f72fd adjusted the types in
preparation for extending the functionality, but only in later commits
that are not merged yet did it make these updates to the types.
In general, `./scripts/restart-server` will already work in any
circumstance where the server is already stopped and needs to be
started. However, it will output a couple minor warnings, and it is
not readily obvious that it *will* work correctly.
Add an alias for `restart-server` named `start-server`, for
parallelism with `stop-server`, which omits the steps of
`restart-server` which would stop the server first.
Using `supervisorctl stop all` to stop the server is not terribly
discoverable, and may stop services which are not part of Zulip
proper.
Add an explicit tool which only stops the relevant services. It also
more carefully controls the order in which services are stopped to
minimize lost requests, and maximally quiesce the server.
Locations which may be stopping _older_ versions of Zulip (without
this script) are left with using `supervisorctl stop all`.
Fixes#14959.
This reverses the policy that was set, but incompletely enforced, by
commit 951514dd7d. The self-closing tag
syntax is clearer, more consistent, simpler to parse, compatible with
XML, preferred by Prettier, and (most importantly now) required by
FormatJS.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Adds a setting UI to list all configured playgrounds
in a realm. The filter functionality can be used to
search playgrounds by its name or language.
Default sort is provided on the 'pygments_language'
field.
Front tests added to maintain server_event_dispatch
coverage. The `settings_playgrounds.js` file is added
to coverage exclusion list since it is majorly UI
based and will be tested using puppeteer tests (in
following commits).
To prevent breaking of the hardcoded playgrounds, we resort
to checking if realm_playgrounds is empty and falling back
to the hard-coded list if so. This logic is removed in the
followup commit which introduces the UI to add a playground.
I have added support for generating integration screenshots remotely by
adding a `realm_uri` parameter to `tools/message-screenshot.js` which we
then pass `realm.uri` to from within
`tools/generate-integration-docs-screenshot`.
I have made `tools/setup/optimize-svg` do the SVG optimization
automatically rather than just telling you the command to run if they
need optimizing. This included adding a `--check` parameter to use in
CI to only check as we previously did rather than actually running the
optimization.
I have also made `tools/setup/optimize-svg` execute
`tools/setup/generate_integration_bots_avatars.py` once it has run the
optimization to ensure it is always ran.
This makes it one less command to run when creating an integration,
but also means that we catch instances where a PNG has just been
copied into the `static/images/integrations/bot_avatars` folder as the
only instance where this won't be run is if `optimize-svg` has not
been run which would be caught in CI.
Fixes#18183. Fixes#18184.
I have updated `tools/run-dev.py` to output the correct subdomain such as
`http://zulip.username.zulipdev.org` so that the user knows the correct
subdomain to access the Zulip Dev realm on.
django.utils.translation.ugettext is a deprecated alias of
django.utils.translation.gettext as of Django 3.0, and will be removed
in Django 4.0.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This widget only filters the user's subscription -- it's only suggest
public streams that the user is not subscribed to. "Filter" is the
correct label for a widget with this use case.
I have suppressed errors for github.com by adding an function to
exclude domains as well as urls; this is necessary because GitHub has
marked this tool's User-Agent as a blocker crawler.
I have also suppressed reoccurring url errors that do definitely exist.
Fixes#17928.
Ideally, we'd print line numbers etc., but that's complicated because
the `translation.json` format doesn't include them, and usually it's
easy to find strings because you just added them anyway.
But adding a bit more of a `git grep` hint should help.
Fixes#14321.
Our aim is to use this library to remove use of bootstrap-tooltip
for showing popovers and tooltips. This will remove our
dependency on bootstrap for showing tooltips. Thus, bootstrap
can be upgrade more independently.
Because the logic in print_listeners doesn't have access to computed
settings in dev_settings.py, we need to duplicate the special
IS_DEV_DROPLET logic for computing the default hostname.
There's still a secondary problem that this URL 404s.
It does not seem like an official version supporting Webpack 4 (to say
nothing of 5) will be released any time soon, and we can reimplement
it in very little code.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
We use GIPHY web SDK to create popover containing GIFs in a
grid format. Simply clicking on the GIFs will insert the GIF in the compose
box.
We add GIPHY logo to compose box action icons which opens the GIPHY
picker popover containing GIFs with "Powered by GIPHY"
attribution.
We should still display the `source` advice when not in Vagrant or a
Droplet, since that's an important hint for anyone using local
installation on Linux.
We move the "If you are using Vagrant..." text a bit after to
highlight things nicely for folks who are running tools outside
Vagrant.
Also tighten text to avoid line-wrapping on an 80 character console.
The only downside of this is that it makes it harder to control the
order of these tests; which isn't that important. And the structure
of naming each with its test order fundamentally requires renaming
files when adding/deleting tests, so if we want to control the default
test order, we'd be better off doing that by just hardcoding a list in
the test runner code.
In `validate_account_and_subdomain` we check
if user's realm is not deactivated. In case
of failure of this check, we raise our standard
JsonableError. While this works well in most
cases but it creates difficulties in handling
of users with deactivated realms for non-browser
clients.
So we register a new REALM_DEACTIVATED error
code so that clients can distinguish if error
is because of deactivated account. Following
these changes `validate_account_and_subdomain`
raises RealmDeactivatedError if user's realm
is deactivated.
This error is also documented in
`/api/rest-error-handling`.
Testing: I have mostly relied on automated
backend tests to test this.
Fixes#17763.
In validate_account_and_subdomain we check if
user's account is not deactivated. In case of
failure of this check we raise our standard
JsonableError. While this works well in most
cases but it creates difficulties in handling
of deactivated accounts for non-browser clients.
So we register a new USER_DEACTIVATED error
code so that clients can distinguish if error
is because of deactivated account. Following
these changes `validate_account_and_subdomain`
raises UserDeactivatedError if user's account
is deactivated.
This error is also documented in
`/api/rest-error-handling`.
Testing: I have mostly relied on automated
backend tests to test this.
Partially addresses issue #17763.
Vishnu KS
Alex Vandiver
Kevin Scott
Anders Kaseorg
rht
Kartik Soneji
Dinesh
Aman Agrawal
Eeshan Garg
sahil839
Sahil Batra
Priyank Patel
Ganesh Pawar
Tim Abbott
Philipp Seßner
Alya Abbott
Priyansh Garg
Priyam Seth
Gaurav Pandey
manavdesai27
m-e-l-u-h-a-n
akshatdalton
Christalee
aryanshridhar
PIG208
Riken Shah
Suyash Vardhan Mathur
Mateusz Mandera
Gilbert Bishop-White
Steve Howell
Adam Birds
Sumanth V Rao
Abhijeet Prasad Bodas
LoopThrough-i-j
Flávio Prado
pilgrim2308
Greg Price
tushar912
100RABHpy
Wesley Aptekar-Cassels
shanukun
YashRE42