Some well-intentioned adblockers also block Sentry client-side error
reporting. Provide an endpoint on the Zulip server which forwards to
the Sentry server, so that these requests are not blocked.
Use the built-in HTML escaping of Markup("…{var}…").format(), in order
to allow Semgrep to detect mistakes like Markup("…{var}…".format())
and Markup(f"…{var}…").
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This commit adds 'zerver/lib/email_notifications.py'
to the FILES_WITH_LEGACY_SUBJECT set.
Because the file can have 'subject' in the email sense,
it should be exempted from the 'avoid subject as a var' lint rule.
Updated the title and description in the 'enable-emoticon-translation'
file and renamed the file accordingly. Added a new bullet point for
'time format' in the 'configure-new-user-settings.md' file and updated
the sidebar index by replacing the title 'Use 24-hour time' with
'Change the time format'.
The Django convention is for __repr__ to include the type and __str__
to omit it. In fact its default __repr__ implementation for models
automatically adds a type prefix to __str__, which has resulted in the
type being duplicated:
>>> UserProfile.objects.first()
<UserProfile: <UserProfile: emailgateway@zulip.com <Realm: zulipinternal 1>>>
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Zulip already has integrations for server-side Sentry integration;
however, it has historically used the Zulip-specific `blueslip`
library for monitoring browser-side errors. However, the latter sends
errors to email, as well optionally to an internal `#errors` stream.
While this is sufficient for low volumes of users, and useful in that
it does not rely on outside services, at higher volumes it is very
difficult to do any analysis or filtering of the errors. Client-side
errors are exceptionally noisy, with many false positives due to
browser extensions or similar, so determining real real errors from a
stream of un-grouped emails or messages in a stream is quite
difficult.
Add a client-side Javascript sentry integration. To provide useful
backtraces, this requires extending the pre-deploy hooks to upload the
source-maps to Sentry. Additional keys are added to the non-public
API of `page_params` to control the DSN, realm identifier, and sample
rates.
This commit renames reset_emails_in_zulip_realm function to
reset_email_visibility_to_everyone_in_zulip_realm which makes
it more clear to understand what the function actually does.
This commit also adds a comment explaining what this function
does.
Updates frontend user-facing strings with "private message" or "PM" to
use "direct message" or "DM" respectively instead.
Note that this updates translated strings as well as a few that
are not translated like search suggestions.
Updates `tools/lib/capitalization.py` for some specific strings
that are impacted by these changes, and removes "PM" and "PMs"
from checked strings.
This lets us simplify the long-ish ‘../../static/js’ paths, and will
remove the need for the ‘zrequire’ wrapper.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Ever since we started bundling the app with webpack, there’s been less
and less overlap between our ‘static’ directory (files belonging to
the frontend app) and Django’s interpretation of the ‘static’
directory (files served directly to the web).
Split the app out to its own ‘web’ directory outside of ‘static’, and
remove all the custom collectstatic --ignore rules. This makes it
much clearer what’s actually being served to the web, and what’s being
bundled by webpack. It also shrinks the release tarball by 3%.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Corepack manages multiple per-project version of Yarn and PNPM, which
means we have to maintain less installation code, and could help us
switch away from Yarn 1 without making the system unusable for
development of other Yarn 1 projects.
https://nodejs.org/api/corepack.html
The Unicode spaces in the timerender test resulted from an ICU
upgrade: https://github.com/nodejs/node/pull/45068.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This commit adds dropdown for move_messages_within_stream_limit_seconds
setting which is used to control for how long the user is allowed to
edit topic.
CZO discussion
[here](https://chat.zulip.org/#narrow/stream/6-frontend/topic/code.20coverage/near/1487773).
This allows `zjquery_element.js` to have test utils that aren't always being used.
`upload.js` is mostly UI focused and has upcoming commits that are hard
to write unit tests for (and for which unit tests wouldn't make much sense).
Previously notifications.clear_compose_notifications was used accross
the codebase. Since introducing the new
compose_banner.clear_message_sent_banners function, the two functions
are similar enough that we can just use clear_message_sent_banners
everywhere. This commit also moves scroll_to_message_banner_message_id
to compose_banner.
- Updates `.prettierignore` for the new directory.
- Updates any reference to the API documentation directory for
markdown files to be `api_docs/` instead of `zerver/api/`.
- Removes a reference link from `docs/documentation/api.md` that
hasn't referenced anything in the text since commit 0542c60.
- Update rendering of API documentation for new directory.
Black 23 enforces some slightly more specific rules about empty line
counts and redundant parenthesis removal, but the result is still
compatible with Black 22.
(This does not actually upgrade our Python environment to Black 23
yet.)
Signed-off-by: Anders Kaseorg <anders@zulip.com>
These files are not Jinja2 templates, so there's no reason that they needed
to be inside `templates/zerver`. Moving them to the top level reflects their
importance and also makes it feel nicer to work on editing the help center content,
without it being unnecessary buried deep in the codebase.
In the very infrequent case that we re-upload a file, the logic for
upload-release would have the existing SHA256SUM take precedence over
the new one -- despite uploading the new file.
Skip existing filenames (i.e. from the new file) when gathering
SHA256SUM metadata used to assemble `SHA256SUMS.txt`.
`check_version` in `install-yarn` had the rather careful check that
the yarn it installed into `/usr/bin/yarn` was the yarn which was
first in the user's `$PATH`. This caused problems when the user had a
pre-existing `/usr/local/bin/yarn`; however, those problems are
limited to the `install-yarn` script itself, since the nearly all
calls to yarn from Zulip's code already hardcode the `/srv/zulip-yarn`
location, and do not depend on what is in `$PATH`.
Remove the checks in `install-yarn` that depend on the local `$PATH`,
and stop installing our `yarn` into it. We also adjust the two
callsites which did not specify the full path to `yarn`, so use
`/srv/zulip-yarn`.
Fixes: #23993
Co-authored-by: Alex Vandiver <alexmv@zulip.com>
This greatly improves the readability of the diffs and in-codebase
translation strings over using ASCII sequences for unicode in the JSON.
We've previously noticed [^1] some JSON translation files ending up with
escaped Unicode sequences on disk, which Transifex indicates is expected
behavior [^2], though it is sometimes fixed by `manage.py
compilemessages` [^3]. Further, as noted in #23932 [^4], some JSON
translation files include HTML-escaped entities like quotation marks.
This script will ingest valid JSON files and output them as proper UTF-8
files with appropriately unescaped (unless otherwise necessary, like
double quotes being backslash-escaped) sequences, except when the key
itself contains HTML escape sequences (as it's presumed the value of
such entries must be pre-escaped before being passed to consumers).
[^1]: https://chat.zulip.org/#narrow/stream/58-translation/topic/Transifex.20client/near/1479205
[^2]: https://chat.zulip.org/#narrow/stream/58-translation/topic/an.20email.20for.20Transifex.20support/near/1481287
[^3]: https://chat.zulip.org/#narrow/stream/58-translation/topic/an.20email.20for.20Transifex.20support/near/1481908
[^4]: Which is not end-to-end fixed yet by this commit: that will
require a new release of Zulip Server.
gitlint-ignore: B1, title-trailing-punctuation, body-min-length, body-is-missing
transifex-client went EOL on November 30, 2022, replaced by
transifex/cli [^1].
Swap this in-place, since per the upstream README [^2]:
> The current version of the client maintains backwards compatibility
> for the tx push and tx pull commands. So, if you have a CI setup that
> uses them, you should not have to change anything.
As the mobile team found out, this is a partial truth if one previously
used some of the more advanced CLI flags, but all workflows referenced
in tools/ and docs/ use forwards-compatible flags to the new version.
[^1]: https://github.com/transifex/transifex-client/
[^2]: a0f28a1cf3/README.md
The module has been renamed, but also the original linter rule was
likely ineffective because of the length of compose_banner calls
anyway.
I'm not particularly worried about future internationalization bugs
here, so removing seems better than replacing it.
Fixes#23625.
The U+FE0F character was appearing in the data supplied by
emoji-datasource-google, but isn't in the CLDR data. By
removing that variation selector character, all remaining
emoji (that were previously marked as "not found in custom
emoji name maps, but also not found in CLDR data. Skipping.")
are now supported.
Since we are switching to tippyjs to display popovers, extracting
this data in a commit of its own, makes further changes easier to
review and merge.
Also, we could add tests with full coverage on the data
provided to popovers, possibly a good first issue for beginners.
Using curl to POST to the CircleCI workflow endpoint on CZO:
- Doesn't work on zulip/zulip@main (CZO runs a revert)
- Sets a bad example for other orgs
- Robs us of an opportunity to dogfood our own zulip/github-actions-zulip
Refactor the Actions workflows in this repo to report failure states
using the Zulip Action, and reimplement the related helper scripts in
Python, since they'd previously mostly shelled out to Python anyway.
Previously, CSS rules were added inline for emails page in
dev server. This commit adds a new file which contains
CSS rules for the emails page in dev server. This will
also help us in adding focus for the radio buttons in
the page, which cannot be added inline to the elements,
when we remove the use of bootstrap for this page.
The script will fail if we process with a droplet object with no
.ip_address, which can sometimes happen if we just droplet.load().
Sometimes that gives droplet.ip_address set to None. If we wait in a
loop, we eventuall get the .ip_address set.
These suffixes suppress some checks in the process, but still generate
and upload a tarball, push a tag, and make a Github prerelease.
`upload-release` already understands that anything with a suffix never
becomes the "latest" release.
Previously, emoji.json was read from
"$ZULIP_PATH/node_modules/emoji-datasource-google/emoji.json".
This path doesn't exist in production when installing from scratch from
a release tarball. And so, we ensure emoji.json exists by copying it to
`static/generated/emoji`.
With tweaks to comments by tabbott.
Fixes: #23469
In 2f547ea, the custom `has_error` logic in BaseDocumentationSpider
was removed in favor of checking whether any errors were logged.
One reference to that method was missed in UnusedImagesLinterSpider
and went unremarked as no images were added to the documentation
that were not used.
Replaces that reference with an error log.
Fixes a warning caused by using the deprecated `2.6` value
of the `REQUEST_FINGERPRINTER_IMPLEMENTATION` setting.
Upgrades `settings.py` to what would have been generated
by starting the documentation_crawler project using the
`startproject` command of scrapy v2.7.
This is still messy, in that it does the `chdir` using a different
library than the rest, but it's at least more consistent with the
codebase and it should be possible to bulk migrate the `ZULIP_PATH`
calculation, which we have in a dozen files, to pathlib later.
Setting `credential_source` is used when assuming role credentials --
that is, when running as one role, use the AssumeRole right to become
someone else.
The AWS command-line tools only do this if `role_arn`, the role to
assume, is also set -- if it is not set, it transparently falls
through to IAM role attached to the EC2 instance profile. However,
with the `aws-sdk-go` package, used by Teleport, this configuration
produces an error.
Remove the `credential_source = Ec2InstanceMetadata` line, which isn't
necessary for the AWS CLI, and interferes with Teleport operation.
This commit introduces the change of rendering private messages
section as collapsible, whose data-fetching logic came with zulip#21357.
We now have separated out `Private messages` from `top_left_corner`
section and shifted it below the `global_filters` in a different
separate section along with stream list with common scroll bar
in left-sidebar.
The new PM section will be opened by-default on loading the page
and will have a toggle-icon in its header, clicking on which makes the
section collapse/expand accordingly.
In default view, only recent 5 PM threads would be shown
and would append the active conversation as the 6th one at last
if not present in those 5, similar to how topics list work.
In PM section with unreads, a maximum of 8 conversations
would be shown and rest of them would be hidden behind
the 'more conversations' li-item, clicking on which takes
to the zoomedIn view of PM section where all the present
PM threads would be visible and rest of the sections of left-sidebar
will get collapsed.
Fixes#20870.
Co-authored-by: Aman Agrawal <amanagr@zulip.com>
Before Zulip 4.9, the Zulip install process left any already-installed
rabbitmq with whatever nodename it had previously configured. Wince
this encodes the name of the host when it was installed, this does not
function well with containers.
Leave rabbitmq-server uninstalled, which lets the Zulip installation
process set the nodename to `localhost`, which ensures that it is
usable across container restarts.
Since /communities/#all was failing the spider test due to `#all`
not being present as an `id` on the page, we exclude it from
this check saying that it is not really a documentation page.
Fixes#22984
Add an `@` icon in unread topics where user is mentioned.
We track a new set of `stream_id:topic` pairs for the unread mentions
so that recent topics instantly knows if a topic is unread and mentioned
or not.
This commit adds private messages to the Recent topics view, to make
it an all-encompassing overview of recent activity visible to the user.
We add a filter "Include PM" to toggle whether PMs should be shown in
recent topics.
Fixes#19449.
This way Puppeteer doesn’t have to re-download Chromium every time we
install new JavaScript dependencies.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This script pulls from our previously custom-written emoji strings
and fills in the rest from CLDR. It also removes 4 custom emoji which
collide with some of the new CLDR names (they will now just be called
by their CLDR name).
Fixes “E713 Test for membership should be `not in`” found by ruff (now
that I’ve fixed it not to ignore scripts lacking a .py extension).
Signed-off-by: Anders Kaseorg <anders@zulip.com>
The implementation is simple, we just check if the
the message sender is a notification bot to decide if we
should show the read receipts list.
We also update the modal content styling to match the padding at the
top of the modal.
Fixes#22905
Transitions the frontend of the web app to no longer use the
user status `away` field for setting a user's activity status
to be 'unavailable' (which is now a deprecated way to access
a user's `presence_enabled` setting).
Instead we now directly use and update the user's `presence_enabled`
setting for this feature.
Renames frontend code related to the feature to `invisible_mode`
vs `away`.
We lose node test coverage in `user_status.js` because we are now
using `channel.patch` to send these user setting updates to the
server.
Removes the temporary updates to `server_events_dispatch.py` (and
related tests) made in a previous commit, since we no longer have
or need the `away_user_ids` set.
Renames the filename so that it accurately reflects its contents
given the changes to the "Recommended setup" page in the previous
commit, and updates all links accordingly.
As a prep-commit for updating the billing / corporate pages for
demo organizations, initialize tippy.js with a default setting
for portico pages to use in general.
Fixes#21037.
This is part of fixing #19371. To bulk-add new emoji regularly,
mobile needs to know which servers support which emoji.
`staticfiles_storage.url` generates a unique URL with a hash
based on the file content, which lets mobile know if it needs
to update its locally stored data.
Django has always expected this, but Django 4.0 added a system check
that spews warnings in production.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
We should rearrange Zulip's developer docs to make it easier to
find the documentation that new contributors need.
Name changes
Rename "Code contribution guide" section -> "Contributing to Zulip".
Rename "Contributing to Zulip" page -> "Contributing guide".
Organizational changes to the newly-named "Contributing to Zulip":
Move up "Contributing to Zulip", as the third link in sidebar index.
Move up renamed "Contributing guide" page to the top of this section.
Move up "Zulip code of Conduct", as the second link of this section.
Move down "Licensing", as the last link of this section.
Move "Accessibility" just below "HTML and CSS" in Subsystems section.
Update all links according to the changes above.
Redirects should be added as needed.
Fixes: #22517.
This is preparatory commit that does basic UI set up for
user group edit in group settings overlay. This allows us to
write proper hashchange logic for user group settings overlay
under diffrent situations.
The work in this commit will be extended in further commits
to add proper UI and group edit logic.
Add support for creation of user groups using right panel
of new user group settings overlay being developed as part
of https://github.com/zulip/zulip/issues/19526.
In further commits we will add support for editing user
groups using right panel of the overlay.
This commit also introduces a minor bug related hashchange
for #groups which would be a quick fix once we have UI
for group edit on #groups overlay.
Dedicated overlay for user group settings is added as part of
addressing zulip#19526.
The newely added overlay is currently empty and more UI
related to settings is to be added in further commits.
A preparatory commit to have legacy user group settings logic
as we move forward to redesign the user group settings.
This is done so that current user group settings are functional
while we are working on the redesign, and also to make it clear
that most of the code in this file will be deleted and developers
should avoid spending much time on it.
Adds an API endpoint for accessing read receipts for other users, as
well as a modal UI for displaying that information.
Enables the previously merged privacy settings UI for managing whether
a user makes read receipts data available to other users.
Documentation is pending, and we'll likely want to link to the
documentation with help_settings_link once it is complete.
Fixes#3618.
Co-authored-by: Tim Abbott <tabbott@zulip.com>
Now that we can assume Python 3.6+, we can use the
email.headerregistry module to replace hacky manual email address
parsing.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
`DiscoverRunner.run_tests` has a return type of `int`. While
`Runner.run_tests` has a wildly different `Tuple[bool, List[str]]`.
This refactors it so that we have the correct return type, by passing
the additional information about failed tests through a side effect to directly
write the failed tests to a file.
Note that we have to make `failed_tests_path` optional as otherwise the method
signature will not be compatible with the supertype.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
Mypy does not know the acccurate return type of `get_runner` that is
determined by the `TEST_RUNNER` setting. We need to cast it to the
correct type to use methods like `get_shallow_tested_templates`. Note
that we import conditionally to avoid adding runtime dependency on
`zerver`.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
Emoji that we specifically choose names or aliases for will be
stored in this new file so that we can generate emoji_names.py
from the custom names and also CLDR (for the rest).
This new file isn't used for anything yet (no user facing change);
it will be used in future commits.
distutils is deprecated in Python 3.10 and will be removed in Python
3.12. We don’t need a full-powered version parser for this anyway.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This fixes having incompatible types of different items in
the dictionary, and makes `**identity_args` type-safe.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
When we were preparing the conversion to ES modules in 2019, the
primary obstacle was that the Node tests extensively relied on the
ability to reach into modules and mutate their CommonJS exports in
order to mock things. ES module bindings are not mutable, so in
commit 173c9cee42 we added
babel-plugin-rewire-ts as a kludgy transpilation-based workaround for
this to unblock the conversion.
However, babel-plugin-rewire-ts is slow, buggy, nonstandard,
confusing, and unmaintained. It’s incompatible with running our ES
modules as native ES modules, and prevents us from taking advantage of
modern tools for ES modules. So we want to excise all use of
__Rewire__ (and the disallow_rewire, override_rewire helper functions
that rely on it) from the tests and remove babel-plugin-rewire-ts.
Commits 64abdc199e and
e17ba5260a (#20730) prepared for this by
letting us see where __Rewire__ is being used. Now we go through and
remove most of the uses that are easy to remove without modifying the
production code at all.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This ensures that anything that looks like a full release will
identify itself as such. It requires that the version have a tag at
the current commit (though it doesn't need to be pushed yet), as well
as the commits exist in `main` or a release branch in the remote.
dmypy is innately incompatible with `local_partial_types` being
`False` as it uses fined-grained incremental mode for caching.
We need to enable `local_partial_types` for mypy as well so that the two
will behave the same way.
This requires us to add additional type annotation in certain siuations
but that's fine.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
mypy daemon performs significantly better than running the regular
mypy cli tool when we type check the entire codebase multiple
times locally.
This adds running mypy daemon as an option for both
`tools/run-mypy` and `tools/lint`.
To ensure daemon messages like "Daemon started", "Daemon stopped"
won't get printed we filter any output that starts with "Daemon".
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
Support for this header was removed in Chrome 78, Safari 15.4, and
Edge 17. It was never supported in Firefox.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Tornado finishes the request automatically. Avoids this error, hidden
until commit 81f7192ca3 (#22301), when
the browser tab is closed:
Traceback (most recent call last):
File "/srv/zulip-py3-venv/lib/python3.8/site-packages/tornado/web.py", line 1683, in _execute
result = await result
File "tools/run-dev.py", line 280, in prepare
await self.finish()
tornado.iostream.StreamClosedError: Stream is closed
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Since python-debian is GPL licensed, our script that imports it should
arguably be GPL licensed as well.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
`postgresql-14.4` is a notable upgrade in the PostgreSQL series, as it
fixes potential database corruption from `CREATE INDEX CONCURRENTLY`
statements which are run while rows are modified[1]. However, it also
requires an upgrade from `libllvm9` to `libllvm10`, which means it is
not installed by a mere `apt-get upgrade`.
Add the `--with-new-pkgs` flag to all of the potentially relevant
`apt-get upgrade` calls, so that this (and similar) packages are
upgraded successfully.
[1]: https://www.postgresql.org/docs/release/14.4/
Add none-checks, rename variables (to avoid redefinition of
the same variable with different types error), add necessary
type annotations.
This is a part of #18777.
Signed-off-by: Zixuan James Li <359101898@qq.com>
The production CI image starts `rabbitmq-server` but does not stop it,
which leaves a stale `/var/run/rabbitmq/pid` file in the image.
`rabbitmqctl wait --timeout 600 /var/run/rabbitmq/pid`, which is run
after starting the rabbitmq node, reads the PID file and waits for the
PID to be running, and for rabbitmq's port to be responding to pings.
If it reads an old PID file before the new PID is written, it
aborts (all but the first and last lines are output from `rabbitmqctl
wait` that is hidden by `/etc/init.d/rabbitmq-server`):
```
* Starting RabbitMQ Messaging Server rabbitmq-server
Waiting for pid file '/var/run/rabbitmq/pid' to appear
pid is 341
Waiting for erlang distribution on node 'rabbit@fc8f64d6acdb' while OS process '341' is running
Error:
process_not_running
* FAILED - check /var/log/rabbitmq/startup_\{log, _err\}
```
If it failed, the `production-upgrade` script tried to start
`rabbitmq` again -- despite it already still starting in the
background. These two attempts conflicted, and often one or both
failed.
Stop `rabbitmq-server` when building the image, which removes the
stale PID file.
Initializing the Zulip client opens a long-lived TCP connection due to
connection pooling in urllib3. In Github Actions, the network kills
such requests after ~270s, making the later `send_message` call fail.
Use a singular call to `zulip.Client()` early on to verify the
credentials, and do not cache the resulting client object. Instead,
re-create it during the final step when it is needed, so we do not run
afoul of bad TCP connection state.
This would ideally be fixed via connection keepalive or retry at the
level of the Zulip module.
The error message a user gets from the linter when using the
data-tippy-allowHTML attribute now conveys the fact that the
<template> tag is supposed to hold the tooltip content. This
might make understanding the correct workflow easier for
someone who encounters this error.
We previously forked tornado.autoreload to work around a problem where
it would crash if you introduce a syntax error and not recover if you
fix it (https://github.com/tornadoweb/tornado/issues/2398).
A much more maintainable workaround for that issue, at least in
current Tornado, is to use tornado.autoreload as the main module.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
The previous commit changed our response for the root domain on a
self-hosted Zulip server containing no realms to be a 404.
This made the HTTP headers verification done here less robust than
would be ideal, in that one could imagine a bug that makes every URL
404 passing that test.
So we instead change the check to verify the /server_settings API,
which doesn't have that problem.
To provide a smoother experience of accessing a web public stream,
we don't ask user to login unless user directly requests a
`/login` URL.
Fixes#21690.
Alex Vandiver
Anders Kaseorg
evykassirer
Prakhar Pratyush
sbansal1999
palashb01
Daniil Fadeev
Sahil Batra
Lauryn Menard
Aman Agrawal
Lauryn Menard
Lalit
Josh Klar
Yuri Sulyma
Greg Price
Tim Abbott
Mateusz Mandera
rht
David Rosa
jai2201
Barissa-Imran
madrix01
Ganesh Pawar
Riken Shah
Purushottam Tiwari
Adam Sah
Dinesh
Julia Bichler
Zixuan James Li
Matt Keller
Zixuan James Li
Raghav Luthra
patricklauer