is_cross_realm_bot_email is just
`email.lower() in settings.CROSS_REALM_BOT_EMAILS` which is the same,
aside of looking at .lower() - which is actually more correct.
Because Slack emoji naming is different from Zulip's.
According to https://emojipedia.org/slack/, Slack's emoji shortcodes are
derived from https://github.com/iamcal/emoji-data.
There are probably some deviations from that dataset, but this PR should
at least catch the ones that are identical to iamcal's.
Fixes “E713 Test for membership should be `not in`” found by
ruff (https://github.com/charliermarsh/ruff).
Signed-off-by: Anders Kaseorg <anders@zulip.com>
These limits don't appear to provide useful security benefits, and
they do impact usability because they prevented email-based users from
replying more than once, or from replying to message more than 5 days
old.
Fixes#2755.
Fixes#19994.
Updates the two UserProfile foreign key fields to have a backward
relation in the MutedUser model by changing the `related_name`
property.
This is a prep commit for removing users with a muted relationship
to the current user from read receipts.
We allow only owners to change the waiting period setting to become
full member. This commit contains only backend changes, frontend
changes will be done separately.
We allow only owners to add, edit or delete the allowed domains.
This commit only contains backend changes, frontend changes will
be done in a separate commit.
We allow only owners to change disallow_disposable_email_addresses
and emails_restricted_to_domains settings. This commit only contains
change in backend part, frontend changes will be done separately.
We allow only owners to change the invite_required setting.
This commit only adds the restriction in backend, frontend
changes will be done separately.
We also add a helper function in test_realm.py to avoid
writing same code repeatedly and this helper will also
be used in tests for other settings to be added in
further commits.
As suggested by the new comments, the cost for a Zulip data export
scales with messages actually included in the export, so an
organizations with 1M private messages but only 50K public stream
messages should not be modeled the same as one with 1M public stream
messages for the purpose of the limits here.
Also improve the comments and variable names more generally.
One should now be able to configure a regex by appending _regex to the
port number:
[tornado_sharding]
9802_regex = ^[l-p].*\.zulipchat\.com$
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Current value of can_remove_subscribers_group field is admins system group
only so behavior is not changed. We would provide support to change this
setting from API and UI in further commits.
This commit adds do_change_can_remove_subscriber_group function for
changing can_remove_subscribers_group field of a stream. We also add
can_remove_subscribers_group_id field to stream and subscription
objects.
This function will be helpful for writing tests in next commit.
We would add API and UI support to change this setting in further
commits.
This commit udpates can_remove_subscribers_group to be not null.
We already added a migration to set the value of this field for
existing streams and also added a commit to set this field to
admins system group for now while creating streams.
This migration sets can_remove_subscribers_group value to admins system
group for all the existing streams. In further commit we would change
can_remove_subscribers_group to be not null and thus we add this migration
to ensure all existing streams have this setting value set.
This commit sets can_remove_subscribers_group to admins system
group while creating streams as it will be the default value
of this setting. In further we would provide an option to set
value of this setting to any user group while creating streams
using API or UI.
We change the import order to import UserGroup objects before
Stream such that we can set can_remove_subscribers_group correctly.
We do not import UserGroupMembership objects here along with
UserGroup since UserProfile objects are not imported and
GroupGroupMembership are also imported later as these are not
required before.
cachify has been removed in 9d448e73d2.
We don't need to keep its tests.
TODO: functools.lru_cache can be replaced by functools.cache when we
drop Python 3.8.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
Updates the base titles so that they begin with the page content,
and end with "| Zulip" + information about the type of doc: "help
center", "API documentation", "terms and policies", or "integrations".
Adds a META_CATEGORY dict for categories that are not best described
as groups of 'tools', so that in the subsequent commit the PAGE_TITLE
can be set accordingly.
Also, removes 'tools' from the 'Miscellaneous' category text and
spells out 'Human resources' instead of using 'HR'.
We do not need direct_members and direct_subgroups field of
UserGroup objects in the export data since we already have
UserGroupMembership and GroupGroupMembership object data.
While importing we keep these fields empty when creating
UserGroup objects and direct_members and direct_subgroups
fields will get set when UserGroupMembership and
GroupGroupMembership objects are created.
This change will also help us in further changes when we
will change the order of importing to import UserGroup
objects just after Realm objects.
Although our POST /messages handler accepts the ‘to’ parameter with or
without JSON encoding, there are two problems with passing it as an
unencoded string.
Firstly, you’d fail to send a message to a stream named ‘true’ or
‘false’ or ‘null’ or ‘2022’, as the JSON interpretation is prioritized
over the plain string interpretation.
Secondly, and more importantly for our tests, it violates our OpenAPI
schema, which requires the parameter to be JSON-encoded. This is
because OpenAPI has no concept of a parameter that’s “optionally
JSON-encoded”, nor should it: such a parameter cannot be unambiguously
decoded for the reason above.
Our version of openapi-core doesn’t currently detect this schema
violation, but after the next upgrade it will.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
The previous schema incorrectly prohibited the string, integer, and
string-array forms that we do in fact accept.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
The previous schema incorrectly prohibited the two-element array form
that we do in fact accept, and didn’t specify anything about the
contents of the object form.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
change the names of "github" and "twitter" external account fields to
"GitHub username" and "Twitter username" respectively and remove the
hints of them.
If there are more than 1 room with the same set of users, the import
will fail due to a unique constraint on the huddle_hash. Figuring out
why and which room is causing this database error is kinda difficult.
We deduplicate those cases here and simply merge the rooms together.
Note however, that the deduplication does not work as expected so we
simply ignore them all together for now and only raise an exception
along some logging output. At least this way, it is pretty clear what is
wrong and you do not have to wait to get a database error during the
actual import.
We also ignore empty huddle rooms since those are the duplicates that
caused problems for me and if they are empty, ignoring them is easier
than trying to get the merge to work.
Not sure where those channels come from since we discovered this with
production data.
Signed-off-by: Florian Pritz <bluewind@xinu.at>
Not sure where those come from since we discovered this with production
data. Somehow there were reactions with usernames that were old and no
longer existed.
Signed-off-by: Florian Pritz <bluewind@xinu.at>
Not sure where those come from since we discovered this with production
data.
There only was a single instance of this in my entire batch of data in
an old message from the time when we started using Rocket.Chat. This
might be an old issue or it might require some special settings that
were later changed.
Signed-off-by: Florian Pritz <bluewind@xinu.at>
Several integration docs instruct the user to create a bot, but don't
specify that the type of bot should be "Incoming webhook".
Renames create-a-bot.md -> create-an-incoming-webhook.md for clarity,
and replaces all incomplete instructions with this macro.
Renames bot_types.png -> bot_types_incoming_webhook.png and updates
the image with a screenshot of the latest UI.
The Yo company shut down in 2016.
https://en.wikipedia.org/wiki/Yo_(app)#History
Removes `yo` instances from `zerver/lib/integrations.py`.
Removes `zerver/webhooks/yo`.
Removes `static/images/integrations/yo-app`.
Added a user_list_style personal user setting to the bottom of
Settings > Display settings > Theme section which controls the look
of the right sidebar user list.
The radio button UI includes a preview of what the styles look like.
The setting is intended to eventually have 3 possible values: COMPACT,
WITH_STATUS and WITH_AVATAR; the final value is not yet implemented.
Co-authored-by: Tim Abbott <tabbott@zulip.com>
SOCIAL_AUTH_GITHUB_TEAM_ID is expected to be the string of an integer.
The requests mock for the bogus /None URL is unused because the
function that would request it is itself mocked.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Since the setup for uptimerobot does not deviate from what we have in
the "create-bot-construct-url" macro, we can reuse it and event
filtering instructions for uptimerobot will be automatically included.
TODO: Add event filtering documentation for buildbot when it supports
custom endpoint URL.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
We create "event-filter-instruction.md" and add it to
"create-bot-construct-url.md". This allows the user to keep track of the
supported event types for most of the integrations that implement this
feature. Note that not all integrations use "create-bot-construct-url.md".
We also need to rename "function" to "view_function" to make this change
type-check.
This is relevant to #18392.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
This handler from commit a696141a25
(#7418) was almost certainly made unreachable when commit
c3e395b7d8 (#13092) removed anything
that might have thrown a UnicodeDecodeError from highlight_string.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Renames article about organization language used for automated
messages and invitation emails. Creates URL redirect and updates
links in repository (web app, help center and api documentation).
Prior to this change, the article was named:
'change-the-default-language-for-your-organization'.
Fixes#21949.
Previously, an active production Zulip server would experience a class
of deadlocks caused by two or more concurrent bulk update operations
on the UserMessage table.
This is because UPDATE ... SET ... WHERE statements that execute in
parallel take row-level UPDATE locks as they get results; since the
query plans may result in getting rows in different orders between two
queries, this can result in deadlocks.
Some databases allow ORDER BY on their UPDATE ... WHERE statements;
PostgreSQL does not. In PostgreSQL, the answer is to do a sub-select
with an ORDER BY ... FOR UPDATE to ensure consistent ordering on row
locks.
We do this all code paths using bitand or bitor as part of bulk
editing message flags, which should ensure that these concurrent
operations obtain row level locks on the table in the same order.
Fixes#19054.
Renames existing template variables starting with `OPEN_GRAPH` to be
either `PAGE` or `PAGE_METADATA` since these variables are used for
adding both open graph metdata and page titles/descriptions for SEO.
Updates `_test` in DocPageTest so that the generic test boolean
parameters will confirm that there is an HTML title element as well
as a meta-description for SEO and meta tags for open graph data.
Sets tests for error pages and dev env pages to `landing_page=False`
since these pages will not have the metadata added in subsequent
commits.
API and integration docs are automated to have this metadata. There
is a specific test for the integrations open graph data. The list of
API endpoints with specific content are tested for this now.
The specific test for portico pages open graph data is removed in
favor of the more generic test style, which will not fail if/when
the template data and text is changed or updated.
In `zerver/tests/test_docs.py`, we split the catch all test for doc
endpoints into more specific test groups: dev environment pages,
error pages, corporate pages. Also, moves the api endpoints being
tested to the specific test for api endpoints.
Expands specific test for new open communities directory page to
test that the zulip dev realm was added to the page. Adds a generic
test for the endpoint to the generic `test_doc_endpoint`
test.
In bbf4c25553, we added support for
triggering user group changes when the waiting_period_threshold realm
setting was changed.
The test_events test did not expect this, and thus would fail if the
last provision was between 10 and 20 days ago.
The simplest fix is to just increase those numbers, since computing
whether the database was too old would be more complex than it is
worth for this test.
This is preparatory commit for #18941.
Importing `do_delete_message` from `message_edit.py` was causing a
circular import error. In order to avoid that, we create a separate
message_delete.py file which has all the functions related to deleting
messages.
The tests for deleting messages are present in
`zerver/tests/test_message_edit.py`.
Fixes a part of #18941
This commit strengthens types by typing the Solano webhook's incoming
payload as WildValue, which eradicates the use of Any within the
incoming webhook integration.
The KeyError exception has been replaced to catch a ValidationError
instead now, since the incoming payload's keys will be tamed before
usage and the non-existence of the key is raised as a
ValidationError in the taming function.
This commit strengthens types by typing the webhook's incoming
payload as WildValue and taming the values of the payload before
usage, which eradicates the use of Any within the UptimeRobot
incoming webhook integration.
The payload's values are now tamed, stored in variables and passed
into message templates as opposed to passing in the payload dict as
keyword arguments.
The KeyError exception has been replaced to catch a ValidationError
instead now, since the incoming payload's keys will be tamed before
usage and the non-existence of the key is raised as a
ValidationError in the taming function.
This commit strengthens types by typing the webhook's incoming
payload as WildValue, which eradicates the use of Any within the
Zabbix incoming webhook integration.
The KeyError exception has been replaced to catch a ValidationError
instead now, since the incoming payload's keys will be tamed before
usage and the non-existence of the key is raised as a
ValidationError in the taming function.
Extends the URL redirect system used for documentation pages to corporate
landing pages. This makes it easier and consistent for contributors who
work on both areas to create new URL redirects when needed.
Creates `zerver.lib.url_redirects.py` to record old and new URLs
for documentation pages that have been renamed/moved and need URL
redirects.
This file is then used by `zproject.urls.py` to redirect links and
by `zerver.test.test_urls.py` to test that all of the old URLs
return a success response with a common page header/text depending
on the type of redirect (help center, policy, or API).
Adds a section to contributor docs on writing documentation for
how to use this redirect system when renaming a help center or api
documentation page.
Fixes#21946. Fixes#17897.
django.request logs responses with 5xx response codes (our configuration
of the logger prevents it from logging 4xx as well which it normally
does too). However, it does it without the traceback which results in
quite unhelpful log message that look like
"Bad Gateway:/api/v1/users/me/apns_device_token" - particularly
confusing when sent via email to server admins.
The solution here is to do the logging ourselves, using Django's
log_response() (which is meant for this purpose), and including the
traceback. Django tracks (via response._has_been_logged attribute) that
the response has already been logged, and knows to not duplicate that
action. See log_response() in django's codebase for these details.
Fixes#19596.
It seems helpful for this to get logged with the traceback rather than
just the general
"<exception name> while trying to connect to push notification bouncer."
The logo were only used in the integration documentation and belong in
static/images/integrations/giphy/; the in-app image is given its own
directory.
Fixes#22464.
Users will only be able to login via GitHub, because imported users
get GitHub's generated noreply email addresses - so this should be the
only auth method enabled at first, to avoid confusion.
In zerver.management.commands.logout_all_users,
we pass a values queryset containing the ids into
this function, which is not actually a list. This
broadens the type annotation so that the ValuesQuerySet
is accepted.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
Only ["id"] is accessed on the dicts (representing the external tool
users). Given that for some tools the id may be under a different name
etc. due to different user dicts format, it's best to just pass those
ids to the function so that it can stay generalized and not reliant
on a specific user dict format.
get_timestamp_from_message was extracted in the previous commit. We can
deduplicate and the code a bit cleaner by using it where appropriate
instead of message["ts"].
message["ts"] is slack-specific. For this to be a general util function
it needs to take a callable that will grab a timestamp from the message
dict (which has varying formats depending on what we're importing from).
Updates in-app and documentation references to automated messages
sent by the notification bot as automated notices (or automated
messages where more appropriate/clear), instead of notifications.
Also, makes some small related revisions / general clean ups to
`resolve-a-topic.md`.
Fixes#22188.
This uses a more specific type `_StrPromise` to replace `Promise`
providing typing information for lazy translation strings.
In places where the callee evaluates the `_StrPromise` object in all
cases we simply force the evaluation with `str()`. This includes
`JsonableError` that ends up handled by the error handler middleware,
and `internal_send_stream_message` that depends on `check_stream_topic`,
requiring the `topic` to be evaluated anyway. In other siuations, the
callee is expected to be able to handle `StrPromise` explicitly.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
Fixes#21037.
This is part of fixing #19371. To bulk-add new emoji regularly,
mobile needs to know which servers support which emoji.
`staticfiles_storage.url` generates a unique URL with a hash
based on the file content, which lets mobile know if it needs
to update its locally stored data.
This commit changes the name of missed message email tests for
personal and huddle messages to be more clear:
- from *_personal_missed_stream_messages to *_missed_personal_messages
- from *_huddle_missed_stream_messages to *_missed_huddle_messages
We add quote prefix ">" to each line of the message in the plain text
missed message emails, which are then rendered as quotes by email
clients. We also move the message content in the next line after sender.
This helps us in clearly showing the message authors in missed message
emails especially in emails with multiple messages and senders.
Fixes#15836.
This ValueError had no test coverage, because the code path wasn't
actually possible with how the caller is constructed.
Rather than writing a highly artificial test for this as proposed in
This also allows us to remove some assertions as we now know that
AVATAR_SALT will never be None.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
This implements get_mandatory_secret that ensures SHARED_SECRET is
set when we hit zerver.decorator.authenticate_notify. To avoid getting
ZulipSettingsError when setting up the secrets, we set an environment
variable DISABLE_MANDATORY_SECRET_CHECK to skip the check and default
its value to an empty string.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
Due to mismatches between the URL parsers in Python and browsers, it
was possible to hoodwink rewrite_local_links_to_relative into
generating links that browsers would interpret as absolute.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
These characters are not allowed and trying to create a Zulip message
with those characters throws a JsonableError in check_stream_topic.
We don't want to reject emails with those chars in the subject, so
it's best to just modify it appropriately.
This commit checks for null values for keys within "attachment" in
the Slack integration's incoming payloads. These keys were expected
to exist optionally previously, and the existence of null values for
these wasn't anticipated. Due to an issue report for such null
values in the payload, their handling is updated appropriately.
The checks for these values are truthiness checks since the strategy
for these values being null or falsy ("", 0) is the same; we don't
process that key-value pair. This is consistent with how Slack handles
this scenario.
For the case where all the attachment fields have null values, Slack
displays this as an empty block with no content, and therefore our
strategy for this is a no-op.
Tests updated.
Since this decorator is only used for methods of
TestServiceBotEventTriggers, we can type the decorated method's
signature accurately without using ParamSpec.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
We can express the type of these decorators with Concatenate and ParamSpec
now for tighter type annotations.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
This removes ViewFuncT and all the associated type casts with ParamSpec
and Concatenate. This provides more accurate type annotation for
decorators at the cost of making the concatenated parameters
positional-only. This change does not intend to introduce any other
behavioral difference. Note that we retype args in process_view as
List[object] because the view functions can not only be called with
arguments of type str.
Note that the first argument of rest_dispatch needs to be made
positional-only because of the presence of **kwargs.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
This module was originally introduced in 2016 to assist adding mypy
annotations to the project. Back then static type checking was not that
established throughout the codebase, so it was helpful to be able to
print out the types for type checking purposes.
This workflow is no longer helpful for improving type annotations right
now, and it has been unused for a while.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
Whether we sent a resolve topic notification or not may be useful in
the caller. It was originally intended to be used in #21712, but may
only be relevant for future logging.
Part of #21712.
This will have no real effect in most situations. However, a user
moves a topic to another stream while also adding/removing the
resolved-topic checkmark from the topic name, then the "This topic was
resolved" notificaiton will now appear just before the "This topic was
moved" notification rather than just after.
This is likely slightly less confusing to users, since the topic
having been moved from somewhere else is likely the most salient fact
to a reader.
We expect to change things to not send both notifications in an
upcoming commit.
This refactoring helps with #21712.
Our seat count calculation is different for guest user than normal users
(a number of initial guests are free, and additional marginal guests are
worth 1/5 of a seat) - so these checks we apply when a user is being
invited or signing up need to know whether it's a guest or non-guest
being added.
Because rate_limit_request_by_ip is the only caller of it, it is safe
for us to inline RateLimitedIpAddr and remove this helper. This ensures
that we have consistent internals for rate limiting functions, which all
have a should_rate_limit check.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
This allows us to use them with HttpResponse objects returned by
calling a view function directly.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
Anders Kaseorg
Mateusz Mandera
rht
Jason Hildebrand
yogesh sirsat
Lauryn Menard
Sahil Batra
Tim Abbott
Zixuan James Li
Florian Pritz
David Rosa
Raghav Luthra
Christopher Chong
Matt Keller
madrix01
Hari Prashant Bhimaraju
Alya Abbott
Khushiyant
Aman Agrawal
Alex Vandiver
evykassirer
sahil839
Adam Sah
Israel Ebonko
Julia Bichler