This monkey-patching approach is not meaningful when what we really need
is just the names of the test, that can already be done in
get_test_names.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
Removes the default title element of "Zulip" from `base.html` and
the default meta-description sentence from `meta_tags.html`. Also
removes default open graph metadata.
For portico templates that would would fail tests, set both
`PAGE_TITLE` and `PAGE_DESCRIPTION` variables with appropriate
content.
Co-authored-by: Alya Abbott <alya@zulip.com>
This makes use of the type parameters of ValuesQuerySet (a.k.a
_QuerySet) to provide a more accurate type annotation for query_for_ids.
Note that QuerySet[ModelT] is equivalent to _QuerySet[ModelT, ModelT].
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
is_cross_realm_bot_email is just
`email.lower() in settings.CROSS_REALM_BOT_EMAILS` which is the same,
aside of looking at .lower() - which is actually more correct.
Because Slack emoji naming is different from Zulip's.
According to https://emojipedia.org/slack/, Slack's emoji shortcodes are
derived from https://github.com/iamcal/emoji-data.
There are probably some deviations from that dataset, but this PR should
at least catch the ones that are identical to iamcal's.
Fixes “E713 Test for membership should be `not in`” found by
ruff (https://github.com/charliermarsh/ruff).
Signed-off-by: Anders Kaseorg <anders@zulip.com>
These limits don't appear to provide useful security benefits, and
they do impact usability because they prevented email-based users from
replying more than once, or from replying to message more than 5 days
old.
Fixes#2755.
Fixes#19994.
Updates the two UserProfile foreign key fields to have a backward
relation in the MutedUser model by changing the `related_name`
property.
This is a prep commit for removing users with a muted relationship
to the current user from read receipts.
We allow only owners to change the waiting period setting to become
full member. This commit contains only backend changes, frontend
changes will be done separately.
We allow only owners to add, edit or delete the allowed domains.
This commit only contains backend changes, frontend changes will
be done in a separate commit.
We allow only owners to change disallow_disposable_email_addresses
and emails_restricted_to_domains settings. This commit only contains
change in backend part, frontend changes will be done separately.
We allow only owners to change the invite_required setting.
This commit only adds the restriction in backend, frontend
changes will be done separately.
We also add a helper function in test_realm.py to avoid
writing same code repeatedly and this helper will also
be used in tests for other settings to be added in
further commits.
As suggested by the new comments, the cost for a Zulip data export
scales with messages actually included in the export, so an
organizations with 1M private messages but only 50K public stream
messages should not be modeled the same as one with 1M public stream
messages for the purpose of the limits here.
Also improve the comments and variable names more generally.
One should now be able to configure a regex by appending _regex to the
port number:
[tornado_sharding]
9802_regex = ^[l-p].*\.zulipchat\.com$
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Current value of can_remove_subscribers_group field is admins system group
only so behavior is not changed. We would provide support to change this
setting from API and UI in further commits.
This commit adds do_change_can_remove_subscriber_group function for
changing can_remove_subscribers_group field of a stream. We also add
can_remove_subscribers_group_id field to stream and subscription
objects.
This function will be helpful for writing tests in next commit.
We would add API and UI support to change this setting in further
commits.
This commit udpates can_remove_subscribers_group to be not null.
We already added a migration to set the value of this field for
existing streams and also added a commit to set this field to
admins system group for now while creating streams.
This migration sets can_remove_subscribers_group value to admins system
group for all the existing streams. In further commit we would change
can_remove_subscribers_group to be not null and thus we add this migration
to ensure all existing streams have this setting value set.
This commit sets can_remove_subscribers_group to admins system
group while creating streams as it will be the default value
of this setting. In further we would provide an option to set
value of this setting to any user group while creating streams
using API or UI.
We change the import order to import UserGroup objects before
Stream such that we can set can_remove_subscribers_group correctly.
We do not import UserGroupMembership objects here along with
UserGroup since UserProfile objects are not imported and
GroupGroupMembership are also imported later as these are not
required before.
cachify has been removed in 9d448e73d2.
We don't need to keep its tests.
TODO: functools.lru_cache can be replaced by functools.cache when we
drop Python 3.8.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
Updates the base titles so that they begin with the page content,
and end with "| Zulip" + information about the type of doc: "help
center", "API documentation", "terms and policies", or "integrations".
Adds a META_CATEGORY dict for categories that are not best described
as groups of 'tools', so that in the subsequent commit the PAGE_TITLE
can be set accordingly.
Also, removes 'tools' from the 'Miscellaneous' category text and
spells out 'Human resources' instead of using 'HR'.
We do not need direct_members and direct_subgroups field of
UserGroup objects in the export data since we already have
UserGroupMembership and GroupGroupMembership object data.
While importing we keep these fields empty when creating
UserGroup objects and direct_members and direct_subgroups
fields will get set when UserGroupMembership and
GroupGroupMembership objects are created.
This change will also help us in further changes when we
will change the order of importing to import UserGroup
objects just after Realm objects.
Although our POST /messages handler accepts the ‘to’ parameter with or
without JSON encoding, there are two problems with passing it as an
unencoded string.
Firstly, you’d fail to send a message to a stream named ‘true’ or
‘false’ or ‘null’ or ‘2022’, as the JSON interpretation is prioritized
over the plain string interpretation.
Secondly, and more importantly for our tests, it violates our OpenAPI
schema, which requires the parameter to be JSON-encoded. This is
because OpenAPI has no concept of a parameter that’s “optionally
JSON-encoded”, nor should it: such a parameter cannot be unambiguously
decoded for the reason above.
Our version of openapi-core doesn’t currently detect this schema
violation, but after the next upgrade it will.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
The previous schema incorrectly prohibited the string, integer, and
string-array forms that we do in fact accept.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
The previous schema incorrectly prohibited the two-element array form
that we do in fact accept, and didn’t specify anything about the
contents of the object form.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
change the names of "github" and "twitter" external account fields to
"GitHub username" and "Twitter username" respectively and remove the
hints of them.
If there are more than 1 room with the same set of users, the import
will fail due to a unique constraint on the huddle_hash. Figuring out
why and which room is causing this database error is kinda difficult.
We deduplicate those cases here and simply merge the rooms together.
Note however, that the deduplication does not work as expected so we
simply ignore them all together for now and only raise an exception
along some logging output. At least this way, it is pretty clear what is
wrong and you do not have to wait to get a database error during the
actual import.
We also ignore empty huddle rooms since those are the duplicates that
caused problems for me and if they are empty, ignoring them is easier
than trying to get the merge to work.
Not sure where those channels come from since we discovered this with
production data.
Signed-off-by: Florian Pritz <bluewind@xinu.at>
Not sure where those come from since we discovered this with production
data. Somehow there were reactions with usernames that were old and no
longer existed.
Signed-off-by: Florian Pritz <bluewind@xinu.at>
Not sure where those come from since we discovered this with production
data.
There only was a single instance of this in my entire batch of data in
an old message from the time when we started using Rocket.Chat. This
might be an old issue or it might require some special settings that
were later changed.
Signed-off-by: Florian Pritz <bluewind@xinu.at>
Several integration docs instruct the user to create a bot, but don't
specify that the type of bot should be "Incoming webhook".
Renames create-a-bot.md -> create-an-incoming-webhook.md for clarity,
and replaces all incomplete instructions with this macro.
Renames bot_types.png -> bot_types_incoming_webhook.png and updates
the image with a screenshot of the latest UI.
The Yo company shut down in 2016.
https://en.wikipedia.org/wiki/Yo_(app)#History
Removes `yo` instances from `zerver/lib/integrations.py`.
Removes `zerver/webhooks/yo`.
Removes `static/images/integrations/yo-app`.
Added a user_list_style personal user setting to the bottom of
Settings > Display settings > Theme section which controls the look
of the right sidebar user list.
The radio button UI includes a preview of what the styles look like.
The setting is intended to eventually have 3 possible values: COMPACT,
WITH_STATUS and WITH_AVATAR; the final value is not yet implemented.
Co-authored-by: Tim Abbott <tabbott@zulip.com>
SOCIAL_AUTH_GITHUB_TEAM_ID is expected to be the string of an integer.
The requests mock for the bogus /None URL is unused because the
function that would request it is itself mocked.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Since the setup for uptimerobot does not deviate from what we have in
the "create-bot-construct-url" macro, we can reuse it and event
filtering instructions for uptimerobot will be automatically included.
TODO: Add event filtering documentation for buildbot when it supports
custom endpoint URL.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
We create "event-filter-instruction.md" and add it to
"create-bot-construct-url.md". This allows the user to keep track of the
supported event types for most of the integrations that implement this
feature. Note that not all integrations use "create-bot-construct-url.md".
We also need to rename "function" to "view_function" to make this change
type-check.
This is relevant to #18392.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
This handler from commit a696141a25
(#7418) was almost certainly made unreachable when commit
c3e395b7d8 (#13092) removed anything
that might have thrown a UnicodeDecodeError from highlight_string.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Renames article about organization language used for automated
messages and invitation emails. Creates URL redirect and updates
links in repository (web app, help center and api documentation).
Prior to this change, the article was named:
'change-the-default-language-for-your-organization'.
Fixes#21949.
Previously, an active production Zulip server would experience a class
of deadlocks caused by two or more concurrent bulk update operations
on the UserMessage table.
This is because UPDATE ... SET ... WHERE statements that execute in
parallel take row-level UPDATE locks as they get results; since the
query plans may result in getting rows in different orders between two
queries, this can result in deadlocks.
Some databases allow ORDER BY on their UPDATE ... WHERE statements;
PostgreSQL does not. In PostgreSQL, the answer is to do a sub-select
with an ORDER BY ... FOR UPDATE to ensure consistent ordering on row
locks.
We do this all code paths using bitand or bitor as part of bulk
editing message flags, which should ensure that these concurrent
operations obtain row level locks on the table in the same order.
Fixes#19054.
Renames existing template variables starting with `OPEN_GRAPH` to be
either `PAGE` or `PAGE_METADATA` since these variables are used for
adding both open graph metdata and page titles/descriptions for SEO.
Updates `_test` in DocPageTest so that the generic test boolean
parameters will confirm that there is an HTML title element as well
as a meta-description for SEO and meta tags for open graph data.
Sets tests for error pages and dev env pages to `landing_page=False`
since these pages will not have the metadata added in subsequent
commits.
API and integration docs are automated to have this metadata. There
is a specific test for the integrations open graph data. The list of
API endpoints with specific content are tested for this now.
The specific test for portico pages open graph data is removed in
favor of the more generic test style, which will not fail if/when
the template data and text is changed or updated.
In `zerver/tests/test_docs.py`, we split the catch all test for doc
endpoints into more specific test groups: dev environment pages,
error pages, corporate pages. Also, moves the api endpoints being
tested to the specific test for api endpoints.
Expands specific test for new open communities directory page to
test that the zulip dev realm was added to the page. Adds a generic
test for the endpoint to the generic `test_doc_endpoint`
test.
In bbf4c25553, we added support for
triggering user group changes when the waiting_period_threshold realm
setting was changed.
The test_events test did not expect this, and thus would fail if the
last provision was between 10 and 20 days ago.
The simplest fix is to just increase those numbers, since computing
whether the database was too old would be more complex than it is
worth for this test.
This is preparatory commit for #18941.
Importing `do_delete_message` from `message_edit.py` was causing a
circular import error. In order to avoid that, we create a separate
message_delete.py file which has all the functions related to deleting
messages.
The tests for deleting messages are present in
`zerver/tests/test_message_edit.py`.
Fixes a part of #18941
This commit strengthens types by typing the Solano webhook's incoming
payload as WildValue, which eradicates the use of Any within the
incoming webhook integration.
The KeyError exception has been replaced to catch a ValidationError
instead now, since the incoming payload's keys will be tamed before
usage and the non-existence of the key is raised as a
ValidationError in the taming function.
This commit strengthens types by typing the webhook's incoming
payload as WildValue and taming the values of the payload before
usage, which eradicates the use of Any within the UptimeRobot
incoming webhook integration.
The payload's values are now tamed, stored in variables and passed
into message templates as opposed to passing in the payload dict as
keyword arguments.
The KeyError exception has been replaced to catch a ValidationError
instead now, since the incoming payload's keys will be tamed before
usage and the non-existence of the key is raised as a
ValidationError in the taming function.
This commit strengthens types by typing the webhook's incoming
payload as WildValue, which eradicates the use of Any within the
Zabbix incoming webhook integration.
The KeyError exception has been replaced to catch a ValidationError
instead now, since the incoming payload's keys will be tamed before
usage and the non-existence of the key is raised as a
ValidationError in the taming function.
Extends the URL redirect system used for documentation pages to corporate
landing pages. This makes it easier and consistent for contributors who
work on both areas to create new URL redirects when needed.
Creates `zerver.lib.url_redirects.py` to record old and new URLs
for documentation pages that have been renamed/moved and need URL
redirects.
This file is then used by `zproject.urls.py` to redirect links and
by `zerver.test.test_urls.py` to test that all of the old URLs
return a success response with a common page header/text depending
on the type of redirect (help center, policy, or API).
Adds a section to contributor docs on writing documentation for
how to use this redirect system when renaming a help center or api
documentation page.
Fixes#21946. Fixes#17897.
django.request logs responses with 5xx response codes (our configuration
of the logger prevents it from logging 4xx as well which it normally
does too). However, it does it without the traceback which results in
quite unhelpful log message that look like
"Bad Gateway:/api/v1/users/me/apns_device_token" - particularly
confusing when sent via email to server admins.
The solution here is to do the logging ourselves, using Django's
log_response() (which is meant for this purpose), and including the
traceback. Django tracks (via response._has_been_logged attribute) that
the response has already been logged, and knows to not duplicate that
action. See log_response() in django's codebase for these details.
Fixes#19596.
It seems helpful for this to get logged with the traceback rather than
just the general
"<exception name> while trying to connect to push notification bouncer."
The logo were only used in the integration documentation and belong in
static/images/integrations/giphy/; the in-app image is given its own
directory.
Fixes#22464.
Users will only be able to login via GitHub, because imported users
get GitHub's generated noreply email addresses - so this should be the
only auth method enabled at first, to avoid confusion.
In zerver.management.commands.logout_all_users,
we pass a values queryset containing the ids into
this function, which is not actually a list. This
broadens the type annotation so that the ValuesQuerySet
is accepted.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
Only ["id"] is accessed on the dicts (representing the external tool
users). Given that for some tools the id may be under a different name
etc. due to different user dicts format, it's best to just pass those
ids to the function so that it can stay generalized and not reliant
on a specific user dict format.
get_timestamp_from_message was extracted in the previous commit. We can
deduplicate and the code a bit cleaner by using it where appropriate
instead of message["ts"].
message["ts"] is slack-specific. For this to be a general util function
it needs to take a callable that will grab a timestamp from the message
dict (which has varying formats depending on what we're importing from).
Updates in-app and documentation references to automated messages
sent by the notification bot as automated notices (or automated
messages where more appropriate/clear), instead of notifications.
Also, makes some small related revisions / general clean ups to
`resolve-a-topic.md`.
Fixes#22188.
This uses a more specific type `_StrPromise` to replace `Promise`
providing typing information for lazy translation strings.
In places where the callee evaluates the `_StrPromise` object in all
cases we simply force the evaluation with `str()`. This includes
`JsonableError` that ends up handled by the error handler middleware,
and `internal_send_stream_message` that depends on `check_stream_topic`,
requiring the `topic` to be evaluated anyway. In other siuations, the
callee is expected to be able to handle `StrPromise` explicitly.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
Fixes#21037.
This is part of fixing #19371. To bulk-add new emoji regularly,
mobile needs to know which servers support which emoji.
`staticfiles_storage.url` generates a unique URL with a hash
based on the file content, which lets mobile know if it needs
to update its locally stored data.
This commit changes the name of missed message email tests for
personal and huddle messages to be more clear:
- from *_personal_missed_stream_messages to *_missed_personal_messages
- from *_huddle_missed_stream_messages to *_missed_huddle_messages
We add quote prefix ">" to each line of the message in the plain text
missed message emails, which are then rendered as quotes by email
clients. We also move the message content in the next line after sender.
This helps us in clearly showing the message authors in missed message
emails especially in emails with multiple messages and senders.
Fixes#15836.
This ValueError had no test coverage, because the code path wasn't
actually possible with how the caller is constructed.
Rather than writing a highly artificial test for this as proposed in
This also allows us to remove some assertions as we now know that
AVATAR_SALT will never be None.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
This implements get_mandatory_secret that ensures SHARED_SECRET is
set when we hit zerver.decorator.authenticate_notify. To avoid getting
ZulipSettingsError when setting up the secrets, we set an environment
variable DISABLE_MANDATORY_SECRET_CHECK to skip the check and default
its value to an empty string.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
Due to mismatches between the URL parsers in Python and browsers, it
was possible to hoodwink rewrite_local_links_to_relative into
generating links that browsers would interpret as absolute.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
These characters are not allowed and trying to create a Zulip message
with those characters throws a JsonableError in check_stream_topic.
We don't want to reject emails with those chars in the subject, so
it's best to just modify it appropriately.
This commit checks for null values for keys within "attachment" in
the Slack integration's incoming payloads. These keys were expected
to exist optionally previously, and the existence of null values for
these wasn't anticipated. Due to an issue report for such null
values in the payload, their handling is updated appropriately.
The checks for these values are truthiness checks since the strategy
for these values being null or falsy ("", 0) is the same; we don't
process that key-value pair. This is consistent with how Slack handles
this scenario.
For the case where all the attachment fields have null values, Slack
displays this as an empty block with no content, and therefore our
strategy for this is a no-op.
Tests updated.
Since this decorator is only used for methods of
TestServiceBotEventTriggers, we can type the decorated method's
signature accurately without using ParamSpec.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
We can express the type of these decorators with Concatenate and ParamSpec
now for tighter type annotations.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
This removes ViewFuncT and all the associated type casts with ParamSpec
and Concatenate. This provides more accurate type annotation for
decorators at the cost of making the concatenated parameters
positional-only. This change does not intend to introduce any other
behavioral difference. Note that we retype args in process_view as
List[object] because the view functions can not only be called with
arguments of type str.
Note that the first argument of rest_dispatch needs to be made
positional-only because of the presence of **kwargs.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
This module was originally introduced in 2016 to assist adding mypy
annotations to the project. Back then static type checking was not that
established throughout the codebase, so it was helpful to be able to
print out the types for type checking purposes.
This workflow is no longer helpful for improving type annotations right
now, and it has been unused for a while.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
Whether we sent a resolve topic notification or not may be useful in
the caller. It was originally intended to be used in #21712, but may
only be relevant for future logging.
Part of #21712.
This will have no real effect in most situations. However, a user
moves a topic to another stream while also adding/removing the
resolved-topic checkmark from the topic name, then the "This topic was
resolved" notificaiton will now appear just before the "This topic was
moved" notification rather than just after.
This is likely slightly less confusing to users, since the topic
having been moved from somewhere else is likely the most salient fact
to a reader.
We expect to change things to not send both notifications in an
upcoming commit.
This refactoring helps with #21712.
Our seat count calculation is different for guest user than normal users
(a number of initial guests are free, and additional marginal guests are
worth 1/5 of a seat) - so these checks we apply when a user is being
invited or signing up need to know whether it's a guest or non-guest
being added.
Because rate_limit_request_by_ip is the only caller of it, it is safe
for us to inline RateLimitedIpAddr and remove this helper. This ensures
that we have consistent internals for rate limiting functions, which all
have a should_rate_limit check.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
This allows us to use them with HttpResponse objects returned by
calling a view function directly.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
This change incorporate should_rate_limit into rate_limit_user and
rate_limit_request_by_ip. Note a slight behavior change to other callers
to rate_limit_request_by_ip is made as we now check if the client is
eligible to be exempted from rate limiting now, which was previously
only done as a part of zerver.lib.rate_limiter.rate_limit.
Now we mock zerver.lib.rate_limiter.RateLimitedUser instead of
zerver.decorator.rate_limit_user in
zerver.tests.test_decorators.RateLimitTestCase, because rate_limit_user
will always be called but rate limit only happens the should_rate_limit
check passes;
we can continue to mock zerver.lib.rate_limiter.rate_limit_ip, because the
decorated view functions call rate_limit_request_by_ip that calls
rate_limit_ip when the should_rate_limit check passes.
We need to mock zerver.decorator.rate_limit_user for SkipRateLimitingTest
now because rate_limit has been removed. We don't need to mock
RateLimitedUser in this case because we are only verifying that
the skip_rate_limiting flag works.
To ensure coverage in add_logging_data, a new test case is added to use
a web_public_view (which decorates the view function with
add_logging_data) with a new flag to check_rate_limit_public_or_user_views.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
This allows us to avoid importing from zilencer conditionally in
zerver.lib.rate_limiter, as we make rate limiting self-contained now.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
- RateLimitTestCase.get_ratelimited_view is replaced by a view
function directly decorated by public_json_view.
- the META dict is initialized with "PATH_INFO": "test" because now the
tests cover the process_client codepath;
- HostRequestMock is initialized with host="zulip.testserver" to pass
the validate_account_and_subdomain check;
- check_rate_limit_public_or_user_views replaces both
test_rate_limiting_happens_in_normal_case and
test_rate_limiting_happens_by_ip_if_unauthed.
Overall, we deduplicate the test cases in this change, and make sure
that they also cover the view function decorators for authentication.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
The test setup for some of the test cases are largely similar, so it
would be cleaner to be able to reuse them.
Note that we use "check" in the name of this helper because later we
will extend it to take a flag to set whether rate limiting is expected.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
This refactors the test case alongside, since normal views accessed by
remote server do not get rate limited by remote server anymore.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
In Zulip 2.1.0, the `is_muted` stream subscription property was
added and replaced the `in_home_view` property. But the server has
still only been sending subscription update events with the
`in_home_view` property.
Updates `do_change_subscription_property` to send a subscription
update event for both `is_muted` and `in_home_view`, so that
clients can fully migrate away from using `in_home_view` allowing
us to eventually remove it completely.
This is a follow-up to #19274. We map the supported event types to a
more suitable format for events.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
By replacing Any with object we enforce type narrowing before using the
kwargs when a more specific type is required.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
The only caller that passes the kwargs argument is the avatar rest_path.
The application of kwargs can be rewritten with a wrapper.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
Mypy considers that "Tuple[Any, ...]" is incompatible with
"Union[Tuple[Callable[..., HttpResponse], Set[str]], HttpResponse]".
handler, view_flags = entry is sufficient to suppress the error, but we
also add assertions for full measure.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
Commit b945aa3443 (#22604) incorrectly
assumed that Django would run the extra EmailField validators if basic
email address validation passed. Actually, it runs all validators
unconditionally and collects all failures. So email_is_not_disposable
needs to catch email address parsing errors.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This allows us to separate the zilencer paths from other JSON paths,
with explicit type annotation expecting `RemoteZulipServer` as the
second parameter of the handler using
authenticated_remote_server_view.
The test case is also updated to remove a test for a situation that no
longer occurs anymore, since we don't perform subdomain checks on
remote servers.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
As noted in the docstring, this is a temporary helper function that
separates routing for paths that support multiple HTTP methods from
`rest_dispatch` itself. We will need to replace this helper with
class-based views in the future. The helper will also be handy to
reduce duplication when splitting up `rest_dispatch` by authentication
methods.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
This is a prep commit such that we can avoid duplicate code when we
unsubscribe bots for inaccessible private streams when changing owner
or reactivating them.
This commit changes the code to consider zero as an invalid value for
message_content_edit_time_limit_seconds. Now to represent the setting that
user can edit the message anytime, the setting value will be "None" in
database and "unlimited" will be passed to API from clients.
We sent the "message_content_delete_limit_seconds"
value to API when we change any setting in that
subsection and thus "do_set_realm_property" is
called even when we do not change that setting
since we handle it separately from most of the
other settings for which we use loop to call
"do_set_realm_property".
This commit changes the code to handle only
the case when the setting value is "None"
separately and rest all the values will be
changed from the "do_set_realm_property"
called in the loop as for other settings.
mypy_django_plugin cannot resolve the type of the custom manager. We add
a type annotation explicitly to skip the type inferrence.
This fixes the following error:
error: Could not resolve manager type for "zerver.models.UserGroup.objects" [django-manager-missing]
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
Technically recipient_id cannot be None when recipient exists. We
actually just want to check if the recipient exists.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
The `message` attirbute doesn't get defined in the class where it is used.
The cleanest solution will be using the `Self` type introduced in PEP
673 and type it as something like `ForeignKey[Union[Self, Combinable],
Self]`.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
This modifies the QueryDict when it is mutable, and assign it to `.POST`
after it is turned immutable, as required by django-stubs for this
attribute.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
Since validators do not affect the database, this migration is a noop.
Removing the migration fails check-database-compatibility. We might
eventually delete it when Django supports a cleaner method for deletion.
TODO:
Remove the type annotation when django-stubs is integrated.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
Doing the dispatch to authenticated_json_view first lets us avoid
messing around with the skip_rate_limiting parameter.
Since rate_limit itself checks user.is_authenticated, there's no
potential downside to doing that check first here.
This refactoring is necessary to separate the expected type annotation
for view functions with different authentication methods. Currently the
signature aren't actually check against view functions because
`rest_path` does not support type checking parameter types, but it will
become useful once we do.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
`authenticated_rest_api_view` and `authenticated_json_view` essentially
remove `UserProfile` from the decorated function.
Note that `authenticated_log_and_execute_json` is removed to avoid
duplicating `ParamT` unnecessarily in the helper.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
This refactors rate limit related functions from `zerver.decorator` to
zerver.lib.rate_limiter.
We conditionally import `RemoteZulipServer`, `RequestNotes`, and
`RateLimitedRemoteZulipServer` to avoid circular dependency.
Most instances of importing these functions from `zerver.decorator` got
updated, with a few exceptions in `zerver.tests.test_decorators`, where
we do want to mock the rate limiting functions imported in
`zerver.decorator`. The same goes with the mocking example in the
"testing-with-django" documentation.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
This refactors `rate_limit` so that we no longer use it as a decorator.
This is a workaround to https://github.com/python/mypy/issues/12909 as
`rate_limit` previous expects different parameters than its callers.
Our approach to test logging handlers also needs to be updated because
the view function is not decorated by `rate_limit`.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
Adds an API endpoint for accessing read receipts for other users, as
well as a modal UI for displaying that information.
Enables the previously merged privacy settings UI for managing whether
a user makes read receipts data available to other users.
Documentation is pending, and we'll likely want to link to the
documentation with help_settings_link once it is complete.
Fixes#3618.
Co-authored-by: Tim Abbott <tabbott@zulip.com>
This commit adds support to change enable_read_receipts
setting through API and also adds the field to response
of "/register" endpoint so that the setting value
is available to clients.
This code is actually a noop (and would be a bug if it wasn't a noop),
because when this runs the server is already initialized, meaning the
internal realm exists and the system bots have been created, so
UserProfile.objects.filter(email=email) is always truthy. Also, system
bots are supposed to live in the internal realm, not in the realm being
imported so this code doesn't make sense currently.
This ensures type safety by not mutating the original queryset values,
that django-stubs to type as a TypedDict without total=False.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
BACKEND_DATABASE_TEMPLATE was introduced in a507a47778.
This setting is only available for the test cases and it is not that
necessary to have it configurable.
We define it as a global variable in zerver.lib.test_fixtures.
This avoids requiring mypy_django_plugin to know the type of
settings.BACKEND_DATABASE_TEMPLATE for type checking purposes, given the fact
that settings.test_extra_settings is not available in production/development
setup.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
We now use EVERYONE_GROUP_NAME instead of writing
the actual group name at multiple places, so that we
can have all the group names coded at one place only.
We now use MEMBERS_GROUP_NAME instead of writing
the actual group name at multiple places, so that we
can have all the group names coded at one place only.
We now use MODERATORS_GROUP_NAME instead of writing
the actual group name at multiple places, so that we
can have all the group names coded at one place only.
We now use ADMINISTRATORS_GROUP_NAME instead of writing
the actual group name at multiple places, so that we can
have all the group names coded at one place only.
We now use OWNERS_GROUP_NAME instead of writing
the actual group name at multiple places, so that
we can have all the group names coded at one place
only.
We now use EVERYONE_ON_INTERNET_GROUP_NAME instead of
writing the actual group name at multiple places, so
that we can have all the group names coded at one place
only.
We now use FULL_MEMBERS_GROUP_NAME instead of
writing the actual full members system group
name at multiple places, so that we can have
all the group names coded at one place only.
There may be some internal realms which were created after applying
"0382_create_role_based_system_groups.py" migration and this migration
is used to create system groups for those realms.
This commit modifies bulk_create_users to add the users to the
respective system groups. And due to this change, now bots in
development environment are also added to system groups.
Tests are changed accordingly as more UserGroupMembeship objects
are created.
Since we include internal realms while creating system groups
in "0382_create_role_based_system_groups.py", we should do it
when creating new internal realms as well to be consistent.
Tests are changed accordingly as UserGroup objects are created.
We also change the user group ids used in api docs examples
such that user groups are of correct realm.
I found the previous model for computing what settings to use for
streams increasingly difficult to understand, which is generally a
recipe for future bugs.
Refactor to have a clear computation of what complete permissions
state the client is requesting, validate that state, and then pass
that state to the do_change_stream_permission.
We now allow changing access to history of the stream by only passing
"history_public_to_subscribers" parameter. Previously, "is_private"
parameter was also required to change history_public_to_subscribers
otherwise the request was silently ignored.
We also raise error when only history_public_to_subscribers parameter
is passed with value False without "is_private: True" for a public
or web-public stream since we do not allow public streams with
protected history.
We raise error when we try to change a public stream (except for
zephyr mirror realms) to be public with protected history, as we do
not support such streams yet.
Previously, in such case we changed nothing and a notification was
sent to the "stream events" topic with message being "stream is
changed from public to public" and was weird.
Note that this commit only handles the case when both is_private and
history_public_to_subscribers parameters are passed to API and commit
not covers the case when only "history_public_to_subscribers" with
value False is passed to API, since we currently ignore requests
which has only history_public_to_subscribers parameter with not None
and not is_private and is_web_public.
We would do this in further commits when we add support for accepting
only history_public_to_subscribers parameter.
This commit removes the unnecessary assertion statements in
do_change_stream_permission for case when "is_web_public" is
True, since we already check those cases in the view function
update_stream_backend and this is the only place from where
do_change_stream_permission is called.
We aim to remove other assertions also from there as mentioned
in the comment and instead check the values in caller itself.
This is a prep commit for changing do_change_stream_permission
to require passing all (invite_only, history_public_to_subscribers
and is_web_public) arguments in further commits.
During a brief period while a custom emoji is being uploaded, it could
be visible to clients even though it was still in the process of being
uploaded, an operation that can fail.
In certain cases, we call `RealmEmoji.save()` before the filename
becomes available. This result in getting invalid urls generated and
flushed. Normally we call it again shortly after, making it harder to
trigger this bug.
Fixes#22552.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
We do not allow changing enable_spectator_access to True using API on
limited plan realms. Frontend changes have been done previously.
This is a follow-up of #22179.
It is apparently possible to have a mention of a user who is not (or
no longer?) in the `users.bson` table.
Skip such mention for the purposes of Zulip import; there's nothing
better for us to do.
This is likely an error somewhere in rocketchat's MongoDB "eventual
consistency," but there is no problem with skipping the chunks at this
step.
In the one case where this was observed so far, the upload-id was not
referenced in any message -- if it is referenced and has chunks, but
has no metadata, we will fail later, at that reference.
This `mimetype` parameter was introduced in c4fa29a and its last
usage removed in 5bab2a3. This parameter was undocumented in the
OpenAPI endpoint documentation for `/user_uploads`, therefore
there shouldn't be client implementations that rely on it's
presence.
Removes the `request.GET` call for the `mimetype` parameter and
replaces it by getting the `content_type` value from the file,
which is an instance of Django's `UploadedFile` class and stores
that file metadata as a property.
If that returns `None` or an empty string, then we try to guess
the `content_type` from the filename, which is the same as the
previous behaviour when `mimetype` was `None` (which we assume
has been true since it's usage was removed; see above).
If unable to guess the `content_type` from the filename, we now
fallback to "application/octet-stream", instead of an empty string
or `None` value.
Also, removes the specific test written for having `mimetype` as
a url parameter in the request, and replaces it with a test that
covers when we try to guess `content_type` from the filename.
Updates `json_change_settings` so that the default value for the `email`,
`full_name`, `new_password` and `old_password` parameters is `None` instead
of an empty string, which also makes the type annotation `Optional[str]`.
Also, updates tests for email and full name changes to include an empty
string as one of the tested invalid values.
Because Django's ContentType objects are, by default, created lazily
when an actual object is created that will use them, this migration
would fail on any server that actually had RealmReactivationStatus
objects already, and had not yet created the ContentType for them.
ContentType objects are very simple:
zulip=> select * from django_content_type where model = 'realmreactivationstatus';
id | app_label | model
----+-----------+-------------------------
85 | zerver | realmreactivationstatus
So we can simply patch this by using get_or_create.
Now the following characters are allowed before @-mentions and stream
references (starting with #) for proper rendering - {, [, /.
This commit makes the markdown rendering consistent with autocomplete
(anything that is autocompleted is also rendered properly).
Initializing a dictionary from an iterable requires the each item to be
a tuple containg a key and a value. `mypy_django_plugin` cannot infer
the number of items in an queryset with annotated values, so we have to
explicitly unpack each row with a dictionary comprehension here.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
With the new signature of has_request_variables, we can now use
`HttpResponseBase` as the return type of the decorated function.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
Decorators like `require_server_admin_api` turns user_profile into a
positional-only parameter, requiring the callers to stop passing it as a
keyword argument.
Functions like `get_chart_data` that gets decorated by both
`require_non_guest_user` and `has_request_variables` now have accurate
type annotation during type checking, with the first two parameters
turned into positional-only, and thus the change in
`analytics.views.stats`.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
This makes `has_request_variables` more generic, in the sense of the return
value, and also makes it more accurate, in the sense of requiring the
first parameter of the decorated function to be `HttpRequest`, and
preserving the function signature without using `cast`.
This affects some callers of `has_request_variables` or the callers of its
decoratedfunctions in the following manners:
- Decorated non-view functions called directly in other functions cannot
use `request` as a keyword argument. Becasue `Concatenate` turns the
concatenated parameters (`request: HttpRequest` in this case) into
positional-only parameters. Callers of `get_chart_data` are thus
refactored.
- Functions to be decorated that accept variadic keyword arguments must
define `request: HttpRequest` as positional-only. Mypy in strict mode
rejects such functions otherwise because it is possible for the caller to
pass a keyword argument that has the same name as `request` for `**kwargs`.
No defining `request: HttpRequest` as positional-only breaks type safety
because function with positional-or-keyword parameters cannot be considered
a subtype of a function with the same parameters in which some of them are
positional-only.
Consider `f(x: int, /, **kwargs: object) -> int` and `g(x: int,
**kwargs: object) -> int`. `f(12, x="asd")` is valid but `g(12, x="asd")`
is not.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
This adds an assertion ensuring the type of `store` before accessing the
`cache_key` attribute that does not exist in the base class. Also note
that `.decode` returns `Dict[str, Any]` instead of a `str`.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
We now send a new user_topic event while muting and unmuting topics.
fetch_initial_state_data now returns an additional user_topics array to
the client that will maintain the user-topic relationship data.
This will support any future addition of new features to modify the
relationship between a user-topic pair.
This commit adds the relevent backend code and schema for the new
event.
Anders Kaseorg
Yogesh Sirsat
Zixuan James Li
Lauryn Menard
Mateusz Mandera
rht
Jason Hildebrand
Sahil Batra
Tim Abbott
Florian Pritz
David Rosa
Raghav Luthra
Christopher Chong
Matt Keller
madrix01
Hari Prashant Bhimaraju
Alya Abbott
Khushiyant
Aman Agrawal
Alex Vandiver
evykassirer
sahil839
Adam Sah
Israel Ebonko
Julia Bichler
Kartik Srivastava
Dinesh
N-Shar-ma