This allows us to avoid importing from zilencer conditionally in
zerver.lib.rate_limiter, as we make rate limiting self-contained now.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
- RateLimitTestCase.get_ratelimited_view is replaced by a view
function directly decorated by public_json_view.
- the META dict is initialized with "PATH_INFO": "test" because now the
tests cover the process_client codepath;
- HostRequestMock is initialized with host="zulip.testserver" to pass
the validate_account_and_subdomain check;
- check_rate_limit_public_or_user_views replaces both
test_rate_limiting_happens_in_normal_case and
test_rate_limiting_happens_by_ip_if_unauthed.
Overall, we deduplicate the test cases in this change, and make sure
that they also cover the view function decorators for authentication.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
The test setup for some of the test cases are largely similar, so it
would be cleaner to be able to reuse them.
Note that we use "check" in the name of this helper because later we
will extend it to take a flag to set whether rate limiting is expected.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
This refactors the test case alongside, since normal views accessed by
remote server do not get rate limited by remote server anymore.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
In Zulip 2.1.0, the `is_muted` stream subscription property was
added and replaced the `in_home_view` property. But the server has
still only been sending subscription update events with the
`in_home_view` property.
Updates `do_change_subscription_property` to send a subscription
update event for both `is_muted` and `in_home_view`, so that
clients can fully migrate away from using `in_home_view` allowing
us to eventually remove it completely.
This is a follow-up to #19274. We map the supported event types to a
more suitable format for events.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
By replacing Any with object we enforce type narrowing before using the
kwargs when a more specific type is required.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
The only caller that passes the kwargs argument is the avatar rest_path.
The application of kwargs can be rewritten with a wrapper.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
Mypy considers that "Tuple[Any, ...]" is incompatible with
"Union[Tuple[Callable[..., HttpResponse], Set[str]], HttpResponse]".
handler, view_flags = entry is sufficient to suppress the error, but we
also add assertions for full measure.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
Commit b945aa3443 (#22604) incorrectly
assumed that Django would run the extra EmailField validators if basic
email address validation passed. Actually, it runs all validators
unconditionally and collects all failures. So email_is_not_disposable
needs to catch email address parsing errors.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This allows us to separate the zilencer paths from other JSON paths,
with explicit type annotation expecting `RemoteZulipServer` as the
second parameter of the handler using
authenticated_remote_server_view.
The test case is also updated to remove a test for a situation that no
longer occurs anymore, since we don't perform subdomain checks on
remote servers.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
As noted in the docstring, this is a temporary helper function that
separates routing for paths that support multiple HTTP methods from
`rest_dispatch` itself. We will need to replace this helper with
class-based views in the future. The helper will also be handy to
reduce duplication when splitting up `rest_dispatch` by authentication
methods.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
This is a prep commit such that we can avoid duplicate code when we
unsubscribe bots for inaccessible private streams when changing owner
or reactivating them.
This commit changes the code to consider zero as an invalid value for
message_content_edit_time_limit_seconds. Now to represent the setting that
user can edit the message anytime, the setting value will be "None" in
database and "unlimited" will be passed to API from clients.
We sent the "message_content_delete_limit_seconds"
value to API when we change any setting in that
subsection and thus "do_set_realm_property" is
called even when we do not change that setting
since we handle it separately from most of the
other settings for which we use loop to call
"do_set_realm_property".
This commit changes the code to handle only
the case when the setting value is "None"
separately and rest all the values will be
changed from the "do_set_realm_property"
called in the loop as for other settings.
mypy_django_plugin cannot resolve the type of the custom manager. We add
a type annotation explicitly to skip the type inferrence.
This fixes the following error:
error: Could not resolve manager type for "zerver.models.UserGroup.objects" [django-manager-missing]
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
Technically recipient_id cannot be None when recipient exists. We
actually just want to check if the recipient exists.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
The `message` attirbute doesn't get defined in the class where it is used.
The cleanest solution will be using the `Self` type introduced in PEP
673 and type it as something like `ForeignKey[Union[Self, Combinable],
Self]`.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
This modifies the QueryDict when it is mutable, and assign it to `.POST`
after it is turned immutable, as required by django-stubs for this
attribute.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
Since validators do not affect the database, this migration is a noop.
Removing the migration fails check-database-compatibility. We might
eventually delete it when Django supports a cleaner method for deletion.
TODO:
Remove the type annotation when django-stubs is integrated.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
Doing the dispatch to authenticated_json_view first lets us avoid
messing around with the skip_rate_limiting parameter.
Since rate_limit itself checks user.is_authenticated, there's no
potential downside to doing that check first here.
This refactoring is necessary to separate the expected type annotation
for view functions with different authentication methods. Currently the
signature aren't actually check against view functions because
`rest_path` does not support type checking parameter types, but it will
become useful once we do.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
`authenticated_rest_api_view` and `authenticated_json_view` essentially
remove `UserProfile` from the decorated function.
Note that `authenticated_log_and_execute_json` is removed to avoid
duplicating `ParamT` unnecessarily in the helper.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
This refactors rate limit related functions from `zerver.decorator` to
zerver.lib.rate_limiter.
We conditionally import `RemoteZulipServer`, `RequestNotes`, and
`RateLimitedRemoteZulipServer` to avoid circular dependency.
Most instances of importing these functions from `zerver.decorator` got
updated, with a few exceptions in `zerver.tests.test_decorators`, where
we do want to mock the rate limiting functions imported in
`zerver.decorator`. The same goes with the mocking example in the
"testing-with-django" documentation.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
This refactors `rate_limit` so that we no longer use it as a decorator.
This is a workaround to https://github.com/python/mypy/issues/12909 as
`rate_limit` previous expects different parameters than its callers.
Our approach to test logging handlers also needs to be updated because
the view function is not decorated by `rate_limit`.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
Adds an API endpoint for accessing read receipts for other users, as
well as a modal UI for displaying that information.
Enables the previously merged privacy settings UI for managing whether
a user makes read receipts data available to other users.
Documentation is pending, and we'll likely want to link to the
documentation with help_settings_link once it is complete.
Fixes#3618.
Co-authored-by: Tim Abbott <tabbott@zulip.com>
This commit adds support to change enable_read_receipts
setting through API and also adds the field to response
of "/register" endpoint so that the setting value
is available to clients.
This code is actually a noop (and would be a bug if it wasn't a noop),
because when this runs the server is already initialized, meaning the
internal realm exists and the system bots have been created, so
UserProfile.objects.filter(email=email) is always truthy. Also, system
bots are supposed to live in the internal realm, not in the realm being
imported so this code doesn't make sense currently.
This ensures type safety by not mutating the original queryset values,
that django-stubs to type as a TypedDict without total=False.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
BACKEND_DATABASE_TEMPLATE was introduced in a507a47778.
This setting is only available for the test cases and it is not that
necessary to have it configurable.
We define it as a global variable in zerver.lib.test_fixtures.
This avoids requiring mypy_django_plugin to know the type of
settings.BACKEND_DATABASE_TEMPLATE for type checking purposes, given the fact
that settings.test_extra_settings is not available in production/development
setup.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
We now use EVERYONE_GROUP_NAME instead of writing
the actual group name at multiple places, so that we
can have all the group names coded at one place only.
We now use MEMBERS_GROUP_NAME instead of writing
the actual group name at multiple places, so that we
can have all the group names coded at one place only.
We now use MODERATORS_GROUP_NAME instead of writing
the actual group name at multiple places, so that we
can have all the group names coded at one place only.
We now use ADMINISTRATORS_GROUP_NAME instead of writing
the actual group name at multiple places, so that we can
have all the group names coded at one place only.
We now use OWNERS_GROUP_NAME instead of writing
the actual group name at multiple places, so that
we can have all the group names coded at one place
only.
We now use EVERYONE_ON_INTERNET_GROUP_NAME instead of
writing the actual group name at multiple places, so
that we can have all the group names coded at one place
only.
We now use FULL_MEMBERS_GROUP_NAME instead of
writing the actual full members system group
name at multiple places, so that we can have
all the group names coded at one place only.
There may be some internal realms which were created after applying
"0382_create_role_based_system_groups.py" migration and this migration
is used to create system groups for those realms.
This commit modifies bulk_create_users to add the users to the
respective system groups. And due to this change, now bots in
development environment are also added to system groups.
Tests are changed accordingly as more UserGroupMembeship objects
are created.
Since we include internal realms while creating system groups
in "0382_create_role_based_system_groups.py", we should do it
when creating new internal realms as well to be consistent.
Tests are changed accordingly as UserGroup objects are created.
We also change the user group ids used in api docs examples
such that user groups are of correct realm.
I found the previous model for computing what settings to use for
streams increasingly difficult to understand, which is generally a
recipe for future bugs.
Refactor to have a clear computation of what complete permissions
state the client is requesting, validate that state, and then pass
that state to the do_change_stream_permission.
We now allow changing access to history of the stream by only passing
"history_public_to_subscribers" parameter. Previously, "is_private"
parameter was also required to change history_public_to_subscribers
otherwise the request was silently ignored.
We also raise error when only history_public_to_subscribers parameter
is passed with value False without "is_private: True" for a public
or web-public stream since we do not allow public streams with
protected history.
We raise error when we try to change a public stream (except for
zephyr mirror realms) to be public with protected history, as we do
not support such streams yet.
Previously, in such case we changed nothing and a notification was
sent to the "stream events" topic with message being "stream is
changed from public to public" and was weird.
Note that this commit only handles the case when both is_private and
history_public_to_subscribers parameters are passed to API and commit
not covers the case when only "history_public_to_subscribers" with
value False is passed to API, since we currently ignore requests
which has only history_public_to_subscribers parameter with not None
and not is_private and is_web_public.
We would do this in further commits when we add support for accepting
only history_public_to_subscribers parameter.
This commit removes the unnecessary assertion statements in
do_change_stream_permission for case when "is_web_public" is
True, since we already check those cases in the view function
update_stream_backend and this is the only place from where
do_change_stream_permission is called.
We aim to remove other assertions also from there as mentioned
in the comment and instead check the values in caller itself.
This is a prep commit for changing do_change_stream_permission
to require passing all (invite_only, history_public_to_subscribers
and is_web_public) arguments in further commits.
During a brief period while a custom emoji is being uploaded, it could
be visible to clients even though it was still in the process of being
uploaded, an operation that can fail.
In certain cases, we call `RealmEmoji.save()` before the filename
becomes available. This result in getting invalid urls generated and
flushed. Normally we call it again shortly after, making it harder to
trigger this bug.
Fixes#22552.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
We do not allow changing enable_spectator_access to True using API on
limited plan realms. Frontend changes have been done previously.
This is a follow-up of #22179.
It is apparently possible to have a mention of a user who is not (or
no longer?) in the `users.bson` table.
Skip such mention for the purposes of Zulip import; there's nothing
better for us to do.
This is likely an error somewhere in rocketchat's MongoDB "eventual
consistency," but there is no problem with skipping the chunks at this
step.
In the one case where this was observed so far, the upload-id was not
referenced in any message -- if it is referenced and has chunks, but
has no metadata, we will fail later, at that reference.
This `mimetype` parameter was introduced in c4fa29a and its last
usage removed in 5bab2a3. This parameter was undocumented in the
OpenAPI endpoint documentation for `/user_uploads`, therefore
there shouldn't be client implementations that rely on it's
presence.
Removes the `request.GET` call for the `mimetype` parameter and
replaces it by getting the `content_type` value from the file,
which is an instance of Django's `UploadedFile` class and stores
that file metadata as a property.
If that returns `None` or an empty string, then we try to guess
the `content_type` from the filename, which is the same as the
previous behaviour when `mimetype` was `None` (which we assume
has been true since it's usage was removed; see above).
If unable to guess the `content_type` from the filename, we now
fallback to "application/octet-stream", instead of an empty string
or `None` value.
Also, removes the specific test written for having `mimetype` as
a url parameter in the request, and replaces it with a test that
covers when we try to guess `content_type` from the filename.
Updates `json_change_settings` so that the default value for the `email`,
`full_name`, `new_password` and `old_password` parameters is `None` instead
of an empty string, which also makes the type annotation `Optional[str]`.
Also, updates tests for email and full name changes to include an empty
string as one of the tested invalid values.
Because Django's ContentType objects are, by default, created lazily
when an actual object is created that will use them, this migration
would fail on any server that actually had RealmReactivationStatus
objects already, and had not yet created the ContentType for them.
ContentType objects are very simple:
zulip=> select * from django_content_type where model = 'realmreactivationstatus';
id | app_label | model
----+-----------+-------------------------
85 | zerver | realmreactivationstatus
So we can simply patch this by using get_or_create.
Now the following characters are allowed before @-mentions and stream
references (starting with #) for proper rendering - {, [, /.
This commit makes the markdown rendering consistent with autocomplete
(anything that is autocompleted is also rendered properly).
Initializing a dictionary from an iterable requires the each item to be
a tuple containg a key and a value. `mypy_django_plugin` cannot infer
the number of items in an queryset with annotated values, so we have to
explicitly unpack each row with a dictionary comprehension here.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
With the new signature of has_request_variables, we can now use
`HttpResponseBase` as the return type of the decorated function.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
Decorators like `require_server_admin_api` turns user_profile into a
positional-only parameter, requiring the callers to stop passing it as a
keyword argument.
Functions like `get_chart_data` that gets decorated by both
`require_non_guest_user` and `has_request_variables` now have accurate
type annotation during type checking, with the first two parameters
turned into positional-only, and thus the change in
`analytics.views.stats`.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
This makes `has_request_variables` more generic, in the sense of the return
value, and also makes it more accurate, in the sense of requiring the
first parameter of the decorated function to be `HttpRequest`, and
preserving the function signature without using `cast`.
This affects some callers of `has_request_variables` or the callers of its
decoratedfunctions in the following manners:
- Decorated non-view functions called directly in other functions cannot
use `request` as a keyword argument. Becasue `Concatenate` turns the
concatenated parameters (`request: HttpRequest` in this case) into
positional-only parameters. Callers of `get_chart_data` are thus
refactored.
- Functions to be decorated that accept variadic keyword arguments must
define `request: HttpRequest` as positional-only. Mypy in strict mode
rejects such functions otherwise because it is possible for the caller to
pass a keyword argument that has the same name as `request` for `**kwargs`.
No defining `request: HttpRequest` as positional-only breaks type safety
because function with positional-or-keyword parameters cannot be considered
a subtype of a function with the same parameters in which some of them are
positional-only.
Consider `f(x: int, /, **kwargs: object) -> int` and `g(x: int,
**kwargs: object) -> int`. `f(12, x="asd")` is valid but `g(12, x="asd")`
is not.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
This adds an assertion ensuring the type of `store` before accessing the
`cache_key` attribute that does not exist in the base class. Also note
that `.decode` returns `Dict[str, Any]` instead of a `str`.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
We now send a new user_topic event while muting and unmuting topics.
fetch_initial_state_data now returns an additional user_topics array to
the client that will maintain the user-topic relationship data.
This will support any future addition of new features to modify the
relationship between a user-topic pair.
This commit adds the relevent backend code and schema for the new
event.
Updates documentation to include information about user presence
objects with `aggregated` key (instead of the user's email) where
appropriate.
Also, cleans up spelling, grammar and formatting errors in the
descriptive text for these objects / endpoints.
`render_table` calls itself recursively when it finds nested
`additionalProperties` (i.e. nested objects) in response schema,
to render their properties.
This fixes `render_table` to call `render_desc` along with
calling itself, to render the description of the nested
`additionalProperties` as well.
We separate the permission checks for content and topic edits
by changing the can_edit_topic_or_content to can_edit_topic
and use it only for checking topic edits and check content
edits separately in check_update_message itself. There is no
change in behavior as of this PR, there will be more changes
as per #21739.
This is a prep commit for #21739. The permission checks for
them are essentially separate except the one that message
sender is allowed to edit content and topic irresepctive of
edit_topic_policy setting, and this will too be changed in
future commit and so it will be better to have these checks
separate for readability.
We can also probably create a new function for checking content
edits but currently we only check the sender is same as the use
who is editing and it does not make sense to have a separate
function for just one check. We can do so in future in case we
do some more refactoring for #21739.
The presence of `auto_signup` in idp_settings_dict in the test case
test_social_auth_registration_auto_signup is incompatible with the
previous type annotation of SOCIAL_AUTH_OIDC_ENABLED_IDPS, where `bool`
is not allowed.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
In 3b4f8cc85b,
we added support to `auto_signup`, but this field was not defined in
`SAMLIdPConfigDict`, causing mypy type error in
`SAMLAuthenBackendTest.test_social_auth_registration_auto_signup`.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
We construct model instances in the import tool solely for the purpose
of serializing them with the `model_to_dict` helper that returns a
dictionary. Passing `float` to these models' DateTimeField is not
accepted by the type checker. Modifying the dictionary instead avoids
this typing issue.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
We pass the realm_emoji dict to notify_realm_emoji instead
of computing it in notify_realm_emoji itself, because its
caller functions will require the dict in further commits
for creating RealmAuditLog entries.
This commit also renames the legacy event variable to
legacy_event from event because same event is sent
twice if we keep only single variable and change its
value. This was happening because the event value was
changed when the first send_event was actually executed.
We pass the realm_linkfiers list to notify_linkifiers instead of
computing it in notify_linkifiers itself, because its caller
functions will require the list in further commits for creating
RealmAuditLog entries.
Since we not allow enabling public access on limited plan realms,
we set the enable_spectator_access setting to False when downgrading
to a limited plan. Setting is still shown in the UI but it is
disabled.
This commit adds code to send stream creation and peer add events
when stream is changed from private to public. These events are
only sent to users who are not susbcribed to the stream and are
not realm admins as subscribers and realm admins already have
the stream data. This will update the stream data with clients
and will remove the need to reload to view the modified stream.
Fixes#22194.
`ParamSpec` can be easily applied to many use cases of ViewFuncT with
`Concatenate` to help us get rid of the `cast` calls. This does not
include decorators that require the second argument being
`UserProfile`.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
As we refactor this, any decorators that `zulip_login_required` depends
on are also refactored to use `ParamSpec`.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
We should not monkey-patch message when unnecessary. Adding
`service_queue_events` to `SendMessageRequests` suits our need to type
safety here.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
Now that we can assume Python 3.6+, we can use the
email.headerregistry module to replace hacky manual email address
parsing.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
We have already checked the size of the file in `upload_file_backend`.
This is the only caller of `upload_message_image_from_request`, and
indirectly the only caller of `get_file_info`. There is no need to
retrieve this information again.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
It is not possible in the codebase to have request.user be None. But
it is possible to have it not present at all. `delattr` is more
appropriate here.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
This makes the test cleaner and we don't have to overwrite the `get_host`
callable on `HttpRequest`.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
`context` as `AccessDeniedError` is incompatible with
`RequestVariableMissingError`. Mypy does not allow such redefinition.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
`DiscoverRunner.run_tests` has a return type of `int`. While
`Runner.run_tests` has a wildly different `Tuple[bool, List[str]]`.
This refactors it so that we have the correct return type, by passing
the additional information about failed tests through a side effect to directly
write the failed tests to a file.
Note that we have to make `failed_tests_path` optional as otherwise the method
signature will not be compatible with the supertype.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
This reverts part of commit 1432067959
(#17047). The spooky warnings foretold by the comment don’t seem to
show up.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
`context` as `AccessDeniedError` is incompatible with
`RequestVariableMissingError`. Mypy does not allow such redefinition.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
It is really a generator of test cases from the test suite. Which should
be typed as an `Iterable` instead.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
`context_data` is only available on `SimpleTemplateResposne`, we can't
narrow `TestHttpResponse` to it because the latter is not in fact a
subtype of `HttpResponse`.
Differently, `redirect_chain` is an attribute that only appears on the
test response when the test client method is called with `follow=True`.
`TestHttpResponse` does not have that by defalut, either.
The occurence of these two cases are rare enough throughout the codebase
and we can't get around that without aggressively overloading the test client
or refactoring `_MonkeyPatchedWSGIResponse` in the upstream.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
This eliminates the possibility of having `request.user` as
`RemoteZulipServer` by refactoring it as an attribute of `RequestNotes`.
So we can effectively narrow the type of `request.user` by testing
`user.is_authenticated` in most cases (except that of `SCIMClient`) in
code paths that require access to `.format_requestor_for_logs` where we
previously expect either `UserProfile` or `RemoteZulipServer` backed by
the implied polymorphism.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
In b46af40bd3,
we set this attribute because back then we might call `rate_limit_user`
on `RemoteZulipServer`.
This is no longer the case as `RemoteZulipServer` now has its own rate
limiting and we never call `rate_limit_user` without an `isinstance` check
for `UserProfile`.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
This verifies that `request_for_logs` is correctly set for requests
with different types of authentication.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
`BaseNotes(str, str).get_notes` does not do anything here.
It was introduced in
53888e5a26
by unintendedly.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
We can express the same idea more simply by not passing `user` in
cases where it isn't valid for UserActivity.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
Fixes some in-app and documentation references to to new user
announcements and new stream announcements that were still using
'notifications' to refer them. These were missed in the original
pass for updating this language.
A user ran into an issue while upgrading where
ContentType.objects.get(model="realmreactivationstatus",
app_label="zerver") fails due to the object being missing. The reason
for that is to be yet figured out, but the immediate solution is clear
in the sense that the migration can just quit early
if not Confirmation.objects.filter(type=REALM_REACTIVATION).exists() and
that'll effectively skip it for almost all servers (because realm
reactivations links are something that's really only useful on Zulip
Cloud).
We are no longer creating confirmation objects associated with realms
directly. This should test for `RealmReactivationStatus` instead.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
Newrelic updated the payload that's sent via the webhook incoming call
causing a bug in the newrelic webhook endpoint.
This fixes the bug by updating the endpoint to respect the new format
of the payload as well as the old format. This should be updated once
the old format is EOLed.
Fixes#22338.
This makes it mandatory to narrow the type of the user to `UserProfile`
before calling this helper.
This effectively removes the `request.user` check. We do not call login_page
anywhere else without getting through the authentication middleware.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
In the test case `test_check_if_every_integration_has_logo_that_exists`,
`urlsplit(integration.logo_url).path` gets inferred as possibly bytes
because `integration.logo_url` might be `None`.
5598b49851/stdlib/urllib/parse.pyi (L166-L169)
TODO:
We might want to ensure that every integration has a `logo_url` with an
explicit assertion in `Integrations` (as noted in the comment).
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
`expected_draft_contents` would be inferred as a list of mutable
mappings that only allow `int` as the value, and thus incompatible with
the `draft_dicts[i]` to be expanded. This is fixed by adding explicit
type annotation.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
This was added in d43b031a32 and was
unused when it was added. This is an error that we want to remove.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
We likely just wanted to check that `validate_password` succeeds without
any exception being raised. A simple call is sufficient to verify that,
since `validate_password` does not return anything and raises an
exception on failure.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
This check was added in 495a8476be.
Now that django-stubs finds that the left operand of the `and` will
always evaluates to `True`, so it makes sense to remove it.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
`report/error` is a path where we allow anonymous user access. This has
to be correctly denoted in the type annotation of the user argument of
the view function.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
This fixes the type annotations of `Set` derived from `QuerySet` objects,
and add necessary assertions.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
`HostRequestMock` has `user` default to `None`, which later gets
initialized as `AnonymousUser`. The separate initialization here is
unnecessary.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
Previously, we had a function named create_add_users_to_system_user_groups
for creating system user groups and adding users to them in case when
exports do not contain these groups when importing from other services.
This commit just separates out the call to create_system_user_groups_for_realm
outside the function and the function is thus renamed to
add_users_to_system_user_group. This change is done because in further
commits we would need to update the import order and user groups will
be created before creating user profile objects.
This commit extracts whether a stream is accessible or not
in a new function such that "Subscription" object is passed
by the caller and thus we can use these functions to check
access of multiple streams in a loop without querying the
database in a loop for subscription objects.
This commit renames admin_access_required parameter of
list_to_streams function to unsubscribing_others since that
parameter is used and passed as True only when calling
the function while unsubscribing others and in further
commits we would allow non-admins too to unsubscribe others
based on can_remove_subscribers_group setting.
This commit removes the instances of using "Stream.objects.create"
in tests with make_stream function. This change will help us to
avoid adding code for things to be done after creating streams in
multiple places. We can instead just add it in make_stream function
only.
The .status value of EmailChangeStatus was not being looked
at anywhere to prevent re-use of email change confirmation links. This
is not a security issue, since the EmailChangeStatus object has a fixed
value for the new_email, while the confirmation link has expiry time of
1 day, which prevents any reasonable malicious scenarios.
We fix this by making get_object_from_key look at
confirmation.content_object.status - which applies
generally to all confirmations where the attached object has the .status
attribute. This is desired, because we never want to
successfully get_object_from_key an object that has already been used or
reused.
This makes the prereg_user.status check in check_prereg_key redundant so
it can be deleted.
Type inference does not work when the default value of `REQ` is
non-optional while `ResultT` is optional. Mypy tries to unify
`json_validator` with `Validator[int]` in `invite_users_backend` instead
of the desired `Validator[Optional[int]]` because of the presence of the
default value `settings.INVITATION_LINK_VALIDITY_MINUTES`, which is
inferred to be an `int`. Mypy does not resort to a less specific type but
instead gives up early.
This issue applies to invite_users_backend and generate_multiuse_invite_backend
in zerver.views.invite.
There might be a way that we can add an overload to get around this, but
it's probably not worth the complexity until it comes up again more frequently.
We do in fact allow `invite_expires_in_minutes` to be `None` in places
like `do_invite_users`, `invite_users_backend`, etc, and we have
`settings.INVITATION_LINK_VALIDITY_MINUTES` as the default for them. So
it makes sense to allow having an optional value for this setting. And
since there isn't a way to independently set the value of this constant,
we move it to a different place.
TODO:
This is a temporary fix that should be refactored when the bug is fixed.
The encountered mypy issue: https://github.com/python/mypy/issues/13234
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
Iterating over ValidatorError does not necessarily return a tuple. This
uses the `message_dict` property on `ValidationError` instead to make
sure that we always get a `dict` (it otherwise raises an `AttributeError`
when the `dict` is not available).
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
The `RateLimited` exception can be caught by `JsonErrorHandler`, so it
is not necessary to have the try...except statement here. It is also invalid
to pass a string to initialize `RateLimited`.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
`extra_data` as a `TextField` expects a `str`, but we had been passing
`dict` instead. This is a temporary solution before #18391 to fix the
type annotation.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
This adds a `check_string_in` validator to ensure that `op` is actually
valid before we finally return `json_success()`.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
Fixes#21266.
We want to tie the prereg_user to the MultiUseInvite directly rather
than to the MultiUserInvite's confirmation object, because the latter is
not possible. This is because the flow is that after going through the
multiuse invite link, the PreregistrationUser is created together with a
Confirmation object, creating a confirmation link (via
create_confirmation_link) to which then the user is redirected to finish
account creation. This means that the PreregistrationUser is already
tied to a Confirmation, so that attribute is occupied.
A standard OpenAPI document has no reason to redundantly include this
information in description fields, as standard generators already
display it.
This uniformly moves the URL above the description, which seems fine.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
The shared fields of `RawUserInfoDict` and `UserInfoDict` could have
been reused if they both require all keys or none. This is unfortunately
not the case, because subclassing does not override `__total__`.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
Since we in fact are using the django test client to generate a response
here, the return type should be `TestHttpResponse` instead.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
`_callback_str` was removed in Django in 1.10, and other logic relevant
to that particular attribute was removed in
32849b80ad, but not to its entirety. It
does not make sense to fall back to `_callback_str`. The
`get_callback_string` helper is no longer needed.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
On registration and login pages on self-hosted Zulip servers,
it is not helpful and confusing to show the full navigation footer
for the Zulip website. Instead, we should show a minimal footer.
Fixes#21776
Due to an incorrect authorization check in Zulip Server 5.4 and
earlier, a member of an organization could craft an API call that
grants organization administrator privileges to one of their bots.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Previously if `test_forward_address_details` failed, the file
created when setting the `forward_address` may not have been
removed, which would then cause an `EmailNotDeliveredException`
to be raised when then creating a new user in the dev environment.
Wraps the test in a try block, with a finally block for the call
to remove the file.
Before this, a link still couldn't be re-used because it would trip up
exception further down user creation codepaths, but that was still a
bug. check_prereg_key is supposed to correctly validate the key - and
trigger an error page being returned if a key (or for any other reason,
the attached PreregistrationUser object) is reused.
test_validate_email_not_already_in_realm needs to be adjusted, because
it was actually re-using a key.
This reverts commit 40fcf5a633.
This commit triggers bug that we haven't fully tracked down, where web
app clients will continually send `update_message_flags` requests,
that then send out via the events system "0 messages were marked as
read" notices, eventually leading to a load spike.
The Tornado part can likely be fixed by checking if
updated_message_ids is empty, but we need to track down the frontend
bug as well.
`_cache` is not an attribute defined on `BaseCache`, but an
implementation detail of django_bmemcache.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
Two endpoints had linked markdown files that were used in
their general descriptions to add warning notes with important
information (`/create-user` and `/get-user-groups`).
This moves the warning content to be inline in the endpoint
descriptions so that the important content is in the OpenAPI
documentation and is still formatted to be rendered in a warning
block.
Deletes `can-create-users-only.md` and `api-members-only.md`
since they were only used for these two endpoint descriptions.
Also, cleans up the other instance of a inline warning block in
an endpoint description (`/fetch-api-key`).
Instead of using `request.POST` to access the `data` parameter used
in the internal `notify_tornado` path, adds `has_request_variables`
decorator and accesses `data` as a `REQ` parameter.
Expands `test_tornado_endpoint` in `test_event_system.py` for
`data` being a required parameter for this path.
Instead of using `request.POST` to access `forward_address` for
the parameter used in `set_forward_address` in `email_page`, adds
`has_request_variable` decorator and an optional `forward_address`
parameter through the `REQ` framework.
Adds an assertion that `forward_address` is not `None` for `POST`
requests.
Previously, automated stream messages for new user signups were not
being translated into the realm's default language for said messages.
Moves `override_language` context manager so that it wraps the
new user message content in `notify_new_user` and topic string in
`send_message_to_signup_notification_stream`.
Fixes#22510.
The supertype contains `*args` and `**kwargs`, this adapts the signature
of the `get` method to make MarkdownDirectoryView compatible with it.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
The calling arguments here are completely wrong. The first argument
should be `request`, and `self` should never get passed to `.get`.
Because `TemplateView` happened to not use `request`, and we happened
to pass `article` as a keyword argument, this error slipped through.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
Updates changelog entry for feature level 1 about GitLab to include
the endpoint with the changes. Also noted that the change updated
a deprecated return value.
Added changes note to the `gitlab` boolean in the
`authentication_methods` return value for the
`/get-server-settings` endpoint.
Part of work on #22102.
Updates the changelog note in feature level 1 about adding None as
a video call provider to include the endpoints where this realm
setting is used.
Updates the OpenAPI doc for the realm setting `video_chat_provider`
to include information about the enum values and meanings.
Part of work on #22102.
Corrects omissions or inconsistencies between the api changelog
and the api documentation for Zulip 3.0, feature level 1,
except for the final two bullet points about GitLab authentication
and adding None as a video call provider option.
The final two bullet points will be addressed in separate commits.
Part of work on #22102.
Initial round of fixes and clean-ups found during audit of
changelog entries for feature levels 1-27, which correspond
to the 3.0 release.
There are a few changes that are not related to those feature
levels, but fit within the context of clean-ups (spelling mistakes
or errors in api documentation formatting/structure/style).
One notable non-3.0 release fix is making all changes notes in
the OpenAPI documentation for 2.x releases use the correct
version numbering-scheme for those releases (e.g. 2.0.0).
Follow-up commits / PRs will address inconsitencies and omissions
for these feature levels found during the audit.
Updates references / language about organization settings that
were previously labeled as "Notifications", but are now labeled
as "Automated messages and emails".
Fixes#22136.
Co-authored by: Lauryn Menard <lauryn@zulip.com>
PostgreSQL's `default_statistics_target` is used to track how many
"most common values" ("MCVs") for a column when performing an
`ANALYZE`. For `tsvector` columns, the number of values is actually
10x this number, because each row contains multiple values for the
column[1]. The `default_statistics_target` defaults to 100[2], and
Zulip does not adjust this at the server level.
This translates to 1000 entries in the MCV for tsvectors. For
large tables like `zerver_messages`, a too-small value can cause
mis-planned query plans. The query planner assumes that any
entry *not* found in the MCV list is *half* as likely as the
least-likely value in it. If the table is large, and the MCV list is
too short (as 1000 values is for large deployments), arbitrary
no-in-the-MCV words will often be estimated by the query planner to
occur comparatively quite frequently in the index. Based on this, the
planner will instead choose to scan all messages accessible by the
user, filtering by word in tsvector, instead of using the tsvector
index and filtering by being accessible to the user. This results in
degraded performance for word searching.
However, PostgreSQL allows adjustment of this value on a per-column
basis. Add a migration to adjust the value up to 10k for
`search_tsvector` on `zerver_message`, which results in 100k entries
in that MCV list.
PostgreSQL's documentation says[3]:
> Raising the limit might allow more accurate planner estimates to be
> made, particularly for columns with irregular data distributions, at
> the price of consuming more space in `pg_statistic` and slightly
> more time to compute the estimates.
These costs seem adequate for the utility of having better search.
In the event that the pgroonga backend is in use, these larger index
statistics are simply wasted space and `VACUUM` computational time,
but the costs are likely still reasonable -- even 100k values are
dwarfed by the size of the database needed to generate 100k unique
entries in tsvectors.
[1]: https://github.com/postgres/postgres/blob/REL_14_4/src/backend/utils/adt/array_typanalyze.c#L261-L267
[2]: https://www.postgresql.org/docs/14/runtime-config-query.html#GUC-DEFAULT-STATISTICS-TARGET
[3]: https://www.postgresql.org/docs/14/planner-stats.html#id-1.5.13.5.3
In `JsonableErrorHandler`, we convert `MissingAuthenticationError` into
a response that has `WWW-Authenticated` set for `/api` or `/json` views.
This covers and verify the value of the header for unauthenticated
access.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
The dangling comment was not very helpful and unclear about the pieces
of code it was referring to.
We expand the part about linking the prereg_user to the created user,
while the part about "revoking other preregistration users" is
redundant, because the relevant code block lower down already has
comments on it with better explanations.
Closes#22274.
This assertion was added in 4b903c5dcd
where it may have made sense, because indeed when doing realm creation
there was always a PreregistrationUser (created because realms were
created via going to a generated realm creation link). With the addition
of the create_realm command that's no longer the case.
It would be unnatural to create a PreregistrationUser in the
realm_creation command, because there is no confirmation link for it to
be tied to - and it just doesn't make sense conceptually.
The intended, correct behavior added in
4b903c5dcd is still maintained - the code
lower down correctly handles the
(prereg_user is None and realm_creation) case.
The type safety of a TypeGuard is unchecked by mypy. While this
particular TypeGuard is safe given the current context, one could
imagine future changes that make it unsafe, so it’s preferable to
avoid unchecked constructs whenever possible.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
The field_data sent from client while creating a select
type field is a dict with a number as key.
In development database the field data for "Favorite editor"
field was of different form where the option label was used
as key in the dict.
This commit fixes it to be of the same as it is when creating
a field from web-app. As a result, we also need to update
the tests and this commit also update field_data for other
select-type fields.
This refactors the test case with more explicit type annotations, fixing
type errors discovered provided type annotations for
`CustomProfileField`.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
This change ensures that we can call the validate and update helper for
custom profile data later.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
We refactor the validator of `update_user_custom_profile_data` to ensure
that the validated variable is properly typed as
a `ProfileDataElementUpdateDict`, so we can call
`validate_user_custom_profile_data` and
`do_update_user_custom_profile_data_if_changed` directly later (unlike
`update_user_backend`, where `value` is allowed to be `None`, the
validator already ensures that no further check is required).
django-stubs types the return type of query.values(...) as a TypedDict.
This makes Dict[str, Any] that we have been using incompatible with it.
We use TypeGuard to ensure that `service_bot_tuples` is correctly
inferred to be `Tuple[int, int]` instead of `Tuple[int, Optional[int]]`.
Given that `bot_type` is optional for `ActiveUserDict`, we need to
narrow `row` to `ActiveBotUserDict` to make sure that `bot_type` is
non-optional. An advantage of this approach is that no assertions or
type casts are needed.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
Otherwise mypy infers the type of `expected_result` to be incompatible
with the first argument of `fix_ordering_of_result`.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
Zixuan James Li
Lauryn Menard
Kartik Srivastava
Mateusz Mandera
Anders Kaseorg
Adam Sah
Sahil Batra
Tim Abbott
Dinesh
Alex Vandiver
Julia Bichler
N-Shar-ma
Lauryn Menard
rodwan-bakkar
Aman Agrawal
Christopher Chong
ritikBhandari