Commit Graph

6763 Commits

Author SHA1 Message Date
Sahil Batra 15ef0daaf8 user_groups: Add EVERYONE_ON_INTERNET_GROUP_NAME constant.
We now use EVERYONE_ON_INTERNET_GROUP_NAME instead of
writing the actual group name at multiple places, so
that we can have all the group names coded at one place
only.
2022-08-11 04:38:36 -07:00
Sahil Batra 31d639160f user_groups: Add FULL_MEMBERS_GROUP_NAME constant.
We now use FULL_MEMBERS_GROUP_NAME instead of
writing the actual full members system group
name at multiple places, so that we can have
all the group names coded at one place only.
2022-08-11 04:38:36 -07:00
Sahil Batra 150f77aea2 bulk_create: Add users to system user groups in bulk_create_users.
This commit modifies bulk_create_users to add the users to the
respective system groups. And due to this change, now bots in
development environment are also added to system groups.

Tests are changed accordingly as more UserGroupMembeship objects
are created.
2022-08-11 04:38:36 -07:00
Sahil Batra 393afc9781 realms: Create default system user groups for internal realm.
Since we include internal realms while creating system groups
in "0382_create_role_based_system_groups.py", we should do it
when creating new internal realms as well to be consistent.

Tests are changed accordingly as UserGroup objects are created.
We also change the user group ids used in api docs examples
such that user groups are of correct realm.
2022-08-11 04:38:36 -07:00
Mateusz Mandera 9b8d9d038a test_rate_limiter: Make test_add_remove_rule clean up its rules.
This test was polluting the rules for other tests, due to adding rate
limiting rules and not removing them.
2022-08-10 16:30:39 -07:00
Sahil Batra 3e6463804e streams: Allow changing history access without is_private parameter.
We now allow changing access to history of the stream by only passing
"history_public_to_subscribers" parameter. Previously, "is_private"
parameter was also required to change history_public_to_subscribers
otherwise the request was silently ignored.

We also raise error when only history_public_to_subscribers parameter
is passed with value False without "is_private: True" for a public
or web-public stream since we do not allow public streams with
protected history.
2022-08-09 17:05:38 -07:00
Sahil Batra 6ccfebac56 streams: Raise error when making public stream with private history.
We raise error when we try to change a public stream (except for
zephyr mirror realms) to be public with protected history, as we do
not support such streams yet.

Previously, in such case we changed nothing and a notification was
sent to the "stream events" topic with message being "stream is
changed from public to public" and was weird.

Note that this commit only handles the case when both is_private and
history_public_to_subscribers parameters are passed to API and commit
not covers the case when only "history_public_to_subscribers" with
value False is passed to API, since we currently ignore requests
which has only history_public_to_subscribers parameter with not None
and not is_private and is_web_public.
We would do this in further commits when we add support for accepting
only history_public_to_subscribers parameter.
2022-08-09 17:05:38 -07:00
Sahil Batra 75768decf1 tests: Pass all arguments in do_change_stream_permission.
This is a prep commit for changing do_change_stream_permission
to require passing all (invite_only, history_public_to_subscribers
and is_web_public) arguments in further commits.
2022-08-09 17:05:38 -07:00
Kartik Srivastava 887233a8eb api: Document /default_streams API endpoint. 2022-08-09 14:55:27 -07:00
Zixuan James Li 172a166159 cache: Avoid flushing invalid cache for realm emoji.
In certain cases, we call `RealmEmoji.save()` before the filename
becomes available. This result in getting invalid urls generated and
flushed. Normally we call it again shortly after, making it harder to
trigger this bug.

Fixes #22552.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-08-09 14:44:28 -07:00
Sahil Batra 5a29f4133b realm: Don't allow changing enable_spectator_access to True on limited plan.
We do not allow changing enable_spectator_access to True using API on
limited plan realms. Frontend changes have been done previously.

This is a follow-up of #22179.
2022-08-09 11:47:33 -07:00
Lauryn Menard aa796af0a8 upload: Remove `mimetype` url parameter in `get_file_info`.
This `mimetype` parameter was introduced in c4fa29a and its last
usage removed in 5bab2a3. This parameter was undocumented in the
OpenAPI endpoint documentation for `/user_uploads`, therefore
there shouldn't be client implementations that rely on it's
presence.

Removes the `request.GET` call for the `mimetype` parameter and
replaces it by getting the `content_type` value from the file,
which is an instance of Django's `UploadedFile` class and stores
that file metadata as a property.

If that returns `None` or an empty string, then we try to guess
the `content_type` from the filename, which is the same as the
previous behaviour when `mimetype` was `None` (which we assume
has been true since it's usage was removed; see above).

If unable to guess the `content_type` from the filename, we now
fallback to "application/octet-stream", instead of an empty string
or `None` value.

Also, removes the specific test written for having `mimetype` as
a url parameter in the request, and replaces it with a test that
covers when we try to guess `content_type` from the filename.
2022-08-08 16:06:09 -07:00
Lauryn Menard df3b8c590f user-settings: Make default `None` for name, email and password changes.
Updates `json_change_settings` so that the default value for the `email`,
`full_name`, `new_password` and `old_password` parameters is `None` instead
of an empty string, which also makes the type annotation `Optional[str]`.

Also, updates tests for email and full name changes to include an empty
string as one of the tested invalid values.
2022-08-08 15:37:41 -07:00
Julia Bichler 0a278c39d2 settings: Send email after deactivating user.
This adds a feature where an admin can choose to send an email
with custom content to an user after they deactivated them.

Fixes #18943.
2022-08-06 21:41:53 -07:00
N-Shar-ma ef044b8697 markdown: Update characters allowed before @ and stream mentions.
Now the following characters are allowed before @-mentions and stream
references (starting with #) for proper rendering - {, [, /.

This commit makes the markdown rendering consistent with autocomplete
(anything that is autocompleted is also rendered properly).
2022-08-06 19:29:39 -07:00
Kartik Srivastava 1291e7000b user_topic: Add user_topic event.
We now send a new user_topic event while muting and unmuting topics.
fetch_initial_state_data now returns an additional user_topics array to
the client that will maintain the user-topic relationship data.
This will support any future addition of new features to modify the
relationship between a user-topic pair.

This commit adds the relevent backend code and schema for the new
event.
2022-08-04 17:44:00 -07:00
Kartik Srivastava e2760a2bf2 api: Document /realm/presence API endpoint. 2022-08-04 16:34:13 -07:00
Zixuan James Li ae949bce04 test_auth_backends: Return an Iterable for idp names.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-08-03 15:03:15 -07:00
Sahil Batra 9d9252ef08 realm: Create RealmAuditLog entries when removing custom emoji.
We also add acting_user parameter to do_remove_realm_emoji in
this commit, so that we can use it while creating RealmAuditLog
objects.
2022-07-31 18:32:28 -07:00
Sahil Batra 90898cc9c4 realm: Create RealmAuditLog entries when adding custom emoji. 2022-07-31 18:32:28 -07:00
Sahil Batra ef48047dfc realm: Create RealmAuditLog entry when removing realm linkifiers.
This commit also adds 'acting_user' parameter to do_remove_linkifier
function.

Fixes a part of #21268.
2022-07-31 18:32:28 -07:00
Sahil Batra 4d62915912 realm: Create RealmAuditLog entry when updating realm linkifiers.
This commit also adds 'acting_user' parameter to do_update_linkifier
function.

Fixes a part of #21268.
2022-07-31 18:32:28 -07:00
Sahil Batra 4f1dd0710c realm: Create RealmAuditLog entry when adding realm linkifiers.
This commit also adds 'acting_user' parameter to do_add_linkifier
function.

Fixes a part of #21268.
2022-07-31 18:32:28 -07:00
Sahil Batra d2f2fbf6b9 realm: Disable enable_spectator_access on changing to limited plan.
Since we not allow enabling public access on limited plan realms,
we set the enable_spectator_access setting to False when downgrading
to a limited plan. Setting is still shown in the UI but it is
disabled.
2022-07-31 18:02:36 -07:00
Sahil Batra f4fcedd072 streams: Send stream creation event when changing stream to public.
This commit adds code to send stream creation and peer add events
when stream is changed from private to public. These events are
only sent to users who are not susbcribed to the stream and are
not realm admins as subscribers and realm admins already have
the stream data. This will update the stream data with clients
and will remove the need to reload to view the modified stream.

Fixes #22194.
2022-07-31 17:33:53 -07:00
Anders Kaseorg b945aa3443 python: Use a real parser for email addresses.
Now that we can assume Python 3.6+, we can use the
email.headerregistry module to replace hacky manual email address
parsing.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-07-29 15:47:33 -07:00
Zixuan James Li bdee079b0d test_logging_handlers: Use delattr instead to unset request.user.
It is not possible in the codebase to have request.user be None. But
it is possible to have it not present at all. `delattr` is more
appropriate here.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-29 14:07:41 -07:00
Zixuan James Li 16b51ef437 test_logging_handlers: Mock get_host instead of replacing it.
This makes the test cleaner and we don't have to overwrite the `get_host`
callable on `HttpRequest`.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-29 14:07:41 -07:00
Zixuan James Li fd8f049a8d typing: Avoid redefinition of variables with a different type.
`context` as `AccessDeniedError` is incompatible with
`RequestVariableMissingError`. Mypy does not allow such redefinition.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-29 13:59:30 -07:00
Zixuan James Li c86b3fd793 test_decorators: Call process_request as an instance method.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-29 13:59:30 -07:00
Zixuan James Li 04d4c6a453 test_decorators: Avoid redefinition of variables with a different type.
`context` as `AccessDeniedError` is incompatible with
`RequestVariableMissingError`. Mypy does not allow such redefinition.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-28 10:26:32 -07:00
Zixuan James Li ba21925b49 test_signup: Add None check for prereg_user.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-28 10:26:32 -07:00
Zixuan James Li f3b2e2b541 test_email_mirror: Add None check for settings.SHARED_SECRET.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-28 10:26:32 -07:00
Zixuan James Li a996228ced test_auth_backends: Add None check for prereg_user.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-28 10:26:31 -07:00
Zixuan James Li 3ad9876d69 test_signup: Use getattr to access potentially not presenting fields.
`context_data` is only available on `SimpleTemplateResposne`, we can't
narrow `TestHttpResponse` to it because the latter is not in fact a
subtype of `HttpResponse`.

Differently, `redirect_chain` is an attribute that only appears on the
test response when the test client method is called with `follow=True`.
`TestHttpResponse` does not have that by defalut, either.

The occurence of these two cases are rare enough throughout the codebase
and we can't get around that without aggressively overloading the test client
or refactoring `_MonkeyPatchedWSGIResponse` in the upstream.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-28 09:48:16 -07:00
Zixuan James Li 0f416c44d0 test_rate_limiter: Define backend on RateLimiterBackendBase.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-28 09:48:16 -07:00
Zixuan James Li b02779c005 request: Refactor remote_server into RequestNotes.
This eliminates the possibility of having `request.user` as
`RemoteZulipServer` by refactoring it as an attribute of `RequestNotes`.

So we can effectively narrow the type of `request.user` by testing
`user.is_authenticated` in most cases (except that of `SCIMClient`) in
code paths that require access to `.format_requestor_for_logs` where we
previously expect either `UserProfile` or `RemoteZulipServer` backed by
the implied polymorphism.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-28 09:38:40 -07:00
Zixuan James Li 42fddd5a1e test_middlware: Add test coverage for request_for_logs.
This verifies that `request_for_logs` is correctly set for requests
with different types of authentication.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-28 09:38:40 -07:00
Zixuan James Li d1acd67897 test_realm: Fix realm confirmation object test case.
We are no longer creating confirmation objects associated with realms
directly. This should test for `RealmReactivationStatus` instead.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-27 17:07:11 -07:00
Adam Sah c6931434e9 email_error: Improve tests for inbound email edge case.
Tests to assist with clarifying #22585.
2022-07-27 13:53:16 -07:00
Zixuan James Li 07eccbde97 test_docs: Add assertion check.
In the test case `test_check_if_every_integration_has_logo_that_exists`,
`urlsplit(integration.logo_url).path` gets inferred as possibly bytes
because `integration.logo_url` might be `None`.

5598b49851/stdlib/urllib/parse.pyi (L166-L169)

TODO:
We might want to ensure that every integration has a `logo_url` with an
explicit assertion in `Integrations` (as noted in the comment).

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-26 18:00:24 -07:00
Zixuan James Li 223a1ad0eb test_auth_backends: Add None checks.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-26 18:00:24 -07:00
Zixuan James Li 51df4031ac test_auth_backends: Extract external_auth_backends.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-26 18:00:24 -07:00
Zixuan James Li d238a448e6 test_draft: Avoid inference with type annotation.
`expected_draft_contents` would be inferred as a list of mutable
mappings that only allow `int` as the value, and thus incompatible with
the `draft_dicts[i]` to be expanded. This is fixed by adding explicit
type annotation.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-26 18:00:24 -07:00
Zixuan James Li 7bf58dbede test_auth_backends: Remove unused self.user_profile.backend.
This was added in d43b031a32 and was
unused when it was added. This is an error that we want to remove.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-26 18:00:24 -07:00
Zixuan James Li 11a23aa5ef test_audit_log: Fix an assertion for a function that never returns.
We likely just wanted to check that `validate_password` succeeds without
any exception being raised. A simple call is sufficient to verify that,
since `validate_password` does not return anything and raises an
exception on failure.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-26 18:00:24 -07:00
Zixuan James Li 4675cbfefd test_message_flags: Remove unused comma.
This was accidentally added in 20a97bdb05,
likely due to some typos.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-26 18:00:24 -07:00
Zixuan James Li b0c5db1c22 test_realm_emoji: Remove redundant assertion.
This check was added in 495a8476be.
Now that django-stubs finds that the left operand of the `and` will
always evaluates to `True`, so it makes sense to remove it.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-26 18:00:24 -07:00
Zixuan James Li 97fd662d48 test_import_export: Enhance typing of getters.
This fixes the type annotations of `Set` derived from `QuerySet` objects,
and add necessary assertions.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-26 18:00:24 -07:00
Zixuan James Li 2925615ee7 test_decorators: Use the default value of user.
`HostRequestMock` has `user` default to `None`, which later gets
initialized as `AnonymousUser`. The separate initialization here is
unnecessary.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-26 18:00:24 -07:00
Sahil Batra bcef35490d test_markdown: Use make_stream function for creating streams.
This commit removes the instances of using "Stream.objects.create"
in tests with make_stream function. This change will help us to
avoid adding code for things to be done after creating streams in
multiple places. We can instead just add it in make_stream function
only.
2022-07-26 17:25:52 -07:00
Mateusz Mandera cf74d7d140 realm_reactivation: Prevent realm reactivation link reuse.
This uses the approach analogical to EmailChangeStatus for email change
confirmation links.
2022-07-26 17:14:26 -07:00
Mateusz Mandera 46c6f33b10 reactivate_realm: Change error status code on invalid links to 404. 2022-07-26 17:14:26 -07:00
Mateusz Mandera 0e2691815e confirmation: Prevent re-use of email change links.
The .status value of EmailChangeStatus was not being looked
at anywhere to prevent re-use of email change confirmation links. This
is not a security issue, since the EmailChangeStatus object has a fixed
value for the new_email, while the confirmation link has expiry time of
1 day, which prevents any reasonable malicious scenarios.

We fix this by making get_object_from_key look at
confirmation.content_object.status - which applies
generally to all confirmations where the attached object has the .status
attribute. This is desired, because we never want to
successfully get_object_from_key an object that has already been used or
reused.
This makes the prereg_user.status check in check_prereg_key redundant so
it can be deleted.
2022-07-26 17:14:26 -07:00
Mateusz Mandera 9992c7b6cc test_email_change: Extract generate_email_change_link helper. 2022-07-26 17:14:26 -07:00
Zixuan James Li fe9ed2e69d settings: Make INVITATION_LINK_VALIDITY_MINUTES optional.
Type inference does not work when the default value of `REQ` is
non-optional while `ResultT` is optional. Mypy tries to unify
`json_validator` with `Validator[int]` in `invite_users_backend` instead
of the desired `Validator[Optional[int]]` because of the presence of the
default value `settings.INVITATION_LINK_VALIDITY_MINUTES`, which is
inferred to be an `int`. Mypy does not resort to a less specific type but
instead gives up early.

This issue applies to invite_users_backend and generate_multiuse_invite_backend
in zerver.views.invite.

There might be a way that we can add an overload to get around this, but
it's probably not worth the complexity until it comes up again more frequently.

We do in fact allow `invite_expires_in_minutes` to be `None` in places
like `do_invite_users`, `invite_users_backend`, etc, and we have
`settings.INVITATION_LINK_VALIDITY_MINUTES` as the default for them. So
it makes sense to allow having an optional value for this setting. And
since there isn't a way to independently set the value of this constant,
we move it to a different place.

TODO:

This is a temporary fix that should be refactored when the bug is fixed.

The encountered mypy issue: https://github.com/python/mypy/issues/13234

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-26 17:07:48 -07:00
Mateusz Mandera 39d8a81e51 registration: Tie PreregistrationUser to the original MultiUseInvite.
Fixes #21266.

We want to tie the prereg_user to the MultiUseInvite directly rather
than to the MultiUserInvite's confirmation object, because the latter is
not possible. This is because the flow is that after going through the
multiuse invite link, the PreregistrationUser is created together with a
Confirmation object, creating a confirmation link (via
create_confirmation_link) to which then the user is redirected to finish
account creation. This means that the PreregistrationUser is already
tied to a Confirmation, so that attribute is occupied.
2022-07-22 17:08:44 -07:00
Zixuan James Li ebfd2b25b1 user_status: Add UserInfoDict.
The shared fields of `RawUserInfoDict` and `UserInfoDict` could have
been reused if they both require all keys or none. This is unfortunately
not the case, because subclassing does not override `__total__`.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-22 16:31:18 -07:00
Zixuan James Li e950b94ab5 test_urls: Remove legacy get_callback_string.
`_callback_str` was removed in Django in 1.10, and other logic relevant
to that particular attribute was removed in
32849b80ad, but not to its entirety. It
does not make sense to fall back to `_callback_str`. The
`get_callback_string` helper is no longer needed.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-22 16:22:47 -07:00
Anders Kaseorg 751b2a03e5 CVE-2022-31168: Fix authorization check for changing bot roles.
Due to an incorrect authorization check in Zulip Server 5.4 and
earlier, a member of an organization could craft an API call that
grants organization administrator privileges to one of their bots.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-07-21 17:59:09 -07:00
Lauryn Menard ad2ca0e668 tests: Remove `FORWARD_ADDRESS_CONFIG_FILE` created in test.
Previously if `test_forward_address_details` failed, the file
created when setting the `forward_address` may not have been
removed, which would then cause an `EmailNotDeliveredException`
to be raised when then creating a new user in the dev environment.

Wraps the test in a try block, with a finally block for the call
to remove the file.
2022-07-21 15:25:11 -07:00
Mateusz Mandera f787ddc7d2 get_object_from_key: Make mark_object_used an obligatory kwarg. 2022-07-21 15:18:15 -07:00
Mateusz Mandera fa7700df11 confirmation: Rename STATUS_ACTIVE to STATUS_USED.
That's much more descriptive of what that value actually means about the
Confirmation objects.
2022-07-21 15:17:37 -07:00
Mateusz Mandera cd1e062878 registration: Return proper error page if a registration link is reused.
Before this, a link still couldn't be re-used because it would trip up
exception further down user creation codepaths, but that was still a
bug. check_prereg_key is supposed to correctly validate the key - and
trigger an error page being returned if a key (or for any other reason,
the attached PreregistrationUser object) is reused.

test_validate_email_not_already_in_realm needs to be adjusted, because
it was actually re-using a key.
2022-07-21 15:17:37 -07:00
Mateusz Mandera 328a1fcf13 test_signup: Fix typo in test name. 2022-07-21 15:17:37 -07:00
Tim Abbott 8faa854267 Revert "message_flags: Filter msgs having (or not) the flag before updating."
This reverts commit 40fcf5a633.

This commit triggers bug that we haven't fully tracked down, where web
app clients will continually send `update_message_flags` requests,
that then send out via the events system "0 messages were marked as
read" notices, eventually leading to a load spike.

The Tornado part can likely be fixed by checking if
updated_message_ids is empty, but we need to track down the frontend
bug as well.
2022-07-21 14:29:54 -07:00
Anders Kaseorg 443b974b3e python: Apply changes from pyupgrade.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-07-20 22:33:28 -07:00
Lauryn Menard 8a0070ace7 decorator: Move tornado notify `data` parameter to `REQ` framework.
Instead of using `request.POST` to access the `data` parameter used
in the internal `notify_tornado` path, adds `has_request_variables`
decorator and accesses `data` as a `REQ` parameter.

Expands `test_tornado_endpoint` in `test_event_system.py` for
`data` being a required parameter for this path.
2022-07-20 14:23:05 -07:00
Zixuan James Li 31c7344979 test_message_fetch: Verify the value of WWW-Authenticate.
In `JsonableErrorHandler`, we convert `MissingAuthenticationError` into
a response that has `WWW-Authenticated` set for `/api` or `/json` views.

This covers and verify the value of the header for unauthenticated
access.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-18 18:01:42 -07:00
Sahil Batra 690420ffa0 populate_db: Fix data for "Favorite editor" custom field.
The field_data sent from client while creating a select
type field is a dict with a number as key.

In development database the field data for "Favorite editor"
field was of different form where the option label was used
as key in the dict.

This commit fixes it to be of the same as it is when creating
a field from web-app. As a result, we also need to update
the tests and this commit also update field_data for other
select-type fields.
2022-07-15 16:51:24 -07:00
Sahil Batra fac5e0605b custom_profile_fields: Delete invalid values for select-type fields.
We delete the user values for options which were deleted for the
field.
2022-07-15 16:33:15 -07:00
Zixuan James Li 4099e56a1f custom_profile: Refactor test_update_profile_data_successfully.
This refactors the test case with more explicit type annotations, fixing
type errors discovered provided type annotations for
`CustomProfileField`.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-15 14:55:03 -07:00
Zixuan James Li 52be020d0c custom_profile: Apply ProfileDataElementUpdateDict.
We explicitly annotate variables or parameters with
`ProfileDataElementUpdateDict` as necessary.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-15 14:55:03 -07:00
Zixuan James Li db7a6d15dc test_auth_backends: Replace handcrafted requests with HostRequestMock.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-15 14:00:56 -07:00
Zixuan James Li 56855e0187 typing: Add assertions for Optional values.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-15 14:00:56 -07:00
Zixuan James Li 7c62af0d73 typing: Add assertions before accessing settings.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-15 14:00:56 -07:00
Zixuan James Li a09c9a4063 test_auth_backends: Remove unused return.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-15 14:00:56 -07:00
Zixuan James Li c5075be9aa test_retention: Add type annotation for expected_result
Otherwise mypy infers the type of `expected_result` to be incompatible
with the first argument of `fix_ordering_of_result`.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-15 14:00:56 -07:00
Zixuan James Li da4654d838 test_send_email: Fix the to_email argument to pass str.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-15 14:00:56 -07:00
Zixuan James Li 45db04a39e test_auth_backends: Fix invalid assertions.
`m.output` is a `list` of `str`s. It does not make sense comparing it to
a `str`. Guessed the intention here is to use `self.assert_length`.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-15 14:00:56 -07:00
Zixuan James Li 2b1820cfcf message_edit: Avoid retyping variables.
Mypy disallows definition of the same variable with incompatible types.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-15 14:00:56 -07:00
Zixuan James Li b775639f42 test: Use list comprehension for dictionary values.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-15 14:00:56 -07:00
Zixuan James Li 7950baafe2 tests: Add type narrowing for EmailMultiAlternatives.
Items in `django.core.mail.outbox` are by default typed as the less
general `EmailMessage` type. Before accessing the attribute
`alternatives`, we need to narrow the type to `EmailMultiAlternatives`.
Then narrow the tuple value we want to access to `str` before using
it in `assertIn` or `self.normalize_string`.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-15 14:00:56 -07:00
Tim Abbott 05b70ba74a decorator: Explicitly require req_secret in internal_notify_view.
It's hard to come up with a realistic story where this would matter:
SHARED_SECRET is generated automatically during server setup at the
same time as SECRET_KEY, which is a required setting, but it seems
preferable to be explicit that this is a required parameter for the
internal_notify authentication model.
2022-07-15 09:20:37 -07:00
Lauryn Menard 855e14272a backend: Migrate `secret` parameter to REQ framework.
Instead of using request.POST to get any potential `secret`
parameter used in `authenticate_notify` for `internal_notify_view`
decorator, moves it to the REQ framework parameters as `req_secret`.

Updates existing tests to explicitly test for a request without
`secret` parameter, which defaults to `None`; this is also tested
in `test_event_system.py`.
2022-07-15 09:20:37 -07:00
Zixuan James Li 0056becd04 test_helpers: Default user_profile to AnonymousUser.
A request that has went through the auth middleware shouldn't have
`.user` being `None`. We should use `AnonymousUser` by default to
represent unauthenticated users.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-14 17:24:24 -07:00
Ganesh Pawar 135854db97 emoji: Use path converter for emoji name in URL.
If the emoji name contains forward slashes, the `str` converter
would treat it as a URL delimiter. Instead use the path converter, so
that forward slashes are included in the emoji name variable.

Fixes #22377
2022-07-14 14:29:40 -07:00
Sahil Batra 093a74bd54 models: Remove "role" field from Subscription.
This commit removes "role" field from subscription
objects since we are not moving forward with stream
administrator concept and instead working on new
permssions model as per #19525.
2022-07-13 17:07:44 -07:00
Sahil Batra 83383090f9 realm: Removed WILDCARD_MENTION_POLICY_STREAM_ADMINS option.
This commit removes WILDCARD_MENTION_POLICY_STREAM_ADMINS
option of wildcard_mention_policy since we are not moving
forward with stream administrator concept and instead working
on new permssions model as per #19525.

We also add a migration to change wildcard_mention_policy of
existing realms to WILDCARD_MENTION_POLICY_ADMINS. This change
is fine since we were already treating both the setting values
as same as stream admin concept was not implemented completely.
2022-07-13 17:07:44 -07:00
Sahil Batra d31fc27912 streams: Remove is_stream_admin property and its uses.
This commit removes the is_stream_admin property of Subscription
model and also updates check_stream_access_for_delete_or_update
to not return true when is_stream_admin is True.

We also removes the relevant tests.

This change is done as we would not be moving forward with the
stream administrator concept as we have decided to modify the
permissions model as per #19525.
2022-07-13 17:07:44 -07:00
Adam Sah cb6382369e merge_streams: Move logic to zerver/actions/streams.py. 2022-07-13 16:12:36 -07:00
Anders Kaseorg 81892df176 requirements: Upgrade to Django 4.0.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-07-13 16:07:17 -07:00
Alya Abbott 7e3388dd9e images: Move /team images into landing-page directory. 2022-07-13 15:14:19 -07:00
Zixuan James Li 74f59bd8d0 2fa: Rename zulip_otp_required for clarity.
The name does not really comply with the actual behavior of
the decorator since it returns True for an unauthenticated user.
This makes it clear that the 2fa check only applies to users that
are already logged in.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-13 14:49:41 -07:00
Zixuan James Li 00bd7513f2 2fa: Verify 2FA authentication status with is_2fa_verified.
This replaces user.is_verified with is_2fa_verified.

The helper does extra checks such that the user being checked for 2fa
authentication status is valid.

`request.user.is_verified` is functionally the same as `is_verified`
from `django_otp.middleware`, except that the former is monkey-patched
onto the user object by the 2FA middleware. We use the latter wrapped
in `is_2fa_verified` instead to avoid accessing the patched attribute.

See also: 6b24d56e59/docs/source/overview.rst (authentication-and-verification)

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-13 14:49:41 -07:00
Zixuan James Li 3367839839 decorator: Add test case for unauthenticated 2fa.
This simulates the situation in which the user is not
authenticated (as an AnonymousUser) and have 2FA enabled.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-13 14:49:41 -07:00
evykassirer 084b054827 emoji: Update emoji validation error messages with more information.
This change tells the user what a valid emoji name looks like, so that
they know what to fix if the name they chose doesn't pass the validator.
2022-07-12 16:43:29 -07:00
Mateusz Mandera e6264cd85b test_import_export: Test exclusion of private uploads from realm export. 2022-07-12 06:08:07 +00:00
Zixuan James Li 5d524aa43d custom_profile_fields: Use cast to assert non-optional items.
The mypy django-stubs plugin incorrectly infers optional value for order,
which makes the `value_list` to be typed as `Iterable[Optional[int]]`.
We use a type cast here to ensure that
`try_reorder_relam_custom_profile_fields` won't causes mypy to complain
about it.

TODO: Remove the cast when https://github.com/typeddjango/django-stubs/issues/444 gets resovled.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-07 11:28:13 -07:00
Zixuan James Li 5a934ef39f test_import_export: Tighten function signatures with generic QuerySet.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-07 11:28:13 -07:00
Zixuan James Li 8fa94138bc realm_domains: Type QuerySet.values as RealmDomainDict.
This ensure that the return type is compatible with the
actual type of `realm.realmdomain_set.values`.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-07 11:27:42 -07:00
Zixuan James Li ab1bbdda65 typing: Broaden type annotations for QuerySet compatibility.
To explain the rationale of this change, for example, there is
`get_user_activity_summary` which accepts either a `Collection[UserActivity]`,
where `QuerySet[T]` is not strictly `Sequence[T]` because its slicing behavior
is different from the `Protocol`, making `Collection` necessary.

Similarily, we should have `Iterable[T]` instead of `List[T]` so that
`QuerySet[T]` will also be an acceptable subtype, or `Sequence[T]` when we
also expect it to be indexed.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-07 11:27:42 -07:00
Aman Agrawal 40fcf5a633 message_flags: Filter msgs having (or not) the flag before updating.
We were blindly adding / removing flag from UserMessages without
check if they even need to be updated.

This caused server to repeatedly update flags for messages which
already had been updated, creating a confusion for other clients
like mobile.

Fixes #22164
2022-07-07 10:52:36 -07:00
Anders Kaseorg feff1d0411 requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-07-05 17:54:17 -07:00
Anders Kaseorg 6c79b8f2f1 test_tornado: Avoid deprecated AsyncHTTPTestCase.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-07-05 17:54:17 -07:00
Zixuan James Li b65401ed47 tests: Tighten signature of the wrapped test client helpers.
We wrap methods of the django test client for the test suite, and
type keyword variadic arguments as `ClientArg` as it might called
with a mix of `bool` and `str`.

This is problematic when we call the original methods on the test
client as we attempt to unpack the dictionary of keyword arguments,
which has no type guarantee that certain keys that the test client
requires to be bool will certainly be bool.

For example, you can call
`self.client_post(url, info, follow="invalid")` without getting a
mypy error while the django test client requires `follow: bool`.

The unsafely typed keyword variadic arguments leads to error within
the body the wrapped test client functions as we call
`django_client.post` with `**kwargs` when django-stubs gets added,
making it necessary to refactor these wrappers for type safety.

The approach here minimizes the need to refactor callers, as we
keep `kwargs` being variadic while change its type from `ClientArg`
to `str` after defining all the possible `bool` arguments that might
previously appear in `kwargs`. We also copy the defaults from the
django test client as they are unlikely to change.

The tornado test cases are also refactored due to the change of
the signature of `set_http_headers` with the `skip_user_agent` being
added as a keyword argument. We want to unconditionally set this flag to
`True` because the `HTTP_USER_AGENT` is not supported. It also removes a
unnecessary duplication of an argument.

This is a part of the django-stubs refactorings.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-06-28 16:12:00 -07:00
Zixuan James Li 4e518a3852 tests: Call client methods with explicit keyword arguments.
This is a prep commit for tightening the types for our wrapped test
client.

The callers of the test client methods are refactored to either call
them without unpacking at all or create a TypedDict for the keyword
arguments to be unpacked. This allows the type checker to know exactly what
keys are present and their corresponding type.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-06-28 16:12:00 -07:00
Zixuan James Li 27be27560b typing: Access LANGUAGE_CODE via `django.utils.translation.get_language`.
We no longer need to access the internal `LANGUAGE_CODE` attribute by
using `django.utils.translation.get_language`.

A test case overriding the translation is added to ensure the password
reset form sending to users requested from a wrong domain is properly
translated.

This is a part of django-stubs refactorings.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-06-28 16:07:18 -07:00
Anders Kaseorg 53231aa9d9 decorator: Type cache_info, cache_clear for ignore_unhashable_lru_cache.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-06-27 10:20:05 -07:00
Anders Kaseorg 0dfde9db94 python: Simplify unnecessary getattr calls.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-06-26 17:38:22 -07:00
Anders Kaseorg dc33a0ae67 markdown: Rewrite include plugin without markdown-include.
markdown-include is GPL licensed.

Also, rewrite it as a block processor, so that it works correctly
inside indented blocks.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-06-26 17:36:31 -07:00
Alex Vandiver 5529639b84 change_realm_subdomain: Support skipping the redirect.
Sometimes (e.g. when moving an old realm out of the way of an import
into that name) we do *not* wish to add a redirect realm.  Add a flag
to support that.
2022-06-25 08:44:24 -07:00
Alex Vandiver 3e4d07952c test_signup: Test redirects more explicitly. 2022-06-25 08:44:24 -07:00
Anders Kaseorg 236ef8a077 test_push_notifications: Simplify with Python 3.8 AsyncMock.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-06-25 08:43:20 -07:00
Anders Kaseorg d03892391a event_queue: Take (typed) keyword parameters for fetch_events.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-06-25 08:42:23 -07:00
Anders Kaseorg 952422d373 test_helpers: Remove default tornado_handler from HostRequestMock.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-06-25 08:42:23 -07:00
Zixuan James Li 6400cd93ea test_middleware: Narrow the type of open_graph_description.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-06-23 22:09:05 -07:00
Zixuan James Li 428dfd2693 typing: Add assertions for report returned from notify_server_error.
We don't have a specific type for the reports returned from the error
logging handlers. The check is necessary as they are currently typed
as `Dict[str, object]` in `run_handler`.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-06-23 22:09:05 -07:00
Zixuan James Li 1860b0b7ac typing: Fix missing attribtute of RedisUtilsTest.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-06-23 22:05:12 -07:00
Zixuan James Li 58e95cbfae typing: Add none-check for test_users.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-06-23 19:25:48 -07:00
Zixuan James Li c567e44408 typing: Avoid redefinition of variables with different types.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-06-23 19:25:48 -07:00
Zixuan James Li d46f7566dd typing: Fix wrong type annotations for mute_stream.
This is a part of djang-stubs refactorings.

Signed-off-by: Zixuan James Li <359101898@qq.com>
2022-06-23 19:25:48 -07:00
Anders Kaseorg 345ed1d09d middleware: Pass unhandled API exceptions through to the test suite.
This results in more useful stack traces in failing tests.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-06-23 19:23:08 -07:00
Anders Kaseorg df69e1d979 mypy: Enable truthy-bool errors.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-06-23 19:22:12 -07:00
Anders Kaseorg 74e94ae78c integrations: Fix config_options type.
The “validator” component of the tuple does not follow the Validator
contract as of 7e9db327b3 (#15498).
Define a separate type for it.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-06-23 19:22:12 -07:00
Anders Kaseorg 0430705d13 test_tornado: Call process_event on first fetch_events return.
The 0.1 second delay was sometimes not long enough to guarantee we hit
the async response path, resulting in a nondeterministic coverage
failure.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-06-08 17:10:38 -07:00
Zixuan James Li 426f8ce385 tests: Replace `HttpResponse` with `TestHttpResponse.`
Since `HttpResponse` is an inaccurate representation of the
monkey-patched response object returned by the Django test client, we
replace it with `_MonkeyPatchedWSGIResponse` as `TestHttpResponse`.

This replaces `HttpResponse` in zerver/tests, analytics/tests, coporate/tests,
zerver/lib/test_classes.py, and zerver/lib/test_helpers.py with
`TestHttpResponse`. Several files in zerver/tests are excluded
from this substitution.

This commit is auto-generated by a script, with manual adjustments on certain
files squashed into it.

This is a part of the django-stubs refactorings.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-06-08 11:25:03 -07:00
Sahil Batra dfacd62d93 streams: Update error message returned for administrative access.
We have now decided to not continue with the stream administrator
concept as we are changing the permissions model to be based on
user groups as per #19525. So, this commit updates the error message
to "Must be an organization administrator".
2022-06-07 09:36:24 -07:00
Zixuan James Li a142fbff85 tests: Refactor away result.json() calls with helpers.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-06-06 23:06:00 -07:00
Mateusz Mandera 09dc166b45 do_delete_old_unclaimed_attachments: Consider ArchivedAttachment rows.
This function is oblivious to the existence of ArchivedAttachment, which
is incorrect. A file can be removed if and only if it is not referenced
by any Messages or ArchivedMessages.
2022-06-02 17:32:23 -07:00
Mateusz Mandera 5ff4754090 test_upload: Fix some URLs to uploaded files.
Using http://localhost:9991 is incorrect - e.g. messages sent with file
urls constructed trigger do_claim_attachments to be called with empty
list in potential_path_ids.

realm.host should be used in all these places, like in the other tests
in the file.
2022-06-02 17:32:23 -07:00
Aman Agrawal 085443c722 home: Get default_language info from cookie for spectator.
This allows us to set default_language as cookie in frontend which
can be used on server load to send the relevant
translation data to the client.
2022-06-01 17:08:00 -07:00
Adam Sah ba5cf331a2 testing: 100% coverage for zerver/tests/test_digest.py. 2022-06-01 16:09:13 -07:00
Anders Kaseorg 9ee636e920 requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-06-01 16:01:54 -07:00
Zixuan James Li 4c6f2ae7be typing: Add assertions for authentication.
Signed-off-by: Zixuan James Li <359101898@qq.com>
2022-05-31 09:43:55 -07:00
Zixuan James Li c572d9be5a typing: Add none-checks for db queries.
Signed-off-by: Zixuan James Li <359101898@qq.com>
2022-05-31 09:43:55 -07:00
Zixuan James Li 4a5043dd6e typing: Add none-checks for miscellaneous cases.
Signed-off-by: Zixuan James Li <359101898@qq.com>
2022-05-31 09:43:55 -07:00
Zixuan James Li e338ada66c typing: Add none-checks for Recipient objects.
Signed-off-by: Zixuan James Li <359101898@qq.com>
2022-05-31 09:43:55 -07:00
Zixuan James Li bb6a934c8d typing: Add appropriate none-checks for LOCAL_UPLOADS_DIR.
This is a part of django-stubs refactorings.

Signed-off-by: Zixuan James Li <359101898@qq.com>
2022-05-31 09:43:55 -07:00
Zixuan James Li 63e9ae8389 typing: Apply trivial fixes to adjust edge cases in typing.
Add none-checks, rename variables (to avoid redefinition of
the same variable with different types error), add necessary
type annotations.

This is a part of #18777.

Signed-off-by: Zixuan James Li <359101898@qq.com>
2022-05-30 12:03:51 -07:00
Zixuan James Li c34ac1fcd4 typing: Access url via key "Location" instead of attribute "url".
This is a part of #18777.

Signed-off-by: Zixuan James Li <359101898@qq.com>
2022-05-30 11:59:47 -07:00
Sahil Batra 9273c314cf invites: Capitalize "ID" in the error raised for invalid stream ids.
This commit changes "id" to "ID" in the error message returned when
invalid stream IDs are passed to multiuse invite endpoint.
2022-05-27 17:06:03 -07:00
Sahil Batra 708204290e streams: Capitalize "ID" in invalid stream errors in API.
This commit changes the error message from "Invalid stream id"
to "Invalid stream ID" for cases where invalid stream IDs are
passed to API endpoints to make it consistent with other similar
error messages.
2022-05-27 17:06:03 -07:00
Zixuan James Li 44ecd66eae types: Better types for API fields.
Signed-off-by: Zixuan James Li <359101898@qq.com>
2022-05-27 14:43:00 -07:00
Zixuan James Li f1ef27cb85 password_reset: Soft reactivate upon password reset.
Fixes #22066

Signed-off-by: Zixuan James Li <359101898@qq.com>
2022-05-27 14:30:34 -07:00
Sahil Batra ce34b585a5 streams: Add endpoint "GET /streams/{stream_id}" to get stream by id.
Fixes #22082.
2022-05-23 15:14:04 -07:00
Sahil Batra 95a2b580a0 streams: Save one DB query in bulk_remove_susbcriptions.
We remove one call to get_occupied_streams to get occupied
streams before unsubscribing because we already know which
streams can become vacant, i.e. the one from which users are
being unsubscribed, and we can directly use the list of streams
from which users are being unsubscribed and get vacant streams
by checking which of these streams are not in get_occupied_streams
called after unsubscribing users.
2022-05-17 15:09:15 -07:00
Sahil Batra dfd7902c77 user_groups: Rename subgroups fields to direct_subgroup_ids.
This commit renames subgroups and subgroup_ids field sent in user
group objects to direct_subgroup_ids for better readability.
2022-05-17 14:51:45 -07:00
Adam Sah 04ccd8c6d8 testing: 100% code coverage for zerver/lib/outgoing_webhook.py. 2022-05-17 13:51:00 -07:00
Alex Vandiver 513fcb7bd5 Revert "settings: Add web-public streams beta subdomain list."
This reverts commit 20368a936c.  It is
no longer in beta, and this configuration is no longer needed.
2022-05-16 14:49:14 -07:00
Alex Vandiver a6d1c41827 email_mirror: Handle invalid `charset="..."` values. 2022-05-16 12:04:38 -07:00
Anders Kaseorg 0043c0b6b2 django: Use HttpRequest.headers.
Fixes #14769.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-05-13 20:42:20 -07:00
Anders Kaseorg d98e3ecb0d tests: Don’t reuse HttpRequest objects.
Django caches some information on HttpRequest objects, including the
headers dict, under the assumption that requests won’t be reused.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-05-13 20:42:20 -07:00
Alex Vandiver 803982e872 message_flags: Short-circuit if no messages changed.
Omit sending an event, and updating the database, if there are no
matching messages.
2022-05-12 21:57:55 -07:00
Sahil Batra 0bfe973b65 streams: Mark messages as read only for unsubscribed streams.
Previously, we were marking messages of all the streams passed
to bulk_remove_subscriptions even if user was not subscribed
to some of them and those streams would ideally not have
any unread messages. This code was added in 766511e519.

This commit changes the code to only mark messages of actually
unsubscribed streams as read.
2022-05-12 18:53:45 -07:00
Anders Kaseorg cce142c61a middleware: Fix URL encoding of next parameter.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-05-12 17:51:51 -07:00
Somesh Ranjan 0b1f8c05e3 org_settings: Add backend support to change bot role.
This commit attempts to add the backend support by extending the
/json/bots/{bot_id}/ url support to accept the role field as a
parameter. This was previously already possible via
`/json/users/{user_id}`, so this change just simplifies client
implementation.
2022-05-10 14:09:57 -07:00
Alya Abbott 86b61d46eb portico: Add Recurse Center case study. 2022-05-05 16:43:00 -07:00
Tim Abbott 24f527cb59 home: Don't send /register response for spectators.
In very large communities, computing page_params can be quite
expensive. Because we've moved the homepage for communities with web
public streams enabled to be the Zulip app, and it's common for
automation to frequently poll the homepage of a Zulip organization,
we'd like to keep those homepages cheap (as the login pages are).

We address this by prototyping something we may end up wanting to do
anyway -- having the web application do a `POST /register` API call in
order to fetch most page_params, and merging those with the mostly
webapp configuration page_params that we leave in the / response for
convenience.

This exact implementation is messy in a few ways:

* We rely on the assumption that ui_init.initialize_everything happens
  before all code that needs to inspect the page_params properties we
  are fetching via /register. This is likely mostly true, but nothing
  in the implementation enforces it.
* The bundle of ~25 keys that are in page_params ideally would be
  considered individually, with some moved to the /register API
  response and perhaps others eliminated or namespaced inside a
  webapp_settings object.
* It's weird to have the spectators network sequence different that
  from logged-in users, and potentially a maintainability risk.
* We might be able to arrange that the initial `/` response be
  cacheable, now that we're no longer embedding our metadata inside
  it. We've made no effort to do that as of yet.

Despite those issues, this commit solves an immediate problem and will
give us helpful experience with a model closer to the one we'll want
in order to happily support a web client that can be run locally
against a production Zulip server's data.

Co-authored-by: Anders Kaseorg <anders@zulip.com>
2022-05-05 15:20:46 -07:00
Tim Abbott 2e86ea6540 events: Add support for spectator access to /register.
This is necessary for the mobile/terminal clients to build spectator
support down the line. We'll also be using it for the web application,
in an upcoming commit.
2022-05-05 15:20:44 -07:00
Tim Abbott 53518e8a24 events: Pass an explicit realm to do_events_register. 2022-05-05 15:17:07 -07:00
Tim Abbott d180f7ccfc right sidebar: Clean up rendered description logic.
Previously, we were masking the realm_description raw Markdown with
rendered Markdown, which was a type error.

When we switch to calling /register explicitly in a few commits, this
results in a bug, since the raw Markdown ends up taking priority.

Fix this by just using a different name for this different concept.
2022-05-05 15:17:07 -07:00
yogesh sirsat 08e8a21da0 custom_profile_fields: Clarify an external account error message.
This error message is for a very precise situation -- the pattern not
having the desired format. We should say that, rather than a generic
"Malformed".
2022-05-04 17:57:44 -07:00
NerdyLucifer 6a5d646739 settings (admin/org): Show error for same choices in "list of options".
Currently an user can create multiple options with same text/label in
the select/"list of options" custom profile field type.

Fix this issue by extending the validator to throw an error if there
are duplicate choices in the "list of options" in custom profile
field.

Tweaked by tabbott to use a simpler check.

Fixes: #21880
2022-05-04 17:55:28 -07:00
Alex Vandiver 21e80e47bd message_edit: Filter UserMessage better, now that mentions are unneeded.
The previous commit removed the need for `render_incoming_message` to
take a list of mentioned users; simplify the callsites accordingly.
2022-05-04 14:45:18 -07:00
Alex Vandiver 74e9b086f9 embed_links: Check that the message still exists before proceeding. 2022-05-04 14:45:18 -07:00
Sahil Batra 9a6886f630 users: Always pass delivery_email in user's own object.
This commit changes the code to always pass delivery_email
field in the user's own object in 'realm_users'.

This commit also fixes the events sent by notify_created_user.
In the "realm_user/add" event sent when creating the user,
the delivery_email field was set according to the access
for the created user itself as the created user was passed as
acting_user to format_user_row. But now since we have changed
the code to always allow the user themselves to have access
to the email, this bug was caught in tests and we fix the person
object in the event to have delivery_email field based on whether
the user receiving the event has access to email or not.
2022-05-04 12:52:43 -07:00
Lauryn Menard 44c9b788f9 settings: Add realm setting for Zulip communities directory.
Adds `want_advertise_in_communities_directory` to the realm model
to track organizations that give permission to be listed on such
a site / directory on zulip.com.

Adds a checkbox to the organization profile admin for
organizations to give permission to be advertised in the
Zulip communities directory.

Adds a help center article about the Zulip communities directory
and uses a shared intro documentation file to create sections in
the articles on creating an organization profile and moderating
open organizations.

Co-authored-by: Alya Abbott <alya@zulip.com>
2022-05-04 11:13:28 -07:00
Mateusz Mandera 163910b765 auth: Don't leak info about accounts in other realms in the login form.
The "wrong subdomain" error leaks information that we shouldn't reveal.
We want to return a generic failure message instead.
2022-05-03 17:06:51 -07:00
Mateusz Mandera 1faebecb7a auth: Change the logger for authing to the wrong subdomain situations.
It's better to use zulip.auth logger here instead of the root logger.
Also the level is adjusted to INFO, since WARNING is excessive.
2022-05-03 17:06:51 -07:00
Mateusz Mandera 71f6f2310c auth: Don't log email when user tried authing to the wrong subdomain.
We want to avoid logging this kind of potentially sensitive information.
Instead, it's more useful to log ids of the matching accounts on
different subdomains.
2022-05-03 17:06:51 -07:00
Adam Sah dbd03b5054 email_backends: Fix concurrent backend testing for generate_emails.
Previously, this command would reliably fail:

```
tools/test-backend --skip-provision-check --parallel=3
  zerver.tests.test_email_log.EmailLogTest.test_forward_address_details
  zerver.tests.test_email_log.EmailLogTest.test_generate_and_clear_email_log
  zerver.tests.test_example.TestDevelopmentEmailsLog
```

and now it reliably succeeds. :-)

After hours of fiddling/googling/hair-tearing, I found that
mocking-away Django Connection.send_messages() was the best:

- We're testing Zulip and not Django.
- Mocking at this lower level exercises more of our code.
- EmailLogBackEnd._do_send_messages() helper method added to simplify mocking.

Fixes #21925.
2022-05-03 08:45:51 -07:00
Anders Kaseorg 7acb642fa5 requirements: Upgrade to Tornado 6.
Fixes #8913.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-05-02 17:41:49 -07:00
Anders Kaseorg 6fd1a558b7 runtornado: Switch to asyncio event loop.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-05-02 17:41:49 -07:00
Siddharth Asthana 9954db4b59 users: Use browser locale to set the default language of new user.
This commit reads the browser locale during user registration, and
sets it as default language of user if it is supported by Zulip.
Otherwise, it is set to realm's default language.
2022-04-28 15:03:26 -07:00
Siddharth Asthana 807c73ef17 i18n: Add function to get default language for new users.
It returns realm's default language if the browser's locale
is not available or is unsupported.
2022-04-28 15:03:26 -07:00
Sahil Batra 476a217087 i18n: Add get_browser_language_code function.
This commit adds get_browser_language_code function
which returns None if there is no Accept-language
header in the request or Accept-languge header contains
only unsupported languages or all languages (meaning
header having value of '*'). Otherwise it returns the
language with highest weight/quality-value.
2022-04-28 15:03:26 -07:00
Aman Agrawal 4e08c737ca home: For web public realms, skip login for spectators.
To provide a smoother experience of accessing a web public stream,
we don't ask user to login unless user directly requests a
`/login` URL.

Fixes #21690.
2022-04-28 12:34:29 -07:00
Alex Vandiver 127108c7d1 workers: Log the exception if the export fails.
We previously just swallowed the exception entirely.
2022-04-28 11:52:47 -07:00
Anders Kaseorg 6331a314d4 Correctly hyphenate “non-”.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-27 22:10:31 -07:00
Anders Kaseorg d58fece832 Correctly hyphenate “web-public”.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-27 22:10:31 -07:00
Aman Agrawal 5ee4f71701 avatar: Add rate limit similar to attachments on medium avatars.
Followup on #20136
2022-04-27 16:51:18 -07:00
Zixuan James Li b4feb673f1 push_notifications: Soft reactivate mentioned users.
Fixes #19861

Signed-off-by: Zixuan James Li <359101898@qq.com>
2022-04-27 16:43:54 -07:00
Zixuan James Li a8fd9eb701 email_notifications: Soft reactivate mentioned users.
Signed-off-by: Zixuan James Li <359101898@qq.com>
2022-04-27 16:43:54 -07:00
Sahil Batra 0f01e3d327 tests: Add checks for user groups in different realms. 2022-04-27 14:17:52 -07:00
Sahil Batra 4d319ff34c tests: Use high enough number for invalid user group and user ids.
We use 1111 for testing invalid user and user group id cases instead
of 25, since we may have a user group in 25 in test data in future.
2022-04-27 14:17:52 -07:00
Anders Kaseorg 7c247a11a9 test_openapi: Use Python 3.8 typing.{get_origin,get_args}.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-27 12:57:49 -07:00
Anders Kaseorg 2afe82ba2e test_link_embed: Convert "".format to f-string.
Generated by pyupgrade.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-27 12:57:49 -07:00
Anders Kaseorg b572b18e70 test_digest: Modernize set literal syntax.
Generated by pyupgrade.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-27 12:57:49 -07:00
Mateusz Mandera 04fdf3e4d9 import_utils: Fix history_public_to_subscribers being set incorrectly.
history_public_to_subscribers wasn't explicitly set when creating
streams via build_stream, thus relying on the model's default of False.
This lead to public streams being created with that value set to False,
which doesn't make sense.

We can solve this by inferring the correct value based on invite_only in
the build_stream funtion itself - rather than needing to add a flag
argument to it.

This commit also includes a migration to fix public stream with the
wrong history_public_to_subscribers value.

Fixes #21784.
2022-04-27 12:08:01 -07:00
Lauryn Menard d2207d4ad5 backend: Add `org_type` to realm settings updates and events.
`org_type` already exists as a field in the Realm model and is
used when organizations are created / updated in Zulip Cloud,
via the `/analytics/support` view.

Extends the `PATCH /realm` view to be able update `org_type` as
other realm / organization settings are updated, but using the
special log / action that was created for the analytics view.

Adds a field to the `realm op: update` / `realm op: update_dict`
events, which also means an event is now sent when and if the
`org_type` is updated via the analytics view. This is similar
to how updates to an organization's `plan_type` trigger events.

Adds `realm_org_type` as a realm setting fetched from the
`POST /register` endpoint.
2022-04-26 16:29:12 -07:00
evykassirer e4c51488c0 populate_db: Add support for creating messages older than 5 days ago.
Fixes #21646.
2022-04-25 16:16:15 -07:00
Sahil Batra 37793434f7 user_groups: Add API endpoint to get subgroups of a user group. 2022-04-25 10:24:03 -07:00
Sahil Batra 1b3c972d9b user_groups: Add API endpoint to get members of a user group.
This commit adds 'GET /user_groups/{user_group_id}/members'
endpoint to get members of a user group. "direct_member_only"
parameter can be passed as True to the endpoint to get only
direct members of the user group and not the members of
subgroup.
2022-04-25 10:24:03 -07:00
Sahil Batra 374d2a66df user_groups: Add endpoint to check whether a user is member of a group.
This commit adds 'GET /user_groups/{id}/members/{id}' endpoint to check
whether a user is member of a group.

This commit also adds for_read parameter to access_user_group_by_id,
which if passed as True will provide access to read user group even
if it a system group or if non-admin acting user is not part of the
group.
2022-04-25 10:24:03 -07:00
Sahil Batra b71067908a user_groups: Add is_user_in_group function.
This commits adds is_user_in_group function
which can be used to check whether a user
is part of a user group or not. It also
supports recursive parameter for including
the members of all the subgroups as well.
2022-04-25 10:24:03 -07:00
Sahil Batra 6f0a7656ac user_groups: Add API endpoint for updating subgroups of a user group. 2022-04-25 10:24:03 -07:00
Sahil Batra b4a9311ef2 actions: Add function to add and remove subgroups from a user group. 2022-04-25 10:24:03 -07:00
Sahil Batra da0b087962 user_groups: Add "subgroups" field to user group objects.
This commit also adds 'subgroups' field to the user_group present
in the event sent on creating a user group. We do not allow passing
the subgroups while creating a user group as of this commit, but added
the field in the event object to pass tests.
2022-04-25 10:24:03 -07:00
Sahil Batra 61365fbe21 invites: Use expiration time in minutes instead of days.
This commit changes the invite API to accept invitation
expiration time in minutes since we are going to add a
custom option in further commits which would allow a user
to set expiration time in minutes, hours and weeks as well.
2022-04-20 13:31:37 -07:00