Commit Graph

7416 Commits

Author SHA1 Message Date
Prakhar Pratyush 35c3724f22 tests: Fix the incomplete exclusion of the notification settings.
'test_change_user_settings' in 'UserDisplayActionTest' excludes
the notification settings and tests only the display settings.

The code block excluding the notification settings doesn't exclude
'modern_notification_settings'. It only excludes the
'notification_settings_legacy'.

This commit replaces 'notification_settings_legacy' with
'notification_setting_types', which consists of all the
notification settings.
2023-09-14 17:16:36 -07:00
Lalit 2b566c778b user_settings: Add new `web_stream_unreads_count_display_policy` field.
This is a backend change that will help us support the new "Show unread counts for"
user display setting.
2023-09-13 18:45:45 -07:00
Anders Kaseorg 28597365da python: Delete superfluous parens.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-09-13 13:40:19 -07:00
Alex Vandiver a8a1f10f3c digest: Clear the cache once we move to a new realm / cutoff value. 2023-09-13 13:25:59 -07:00
Alex Vandiver b9f72bdd68 digest: Switch loop to early-abort for clarity. 2023-09-13 13:25:59 -07:00
Alex Vandiver b555d3f553 digest: Cache per-stream recent topics, rather than batching.
The query plan for fetching recent messages from the arbitrary set of
streams formed by the intersection of 30 random users can be quite
bad, and can descend into a sequential scan on `zerver_recipient`.
Worse, this work of pulling recent messages out is redone if the
stream appears in the next batch of 30 users.

Instead, pull the recent messages for a stream on a one-by-one basis,
but cache them in an in-memory cache.  Since digests are enqueued in
30-user batches but still one-realm-at-a-time, work will be saved both
in terms of faster query plans whose results can also be reused across
batches.

This requires that we pull the stream-id to stream-name mapping for
_all_ streams in the realm at once, but that is well-indexed and
unlikely to cause performance issues -- in fact, it may be faster
than pulling a random subset of the streams in the realm.
2023-09-13 13:25:59 -07:00
Alex Vandiver bca9821c89 digest: Rename get_recent_streams for clarity. 2023-09-13 13:25:59 -07:00
Alex Vandiver 524d4913b3 digest: Filter out users who have joined recently in SQL. 2023-09-13 13:25:59 -07:00
Alex Vandiver 584c202d36 digest: Remove unnecessary should_process_digest function. 2023-09-13 13:25:59 -07:00
N-Shar-ma 89739ba3c3 widgets: Add test for tokenizing widget type on newline.
Follow up to #26643, adds test case for the bug fix.
2023-09-13 12:42:35 -07:00
Anders Kaseorg 2665a3ce2b python: Elide unnecessary list wrappers.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-09-13 12:41:23 -07:00
Anders Kaseorg 6988622fe8 ruff: Enable B023 Function definition does not bind loop variable.
Python’s loop scoping is misdesigned, resulting in a very common
gotcha for functions that close over loop variables [1].  The general
problem is so bad that even the Go developers plan to break
compatibility in order to fix the same design mistake in their
language [2].

Enable the Ruff rule function-uses-loop-variable (B023) [3], which
conservatively prohibits functions from binding loop variables at all.

[1] https://docs.python-guide.org/writing/gotchas/#late-binding-closures
[2] https://go.dev/s/loopvar-design
[3] https://beta.ruff.rs/docs/rules/function-uses-loop-variable/

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-09-11 18:03:45 -07:00
Alex Vandiver b94402152d models: Always search Messages with a realm_id or id limit.
Unless there is a limit on `id`, always provide a `realm_id` limit as
well.  We also notate which index is expected to be used in each
query.
2023-09-11 15:00:37 -07:00
Alex Vandiver f9dd2549eb narrow: Set a realm_id limit on messages in user searches. 2023-09-11 15:00:37 -07:00
Alex Vandiver 3518d31797 migrations: Add indexes with realm_id.
This is designed to help PostgreSQL have better specificity and
locality in its indexes.  Subsequent commits will adjust the code to
make sure that we use these indexes rather than the `realm_id`-less
versions.

We do not add a `realm_id` variation to the full-text index, since
it is a GIN index; multi-column GIN indexes are not terribly
performant, require the `btree_gin` extension for `int` types (which
requires superuser privileges on PostgreSQL 12 and earlier), and
cannot be consistently added concurrently on running instances.

After all indexes have been made, we also run `CREATE STATISTICS` in
order to give PostgreSQL the opportunity to realize that recipient and
sender are highly correlated with message realm, allowing it to
estimate that `(realm_id, recipient_id)` is likely as specific as
matching a given `recipient_id`, instead of as likely as matching
`realm_id` times matching a `recipient_id`.  Finally, those statistics
must be filled by `ANALYZE zerver_message`, which is run last.
2023-09-11 15:00:37 -07:00
Alex Vandiver d6745209f2 django: Use .exists() instead of .count() when possible. 2023-09-11 15:00:37 -07:00
Alex Vandiver 710524465a messages: Switch limits from sender__realm to realm.
We now have a `realm_id` on Message; use it, rather than having to
check the sender's realm.  This is theoretically different for
cross-realm bots, but these changes are all in tests where that does
not apply.
2023-09-11 15:00:37 -07:00
Adrián Oliva 732ad89f3d markdown: Fix URL link topic skipping query.
When searching for links inside a topic name, the question mark (?)
was used to split the topic. If a URL had a query after the URL
(e.g., "?foo=bar"), then the query was trimmed from the URL.

Removing the question mark from `basic_link_splitter` is sufficient
to fix this issue. The `get_web_link_regex` function then removes
the trailing punctuation if any, including literal question marks.

Fixes #26368.
2023-09-08 16:17:11 -07:00
Zixuan James Li 9fef12950a webhooks: Migrate transifex to use endpoint to use @typed_endpoint.
Transifex has parameters that need to be parsed from JSON and converted
to int. Note that we use Optional[Json[int]] instead of
Json[Optional[int]] to replicate the behavior of json_validator. This
caveat is explained in a new test called test_json_optional.
2023-09-08 08:20:17 -07:00
Zixuan James Li 910f69465c drafts: Migrate drafts to use @typed_endpoint.
This demonstrates the use of BaseModel to replace a check_dict_only
validator.

We also add support to referring to $defs in the OpenAPI tests. In the
future, we can descend down each object instead of mapping them to dict
for more accurate checks.
2023-09-08 08:20:17 -07:00
Zixuan James Li 6201914fd3 message_edit: Migrate message_edit to use @typed_endpoint.
This demonstrates how an alias is created and its suitable use case, the
use of PathOnly, NonNegativeInt, and Literal.
2023-09-08 08:20:17 -07:00
Zixuan James Li 9c53995830 alert_words: Migrate alert_words to use @typed_endpoint.
This demonstrates some basic use cases of the Json[...] wrapper with
@typed_endpoint.

Along with this change we extend test_openapi so that schema checking
based on function signatures will still work with this new decorator.
Pydantic's TypeAdapter supports dumping the JSON schema of any given type,
which is leveraged here to validate against our own OpenAPI definitions.
Parts of the implementation will be covered in later commits as we
migrate more functions to use @typed_endpoint.

See also:
https://docs.pydantic.dev/latest/api/type_adapter/#pydantic.type_adapter.TypeAdapter.json_schema

For the OpenAPI schema, we preprocess it mostly the same way. For the
parameter types though, we no longer need to use
get_standardized_argument_type to normalize type annotation, because
Pydantic dumps a JSON schema that is compliant with OpenAPI schema
already, which makes it a lot convenient for us to compare the types
with our OpenAPI definitions.

Do note that there are some exceptions where our definitions do not match
the generated one. For example, we use JSON to parse int and bool parameters,
but we don't mark them to use "application/json" in our definitions.
2023-09-08 08:20:17 -07:00
Zixuan James Li c336bf0398 api: Avoid programming errors due to nested Annotated types.
We want to reject ambiguous type annotations that set ApiParamConfig
inside a Union. If a parameter is Optional and has a default of None, we
prefer Annotated[Optional[T], ...] over Optional[Annotated[T, ...]].

This implements a check that detects Optional[Annotated[T, ...]] and
raise an assertion error if ApiParamConfig is in the annotation. It also
checks if the type annotation contains any ApiParamConfig objects that
are ignored, which can happen if the Annotated type is nested inside
another type like List, Union, etc.

Note that because
param: Annotated[Optional[T], ...] = None
and
param: Optional[Annotated[Optional[T], ...]] = None
are equivalent in runtime prior to Python 3.11, there is no way for us
to distinguish the two. So we cannot detect that in runtime.
See also: https://github.com/python/cpython/issues/90353
2023-09-08 08:20:17 -07:00
Zixuan James Li 5a7b1065e5 api: Rewrite argument type test for clarity.
We refactor HostRequestMock so that it now proper populates the request
body given the post data, assuming that the request is JSON encoded.
2023-09-08 08:20:17 -07:00
Zixuan James Li f4caf9dd79 api: Add new typed_endpoint decorators.
The goal of typed_endpoint is to replicate most features supported by
has_request_variables, and to improve on top of it. There are some
unresolved issues that we don't plan to work on currently. For example,
typed_endpoint does not support ignored_parameters_supported for 400
responses, and it does not run validators on path-only arguments.

Unlike has_request_variables, typed_endpoint supports error handling by
processing validation errors from Pydantic.

Most features supported by has_request_variables are supported by
typed_endpoint in various ways.

To define a function, use a syntax like this with Annotated if there is
any metadata you want to associate with a parameter, do note that
parameters that are not keyword-only are ignored from the request:
```
@typed_endpoint
def view(
    request: HttpRequest,
    user_profile: UserProfile,
    *,
    foo: Annotated[int, ApiParamConfig(path_only=True)],
    bar: Json[int],
    other: Annotated[
        Json[int],
        ApiParamConfig(
            whence="lorem",
            documentation_status=NTENTIONALLY_UNDOCUMENTED
        )
    ] = 10,
) -> HttpResponse:
    ....
```

There are also some shorthands for the commonly used annotated types,
which are encouraged when applicable for better readability and less
typing:
```
WebhookPayload = Annotated[Json[T], ApiParamConfig(argument_type_is_body=True)]
PathOnly = Annotated[T, ApiParamConfig(path_only=True)]
```

Then the view function above can be rewritten as:
```
@typed_endpoint
def view(
    request: HttpRequest,
    user_profile: UserProfile,
    *,
    foo: PathOnly[int],
    bar: Json[int],
    other: Annotated[
        Json[int],
        ApiParamConfig(
            whence="lorem",
            documentation_status=INTENTIONALLY_UNDOCUMENTED
        )
    ] = 10,
) -> HttpResponse:
    ....
```

There are some intentional restrictions:
- A single parameter cannot have more than one ApiParamConfig
- Path-only parameters cannot have default values
- argument_type_is_body is incompatible with whence
- Arguments of name "request", "user_profile", "args", and "kwargs" and
  etc. are ignored by typed_endpoint.
- positional-only arguments are not supported by typed_endpoint. Only
  keyword-only parameters are expected to be parsed from the request.
- Pydantic's strict mode is always enabled, because we don't want to
  coerce input parsed from JSON into other types unnecessarily.
- Using strict mode all the time also means that we should always use
  Json[int] instead of int, because it is only possible for the request
  to have data of type str, and a type annotation of int will always
  reject such data.

typed_endpoint's handling of ignored_parameters_unsupported is mostly
identical to that of has_request_variables.
2023-09-08 08:20:17 -07:00
Tim Abbott 6c83bbcbdb settings: Disallow everyone group for new setting.
This is important because the "guests" value isn't one that we'd
expect anyone to pick intentionally, and in particular isn't an
available option for the similar/adjacent "email invitations" setting.
2023-09-07 14:21:01 -07:00
Ujjawal Modi 5e31a6b1c0 invites: Make it possible for non-admins to revoke multiuse invites.
This commit makes changes to allow non-admins to revoke multiuse
invitations created by them.
2023-09-07 14:21:01 -07:00
Ujjawal Modi ec49c3acc8 invites: Rename `can_invite_others_to_realm` local variables.
This commit rename the existing setting `Who can invite users to this
organization` to `Who can send email invitations to new users` and
also renames all the variables related to this setting that do not
require a change to the API.

This was done for better code readability as a new setting
`Who can create invite links` will be added in future commits.
2023-09-07 14:21:01 -07:00
Ujjawal Modi f67cef8885 invite: Add new setting for "Who can create multiuse invite links".
This commit does the backend changes required for adding a realm
setting based on groups permission model and does the API changes
required for the new setting `Who can create multiuse invite link`.
2023-09-07 14:21:01 -07:00
Ujjawal Modi 2e59b1f30e tests: Use function to create realm rather than django ORM.
This commit makes changes in backend tests to use
`do_create_realm` function to create realm.
2023-09-07 14:21:01 -07:00
Anders Kaseorg 48a3588cdb docs: Fix typos caught by ‘typos’.
https://github.com/crate-ci/typos

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-09-06 18:59:05 -07:00
Anders Kaseorg 6c76bad65a middleware: Fix exception logging format on JSON views.
Previously (with ERROR_REPORTING = True), we’d stuff the entire
traceback of the initial exception into the subject line of an error
email, and then also send a separate email for the JSON 500 response.
Instead, log one error with the standard Django format.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-09-06 09:14:49 -07:00
Anders Kaseorg 81bd63cb46 ruff: Fix PIE808 Unnecessary `start` argument in `range`.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-09-01 14:57:01 -07:00
Prakhar Pratyush 9cff7e14c4 push_notifications: Test message content truncated in a simple manner.
'test_get_message_payload_gcm_stream_message' verifies the payload
for notifications generated (for stream messages) due to any of the
push notification triggers, including
'NotificationTriggers.STREAM_PUSH'.

Earlier, 'test_get_message_payload_gcm_stream_notifications' tested
the same thing as 'test_get_message_payload_gcm_stream_message' with
the only difference that it included content that was not truncated.

This commit removes the test
'test_get_message_payload_gcm_stream_notifications' and updates
the test 'test_get_message_payload_gcm_stream_message' to cover
both the cases, i.e., truncated as well as not truncated content.
2023-09-01 10:46:16 -07:00
Prakhar Pratyush 5d8897b909 push_notifications: Remove 'alert' field from the payload for android.
This commit removes the 'alert' field from the payload for
Android via GCM/FCM.

The alert strings generated do not get used at all and have
not been used since at least 2019. On Android, we construct
the notification UI ourselves in the client, and we ignore
the alert string.
2023-09-01 10:46:16 -07:00
Lauryn Menard 91b40a45fe demo-orgs: Add email and password process for demo organization owners.
Creates process for demo organization owners to add an email address
and password to their account.

Uses the same flow as changing an email (via user settings) at the
beginning, but then sends a different email template to the user
for the email confirmation process.

We also encourage users to set their full name field in the modal for
adding an email in a demo organization. We disable the submit button
on the form if either input is empty, email or full name.

When the user clicks the 'confirm and set password' button in the
email sent to confirm the email address sent via the form, their
email is updated via confirm_email_change, but the user is redirected
to the reset password page for their account (instead of the page for
confirming an email change has happened).

Once the user successfully sets a password, then they will be
prompted to log in with their newly configured email and password.
2023-08-31 15:02:16 -07:00
Lauryn Menard a9eb70ac68 demo-orgs: Set owner `email_address_visibility` on account creation.
Since an email address is not required to create a demo organization,
we need a Zulip API email address for the web-app to use until the
owner configures an email for their account.

Here, we set the owner's `email_address_visibility` to "Nobody" when
the owner's account is created so that the Zulip API email field in
their profile is a fake email address string.
2023-08-31 15:02:16 -07:00
Eeshan Garg 5e33ae8adf demo-orgs: Create dev environment demo organization without email.
To make creation of demo organizations feel lightweight for users,
we do not want to require an email address at sign-up. Instead an
empty string will used for the new realm owner's email. Currently
implements that for new demo organizations in the development
environment.

Because the user's email address does not exist, we don't enqueue
any of the welcome emails upon account/realm creation, and we
don't create/send new login emails.

This is a part of #19523.

Co-authored by: Tim Abbott <tabbott@zulip.com>
Co-authored by: Lauryn Menard <lauryn@zulip.com>
2023-08-31 15:02:16 -07:00
Lauryn Menard ebfe9637c8 subscribe-unsubscribe: Improve error response for unexpected users.
Updates the API error response when there is an unknown or
deactivated user in the `principals` parameter for either the
`/api/subscribe` or `/api/unsubscribe` endpoints. We now use
the `access_user_by_email` and `access_user_by_id` code paths,
which return an HTTP response of 400 and a "BAD_REQUEST" code.

Previously, an HTTP response of 403 was returned with a special
"UNAUTHORIZED_PRINCIPAL" code in the error response. This code
was not documented in the API documentation and is removed as
a potential JsonableError code with these changes.

Fixes #26593.
2023-08-31 13:48:39 -07:00
David Rosa 4b8c99b01a widgets: Rename confusing attribute name in `tabbed_sections.py`.
Renames misleading attribute in HTML template using `code-section`
to refer to both language toggles in API docs and app toggles in
help center docs.
2023-08-31 11:55:28 -07:00
Alex Vandiver 7787fe3f49 push_notifications: Send all APNS devices in parallel.
Instead of starting up one event loop for every device send, use
asyncio.gather to send to all of a user's devices at once.
2023-08-30 11:56:52 -07:00
Alex Vandiver 69825cd54c push_notifications: Drop error messages from failure to send.
We handle, and possibly log, these errors ourselves.
2023-08-30 11:56:52 -07:00
Alex Vandiver 733c6da298 push_notifications: Upgrade aioapns. 2023-08-30 11:56:52 -07:00
Alex Vandiver c2c67a7640 send_email: Broaden type of context dict.
jinja2 can take objects and other data structures.
2023-08-30 11:54:28 -07:00
Akshat 3977514dec fenced_code: Enable code-highlighting without language markers.
The 'startinline' option is utilized in the `CodeHilite` instantiation
to indicate that the provided PHP code snippet should be highlighted
even if it doesn't start with the opening tag or marker of the
associated programming language (which will rarely be the case in
Zulip, since one discusses a section of a file much more often than a
whole file).

Fixes: https://chat.zulip.org/#narrow/stream/137-feedback/topic/php.20syntax.20highlighting.20should.20not.20require.20.60.3C.3Fphp.60.

Signed-off-by: Akshat <akshat25iiit@gmail.com>
2023-08-29 18:16:11 -07:00
Prakhar Pratyush 46446d0e1d test_push_notifications: Add a test for FOLLOWED_TOPIC_PUSH trigger.
This commit adds a test to verify the payload
'get_message_payload_apns' returns when the notification trigger is
'NotificationTriggers.FOLLOWED_TOPIC_PUSH'.
2023-08-29 17:12:21 -07:00
Prakhar Pratyush 6dc3b1a052 push_notifications: Return a common subtitle for wildcard mentions.
This commit updates the 'get_apns_alert_subtitle' function to
return a common subtitle, i.e., "{full_name} mentioned everyone:"
for wildcard mentions.

The triggers for the stream or topic wildcard mentions include:
* NotificationTriggers.TOPIC_WILDCARD_MENTION_IN_FOLLOWED_TOPIC
* NotificationTriggers.STREAM_WILDCARD_MENTION_IN_FOLLOWED_TOPIC
* NotificationTriggers.TOPIC_WILDCARD_MENTION
* NotificationTriggers.STREAM_WILDCARD_MENTION
2023-08-29 17:12:21 -07:00
Danny Su 2414ddd2d4 compose: Verify request payload in Zoom meeting creation test
Verify that Zoom meeting creation logic sends the expected request payload for
configuring host_video and participant video.
2023-08-29 17:09:43 -07:00
Danny Su 66b9c06de6 compose: Add support for Zoom audio call
This PR implements the audio call feature for Zoom. This is done by explicitly
telling Zoom to create a meeting where the host's video and participants' video
are off by default.

Another key change is that when creating a video call, the host's and
participants' video will be on by default. The old code doesn't specify that
setting, so meetings actually start with video being off. This new behavior has
less work for users to do. They don't have to turn on video when joining a call
advertised as "video call". It still respects users' preferences because they
can still configure their own personal setting that overrides the meeting
defaults.

The Zoom API documentation can be found at
https://developers.zoom.us/docs/api/rest/reference/zoom-api/methods/#operation/meetingCreate

Fixes #26549.
2023-08-28 18:32:20 -07:00
Sahil Batra ada2991f1c users: Send stream creation/deletion events on role change.
We now send stream creation and stream deletion events on
changing a user's role because a user can gain or lose
access to some streams on changing their role.
2023-08-25 12:56:36 -07:00
Sahil Batra 5e1eb3cd44 events: Fix applying stream creation events in apply_event.
There was a bug in apply_event code where only a stream which
is not private is added to the "never_subscribed" data after
a stream creation event. Instead, it should be added to the
"never_subscribed" data irrespective of permission policy of
the stream as we already send stream creation events only to
those users who can access the stream. Due to the current
bug, private streams were not being added to "never_subscribed"
data in apply_event for admins as well. This commit fixes it
and also makes sure the "never_subscribed" list is sorted
which was not done before and was also a bug.

The bugs mentioned above were unnoticed as the tests did not
cover these cases and this commit also adds tests for those
cases.
2023-08-25 12:56:36 -07:00
Mateusz Mandera c908b518ef CVE-2023-32678: Prevent unauthorized editing/deletion in priv streams.
Users who used to be subscribed to a private stream and have been
removed from it since retain the ability to edit messages/topics, and
delete messages that they used to have access to, if other relevant
organization permissions allow these actions. For example, a user may be
able to edit or delete their old messages they posted in such a private
stream. An administrator will be able to delete old messages (that they
had access to) from the private stream.

We fix this by fixing the logic in has_message_access (which lies at the
core of our message access checks - access_message() and
bulk_access_messages())
to not rely on only a UserMessage row for checking access but also
verify stream type and subscription status.
2023-08-25 14:10:27 -04:00
Zixuan James Li a081428ad2 user_groups: Make locks required for updating user group memberships.
**Background**

User groups are expected to comply with the DAG constraint for the
many-to-many inter-group membership. The check for this constraint has
to be performed recursively so that we can find all direct and indirect
subgroups of the user group to be added.

This kind of check is vulnerable to phantom reads which is possible at
the default read committed isolation level because we cannot guarantee
that the check is still valid when we are adding the subgroups to the
user group.

**Solution**

To avoid having another transaction concurrently update one of the
to-be-subgroup after the recursive check is done, and before the subgroup
is added, we use SELECT FOR UPDATE to lock the user group rows.

The lock needs to be acquired before a group membership change is about
to occur before any check has been conducted.

Suppose that we are adding subgroup B to supergroup A, the locking protocol
is specified as follows:

1. Acquire a lock for B and all its direct and indirect subgroups.
2. Acquire a lock for A.

For the removal of user groups, we acquire a lock for the user group to
be removed with all its direct and indirect subgroups. This is the special
case A=B, which is still complaint with the protocol.

**Error handling**

We currently rely on Postgres' deadlock detection to abort transactions
and show an error for the users. In the future, we might need some
recovery mechanism or at least better error handling.

**Notes**

An important note is that we need to reuse the recursive CTE query that
finds the direct and indirect subgroups when applying the lock on the
rows. And the lock needs to be acquired the same way for the addition and
removal of direct subgroups.

User membership change (as opposed to user group membership) is not
affected. Read-only queries aren't either. The locks only protect
critical regions where the user group dependency graph might violate
the DAG constraint, where users are not participating.

**Testing**

We implement a transaction test case targeting some typical scenarios
when an internal server error is expected to happen (this means that the
user group view makes the correct decision to abort the transaction when
something goes wrong with locks).

To achieve this, we add a development view intended only for unit tests.
It has a global BARRIER that can be shared across threads, so that we
can synchronize them to consistently reproduce certain potential race
conditions prevented by the database locks.

The transaction test case lanuches pairs of threads initiating possibly
conflicting requests at the same time. The tests are set up such that exactly N
of them are expected to succeed with a certain error message (while we don't
know each one).

**Security notes**

get_recursive_subgroups_for_groups will no longer fetch user groups from
other realms. As a result, trying to add/remove a subgroup from another
realm results in a UserGroup not found error response.

We also implement subgroup-specific checks in has_user_group_access to
keep permission managing in a single place. Do note that the API
currently don't have a way to violate that check because we are only
checking the realm ID now.
2023-08-24 17:21:08 -07:00
Zixuan James Li 006b2acd5d user_groups: Make acting user requried for deletion.
When doing a checked deletion, the acting user is always required.
It is effectively the user_profile parameter we had been previously
requiring.
2023-08-24 17:21:08 -07:00
Samuel 3ce7b77092 typing: Add typing constants to the post register api response.
Adds typing notification constants to the response given by
`POST /register`. Until now, these were hardcoded by clients
based on the documentation for implementing typing notifications
in the main endpoint description for `api/set-typing-status`.

This change also reflects updating the web-app frontend code
to use the new constants from the register response.

Co-authored-by: Samuel Kabuya <samuel.mwangikabuya@kibo.school>
Co-authored-by: Wilhelmina Asante <wilhelmina.asante@kibo.school>
2023-08-23 16:36:44 -07:00
evykassirer 0289beb784 emoji: Match emoji sequences in markdown.
Fixes #11767.

Previously multi-character emoji sequences weren't matched in the
emoji regex, so we'd convert the characters to separate images,
breaking the intended display.

This change allows us to match the full emoji sequence, and
therefore show the correct image.
2023-08-23 16:18:15 -07:00
Sahil Batra e9a6f69849 tests: Access realm directly from ScheduledMessage object.
There is no need to get realm for sender as ScheduledMessage
object also has realm field.

There is no direct benefit of this change but it is nice to
maintain the pattern which we want to follow in the code
in tests as well.
2023-08-23 11:38:32 -07:00
Sahil Batra 7295028194 message: Access realm object directly from message.
We can directly get the realm object from Message object now
and there is no need to get the realm object from "sender"
field of Message object.

After this change, we would not need to fetch "sender__realm"
field using "select_related" and instead only passing "realm"
to select_related when querying Message objects would be enough.

This commit also updates a couple of cases to directly access
realm ID from message object and not message.sender. Although
we have fetched sender object already, so accessing realm_id
from message directly or from message.sender should not matter,
but we can be consistent to directly get realm from Message
object whenever possible.
2023-08-23 11:38:32 -07:00
Sahil Batra 4f30447b95 test_markdown: Set realm for Message objects.
We do not set realm to Message objects defined for markdown tests
and this works because we currently access realm from sender object.

This commit changes the code to set realm in Message objects as
we would be accessing realm from Message object directly in further
commits.
2023-08-23 11:38:32 -07:00
Sahil Batra 7137eba222 streams: Don't compute traffic data for sub objects in zephyr realm.
We set stream_weekly_traffic field to "null" for Subscription
objects in zephyr mirror realm as we do not need stream traffic
data in zephyr mirror realm. This makes the subscription data
consistent with steams data.

This commit also udpates test to check never_subscribed data for
zephyr mirror realm.
2023-08-21 15:21:58 -07:00
Lauryn Menard 5d5a578d6c scheduled-mails: Migrate existing scheduled emails to new templates.
Migrates existing ScheduledEmails for onboarding emails that have
either "zerver/emails/followup_day1" or "zerver/emails/followup_day2"
as the email template prefix to instead use the new template
prefixes "zerver/emails/account_registered" and
"zerver/emails/zulip_onboarding_topics".
2023-08-18 16:51:46 -07:00
Lauryn Menard 5e29e025c5 email-templates: Add zulip_onboarding_topics email templates.
The "followup_day2" email template name is not clear or descriptive
about the purpose of the email. Creates a duplicate of those email
template files with the template name "zulip_onboarding_topics".

Because any existing scheduled emails that use the "followup_day2"
templates will need to be updated before the current templates can
be removed, we don't do a simple file rename here.
2023-08-18 16:25:48 -07:00
Lauryn Menard c491bef07b email-templates: Add account_registered email templates.
The "followup_day1" email template name is not clear or descriptive
about the purpose of the email. Creates a duplicate of those email
template files with the template name "account_registered".

Because any existing scheduled emails that use the "followup_day1"
templates will need to be updated before the current templates can
be removed, we don't do a simple file rename here.
2023-08-18 16:25:48 -07:00
Anders Kaseorg c43629a222 ruff: Fix PLW1510 `subprocess.run` without explicit `check` argument.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-08-17 17:05:34 -07:00
Anders Kaseorg 1df3db286b test_import_export: Fix incorrect type annotation.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-08-17 17:05:34 -07:00
Zixuan James Li 30495cec58 migration: Rename extra_data_json to extra_data in audit log models.
This migration applies under the assumption that extra_data_json has
been populated for all existing and coming audit log entries.

- This removes the manual conversions back and forth for extra_data
throughout the codebase including the orjson.loads(), orjson.dumps(),
and str() calls.

- The custom handler used for converting Decimal is removed since
DjangoJSONEncoder handles that for extra_data.

- We remove None-checks for extra_data because it is now no longer
nullable.

- Meanwhile, we want the bouncer to support processing RealmAuditLog entries for
remote servers before and after the JSONField migration on extra_data.

- Since now extra_data should always be a dict for the newer remote
server, which is now migrated, the test cases are updated to create
RealmAuditLog objects by passing a dict for extra_data before
sending over the analytics data. Note that while JSONField allows for
non-dict values, a proper remote server always passes a dict for
extra_data.

- We still test out the legacy extra_data format because not all
remote servers have migrated to use JSONField extra_data.
This verifies that support for extra_data being a string or None has not
been dropped.

Co-authored-by: Siddharth Asthana <siddharthasthana31@gmail.com>
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2023-08-16 17:18:14 -07:00
Sahil Batra 98b0fa8ae2 bots: Remove subscription from inaccessible streams on reactivating bot.
This commit removes the private stream suscriptions of the bot if the
original owner is deactivated and we change the owner to the user who
is reactivating the bot. We unsusbcribe the bot from private streams
that the new owner is not subscribed to.

Fixes part of #21700.
2023-08-16 15:37:37 -07:00
Sahil Batra 35d5609996 bots: Remove private stream subscriptions on changing bot owner.
We remove bot's subscriptions for private streams to which the
new owner is not subscribed and keep the ones to which the new
owner is subscribed on changing owner.

This commit also changes the code for sending subscription
remove events to use transaction.on_commit since we call
the function inside a transactopn in do_change_bot_owner and
this also requires some changes in tests in test_events.
2023-08-16 15:37:37 -07:00
Sahil Batra ba00907946 bots: Re-parent bot while reactivating if original owner is deactivated.
Since the original owner is deactivated, we change the owner to the
admin who is reactivating the bot.
2023-08-16 15:37:37 -07:00
Lauryn Menard 1713449639 api-docs: Fix documentation of `realm_emoji` in register response.
Fixes the `/api/register-queue` endpoint documentation so that the
`realm_emoji` has the correct type, object that contains objects.

By correcting the API documentation, we also fix an error in the
test for the events system, which had been relying on the API
documentation having a list as a possible type for `realm_emoji`
in the register response.
2023-08-16 14:32:20 -07:00
Prakhar Pratyush 665f491f22 message_edit: Fix wildcard_mentioned flag set for all user-messages.
Earlier, for topic wildcard mentions, the 'wildcard_mentioned'
flag was set for all the user-messages. (similar to stream wildcard
mention).

The flag should be set for the topic participants only.

The bug was introduced in 4c9d26c.
2023-08-16 11:31:56 -07:00
Prakhar Pratyush 379a08eb1e message_send: Fix wildcard_mentioned flag unset for few participants.
For topic wildcard mentions, the 'wildcard_mentioned' flag is set
for those user messages having 'user_profile_id' in
'topic_participant_user_ids', i.e. all topic participants.

Earlier, the flag was set if the 'user_profile_id' exists in
'all_topic_wildcard_mention_user_ids'.
'all_topic_wildcard_mention_user_ids' contains the ids of those
users who are topic participants and have enabled notifications
for '@topic' mentions.

The earlier approach was incorrect, as it would set the
'wildcard_mentioned' flag only for those topic participants
who have enabled the notifications for '@topic' mention instead
of setting the flag for all the topic participants.

The bug was introduced in 4c9d26c.
2023-08-16 11:31:56 -07:00
Tim Abbott ea83c911e9 Revert "narrow: Fix topic highlighting issue with apostrophes in search results."
This reverts commit baede93f69.

This failed tests after rebasing it on top of
5151dd7ff8.
2023-08-15 17:51:03 -07:00
Akshat baede93f69 narrow: Fix topic highlighting issue with apostrophes in search results.
This commit addresses the issue where the topic highlighting
in search results was offset by one character when an
apostrophe was present. The problem stemmed from the disparity
in HTML escaping generated by the function `func.escape_html` which
is used to obtain `topic_matches` differs from the escaping performed
by the function `django.utils.html.escape` for apostrophes (').

func.escape_html | django.utils.html.escape
-----------------+--------------------------
      &#39;      |           &#x27;

To fix this SQL query is changed to return the HTML-escaped
topic name generated by the function `func.escape_html`.

Fixes: #25633.
2023-08-15 17:29:20 -07:00
Akshat 5151dd7ff8 messages_in_narrow: Use `add_narrow_conditions`.
The `messages_in_narrow_backend` function was directly
calling `NarrowBuilder` instead of utilizing the
`add_narrow_conditions` method like `fetch_messages` does.

This behaviour was not combing any search operands together
as it happens inside the `add_narrow_conditions`.

Fixes: https://chat.zulip.org/#narrow/stream/3-backend/topic/messages_in_narrow_backend.20calling.20NarrowBuilder.20directly/near/1611193.

Test added by tabbott.

Signed-off-by: Akshat <akshat25iiit@gmail.com>
2023-08-15 13:18:10 -07:00
Joelute eb78264162 navbar_alerts: Delay showing "Complete the organization profile" banner.
Currently, we are displaying the "Complete the organization profile"
banner immediately after the organization was created. It's important to
strongly encourage orgs to configure their profile, so we should delay
showing the banner if the profile has not been configured after 15 days.
Thus also allows the users to check out Zulip and see how it works before
configuring the organization settings.

Fixes: #24122.
2023-08-15 10:46:33 -07:00
Lauryn Menard 571bd16ba4 test-openapi: Add coverage for documented fetch API key endpoints.
Previoulsy, test_openapi_arguments had assumed that an endpoint
not using rest_dispatch used the GET method for the request. This
was not the case for the "/fetch_api_key" and "/dev_fetch_api_key"
endpoints, which is why those endpoints were marked as pending
even though they were documented in `zerver/openapi/zulip.yaml`.

Updates test_openapi_arguments to check a set of endpoints that
are documented and don't use the GET method so that these endpoints
can be tested and removed from the pending_endpoints set.
2023-08-15 09:52:55 -07:00
Zixuan James Li 37660dd0e7 linkifier: Support reordering linkifiers.
This adds API support to reorder linkifiers and makes sure that the
returned lists of linkifiers from `GET /events`, `POST /register`, and
`GET /realm/linkifiers` are always sorted with the order that they
should processed when rendering linkifiers.

We set the new `order` field to the ID with the migration. This
preserves the order of the existing linkifiers.

New linkifiers added will always be ordered the last. When reordering,
the `order` field of all linkifiers in the same realm is updated, in
a manner similar to how we implement ordering for
`custom_profile_fields`.
2023-08-14 15:21:48 -07:00
Zixuan James Li 011b4c1f7a populate_db: Populate linkifiers.
The curl examples of reordering linkifiers require there to be some
linkifiers in the database to be reordered. This adjusts some test cases
so they do not assume that there is no linkifier in the test db.
2023-08-14 15:21:48 -07:00
Zixuan James Li aa5765ca64 test_openapi: Use subtests for arguments test.
Each unittest subTest can fail without interrupting the other subTests.
By wrapping the test for each view function, we can get all validation
errors at once, which can be useful if multiple endpoints are updated.

More importantly, if the test fails anywhere inside test_openapi but
before the formatted output is printed, we will not lose the information
of which view function fails the validation. Because we attach the name
of the function to the subTest:

```
FAIL: test_openapi_arguments (zerver.tests.test_openapi.OpenAPIArgumentsTest) [zerver.views.alert_words.add_alert_words]
```
2023-08-14 14:24:16 -07:00
Zixuan James Li 6c9e89e6c5 test_openapi: Extract test_openapi_arguments_for_endpoint. 2023-08-14 14:24:16 -07:00
Zixuan James Li e8b7aad462 requests: Split out test_has_request_variables. 2023-08-11 16:43:12 -07:00
Zixuan James Li c9a299a8f8 validators: Split out test_validators. 2023-08-11 16:43:12 -07:00
Steve Howell 51db22c86c per-request caches: Add per_request_cache library.
We have historically cached two types of values
on a per-request basis inside of memory:

    * linkifiers
    * display recipients

Both of these caches were hand-written, and they
both actually cache values that are also in memcached,
so the per-request cache essentially only saves us
from a few memcached hits.

I think the linkifier per-request cache is a necessary
evil. It's an important part of message rendering, and
it's not super easy to structure the code to just get
a single value up front and pass it down the stack.

I'm not so sure we even need the display recipient
per-request cache any more, as we are generally pretty
smart now about hydrating recipient data in terms of
how the code is organized. But I haven't done thorough
research on that hypotheseis.

Fortunately, it's not rocket science to just write
a glorified memoize decorator and tie it into key
places in the code:

    * middleware
    * tests (e.g. asserting db counts)
    * queue processors

That's what I did in this commit.

This commit definitely reduces the amount of code
to maintain. I think it also gets us closer to
possibly phasing out this whole technique, but that
effort is beyond the scope of this PR. We could
add some instrumentation to the decorator to see
how often we get a non-trivial number of saved
round trips to memcached.

Note that when we flush linkifiers, we just use
a big hammer and flush the entire per-request
cache for linkifiers, since there is only ever
one realm in the cache.
2023-08-11 11:09:34 -07:00
Steve Howell 751b8b5bb5 tests: Flush per-request caches automatically for query counts. 2023-08-11 11:09:34 -07:00
Steve Howell 730ae61ce5 tests: Improve linkifiers test.
We test at a higher level now.
2023-08-11 11:09:34 -07:00
Steve Howell 549891266d tests: Add assert_memcached_count.
We use a specific name to distinguish from other caches
like per-request caches.
2023-08-11 11:09:34 -07:00
Tim Abbott 0a181bca86 tests: Fix a query count incorrect due to rebase. 2023-08-10 18:32:10 -07:00
Steve Howell f8ec00b895 mypy: Improve type checks for user display recipients. 2023-08-10 18:13:43 -07:00
Steve Howell 63c0ed303d tests: Mock Recipient.label.
We may eventually want to decouple how we send recipients
over the wire from how we represent them in debugging.
2023-08-10 18:13:43 -07:00
Steve Howell a54760da0e tests: Add assert_message_stream_name
The get_display_recipient helper is a clumsy way to get
stream names, and it's not even representative of how
most of our code retrieves stream names.

The new helper also double-checks that the Stream
object has the correct recipient id.
2023-08-10 18:13:43 -07:00
Steve Howell df068ae7a5 tests: Test directly for stream name. 2023-08-10 18:13:43 -07:00
Steve Howell 6ff7c17f82 tests: Avoid Union type to verify stream names.
There's also no need to fetch a full Stream object when
the thing being verified is just that the display_recipient
field matches the stream name.
2023-08-10 18:13:43 -07:00
Steve Howell 538f498447 tests: Fix clumsy narrow test.
We now explicitly write messages to three different streams,
as well a DM, to make sure each narrow result filters out
all the noise.
2023-08-10 18:13:43 -07:00
Prakhar Pratyush 860eee94fd notifications: Rename 'pm' to 'dm' in 'RecipientInfoResult' dataclass.
This commit renames the keyword 'pm' to 'dm' in the
'pm_mention_email_disabled_user_ids' and
'pm_mention_push_disabled_user_ids' attributes of the
'RecipientInfoResult' dataclass.

'pm' and 'dm' are the acronyms for 'private message' and
'direct message' respectively.

It includes 'TODO/compatibility' code to support the old format
fields in the tornado queues during the Zulip server upgrades.
2023-08-10 17:41:49 -07:00
Prakhar Pratyush c4e4737cc6 notification_trigger: Rename `private_message` to `direct_message`.
This commit renames the 'PRIVATE_MESSAGE' attribute of the
'NotificationTriggers' class to 'DIRECT_MESSAGE'.

Custom migration to update the existing value in the database.

It includes 'TODO/compatibility' code to support the old
notification trigger value 'private_message' in the
push notification queue during the Zulip server upgrades.

Earlier 'private_message' was one of the possible values for the
'trigger' property of the '[`POST /zulip-outgoing-webhook`]' response;
Update the docs to reflect the change in the above-mentioned trigger
value.
2023-08-10 17:41:49 -07:00
Sahil Batra 0e23280b4f tests: Pass required args to select_related call for Message objects.
This commit adds code to pass all the required arguments to
select_related call for Message objects such that only the
required related fields are fetched from the database.

Previously, we did not pass any arguments to select_related,
so all the directly and indirectly related fields were fetched
when many of them were actually not being used and made the
query unnecessarily complex.
2023-08-10 17:35:43 -07:00
Sahil Batra ebd91b152c email_mirror: Pass required args to select_related.
This commit adds code to pass all the required arguments to
select_related call for MissedMessageEmailAddress such that
only the required related fields are fetched from the database.

Previously, we did not pass any arguments to select_related,
so all the directly and indirectly related fields were fetched
when many of them were actually not being used and made the
query unnecessarily complex.
2023-08-10 17:35:43 -07:00
Sahil Batra ab488010b3 models: Pass args to select_related in get_stream_by_id_in_realm.
This commit updates the code to pass "realm" and "recipient" as
arguments to select_related call in get_stream_by_id_in_realm.

Previously, since there was no arguments, it fetched
can_remove_subscribers_group and the related fields of
"Realm" model as well which were not being used, but
did not fetch "recipient" as it is a nullable field.
2023-08-10 17:35:43 -07:00
Sahil Batra 91a58d026b models: Remove get_huddle_recipient and use get_or_create_huddle.
This commit removes get_huddle_recipient function and we now use
get_or_create_huddle in get_recipient_from_user_profiles.

As a result of this change, we do not fetch the recipient from
Huddle object but instead get it using the "id" and "recipient_id"
fields available from Huddle object like we do for a personal
message. This change allows us to not fetch recipient object
using select_related when querying the Huddle object.
2023-08-10 17:35:43 -07:00