Commit Graph

17786 Commits

Author SHA1 Message Date
Anders Kaseorg a688e753de test_helpers: Fix logging in cursor_executemany mock.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-11-15 15:27:54 -08:00
Anders Kaseorg cb9a04d3e3 test_create_video_call: Add missing not None assertions.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-11-15 15:27:54 -08:00
Anders Kaseorg 2c9c105bbb db: Check connection type in wrapper_execute.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-11-15 15:27:54 -08:00
Anders Kaseorg 248ee82db0 migrations: Remove useless null argument for ManyToManyField.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-11-15 15:27:54 -08:00
Anders Kaseorg 9552014047 models: Include explicit app name in ManyToManyField reference strings.
https://github.com/typeddjango/django-stubs/issues/1802

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-11-15 15:27:54 -08:00
Tim Abbott 2e2997bd7d typing: Limit typing notifications in large streams. 2023-11-15 09:42:25 -08:00
Alex Vandiver 091e2f177b linkifiers: Match JS implementation to server implementation.
Since the server-side implementation no longer uses look-ahead
or (more importantly) look-behind, it is possible to exactly implement
in Javascript.  This removes a common class which would prevent local
echo.

This requires reworking the topic linking algorithm, to march the
server's as well.  The tests and behaviour are adjusted in so doing --
previously, the JS implementation would have linked `#foo` with a
`foo` regex on the linkifier, but the server implementation would not
have.
2023-11-14 20:43:39 -08:00
Alex Vandiver 70b20e9d2b markdown: Use \p{White_Space} equivalent for linkifier boundaries.
We do not use \p{White_Space} itself because re2 does not support it.
2023-11-14 20:43:39 -08:00
Alex Vandiver 549dd8a4c4 markdown: Expand prepare_linkifier_pattern to make it more legible. 2023-11-14 20:43:39 -08:00
Sahil Batra 532293c4cf streams: Unset is_web_public and is_realm_public fields on attachments.
This commit adds code to unset is_web_public and is_realm_public fields
on attachments when deactivating a stream as we do not want to allow
spectators to access them after the stream is deactivated.

This commit also adds a comment explaining why we don't use
do_change_stream_permission to set the privacy fields on deactivating
a stream.

Fixes #27634.
2023-11-13 10:26:14 -08:00
Sahil Batra 30e2c14f4a streams: Unset is_realm_public on attachment when unarchiving streams.
We did not unset is_realm_public field on attachements when unarchiving
streams, but we do unset is_web_public field. This commit adds code to
unset the is_realm_public field as well as we make the stream private
while unarchiving it.
2023-11-13 10:26:14 -08:00
Alex Vandiver 1f3a53db2d cache: Drop the huddle cache.
This cache was only used in one place, which is infrequently
called (only when sending messages, or searching explicitly for a list
of users) and the overhead of maintaining the cache is not worth
trying to avoid the well-indexed lookup of the huddle.
2023-11-13 10:01:29 -08:00
Sahil Batra 8f79cec51a users: Pass bogus data for inaccessible users.
We now pass bogus data for inaccessible users when sending
the users data in "realm_users" field of "register" response
or when using endpoints like "GET /users" to get data of
all the users in realm.

We would add a client capability field in future commits
such that new clients would receive data only for accessible
users and they can form the bogus data by themselves.
2023-11-13 08:04:45 -08:00
Sahil Batra aedb9b23b4 models: Pass "realm__can_access_all_users_group" to select_related. 2023-11-13 08:04:45 -08:00
Sahil Batra c82bb3ec76 models: Add can_access_all_users_group setting.
This commit adds new setting for controlling who can access
all users in the realm which would have "Everyone" and
"Members only" option.

Fixes part of #10970.
2023-11-13 08:04:45 -08:00
Sahil Batra f20d427a4d users: Remove unnecessary comment.
This comment can be removed as the code was updated in 2ab71fb67d.
2023-11-13 08:04:45 -08:00
Sahil Batra df01d0725c message_send: Remove select_related call for stream.
We do not access any related field from the stream object,
so there is no need for select_related.
2023-11-13 08:04:45 -08:00
Mateusz Mandera 48db4bf854 counts: Add new mobile_pushes RemoteRealmCount stats.
This requires a bit of complexity to avoid a name collision in
COUNT_STATS with the RemoteInstallationCount stats with the same name.
2023-11-10 16:09:11 -08:00
Mateusz Mandera 8a6d5b4997 counts: Add new Add new mobile_pushes_sent::day LoggingCountStat.
This is a CountStat for tracking how many mobile notifications the
server requested.
1. On a self-hosted server, that means requesting from the push bouncer.
2. On a server that's its own push bouncer, that's just the number
   directly sent.

This number has room for inaccuracy due to incrementing by the number of
user devices on a self-hosted server, as it doesn't account for errors
that may occur in the GCM/APNs low-level sending codepaths on the bouncer.

Also tests that a server that's its own push bouncer correctly
increments its mobile_pushes_sent::day CountStat, by basing it on the
values returned from the send_apple/android_push_notification functions
which tell us the actual number of successfully sent notifications.

Since the return values of send_..._push_notification are now
used in those codepaths, we need to tweak our mocks in some unrelated
tests to set up some return value to avoid errors.
2023-11-10 16:09:11 -08:00
Prakhar Pratyush c597de6a1d topic_mentions: Rename wildcard_mentioned to stream_wildcard_mentioned.
Rename the existing 'wildcard_mentioned' flag to
'stream_wildcard_mentioned'.

The 'wildcard_mentioned' flag is deprecated and exists for
backwards compatibility.

We have two separate flags for stream and topic wildcard mentions,
i.e., 'stream_wildcard_mentioned' and 'topic_wildcard_mentioned',
respectively.

* stream wildcard mentions: `@all`, `@everyone`, and `@stream`
* topic wildcard mentions: `@topic`

The `wildcard_mentioned` flag is included in the events and
API response if either `stream_wildcard_mentioned` or
`topic_wildcard_mentioned` is set.
2023-11-10 11:06:26 -08:00
Prakhar Pratyush 24fa361f40 migration: Clear old data for unused usermessage flags.
In c37871ac3a, we renamed the
two unused and historical bits of the 'flags' bitfield of
the 'UserMessage' table:

* 'summarize_in_home' to 'topic_wildcard_mentioned'
* 'summarize_in_stream' to 'group_mentioned'

This commit clears out the old data for those bits.

Additionally, we are clearing 'force_expand' and 'force_collapse'
unused flags to save future work.
2023-11-10 11:06:26 -08:00
Mateusz Mandera 1312c7ccd7 zilencer: Add mechanism to update RemoteRealm when Realm is changed.
This requires a migration to allow RemoteRealmAuditLog.remote_id to be
NULL, and to add a RemoteRealmAuditLog.remote_realm.
2023-11-08 15:54:22 -08:00
Mateusz Mandera 76e0511481 zilencer: Add new model RemoteRealm and send the data to the bouncer.
Add the new model for recording basic information about Realms on remote
server, to go with the other analytics data. Also adds necessary changes
to the bouncer endpoint and the send_analytics_to_push_bouncer()
function to submit such Realm information.
2023-11-08 15:54:22 -08:00
Tim Abbott 0d525cf644 integrations: Recommend new integration URL tool.
Fixes part of #25976.
2023-11-08 14:19:05 -08:00
David Rosa 0c4ef58acd help: Document new help menu.
Updates help center docs to reflect the new dedicated help menu.

- Adds `keyboard_tip` to document the `?` keyboard shortcut.

Fixes #27384.
2023-11-08 14:17:13 -08:00
Alex Vandiver acb4e9e783 cache_helpers: Show item and DB query counts. 2023-11-08 11:00:00 -08:00
roanster007 dc492867af user_mention: Fix mentions of deactivated users.
Previously, when a deactivated user was mentioned, he wasn't
rendered as a Pill. This is because the dataset for validating mentions
only included active users, which is fixed by removing that filter.

To allow only silent mentions of them, an extra is_active property
added to FullNameInfo class, which is populated from the query,
which tells if user is deactivated. This is used to convert any
mentions of them to silent mentions in the backend markdown.

Fixes #26857
2023-11-08 09:48:31 -08:00
Sahil Batra 617d2d509c users: Update format_user_row to return a TypedDict.
This commit updates format_user_row to return a TypedDict.

This commit is a prep commit for feature of restricting user
access such that code can be easy to read and understand when
we add that feature.
2023-11-08 09:22:26 -08:00
Sahil Batra 2ab71fb67d users: Update user_profile_to_user_row to return a TypedDict.
This commit updates user_profile_to_user_row to return a TypedDict
and also updates the return type of get_realm_user_dicts to be a
TypedDict.

This commit is a prep commit for feature of restricting user
access such that code can be easy to read and understand when
we add that feature.
2023-11-08 09:22:26 -08:00
Sahil Batra deb8431a47 users: Rename get_raw_user_data to get_users_for_api.
This is a prep commit for adding feature of restricting
user access to guests such that we can keep the code
easy to read and understand when that feature is added.
2023-11-08 09:22:26 -08:00
Greg Price 7a0ebc0d18 push_notifs: Log an error if ios_app_id missing on any APNs token. 2023-11-07 16:19:42 -08:00
Greg Price f109e3b598 push_notifs: Backfill ios_app_id on bouncer. 2023-11-07 16:19:42 -08:00
Greg Price be2a9a03d0 push_notifs: Check app ID has a plausible shape at bouncer. 2023-11-07 16:19:42 -08:00
Greg Price 1b2178f558 push_notifs: Require ios_app_id on register at bouncer, too.
We'll need this information in order to properly direct APNs
notifications.  Happily, the Zulip server always sends it when
registering an APNs token; and it appears it always has done so
since the commit:

cddee49e7 Add support infrastructure for push notification bouncer service.

back in 2016.  So there's no compatibility issue from requiring it.
2023-11-07 16:19:42 -08:00
Greg Price 9c2d53bcef push_notifs: Stop dropping ios_app_id on floor at bouncer.
This missing `REQ` call has meant we just drop this parameter:
even though the remote Zulip server passes it (for all APNs tokens),
we never notice and never store it.  Fix that.
2023-11-07 16:19:42 -08:00
Greg Price 9072ddda05 push_notifs: Remove remaining references to ZULIP_IOS_APP_ID setting. 2023-11-07 16:19:42 -08:00
Greg Price a9796ec503 push_notifs: Check app ID has a plausible shape. 2023-11-07 16:19:42 -08:00
Greg Price fb3af7fbcb push_notifs: Make appid required on add_apns_device_token.
We're going to need to use this information, so we shouldn't just
assume a value; the client should tell us the actual value.

Conveniently, the Zulip mobile app does already pass this parameter
and has since forever.  So we can just start requiring it, with no
compatibility constraint.
2023-11-07 16:19:42 -08:00
Greg Price 5d0d35d3f5 push_notifs: Always supply appid to add_apns_device_token in tests.
We already always pass this parameter from the mobile client,
so this makes the tests more realistic already.  And we'll shortly
be making this parameter required.
2023-11-07 16:19:42 -08:00
Greg Price 3893a70888 push_notifs: Cut spurious token_kind arguments in some test data.
These endpoints distinguish the token kinds by having different
URL paths, and don't take any argument by this name.
2023-11-07 16:19:42 -08:00
Alya Abbott 8fd631bf06 integrations-docs: Update intro content for Slack incoming webhooks. 2023-11-07 12:59:24 -08:00
Lauryn Menard 1618060a3e integrations-docs: Update both Slack integration articles.
Updates the Slack integration page to not describe adding a stream
or topic parameter to the URL query since that's not supported by
the current integration implementation.

Updates the Slack-compatible webhook integration page to have the
extra notes about the integration at the top of the page. Also,
removes the reference to a screenshot of the webhook since there
isn't one.
2023-11-07 12:59:24 -08:00
Prakhar Pratyush 94679d590f email_notification: Include prior message context only when mentioned.
Earlier, email message notifications included prior messages sent
to the same topic for context. This is more confusing than helpful
for messages that the user is likely to have received notifications
for all the prior messages in the conversation already (or read them
in the Zulip UI).

Now, we include prior context only when the user is mentioned via
personal, group, stream or topic wildcard mention.

Fixes #27479.
2023-11-07 11:44:38 -08:00
Prakhar Pratyush c0f445294c test_message_notif_email: Verify email has multiple stream messages.
This commit improves the test to explicitly verify that multiple
messages that were sent in quick succession to a topic are included
in the email body when we have email notifications enabled for a
given stream.

Earlier, the test was only verifying the email subject and the fact
that only one email was sent.

It is important to verify the fact that all the messages sent to a
topic in quick succession should be included in the email body.
2023-11-07 11:44:38 -08:00
Lauryn Menard fff2c9e47b integrations-docs: Update GitHub doc for new URL instructions. 2023-11-07 11:21:05 -08:00
Prakhar Pratyush e6e156709a typing_notifications: Don't notify long_term_idle subscribers.
The event for stream typing notifications is no longer sent
to the long_term_idle subscribers of the stream.

This helps to reduce the tornado's work of parsing super-long
JSON-encoded lists of user IDs in large streams. Now the lists
are shorter.
2023-11-07 09:30:27 -08:00
Aman Agrawal be6f467f42 home: Add `sponsorship_pending` page_param.
This will be used in gear menu to inform admin of their
sponsorship application status.

This includes some additional tweaks for when to show
billing and plans to users.
2023-11-06 14:55:14 -08:00
Aman Agrawal 00df9c6c18 urls: Show sponsorship page on a new URL.
Fixes #27377

Replaced #sponsorship links with sponsorship/ links and in other
places liking gear menu.
2023-11-06 14:55:14 -08:00
David Rosa 56f725e39b help: Document ability to drag-and-drop anywhere to upload a file.
- Replaces the "Via Markdown" tab with "Via drag-and-drop", and
modifies the instructions to explain that you can drag and drop
anywhere in the app, whether or not the compose box is open.
- Adds "Via paste" tab for the copy-pasting instructions.

Fixes #26894.
2023-11-04 12:49:54 -07:00
Aman Agrawal ddab0e6cd2 test_user_groups: Fix flaky tests.
There order of group ids doesn't matter here and thus the
compared values can have the ids in different order and test
should still pass. So, using `set` for comparing unordered
lists seems like the right fix here.
2023-11-04 12:46:06 -07:00
roanster007 ac084ddb0e mention_dataset: Add query to mention cross realm bots as pill.
Previously, cross realm bots were not displayed as mention Pills.

This is because, the data set for validating mentions considers
only the realm id which is None in case of cross realm bots.

Hence, adding an or Q object to it, to also check if
the email is a part of the cross realm bots email, in case the
realm id returns None.

Fixes #26913
2023-11-03 12:29:20 -07:00
Aman Agrawal 9ebb3f06b3 populate_billing_realms: Add script to populate different realms.
These realms are based on different cases of realms which can
exist while billing them and hence are useful for testing.
2023-11-03 08:44:23 -07:00
David Rosa c486ae37d3 help: Document option to paste URL to create a named link.
- Adds a "Via paste" tab to document the new functionality.

Fixes #26892.
2023-11-02 12:44:33 -07:00
Prakhar Pratyush b0ef76bf27 topic_mentions: Set 'topic_wildcard_mentioned' flag on @topic mention.
Earlier, the 'wildcard_mentioned' flag was set for both the
stream and topic wildcard mentions.

Now, the 'topic_wildcard_mentioned' flag is set for topic
wildcard mentions, and the 'wildcard_mentioned' flag is set for
stream wildcard mentions.

We will rename the 'wildcard_mentioned' flag to
'stream_wildcard_mentioned' in a later commit.
2023-11-02 09:25:51 -07:00
Prakhar Pratyush c37871ac3a user_message: Rename unused flags and create an index.
This commit renames the two unused and historical bits of the
'fields' bitfield of the 'UserMessage' and 'ArchivedUserMessage'
tables.

* 'summarize_in_home' to 'topic_wildcard_mentioned'
* 'summarize_in_stream' to 'group_mentioned'

The 'group_mentioned' flag doesn't affect the feature,
but completing the work here helps to save future migration
and indexing efforts on the UserMessage table, as we plan to
use this flag in the future for group mentions.

The unused bits may have old data; we'll clear that in
a separate commit.

It creates the 'zerver_usermessage_any_mentioned_message_id'
index concurrently.
2023-11-02 09:25:51 -07:00
Alex Vandiver 5a44a6624b gitlab: Description can be none. 2023-11-02 09:24:31 -07:00
Sahil Batra bb15b2d708 users: Send "update" events when deactivating or reactivating users.
We now send "realm_user/update" (and "realm_bot/update" for bots)
events with "is_active" field when deactivating and reactivating
users, including bots.

We would want to use "remove" event for a user losing access
to another user for #10970, so it is better to use "update"
event for deactivation as we only update "is_active" field
in the user objects and the clients still have the data for
deactivated users.

Previously, we used to send "add" event for reactivation along
with complete user objects, but clients should have the data
for deactivated users as well, so an "update" event is enough
like we do when deactivating users.
2023-11-02 08:31:12 -07:00
Mateusz Mandera 6819ecee92 zilencer: Add new LoggingCountStat mobile_pushes_forwarded.
This one counts actual successful deliveries.
2023-11-01 17:26:10 -07:00
Mateusz Mandera b7117d51b2 zilencer: Don't allow syncing mobile_pushes_received::day count. 2023-11-01 17:26:10 -07:00
Mateusz Mandera 7eb24ff237 send_test_push_notification_api: Fixes to the API endpoint doc.
Fixes misspellings and bad wording in the original zulip.yaml doc for
this endpoint.
2023-11-01 16:57:04 -07:00
Alex Vandiver 779268c3a4 sentry: Circuit-break on 429s and 500s from Sentry. 2023-11-01 16:16:37 -07:00
Alex Vandiver 631a1def13 portico: Skip footer and adjust header on invalid realm. 2023-11-01 11:24:12 -07:00
Alex Vandiver 54605e54b6 tests: Switch test_i18n fetch helper to use client_foo methods.
38f2a2f475 updated the comment but not the code.  Using
`self.client.post` instead of `self.client_post` means that we do not
set the host headers correctly.
2023-11-01 11:24:12 -07:00
Sahil Batra 47c8e369cf register: Pass the configuration objects for group settings in response.
This commit adds code to pass configuration objects for group
permission settings in register response to clients such that
we do need to duplicate that data in clients and can avoid
future bugs due to inconsistency.

The "server_supported_permission_settings" field is included
in the response if "realm" is present in "fetch_event_types",
as this is what we do for other server-related fields.
2023-11-01 10:42:56 -07:00
Sahil Batra e458b73a01 user_groups: Move constants for system group names to a new class.
This commit moves constants for system group names to a new
"SystemGroups" class so that we can use these group names
in multiple classes in models.py without worrying about the
order of defining them.
2023-11-01 10:42:56 -07:00
Sahil Batra 2c43c2a4db user_groups: Pass config object to access_user_group_for_setting.
We now pass the complete configuration object for a setting to
access_user_group_for_setting instead of passing the configuration
object's fields as different variables.
2023-11-01 10:42:56 -07:00
Sahil Batra 08d41f9eb3 settings: Rename permissions_configuration variable.
This commit renames permissions_configuration variable to
permission_configuration since the object contains config for
a single permission setting and thus permission_configuration
seems like a better name.
2023-11-01 10:42:56 -07:00
Lauryn Menard 6671a4a3f5 api-docs: Update description of update_message_flags op: add event.
Updates the description of update_message_flags op: add event for
details about actions that send/trigger the event and other details
that are useful for client implementations.

Also, links to the above updates in the op: remove variant of the
update_message_flags event.
2023-10-31 14:28:30 -07:00
Lauryn Menard afa5119303 api-docs: Update delete_message event for moved messages case. 2023-10-31 14:28:30 -07:00
Sahil Batra 9a6cf82adc streams: Fix sending stream-related events to guests.
Previous behavior-
- Guest did not receive stream creation events for new
web-public streams.
- Guest did not receive peer_add and peer_remove events
for web-public and subscribed public streams.

This commit fixes the behavior to be -
- Guests now receive stream creation events for new
web-public streams.
- Guest now receive peer_add and peer_remove events for
web-public and subscribed public streams.
2023-10-31 10:54:21 -07:00
Sahil Batra 71b8f49614 streams: Return early if there is no change in subscriptions.
This commit updates code in bulk_remove_subscriptions and
bulk_add_subscriptions to return early if there are no
subscribers to remove or add to the streams.

This change helps us in avoiding unnecessary queries like the
one used to get subscribers list of streams, which is then used
to send events but we would not send any events if no subscribers
are added or removed and some more similar queries.
2023-10-31 10:54:21 -07:00
Sahil Batra cdd15b4a69 test_events: Fix typo in comments. 2023-10-31 10:54:21 -07:00
Chandan Yadav 951e31a154 user_status: Remove `status` field from `UserInfoDict` type. 2023-10-30 11:10:47 -07:00
Anders Kaseorg 14b7d097a0 compilemessages: Normalize language names to NFC.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-10-26 16:27:35 -07:00
Hemant Umre d27a50852e onboarding: Internationalize the introductory messages in streams.
We use `Realm.default_language` value, which is set by selecting
the 'Organization language', to internationalize the introductory
messages of the initial streams.

Fixes #25729.
2023-10-26 16:27:35 -07:00
Hemant Umre ac1f711fef registration: Set the organization language at creation time.
In this commit, we add a new dropdown 'Organization language' on
the `/new` and `/realm/register` pages. This dropdown allows setting
the language of the organization during its creation. This allows
messages from Welcome Bot and introductory messages in streams to be
internationalized.

Fixes a part of #25729.
2023-10-26 16:27:35 -07:00
Sahil Batra c9fccf476a settings: Rename default_view and escape_navigates_to_default_view.
This commit renames default_view and escape_navigates_to_default_view
settings to web_home_view and web_escape_navigates_to_home_view in
database and API to match with our recent renaming of user facing
strings related to this.

We also rename the variables, functions, comments in code and class
names and IDs for elements related to this.
2023-10-26 11:00:12 -07:00
Lauryn Menard 02d6b3e16d onboarding-emails: Add new onboarding email for organization creator.
Adds a new onboarding email `onboarding_team_to_zulip` for the user
who created the new Zulip organization.

Co-authored by: Alya Abbott <alya@zulip.com>
2023-10-25 13:51:03 -07:00
Lauryn Menard f755fd1a3a api-docs: Check for deprecated at top level of endpoint docs.
Previously, we had checked that deprecated parameters and return
values had been marked as `deprecated: true` in the OpenAPI
documentation and had a description with a deprecated note.

Here we extend that check at the top level to deprecated endpoints.

The backend test that catches a failed assertion for this check
is `test_api_doc_endpoints` in zerver/tests/test_docs.py as that
test checks for a success response all pages linked the sidebar of
the API documentation.
2023-10-25 12:21:53 -07:00
Lauryn Menard 94088a7709 api-docs: Add deprecated true to /api/mute-topic endpoint. 2023-10-25 12:21:53 -07:00
Lauryn Menard 8905d918ae api-docs: Document bulk mark as read endpoints as deprecated. 2023-10-25 12:21:53 -07:00
Mateusz Mandera bb63aca8bb send_analytics_to_push_bouncer: Fix a stale comment.
The comment has drifted away from where it should be placed within the
code and also talks about RealmCounts specifically, while we have other
object types that this equally applies to.
2023-10-25 11:09:49 -07:00
Mateusz Mandera 986f2fd962 send_analytics_to_remote_server: Rename to ..._to_push_bouncer.
The former name is kind of misleading - this function is for the remote
server to send analytics to the push bouncer. Under our usual
terminology, a "remote server" is a self-hosted Zulip server. So data is
sent FROM not TO a remote server.
2023-10-25 11:09:49 -07:00
Prakhar Pratyush 3f1cc1b69d api_docs: Update the docs for the `POST /messages` response.
This commit updates the API docs for the optional parameter
'automatic_new_visibility_policy' in the `POST /messages` response.

Changes include:
* Explicitly mention that the new visibility policy is still
sent as `user_topic` event.
* Adds a link as a way to understand the meaning of the enum
values '2' and '3'.

Co-authored-by: Tim Abbott <tabbott@zulip.com>
2023-10-25 10:28:15 -07:00
Prakhar Pratyush b0d588bd98 models: Set defaults for the user_topic policies.
Now that we're enabling the feature in the UI, we should set
these to the planned long-term defaults for these settings.

Also, this commit cleans up the '0476' and '0477' migration
files related to user_topic policies.

'0476' sets 'null=True'
'0477' is noop
'0482' sets the default values and performs backfilling.

Co-authored-by: Tim Abbott <tabbott@zulip.com>
2023-10-25 02:21:03 +00:00
Lauryn Menard a8e3606c2c api-docs: Note that unread_message_ids are sorted in unread_msgs data. 2023-10-24 10:47:14 -07:00
Sayyed Arib Hussain 7e38d06130
create_user: Use silent mentions in user join notices.
This provides more information (email, etc.) when clicked on
while being better adapted to email_address_visibility existing.

Fixes #27243.
2023-10-24 09:58:55 -07:00
David Rosa 447b853ef1 help: Rename finding-a-topic-to-read -> finding-a-conversation-to-read
- Renames filenames to match page title.

Fixes part of #27191.
2023-10-23 16:52:41 -07:00
David Rosa d32a0601cd help: Rename help/reading-topics > help/reading-conversations.
- Renames filenames to match page title.

Fixes part of #27191.
2023-10-23 16:52:41 -07:00
ecxtacy ad63324dce help: Improve user stream management docs.
Documents the procedure to subscribe / unsubscribe a user via their
profile and general stream settings.

Both methods are separated into tabs in the documentation.

Fixes #26902
2023-10-23 14:17:21 -07:00
Anders Kaseorg d2faae36c6 openapi: Remove fix_events.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-10-23 09:07:07 -07:00
Anders Kaseorg ce0040b69f openapi: Add missing user property to deprecated user settings events.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-10-23 09:07:07 -07:00
Anders Kaseorg f256a5a536 openapi: Separate EmojiReactionEvent from EmojiReaction.
EmojiReactionEvent has user.user_id while EmojiReaction has user.id.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-10-23 09:07:07 -07:00
Anders Kaseorg b1f6ff20ef attachments: Correct attachment_id type from string to integer.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-10-22 17:06:34 -07:00
Esther Anierobi e957decd12
email: Drop support for SEND_MISSED_MESSAGE_EMAILS_AS_USER. (#27223)
Originally, this was how the notification emails worked, but that was changed
in 797a7ef97b, with this old behavior 
available as an option.

The footer and from address of emails that are sent when this
setting is set to True are confusing, especially when more people
are involved in a stream and since we have changed the way we send
emails, it should be removed. It’s also not widely used.

Fixes #26609.
2023-10-20 16:38:43 -07:00
Anders Kaseorg 9ba102f3ea openapi: Remove unused display_brief_error parameter.
It’s unused since commit a881918a05
(#24979).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-10-20 10:05:29 -07:00
Sahil Batra 760d793d6b settings: Rename "default" views to "home" views in settings and docs.
This commit renames "default" views to "home" views in the setting
labels, keyboard shortcuts list, help documentation and its urls.

This commit does not do changes in variable and class names, setting
field in database, API docs and changelog.

Fixes part of #27251.
2023-10-18 11:25:20 -07:00
Mateusz Mandera ff81e6bf32 import_util: Remove uuid and uuid_owner_secret from realm dict. 2023-10-18 11:00:49 -07:00
Mateusz Mandera 73cc680289 realm: Add uuid and associated secret columns - backfill migrations. 2023-10-18 11:00:49 -07:00
Mateusz Mandera 3afe585922 realm: Add uuid and associated secret columns.
Thisi and the following commit follow the approach used in
3e2ad84bbe.

First migration requires a server restart - after that any new realms
will be created with the columns set.

The following migrations are in the next commit:

Second migration does a backfill for older realms and can run in the
background while the server is operating normally.

Third migration enforces null=False now that all realms have the columns
set.
2023-10-18 11:00:49 -07:00
Prakhar Pratyush 17a0304309 send_message: Add an optional parameter in the success response.
Add an optional `automatic_new_visibility_policy` enum field
in the success response to indicate the new visibility policy
value due to the `automatically_follow_topics_policy` and
`automatically_unmute_topics_in_muted_streams_policy` user settings
during the send message action.

Only present if there is a change in the visibility policy.
2023-10-17 15:38:16 -07:00
Anders Kaseorg 70af7d7c58 test_user_groups: Fix database access outside test function.
Python evaluates function parameter defaults at definition time, not
call time.  This function wouldn’t work with other realms anyway.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-10-16 15:46:03 -07:00
Anders Kaseorg 94934f49f9 webpack_public_path: Replace with webpack 5 publicPath: "auto" setting.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-10-16 10:36:27 -07:00
Mateusz Mandera 7604c7935c push_notifs: Improve error responses from /test_notification endpoint. 2023-10-12 18:15:00 -07:00
Mateusz Mandera d43be2b7c4 push_notifs: Add endpoint for sending a test notification.
Fixes #23997
2023-10-12 18:15:00 -07:00
Alya Abbott b1e0fee101 help: Add keyboard documentation for default views. 2023-10-12 12:58:20 -07:00
David Rosa 69b8dbeb28 help: Document Inbox view.
- Adds desktop/web instructions.
- Adds #inbox relative link for logged-in users.
- Moves Inbox up in the left sidebar just under "Reading strategies".
- Moves Inbox article content to Markdown include.
- Adds "From the Inbox view" section to "Finding a topic to read",
  "Getting started with Zulip", and "Reading strategies".
- Documents Inbox as a new option for the default web app view.
- Removes unused Markdown link.
- Tweaks subheading to better match help center patterns.
- Add Inbox option in "Configure default settings for new users".
- Adds new tabbed section and instructions for marking messages as
  read and reading topics via the Inbox view.

Fixes #26903.

Co-authored-by: Alya Abbott <alya@zulip.com>
2023-10-12 12:58:20 -07:00
Anders Kaseorg a50eb2e809 mypy: Enable new error explicit-override.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-10-12 12:28:41 -07:00
Anders Kaseorg e245f51974 test_helpers: Remove unnecessary HostRequestMock.body override.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-10-12 12:28:41 -07:00
lapaz 417b29638c messages: Add support to allow bot-owners to delete messages.
This commit adds support to allow bot-owners to delete messages
sent by their bots if they are allowed to delete their own messages
as per "delete_own_message_policy" setting and the message delete
time limit has not passed.
2023-10-12 12:13:39 -07:00
Sahil Batra c51c1d5135 realm: Add setting to control adding "(guest)" indicator to name.
This commit adds new realm setting to control adding "(guest)"
indicator to names of guest users in the UI.

Fixes part of #26700.
2023-10-12 12:06:10 -07:00
Tim Abbott 07e6bcda7c tests: Fix missing setUp calls in ZulipTestCase subclasses.
This may save us from bugs like the ones fixed in
78683c1b9c in the future.
2023-10-12 10:28:04 -07:00
Alex Vandiver 869f96e31b decorator: Drop the stack_info on most WebhookErrors.
`stack_info` shows the stack between where the error was raised and
where it was captured -- which is not interesting when we
intentionally raised it, and know where it will be captured.

Omit the `stack_info` when it will just fill the logs with
uninteresting data.
2023-10-12 10:06:31 -07:00
Alex Vandiver 05d8c9d49e middleware: Don't double-log AnomalousWebhookPayloadError.
These are `WebhookError`s which have a 500 status code; we have logged
them already in the webhook decorator, so just return the content directly.
2023-10-12 10:06:31 -07:00
Alex Vandiver 8a6f96acad decorator: Stop stringifying exceptions unnecessarily. 2023-10-12 10:06:31 -07:00
Alex Vandiver 42b6b03ee7 decorator: Import notify_bot_owner_about_invalid_json.
This no longer causes an import cycle.
2023-10-12 10:06:31 -07:00
Alex Vandiver ea46478686 decorator: Reorganize and comment to clarify behaviour.
This is not a behaviour change, merely a hopefully cleaner way of
expressing the same logic.
2023-10-12 10:06:31 -07:00
Alex Vandiver 0a2ceda199 decorator: Let UnsupportedWebhookEventTypeError go through the logger.
It is not clear why 84723654c8 added these lines, are they are not
related to status codes.

Using an explicit `capture_exception` causes the exception in Sentry
to not have a `logger` field, which is quite useful for filtering.
2023-10-12 10:06:31 -07:00
Alex Vandiver 1e09de1da6 webhooks: Pass helpful strings to UnsupportedWebhookEventTypeError.
This helps understand _what_ event type was not supported.
2023-10-12 10:06:31 -07:00
Alex Vandiver 69326930ff freshping: Check state is supported before using it. 2023-10-12 10:06:31 -07:00
Alex Vandiver 204535926f transifex: Use "event" parameter.
Transifex's webhook documentation[^1] describes an `event` parameter
which is used to distinguish which event type was received.  Dispatch
based on that, and pass that value to UnsupportedWebhookEventTypeError
if need be.

[^1]: https://developers.transifex.com/docs/webhooks
2023-10-12 10:06:31 -07:00
Alex Vandiver c4085d1b8a statuspage: Raise AnomalousWebhookPayloadError on other events.
Their documentation only describes these two event types.
2023-10-12 10:06:31 -07:00
Alex Vandiver bda6a4b7e4 webhooks: Make MissingHTTPEventHeaderError a WebhookError.
Specifically, an AnomalousWebhookPayloadError.
2023-10-12 10:06:31 -07:00
Alex Vandiver 558a47af82 webhooks: All validate_extract_webhook_http_header are fatal.
85f453998e removed the one and only callsite which passed
`fatal=False`; simplify the type for this function.
2023-10-12 10:06:31 -07:00
Aman Agrawal b8b7938eb9 portico: Remove orgs who haven't set their own description. 2023-10-12 09:58:47 -07:00
Prakhar Pratyush c520a96354 typing_indicator: Add a 'stream_id' parameter to 'POST /typing'.
This commit adds a 'stream_id' parameter to the 'POST /typing'
endpoint.

Now, 'to' is used only for "direct" type. In the case of
"stream" type, 'stream_id' and 'topic' are used.
2023-10-12 09:53:09 -07:00
Prakhar Pratyush 78683c1b9c test_event_queue: Add missing "super call" in 'MissedMessageHookTest'.
Earlier, 'MissedMessageHookTest' didn't have 'super().setUp()'
and 'super().tearDown()' in the overrided methods 'setUp' and
'tearDown', respectively, that resulted in cached objects being
used between tests and hence flaky test failures.

This commit adds 'super().setUp()' and 'super().tearDown()'.
2023-10-12 09:46:13 -07:00
Esther Anierobi b2ea3125b2 exports: Improve notifications about completed data exports.
Change the url in the notification message to point to the settings
interface rather than linking to the export directly.

This is a much better user experience in the case that the export has
been deleted since the time the export was requested.

Fixes: #26923.
2023-10-11 17:42:32 -07:00
Alex Vandiver 7c80cbbc77 registration: On Zulip Cloud, show a 500 instead of a config page. 2023-10-11 17:13:01 -07:00
Alex Vandiver db38d43473 registration: Provide a better exception message. 2023-10-11 17:13:01 -07:00
Alex Vandiver e45f74dafb registration: Show SMTP failure page correctly.
232eb8b7cf changed how these pages work, to render inline instead of
serving from a URL, but did not update the SMTP use case; this made
SMTP failures redirect to a 404.
2023-10-11 17:13:01 -07:00
Alex Vandiver 2536dce136 test_helpers: Exempt testing-only endpoint from URL coverage. 2023-10-11 17:13:01 -07:00
Alex Vandiver d98aff26c9 config_error: Return status code 500. 2023-10-11 17:13:01 -07:00
Alex Vandiver a21971ceae config_error: Split error pages into their own pages.
This is much easier to maintain than a giant set of if statements.
2023-10-11 17:13:01 -07:00
Alex Vandiver 5506dffa36 config_error: Coalesce redundant dicts. 2023-10-11 17:13:01 -07:00
Alex Vandiver 536aa32255 views: Extract config_error view function. 2023-10-11 17:13:01 -07:00
Alex Vandiver 50c80c6ff9 test_signup: Use verify_signup helper when possible. 2023-10-11 16:08:43 -07:00
Alex Vandiver e8405c6f7b test_signup: Fix docstring on a test. 2023-10-11 16:08:43 -07:00
Alex Vandiver 2f61a70102 test_signup: Use verify_signup helper. 2023-10-11 16:08:43 -07:00
Alex Vandiver 2952595234 test_signup: Clean up test name and docstring. 2023-10-11 16:08:43 -07:00
Alex Vandiver 89b7f9cbd0 registration: Handle race conditions with duplicate emails.
Two registration requests for the same email address can race,
leading to an IntegrityError when making the second user.

Catch this and redirect them to the login page for their existing
email.
2023-10-11 16:08:43 -07:00
Alex Vandiver b49938265f backup: Only pass --host and --port if non-empty.
This works around the `/usr/bin/pg_dump` failure described in the
previous commit.  Since we are now calling the appropriately-versioned
`pg_dump` binary directly, it is no longer "necessary", but is added
as a defense-in-depth.
2023-10-11 11:50:56 -07:00
Alex Vandiver 3a8b4b0205 backup: Call the pg_dump binary whose version we are running.
`/usr/bin/pg_dump` on Ubuntu and Debian is actually a tool which
attempts to choose which `pg_dump` binary from all of the
`postgresql-client-*` packages that are installed to run.  However,
its logic is confused by passing empty `--host` and `--port` options
-- instead of looking at the running server instance on the server, it
instead assumes some remote host and chooses the highest versioned
`pg_dump` which is installed.

Because Zulip writes binary database backups, they are sensitive to
the version of the client `pg_dump` binary is used -- and the output
may not be backwards compatible.  Using a PostgreSQL 16 `pg_dump`
writes archive format 1.15, which cannot be read by a PostgreSQL 15
`pg_restore`.

Zulip does not currently support PostgreSQL 16 as a server.  This
means that backups on servers with `postgresql-client-16` installed
did not successfully round-trip Zulip backups -- their backups are
written using PostgreSQL 16's client, and the `pg_restore` chosen on
restore was correctly chosen as the one whose version matched the
server (PostgreSQL 15 or below), and thus did not understand the new
archive format.

Existing `./manage.py backups` taken since `postgresql-client-16` were
installed are thus not directly usable by the `restore-backup` script.
They are not useless, however, since they can theoretically be
converted into a format readable by PostgreSQL 15 -- by importing into
a PostgreSQL 16 instance, and re-dumping with a PostgreSQL 15
`pg_dump`.

Fix this issue by hard-coding path to the binary whose version matches
the version of the server we are connected to.  This may theoretically
fail if we are connected to a remote PostgreSQL instance and we do not
have a `postgresql-client` package locally installed which matches the
remote PostgreSQL server's version.  However, choosing a matching
version is the only way to ensure that it will be able to be imported
cleanly -- and it is preferable that we fail the backup process rather
than write backups that we cannot easily restore from.

Fixes: #27160.
2023-10-11 11:50:56 -07:00
Alex Vandiver 9d46b41e25 backup: Use simpler api for server version. 2023-10-11 11:50:56 -07:00
Alex Vandiver cdc3413a4f sentry: Add a circuit-breaker around attempts to report to Sentry.
The goal is to reduce load on Sentry if the service is timing out, and
to reduce uwsgi load from long requests.  This circuit-breaker is
per-Django-process, so may require more than 2 failures overall before
it trips, and may also "partially" trip for some (but not all)
workers.  Since all of this is best-effort, this is fine.

Because this is only for load reduction, we only circuit-breaker on
timeouts, and not unexpected HTTP response codes or the like.

See also #26229, which would move all browser-submitted Sentry
reporting into a single process, which would allow circuit-breaking to
be more effective.
2023-10-11 09:44:18 -07:00
Alex Vandiver 9519945dc0 sentry: Do not report errors tlaking to Sentry, to Sentry.
This prevents failure to submit a client-side Sentry trace from
turning into a server-side client trace.  If Sentry is down, we merely
log the error to our error logs and carry on.
2023-10-11 09:44:18 -07:00
Dinesh fccdc054a1 typing_notifications: Frontend work to display notifs for streams.
Display/hide "X is typing" notification
on receiving start/stop typing event for streams.

Co-authored-by: Prakhar Pratyush <prakhar841301@gmail.com>
2023-10-10 17:36:55 -07:00
Prakhar Pratyush e23f3ac89b typing_indicator: Update the 'to' parameter to accept stream ID.
When the `type` of the message being composed is "stream",
this commit updates the `to` parameter to accept the ID of
the stream in which the message is being typed.

Earlier, it accepted a single-element list containing the ID
of the stream.

Sending the element instead of a list containing the single element
makes more sense.
2023-10-10 17:15:28 -07:00
Prakhar Pratyush 8b12cc606a request: Extract out methods from 'scheduled_messages' to reuse.
This is a prep commit that extracts the following two methods
from '/actions/scheduled_messages' to reuse in the next commit.
* extract_stream_id
* extract_direct_message_recipient_ids

The 'to' parameter for 'POST /typing' will follow the same pattern
in the next commit as we currently have for the 'to' parameter in
'POST /scheduled_messages', so we can reuse these functions.
2023-10-10 17:15:28 -07:00
Prakhar Pratyush 39ec5b9f66 test_typing: Replace 'private' with 'direct'.
This commit replaces the keyword 'private' with 'direct'.

'pm' has been changed to 'direct message' to be more verbose.
2023-10-10 17:15:28 -07:00
Prakhar Pratyush 8d8b6ef156 typing_indicator: Remove the "private" compatibility support.
This commit removes the compatibility support for "private"
being a valid value for the 'type' parameter in 'POST /typing'.

"direct" and "stream" are the only valid values.
2023-10-10 17:15:28 -07:00
Prakhar Pratyush a22f637769 typing_indicator: Replace private with direct in 'message_type' field.
This commit replaces the value `private` with `direct` in the
`message_type` field for the `typing` events sent when a user
starts or stops typing a message.
2023-10-10 17:15:28 -07:00
Anders Kaseorg 835ee69c80 docs: Fix grammar errors found by mwic.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-10-09 13:24:09 -07:00
Anders Kaseorg f338c3af07 rate_limiter: Fix PIE790 Unnecessary `pass` statement.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-10-09 11:55:16 -07:00
Anders Kaseorg 4cb2eded68 typos: Fix typos caught by typos.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-10-09 11:55:16 -07:00
Anders Kaseorg 7b4a74cc4d codespell: Fix typos caught by codespell.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-10-09 11:55:15 -07:00
Alya Abbott f214ba7acc integrations docs: Clarify instructions for direct messages from Zapier. 2023-10-09 11:15:25 -07:00
Prakhar Pratyush 02f3f73b0c recipient_info: Include 'sender_id' in 'topic_participant_user_ids'.
This commit includes the message's sender id in the
'topic_participant_user_ids' set.

The 'participants_for_topic' function doesn't include the sender_id,
if the user is sending their first message in the topic, because
'participants_for_topic' queries the 'Message' table, but the message
is actually sent at a later stage in the codepath, resulting in
missing the sender_id in this case.

This is needed to set the 'wildcard_mentioned' flag for the sender's
user message in the case of topic wildcard mentions.

This doesn't lead to sending email and push notifications to the
sender because we have a check to skip notifications if the user
to receive notifications is the sender itself.

This should have been included in c0c30bc.
2023-10-06 11:38:37 -07:00
Simon Michalke b6a25840a1 zerver: Implement ldap group synchronization.
Fixes #9957.

Co-authored-by: Mateusz Mandera <mateusz.mandera@zulip.com>
2023-10-05 16:12:52 -07:00
Tim Abbott 2bfbbf0035 migrations: Fix initial value for automatic policy migrations.
Migration 0476 was incorrectly not updated for changes in the
defaults -- we intended the "never" value.
2023-10-05 16:11:25 -07:00
Alex Vandiver db203cfc2d newrelic: ids are uuids, not integers.
The previous fixtures were not generated from New Relic directly, so
incorrectly implied that `id` was an integer.  See the examples for
`issueId` in their documentation[^1].

[^1]: https://docs.newrelic.com/docs/alerts-applied-intelligence/notifications/message-templates/
2023-10-05 09:44:51 -07:00
Anders Kaseorg f99cce91bf middleware: Send got_request_exception signal for JSON 500 errors.
This is ordinarily emitted by Django at
  https://github.com/django/django/blob/4.2.6/django/core/handlers/exception.py#L139
and received by Sentry at
  https://github.com/getsentry/sentry-python/blob/1.31.0/sentry_sdk/integrations/django/__init__.py#L166

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-10-04 17:25:20 -07:00
Prakhar Pratyush 58568a60d6 settings: Add automatically follow and unmute topics policy settings.
This commit adds two user settings, named
* `automatically_follow_topics_policy`
* `automatically_unmute_topics_in_muted_streams_policy`

The settings control the user's preference on which topics they
will automatically 'follow' or 'unmute in muted streams'.

The policies offer four options:
1. Topics I participate in
2. Topics I send a message to
3. Topics I start
4. Never (default)

There is no support for configuring the settings through the UI yet.
2023-10-04 13:04:29 -07:00
Prakhar Pratyush c349d1137c sent_by_human: Add "test suite" to the set of Zulip UI-style clients.
Earlier, when we used 'self.send_message()' in the backend tests,
the sent message was not marked as read for the sender.

Reason: To set the read flag, we have to check if
'message.sent_by_human()'. It returns False because the
'sending_client' for tests is "test suite" and the 'sent_by_human'
function doesn't enlist the "test suite" client name as a human client.

This commit adds "test suite" to that list.

Also fixes a bug in when apply_unread_message_event was called that
was revealed by this change.
2023-10-04 13:04:29 -07:00
Prakhar Pratyush 68abc6af21 unread_msgs: Fix 'raw_unread_messages["muted_stream_ids"]' not set.
The raw_unread_messages["muted_stream_ids"] was not properly
updated with the calculated value.

The bug was introduced in 72e0084692.
2023-10-04 13:04:29 -07:00
Alya Abbott 456f3a0d94 integrations: Update Zapier documentation. 2023-10-03 14:21:36 -07:00
Lauryn Menard a1de07ba98 demo-orgs: Set organization type to "unspecified".
Instead of having "business" as the default organization type
for demo organizations in the dev environment, we set it to
"unspecified". This way a more generic zulip guide email will
be sent as part of the onboarding process for users invited
to try out the demo organization if the owner has not yet
updated the organization type.
2023-10-03 09:11:41 -07:00
Lauryn Menard f2ee4e8288 tests: Update events and schemas for drafts events.
Updates the testing for draft event schemas to be fully checked by
`zerver/tests/test_events.py` and `tools/check-schema`.

Also, corrects the type for the timestamp field in Draft objects
in the OpenAPI documentation.
2023-10-03 09:05:22 -07:00
Lauryn Menard ca5d1c1265 tests: Update events and schemas for scheduled message events.
Updates the testing for scheduled message event schemas to be fully
checked by `zerver/tests/test_events.py` and `tools/check-schema`.

Adds the missing 'failed' field to the scheduled message events
in `web/tests/lib/events.js` as well.
2023-10-03 09:05:22 -07:00
Chris Bobbe ad7dcc7d83 api docs: Use enum for reaction_type.
It's not currently rendered in the docs, but good to have this in
place anyway:
  https://github.com/zulip/zulip/pull/21005#discussion_r797823366
2023-10-03 08:13:07 -07:00
Aman Agrawal 9042cd989b base: Play videos in new tab instead of downloading them.
We add `Content-Disposition: inline` header to commonly supported
video MIME types so that when we `Open` them in lightbox, they
play in new tab.

This will require a follow-up database migration to apply to
previously uploaded videos.
2023-10-02 22:39:39 -07:00
Aman Agrawal 8ef52d55d3 markdown: Add support for inline video thumbnails. 2023-10-02 22:39:02 -07:00
Aman Agrawal 75a1a74adf hello: Redesign page.
Fixes #24082
2023-10-02 22:30:03 -07:00
Satyam Bansal 1e2f6a06a9 events: Exclude legacy integrations from realm_incoming_webhook_bots.
This excludes the legacy webhook from the
"realm_incoming_webhook_bots" object as those do not have the same URL
format as modern webhook integrations.
2023-10-02 12:29:28 -07:00
Satyam Bansal 02bc17e5dc api-docs: Minor grammar fix. 2023-10-02 12:26:44 -07:00
Tim Abbott d4d5cd1daf test_realm: Fix incorrect comparison of sort() return values.
Fixes #22105.
2023-10-01 21:49:48 -07:00
Pedro Madriz 0ef8ec698a Add support for importing guest users from Mattermost.
This change adds support for importing guest users from a Mattermost
export file into Zulip. The function now checks the user's teams and
roles to determine whether the user is a guest on the team, and sets
the user's role accordingly. This ensures that the imported user data
includes the correct role for each user.

Fixes #23720.
2023-10-01 17:13:23 -07:00
Tim Abbott c6fe799369 i18n: Fix default language for users created via API/LDAP.
This fixes a regression introduced in
9954db4b59, where the realm's default
language would be ignored for users created via API/LDAP/SAML,
resulting in all such users having English as their default language.

The API/LDAP/SAML account creation code paths don't have a request,
and thus cannot pull default language from the user's browser.

We have the `realm.default_language` field intended for this use case,
but it was not being passed through the system.

Rather than pass `realm.default_language` through from each caller, we
make the low-level user creation code set this field, as that seems
more robust to the creation of future callers.
2023-10-01 21:10:13 +02:00
Mateusz Mandera 7669fe8446 i18n: Tweak args/kwargs for get_default_language_for_new_user.
Making request a mandatory kwarg avoids confusion about the meaning of
parameters, especially with `request` acquiring the ability to be None
in the upcoming next commit.
2023-10-01 21:10:13 +02:00
Alex Vandiver c5373e9934 users: Prevent mirror dummy users from being "reactivated."
These do not represent users who were ever active, so they should not
be able to be reactivated.

See also #26703.
2023-10-01 09:29:23 -07:00
Mateusz Mandera 11390b70be tests: Eliminate instances of time_machine.travel without tick=False.
None of these tests seem to want to have tick=True, which is the
default. Letting the clock tick without a reason introduces the
possibility of nondeterministic test failures depending on the execution
time.
2023-10-01 09:28:42 -07:00
Alex Vandiver 723a4c2fff push_notifications: Send timestamp with only second granularity.
This reverts b8581e2895.  The mobile
client on Android parses this field using:

```kotlin
                timeMs = data.require("time").parseLong("time") * 1000
```

This throws an error if value is not `long` (i.e. an integer),
resulting in dropped notifications on Android from servers which had
deployed b8581e2895.

Switch back to sending an integer, but keep the behaviour from
fd6091ad17 where we send the timestamp in the payload of both
Android and Apple push notifications.
2023-09-29 14:33:44 -07:00
Alex Vandiver c3cf1cd9d8 data_import: Skip Slack "Posts" which we don't otherwise support. 2023-09-29 12:34:07 -07:00
Alex Vandiver 325ff11ecf soft_deactivation: Stop computing "new messages" in Python.
Rather than fetch all UserMessage rows for all streams, and subtract
those out in Python-space from the list of all Message rows the user
may have received -- do this via a "NOT EXISTS" subquery.  This is
much better indexed (performing in fractions of milliseconds rather
than hundreds), and also consumes much less memory.
2023-09-29 10:26:04 -07:00
Alex Vandiver f3cd1c4b08 delete_old_unclaimed_attachments: Update docs on default max age.
42f1cb3444 updated the default up, from 1 week to 5 weeks, but did
not adjust the documentation.
2023-09-29 09:49:03 -07:00
Tim Abbott ef5fee4778 user_groups: Improve bulk membership change logic.
Adds support for bulk-adjusting a single user's membership in multiple
user groups in a single transaction in the low-level actions
functions, for future use by work on #9957.
2023-09-29 15:08:02 +02:00
Lauryn Menard 5774486c12 api-docs: Correct error documentation for api/deactivate-own-user.
In commit 3e369bcf9, the `code` field for api/deactivate-own-user
was incorrectly documented as "BAD_REQUEST", which is the code for
the similar error returned by api/deactivate-user.

Corrects the error code to be "CANNOT_DEACTIVATE_LAST_USER" and
adds documentation for the two other fields returned by this
error response.

Note that the descriptions for the fields are added in the error
response schema will not be rendered in our current documentation.

They are rendered in other third-party tools and are therefore
good to have in our OpenAPI documentation. The description that
will be rendered in our documentation is the general error response
schema description and that is also updated for details about the
extra fields in this error response.
2023-09-28 08:39:29 -07:00
Lauryn Menard 84f1ee84ff email-notifications: Add helper function to TestFollowupEmailDelay.
Adds a helper function to main test in TestFollowupEmailDelay so
that adding additional onboarding emails to the schedule is clearer.
2023-09-27 17:38:05 -07:00
Mateusz Mandera ac14a8bcf5 typed_endpoint: Rename WebhookPayload to JsonBodyPayload.
This kind of payload that's loaded from json in the body of the request
is not only used for webhooks, but also in the push bouncer, and may get
used elsewhere too - so a general name is better.
2023-09-27 14:21:42 -07:00
Tim Abbott 9e3133e820 openapi: Add inline code block around count parameter. 2023-09-27 14:02:27 -07:00
Prakhar Pratyush 49092dfa79 unread_msgs: Fix all unreads in muted stream being treated as muted.
Earlier, 'is_row_muted' returned 'true' if the message was in
a muted stream or muted topic.

If the message is in an unmuted or followed topic in a muted
stream, such topics should be treated as not muted topics
in an unmuted stream.

This commit fixes the incorrect behavior.

Now, for wildcard mentions, 'unread_msgs.mentions' exclude
the IDs in muted streams only if the message is in default or
muted topic.

Also, 'unread_msgs.count' takes into account the unreads in unmuted
or followed topics in muted streams too.

Documents that this bug was fixed in the API changelog.
2023-09-27 13:11:20 -07:00
Prakhar Pratyush a18a526427 muted_stream_ids: Use set for O(1) search operation.
Update 'get_muted_stream_ids' to return a set of IDs
instead of a list.

This will help to avoid linear time search operations later
while using 'if stream_id in muted_streams_ids'.
2023-09-27 13:11:20 -07:00
Prakhar Pratyush faa98317bc user_topics: Refactor build_topic_mute_checker to support each policy.
This prep commit renames the 'build_topic_mute_checker' function
to 'build_get_topic_visibility_policy' and updates it to support
all the visibility policies.

The function prefetches the visibility policies the user has
configured for various topics and prepares a dict named
'topic_to_visibility_policy' to be used later on.
2023-09-27 13:11:20 -07:00
Prakhar Pratyush b5232c56ad test_unread_msgs: Update comment regarding unread message count.
A comment was added in f797604 to convey that the unread count
at that time doesn't exclude the unreads in muted topics.

848c080 added the support to exclude the muted topic;
however, the comment was not updated.

This commit updates the comment to reflect the current behavior.
2023-09-27 13:11:20 -07:00
Tim Abbott efb3059fdb models: Add more comments documenting AbstractMessage. 2023-09-27 11:49:04 -07:00
Deepayan Mukherjee 79916b9c24
onboarding: Finish renaming display settings -> preferences.
Fixes part of #26874.
2023-09-27 11:30:11 -07:00
Mateusz Mandera a095c34503 saml: Add OneLogin_Saml2_ValidationError to caught parsing exceptions.
This is an exception that we should be generally catching like the
others, which will give our standard /login/ redirect and proper logging
- as opposed to a 500 if we don't catch.

Addresses directly a bug we occurred in the wild, where a SAMLResponse
was submitted without issuers specified in a valid way, causing this
exception. The added test tests this specific type of scenario.
2023-09-27 10:32:18 -07:00
Alex Vandiver e1aca65d7d narrow: Add usermessage "private" flag limit to direct message queries.
This flag exists to accelerate searches -- use it.
2023-09-27 10:22:42 -07:00
Alex Vandiver 2405b23ad0 narrow: Add realm_id limits for direct message queries.
These queries benefit from the increased specificity of using the
realm / recipient / sender indexes.  The argument from 11a1cb9630
does not apply in these cases, since there are only 2 usermessage rows
for each matching message row for DMs, and few more than that for
huddles.
2023-09-27 10:22:42 -07:00
Alex Vandiver d55240e543 topic: Add comments calling out case-sensitive index usage. 2023-09-27 10:22:42 -07:00
Alex Vandiver 436e9c8a0c topic: Add realm limits to topic history queries. 2023-09-27 10:22:42 -07:00
Alex Vandiver ae822103a7 message: Add realm limit to private-message summary query.
This query has two halves; messages set by the user, and messages
received by the user.  The former uses the already-specific
usermessage privatemessage flag index; the latter relies on the
recipient index on messages.

Add the realm_id to the latter half, so that the recipient_id is
paired with the realm_id.
2023-09-27 10:22:42 -07:00
Alex Vandiver d28d347c35 narrow: Remove an out-of-date comment.
a9b3a9c673 changed this to use an explicit map, instead of getattr
magic.
2023-09-27 10:22:42 -07:00
Hardik Dharmani 6c1f993855 user_preferences: Change option text for "Show unread counts for".
This commit updates the text for a dropdown option `Unmuted streams`
to `Unmuted streams and topics` for `Show unread counts for` user
preference settings for better clarity.
2023-09-26 15:10:51 -07:00
Lauryn Menard e5da17459d emails: Update out-of-date references to day1/day2 onboarding emails. 2023-09-26 11:46:25 -07:00
Alex Vandiver 71e297efb4 realm: Differentiate reserved realms from in-use realms.
Fixes: #23896.
2023-09-25 12:48:14 -07:00
Anders Kaseorg dd7b09d71a ruff: Fix C416 Unnecessary `list` comprehension.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-09-24 23:25:52 -07:00
Alex Vandiver 4cef665e98 zilencer: Log how long the remote server delayed the push notification. 2023-09-24 15:24:02 -07:00
Alex Vandiver fd6091ad17 push_notifications: Provide message timestamp in both APNS and GCM notifications. 2023-09-24 15:24:02 -07:00
Alex Vandiver b8581e2895 push_notifications: Provide subsecond granularity on the sent time. 2023-09-24 15:24:02 -07:00
Lauryn Menard 48a1cf04d0 api-docs: Update the add and remove update_message_flags events.
Clarifies that the `all` field in the `op: "add"` event is only
relevant for the `"read"` message flag, and that it will be false
for all other specified flags in theses events.

Deprecates the `all` field in the `op: "remove"` event and document
that it is false for all specified flags.

Updates the deprecated `operation` field description and makes
a few other small revisions to the event text for clarity and
accuracy.
2023-09-22 11:28:09 -07:00
Hemant Umre be653dd5b4 org_settings: Add backend for `realm_jitsi_server_url` setting.
This commit adds a `jitsi_server_url` field to the Realm model, which
will be used to save the URL of the custom Jitsi Meet server. In
the database, `None` will encode the server-level default. We can't
readily use `None` in the API, as it could be confused with "field not
sent". Therefore, we will use the string "default" for this purpose.

We have also introduced `server_jitsi_server_url` in the `/register`
API. This will be used to display the server's default Jitsi server
URL in the settings UI.

The existing `jitsi_server_url` will now be calculated as
`realm_jitsi_server_url || server_jitsi_server_url`.

Fixes a part of #17914.

Co-authored-by: Gaurav Pandey <gauravguitarrocks@gmail.com>
2023-09-21 17:39:10 -07:00
Alex Vandiver 11a1cb9630 narrow: Skip the realm_id limit on joins across to usermessage.
The unique index on `(user_id, message_id)` that is the
`zerver_usermessage` table is rather specific, and even the PostgreSQL
extended statistics are not enough for it to realize there is a
correlation between the `realm_id` in the message table and the
`user_id` in the usermessage table.  This means that adding the
`realm_id` limit when there is a join to `zerver_usermessage` flips
the query plan from a nested loop of unique usermessage index-only
scan, with an index scan of the messages pkey -- to a parallel hash
join of the messages limit with a index scan of just the user_id limit
on usermessages.  It thinks this is necessary because it thinks that
the `realm_id` limit may remove a large number of messages from the
usermessage set -- which is totally untrue.

Remove the `realm_id` limit if we have a usermessage join.
2023-09-21 13:58:34 -07:00
Mateusz Mandera 00f4f77358 zilencer: Remove /json versions of push bouncer endpoints.
These don't make sense (because authentication here is not based on
session) and aren't used.
2023-09-21 10:42:15 -07:00
David Rosa 6588190918 help: Delete "Start a new topic" page in favor of "Starting a new topic".
- Deletes page, updates references, and adds URL redirect.
2023-09-21 08:41:40 -07:00
Sahil Batra cb203fbe9a user_groups: Do not allow empty group names in backend.
We now raise error if a user tries to create a group with
empty name or tries to update a group name to be empty.
2023-09-20 15:35:26 -07:00
Lauryn Menard f3a2927e7a api-docs: Remove unused PartiallyCompleted schema.
This was removed as a potential success response in #26658, but
removing the schema was missed in those changes.
2023-09-20 10:38:01 -07:00
Lauryn Menard ef1ac399e6 api-docs: Clean up shared schemas for error responses.
Removes the JsonErrorBase and JsonError schemas as all error
responses in the API docs use the CodedErrorBase or CodedError
schemas.

Removes the AddSubscriptionsResponse schema since it's no longer
incorrectly used as a shared schema for error responses, and
instead documents the specific success response properties in the
endpoint.
2023-09-20 10:38:01 -07:00
Lauryn Menard c0a4ecd5aa api-docs: Move ignored params example to IgnoredParametersSuccess. 2023-09-20 10:38:01 -07:00
Lauryn Menard f0fe86d5e3 api-docs: Move simple success example up to JsonSuccess schema. 2023-09-20 10:38:01 -07:00
Lauryn Menard eb99efccd8 api-docs: Make "code" field required for CodedErrorBase schema. 2023-09-20 10:38:01 -07:00
Lauryn Menard 6fac83d54f api-docs: Fix RateLimitedError schema. 2023-09-20 10:38:01 -07:00
Lauryn Menard ee2f58e67a api-docs: Replace incorrect JsonError schema refs with CodedError. 2023-09-20 10:38:01 -07:00
Lauryn Menard d3e66b4f4f api-docs: Add 'code' field to emoji does not exist error. 2023-09-20 10:38:01 -07:00
Lauryn Menard 28a2893a41 api-docs: Update mute/unmute user error documentation for 'code' fields. 2023-09-20 10:38:01 -07:00
Lauryn Menard c4d4dfda4d api-docs: Add shared InvalidStreamError schema.
Adds an InvalidStreamError schema for errors that return a 'msg'
field with the string: "Invalid stream ID". Updates endpoints that
have this error 'str' documented to use the shared schema.
2023-09-20 10:38:01 -07:00
Lauryn Menard 3e369bcf96 api-docs: Add 'code' field to errors for deleting only org owner. 2023-09-20 10:38:01 -07:00
Lauryn Menard cb921300ce api-docs: Add 'code' field to create user email already exists. 2023-09-20 10:38:01 -07:00
Lauryn Menard 1b14745b17 api-docs: Update InvalidMessageError and use for read receipts doc. 2023-09-20 10:38:01 -07:00
Lauryn Menard 767c6a506c api-docs: Update drafts/scheduled message ResourceNotFoundErrors.
Updates documentation of ResourceNotFoundErrors for unknown draft
and scheduled message IDs to include the 'code' field, have an
HTTP status code of 404 in the documentation, and to follow the
general description format of errors in the API documentation.
2023-09-20 10:38:01 -07:00
Lauryn Menard 03ba4df8ee api-docs: Add 'code' field to unauthorized error in delete attachments. 2023-09-20 10:38:01 -07:00
Lauryn Menard 68fcd0a299 api-docs: Add 'code' field to InvalidApiKeyError response example. 2023-09-20 10:38:01 -07:00
Mateusz Mandera 41d76969cb build_message_send_dict: Remove redundant realm arg.
Since the function asserts that the realm arg matches message.realm
anyway, having the argument is redundant here.
2023-09-20 09:55:25 -07:00
Alex Vandiver 5ee4b642ad views: Add a /health healthcheck endpoint.
This endpoint verifies that the services that Zulip needs to function
are running, and Django can talk to them.  It is designed to be used
as a readiness probe[^1] for Zulip, either by Kubernetes, or some other
reverse-proxy load-balancer in front of Zulip.  Because of this, it
limits access to only localhost and the IP addresses of configured
reverse proxies.

Tests are limited because we cannot stop running services (which would
impact other concurrent tests) and there would be extremely limited
utility to mocking the very specific methods we're calling to raising
the exceptions that we're looking for.

[^1]: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/
2023-09-20 09:53:59 -07:00
David Rosa cb123d4f15 management: Rename command reactivate_stream -> unarchive_stream. 2023-09-19 15:03:33 -07:00
Steve Howell a8f5836ee6 tests: Make soft-reactivation tests readable.
The `expected` flag was incredibly confusing, as you
couldn't tell from the calling code what you were
actually expecting to happen.

I avoid the context manager idiom in order to force
the callers to create simple helper functions, and
I de-duplicate some code in some places.

I also force the caller to explicitly soft-deactivate
the user with one simple line of code, so that the
person reading the test doesn't have to research
the side effects of the helper. (And I make it
very easy for new authors to follow the practice
going forward.)

This is also somewhat of a prep commit to avoid
the obfuscated use of refresh_from_db.
2023-09-18 16:55:06 -07:00
Steve Howell 31d2660ee2 tests: Simplify policy-related tests.
The helper here was no longer a useful abstraction.
2023-09-18 16:55:06 -07:00
Steve Howell 0e261f6ec4 tests: Wrap get_user method.
The get_user function is poorly named, but I don't want to
sweep the entire codebase yet.

It's also nice to have a test wrapper for little experiments
like profiling tests or hunting down calls to refresh_from_db.

It's possible that we would also just change the new wrapper
to more directly call Django. The `get_user` function isn't
used in a ton of real-world places, so we might want the test
code to just bypass the cache.
2023-09-18 16:55:06 -07:00
Steve Howell df43f86cbc tests: Clean up check_has_permission_policies.
I add a bunch of cute helper methods to make
the test a bit more readable.

And then I make sure to get clean objects,
which precludes the need for our callback
functions to refresh the user objects.

And finally I make sure that our validation
functions don't cause any round trips (assuming
we have fetched objects using a standard
Zulip helper, which example_user ensures.)
2023-09-18 16:55:05 -07:00
Lauryn Menard 31daef7f79 response: Remove "result: partially_completed" for success responses.
In feature levels 153 and 154, a new value of "partially_completed"
for `result` in a success (HTTP status code 200) was added for two
endpoints that process messages in batches: /api/delete-topic and
/api/mark-all-as-read.

Prior to these changes, `result` was either "success" or "error" for
all responses, which was a useful API invariant to have for clients.

So, here we remove "partially_completed" as a potential value for
"result" in a response. And instead, for the two endpoints noted
above, we return a boolean field "complete" to indicate if the
response successfully deleted/marked as read all the targeted
messages (complete: true) or if only some of the targeted messages
were processed (complete: false).

The "code" field for an error string that was also returned as part
of a partially completed response is removed in these changes as
well.

The web app does not currently use the /api/mark-all-as-read
endpoint, but it does use the /api/delete-topic endpoint, so these
changes update that to check the `complete` boolean instead of the
string value for `result`.
2023-09-18 13:18:24 -07:00
Aman Agrawal 7e11f95716 inbox: Add inbox as default view.
Fixes #26736
2023-09-18 12:17:49 -07:00
Lauryn Menard 7c618b57cf emails: Update subject for confirm_new_email to have realm host.
Updates the email subject for confirming an email change to include
the realm host.
2023-09-17 15:12:21 -07:00
Lauryn Menard ab54a3b792 api-docs: Clean up descriptions in unread_msgs register response.
For arrays of objects in return values of API endpoints, any
general description of the objects in the arrays should be
documented in the description of the array. A description at the
level of the items in the array will not be rendered in the API
documentation. Descriptions of each property of the object will
be rendered, but these are specific to the property and not the
object as a whole.

Updates the pms, streams and huddles arrays of objects included
in the unread_msgs object of the register response so that the
descriptions are at the array level in the OpenAPI documentation.
2023-09-15 13:20:03 -07:00
Lauryn Menard 0f5353486b api-docs: Document order of user IDs in huddles unread messages data.
When unread_msgs data was added to the register queue response, see
commit 4f0110e, the `user_ids_string` field in the `huddles` array
of objects with information about unread group direct messages, had
the user IDs in the string sorted numerically.

Documents that these strings include the current users's ID and are
sorted numerically and separated by commas so that the documentation
is clear for client implementations.
2023-09-15 13:20:03 -07:00
Mateusz Mandera 3e15ea3f3f scim: Add supporting for syncing the user role.
This adds support for syncing user role via the newly added "role"
attribute, which can be set to either of
['owner', 'administrator', 'moderator', 'member', 'guest'].

Removes durable=True from the atomic decorator of do_change_user_role,
as django-scim2 runs PATCH operations in an atomic block.
2023-09-15 13:15:45 -07:00
Prakhar Pratyush db8229ae32 test_events: Separate tests for send and update message events.
This is a prep commit to separate the single test
'test_stream_send_message_events' into two separate tests named
'test_stream_send_message_events' & test_stream_update_message_events'
to verify the events related to send and update message, respectively.

As a part of introducing two new user settings
* 'automatically_follow_topics_policy'
* 'automatically_unmute_topics_policy'
in the next commit, we will extend 'test_stream_send_message_events'.

This logical separation helps in avoiding a single, super-long test.
2023-09-14 17:16:36 -07:00
Prakhar Pratyush cf804200c7 tests: Remove the usage of stray desktop_icon_count_display values.
This commit removes the stray values, i.e., [1, 2, 3], used
in the tests for desktop_icon_count_display.

We use 'UserProfile.DESKTOP_ICON_COUNT_DISPLAY_CHOICES' instead.
2023-09-14 17:16:36 -07:00
Prakhar Pratyush 35c3724f22 tests: Fix the incomplete exclusion of the notification settings.
'test_change_user_settings' in 'UserDisplayActionTest' excludes
the notification settings and tests only the display settings.

The code block excluding the notification settings doesn't exclude
'modern_notification_settings'. It only excludes the
'notification_settings_legacy'.

This commit replaces 'notification_settings_legacy' with
'notification_setting_types', which consists of all the
notification settings.
2023-09-14 17:16:36 -07:00
Alex Vandiver 4f4627b79b send_email: Use a consistent order when sending custom emails to users. 2023-09-14 17:16:36 -07:00
Lauryn Menard 3cd2b6886d api-docs: Expands API changelog feature level 134 entry.
Expands API changelog feature level 134 entry and adds the related
Changes notes to the events documentation for the updates made in
commit f4fcedd: "stream op: create" and "subscription op: peer_add"
events being sent when a private stream is made public.

Those changes were made after the feature level 133 updates, but
before the feature level 134 updates, which is why 134 is the
feature level for the change that is documented for clients.
2023-09-14 09:50:41 -07:00
Lauryn Menard fdf41774db api-docs: Update subscription event for peer_add feature level 205.
In commit ada2991f1c, when a user gains access to a stream due to
a role change, in addition to sending "stream op: create" events,
"subscription op: peer_add" events are sent for streams that the
user gains access to due to their role change. Updates the API
changelog entry for feature level 205.

Updates the "subcription op: peer_add" event documentation to be
more accurate in for the general use cases of this event, which
are to provide updated subscriber information for streams that
a user has access to.
2023-09-14 09:50:41 -07:00
Lalit 2b566c778b user_settings: Add new `web_stream_unreads_count_display_policy` field.
This is a backend change that will help us support the new "Show unread counts for"
user display setting.
2023-09-13 18:45:45 -07:00
Anders Kaseorg 28597365da python: Delete superfluous parens.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-09-13 13:40:19 -07:00
Alex Vandiver e0d3176098 digest: Increase size of stream cache.
Since the cache is flushed when the cutoff or realm changes, the
maximum size of the cache should cap out at the number of streams in
the realm.  Raise the max cache size, now that this will not simply
lead to useless cache space for smaller servers.
2023-09-13 13:25:59 -07:00
Alex Vandiver a8a1f10f3c digest: Clear the cache once we move to a new realm / cutoff value. 2023-09-13 13:25:59 -07:00
Alex Vandiver 39358f77dd digest: Enqueue emails as we generate the contexts.
There is now no longer any reason to have the scheduled_email
enqueuing wait until all of the users' contexts have been generated.
Switch to returning the contexts as an iterator, and send them as we
compute them.
2023-09-13 13:25:59 -07:00
Alex Vandiver b9f72bdd68 digest: Switch loop to early-abort for clarity. 2023-09-13 13:25:59 -07:00
Alex Vandiver b555d3f553 digest: Cache per-stream recent topics, rather than batching.
The query plan for fetching recent messages from the arbitrary set of
streams formed by the intersection of 30 random users can be quite
bad, and can descend into a sequential scan on `zerver_recipient`.
Worse, this work of pulling recent messages out is redone if the
stream appears in the next batch of 30 users.

Instead, pull the recent messages for a stream on a one-by-one basis,
but cache them in an in-memory cache.  Since digests are enqueued in
30-user batches but still one-realm-at-a-time, work will be saved both
in terms of faster query plans whose results can also be reused across
batches.

This requires that we pull the stream-id to stream-name mapping for
_all_ streams in the realm at once, but that is well-indexed and
unlikely to cause performance issues -- in fact, it may be faster
than pulling a random subset of the streams in the realm.
2023-09-13 13:25:59 -07:00
Alex Vandiver ffb6c95bba email_notifications: Make stream_id_map optional to build_message_list.
This feels cleaner than passing an empty dict.
2023-09-13 13:25:59 -07:00
Alex Vandiver f8a9779b54 digest: Rename get_slim_stream_map slightly and explain its name more. 2023-09-13 13:25:59 -07:00
Alex Vandiver bca9821c89 digest: Rename get_recent_streams for clarity. 2023-09-13 13:25:59 -07:00
Alex Vandiver 524d4913b3 digest: Filter out users who have joined recently in SQL. 2023-09-13 13:25:59 -07:00
Alex Vandiver d8668ab242 digest: Narrow the query by only fetching the sender full name. 2023-09-13 13:25:59 -07:00
Alex Vandiver 058a168bfe digest: Rewrite target-user algorithm as one query.
There is no reason to do this set manipulation in Python.
2023-09-13 13:25:59 -07:00
Alex Vandiver 584c202d36 digest: Remove unnecessary should_process_digest function. 2023-09-13 13:25:59 -07:00
David Rosa 6505583b7e help: Update user management pages.
- Documents how to access the "Manage user" tab via a user's profile.
2023-09-13 13:06:04 -07:00
N-Shar-ma 89739ba3c3 widgets: Add test for tokenizing widget type on newline.
Follow up to #26643, adds test case for the bug fix.
2023-09-13 12:42:35 -07:00
Anders Kaseorg 2665a3ce2b python: Elide unnecessary list wrappers.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-09-13 12:41:23 -07:00
Anders Kaseorg 8efa418205 typed_endpoint: Add missing tuple comma for OptionalTopic aliases.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-09-12 11:49:11 -07:00
Alex Vandiver 135acfea93 nginx: Suppress proxy warnings when the proxy itself sent the request.
This is common in cases where the reverse proxy itself is making
health-check requests to the Zulip server; these requests have no
X-Forwarded-* headers, so would normally hit the error case of
"request through the proxy, but no X-Forwarded-Proto header".

Add an additional special-case for when the request's originating IP
address is resolved to be the reverse proxy itself; in these cases,
HTTP requests with no X-Forwarded-Proto are acceptable.
2023-09-12 10:10:58 -07:00
Anders Kaseorg 6988622fe8 ruff: Enable B023 Function definition does not bind loop variable.
Python’s loop scoping is misdesigned, resulting in a very common
gotcha for functions that close over loop variables [1].  The general
problem is so bad that even the Go developers plan to break
compatibility in order to fix the same design mistake in their
language [2].

Enable the Ruff rule function-uses-loop-variable (B023) [3], which
conservatively prohibits functions from binding loop variables at all.

[1] https://docs.python-guide.org/writing/gotchas/#late-binding-closures
[2] https://go.dev/s/loopvar-design
[3] https://beta.ruff.rs/docs/rules/function-uses-loop-variable/

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-09-11 18:03:45 -07:00
Anders Kaseorg cf4791264c python: Replace functools.partial with type-safe returns.curry.partial.
The type annotation for functools.partial uses unchecked Any for all
the function parameters (both early and late).  returns.curry.partial
uses a mypy plugin to check the parameters safely.

https://returns.readthedocs.io/en/latest/pages/curry.html

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-09-11 18:03:45 -07:00
Alex Vandiver 5874c6542f migrations: Remove indexes on Message without realm_id.
These indexes should no longer be necessary after the changes in the
previous commit.
2023-09-11 15:00:37 -07:00
Alex Vandiver b94402152d models: Always search Messages with a realm_id or id limit.
Unless there is a limit on `id`, always provide a `realm_id` limit as
well.  We also notate which index is expected to be used in each
query.
2023-09-11 15:00:37 -07:00
Alex Vandiver f9dd2549eb narrow: Set a realm_id limit on messages in user searches. 2023-09-11 15:00:37 -07:00
Alex Vandiver 3518d31797 migrations: Add indexes with realm_id.
This is designed to help PostgreSQL have better specificity and
locality in its indexes.  Subsequent commits will adjust the code to
make sure that we use these indexes rather than the `realm_id`-less
versions.

We do not add a `realm_id` variation to the full-text index, since
it is a GIN index; multi-column GIN indexes are not terribly
performant, require the `btree_gin` extension for `int` types (which
requires superuser privileges on PostgreSQL 12 and earlier), and
cannot be consistently added concurrently on running instances.

After all indexes have been made, we also run `CREATE STATISTICS` in
order to give PostgreSQL the opportunity to realize that recipient and
sender are highly correlated with message realm, allowing it to
estimate that `(realm_id, recipient_id)` is likely as specific as
matching a given `recipient_id`, instead of as likely as matching
`realm_id` times matching a `recipient_id`.  Finally, those statistics
must be filled by `ANALYZE zerver_message`, which is run last.
2023-09-11 15:00:37 -07:00
Alex Vandiver 067de6f948 coverage: Skip zerver.lib.migrate coverage.
It is only covered when we run migration tests, which we are not
guaranteed to always be able to do.
2023-09-11 15:00:37 -07:00
Alex Vandiver d6745209f2 django: Use .exists() instead of .count() when possible. 2023-09-11 15:00:37 -07:00
Alex Vandiver 9d3d57e786 message_send: Inline single use of filter_by_exact_message_topic.
Matching the topic exactly, as opposed to case-insensitively, is not a
common operation, and one that we want to make difficult to do
accidentally.  Inline the single use case of it.
2023-09-11 15:00:37 -07:00
Alex Vandiver 710524465a messages: Switch limits from sender__realm to realm.
We now have a `realm_id` on Message; use it, rather than having to
check the sender's realm.  This is theoretically different for
cross-realm bots, but these changes are all in tests where that does
not apply.
2023-09-11 15:00:37 -07:00
Alex Vandiver 5a0f4a1a22 messages: Limit to "id" column for max-message-id computation.
This lets PostgreSQL use an "Index Only Scan" which is slightly faster
than an "Index scan".
2023-09-11 15:00:37 -07:00
Alex Vandiver 631868a05b users: Refactor and optimize max_message_id_for_user by removing a join.
This algorithm existed in multiple places, with different queries.
Since we only access properties in the UserMessage table, we
standardize on the much simpler and faster Index Only Scan, rather
than a merge join.
2023-09-11 15:00:37 -07:00
Anders Kaseorg 1905df2342 requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-09-09 12:53:39 -07:00
Adrián Oliva 732ad89f3d markdown: Fix URL link topic skipping query.
When searching for links inside a topic name, the question mark (?)
was used to split the topic. If a URL had a query after the URL
(e.g., "?foo=bar"), then the query was trimmed from the URL.

Removing the question mark from `basic_link_splitter` is sufficient
to fix this issue. The `get_web_link_regex` function then removes
the trailing punctuation if any, including literal question marks.

Fixes #26368.
2023-09-08 16:17:11 -07:00
N-Shar-ma 8c91c91d86 widgets: Fix bug where a new line right after /todo broke rendering.
When there was no space right after `/todo` but there was content on a
new line, the message would be rendered plainly, not as a todo widget.
This was because we split on only the space character to then check if
the first token was a valid widget.

Now we split on both spaces and newlines to extract the widget name,
irrespective of whether it is followed by a space or a newline. This
results in the message being rendered as a todo widget as expected.
2023-09-08 15:39:07 -07:00
Lauryn Menard 11adc0f37d demo-organizations: Rename shortend versions of 'demo organization'.
Rename existing shortened references to demo organizations, like
`is_demo_org` or `demo-org-warning`, that have been used in the
codebase so far and replace them to be like the `models.py`
variable: `Realm.demo_organization_scheduled_deletion_date`.
2023-09-08 15:17:23 -07:00
Alex Vandiver 61262c7b9a tabbed_sections: Fix a backtrack-able regex.
This REDOS was not exploitable, as its content is only read from
checked-in files; regardless, simplify it to not backtrack.  We also
do not actually have any location which use leading or trailing
whitespace, so remove those optional bits.
2023-09-08 14:51:51 -07:00
Tim Abbott bcff5580d1 makemessages: Fix handling of handlebars whitespace control.
Our logic for extracting strings from templates did not properly
handle the syntax for code containing whitespace control characters,
resulting in a couple strings from subscribe_to_more_streams.hbs not
being processed.
2023-09-08 09:09:46 -07:00
Zixuan James Li 7d683018bd webhooks: Migrate travis to use @typed_endpoint.
To perform the same check, we define a Pydantic model. This includes
some keys "build_url" and "type" that we did not check for previously.
2023-09-08 08:20:17 -07:00
Zixuan James Li 4037196fb2 webhooks: Migrate librato to use @typed_endpoint.
The Librato webhook requires a mapping (which should be considered
immutable) with a default value. Ruff reports a false-positive due to
the Json wrapper.
2023-09-08 08:20:17 -07:00
Zixuan James Li a33607d8ad webhooks: Convert gitlab to use @typed_endpoint.
The GitLab webhook has a mix of different types of parameters each
requring a unique set of configurations.
2023-09-08 08:20:17 -07:00
Zixuan James Li b163f2fe4e webhooks: Convert non-body payload webhooks to use @typed_endpoint.
These webhooks do not use argument_type_is_body, so they are parsing the
payload from a query parameter directly into WildValue.
2023-09-08 08:20:17 -07:00
Zixuan James Li 318a9316a7 webhooks: Migrate webhooks with special payload types to use @typed_endpoint.
Instead of a WildValue, the JSON/Sentry webhook expect the request body to be a
dict.

For the JSON webhook, json.dumps accepts other types of input as well and the
constraint is not necessary, but this serve as a good example of an alternative
use of WebhookPayload to describe a payload that is intended to be parsed from
the entire request body from JSON, into a type other than WildValue.
2023-09-08 08:20:17 -07:00
Zixuan James Li ece6b98699 webhooks: Migrate helloworld to use WildValue to use @typed_endpoint.
We owe more documentation on the use of WildValue. A follow-up on
updating it with examples of WildValue and endpoint will be desirable.
2023-09-08 08:20:17 -07:00
Zixuan James Li 9fef12950a webhooks: Migrate transifex to use endpoint to use @typed_endpoint.
Transifex has parameters that need to be parsed from JSON and converted
to int. Note that we use Optional[Json[int]] instead of
Json[Optional[int]] to replicate the behavior of json_validator. This
caveat is explained in a new test called test_json_optional.
2023-09-08 08:20:17 -07:00
Zixuan James Li 1329284848 webhooks: Migrate webhooks with str parameters to use @typed_endpoint.
These webhooks have some URL query params that do not need additional
validation or parsing from JSON. So WebhookPaylaod is not applicable to
these webhooks.
2023-09-08 08:20:17 -07:00
Zixuan James Li 9377080f1f webhooks: Migrate most webhooks to use @typed_endpoint.
This converts most webhook integration views to use @typed_endpoint instead
of @has_request_variables, rewriting REQ parameters. For these
webhooks, it simply requires switching the decorator, rewriting the
type annotation of payload/message to WebhookPayload[WildValue], and
removing the REQ default that defines the to_wild_value converter.
2023-09-08 08:20:17 -07:00
Zixuan James Li 574740dda4 webhooks: Migrate check_send_webhook_message to use @typed_endpoint.
This function is used by almost all webhooks.

To support it, we use the "api_ignore_parameter" flag so that positional
arguments like topic and body that are not intended to be parsed from
the request can be ignored.
2023-09-08 08:20:17 -07:00
Zixuan James Li 910f69465c drafts: Migrate drafts to use @typed_endpoint.
This demonstrates the use of BaseModel to replace a check_dict_only
validator.

We also add support to referring to $defs in the OpenAPI tests. In the
future, we can descend down each object instead of mapping them to dict
for more accurate checks.
2023-09-08 08:20:17 -07:00
Zixuan James Li 4701f290f7 presence: Migrate presence to use @typed_endpoint.
This demonstrates a use of StringConstraints.
2023-09-08 08:20:17 -07:00
Zixuan James Li 6201914fd3 message_edit: Migrate message_edit to use @typed_endpoint.
This demonstrates how an alias is created and its suitable use case, the
use of PathOnly, NonNegativeInt, and Literal.
2023-09-08 08:20:17 -07:00
Zixuan James Li 9c53995830 alert_words: Migrate alert_words to use @typed_endpoint.
This demonstrates some basic use cases of the Json[...] wrapper with
@typed_endpoint.

Along with this change we extend test_openapi so that schema checking
based on function signatures will still work with this new decorator.
Pydantic's TypeAdapter supports dumping the JSON schema of any given type,
which is leveraged here to validate against our own OpenAPI definitions.
Parts of the implementation will be covered in later commits as we
migrate more functions to use @typed_endpoint.

See also:
https://docs.pydantic.dev/latest/api/type_adapter/#pydantic.type_adapter.TypeAdapter.json_schema

For the OpenAPI schema, we preprocess it mostly the same way. For the
parameter types though, we no longer need to use
get_standardized_argument_type to normalize type annotation, because
Pydantic dumps a JSON schema that is compliant with OpenAPI schema
already, which makes it a lot convenient for us to compare the types
with our OpenAPI definitions.

Do note that there are some exceptions where our definitions do not match
the generated one. For example, we use JSON to parse int and bool parameters,
but we don't mark them to use "application/json" in our definitions.
2023-09-08 08:20:17 -07:00
Zixuan James Li c336bf0398 api: Avoid programming errors due to nested Annotated types.
We want to reject ambiguous type annotations that set ApiParamConfig
inside a Union. If a parameter is Optional and has a default of None, we
prefer Annotated[Optional[T], ...] over Optional[Annotated[T, ...]].

This implements a check that detects Optional[Annotated[T, ...]] and
raise an assertion error if ApiParamConfig is in the annotation. It also
checks if the type annotation contains any ApiParamConfig objects that
are ignored, which can happen if the Annotated type is nested inside
another type like List, Union, etc.

Note that because
param: Annotated[Optional[T], ...] = None
and
param: Optional[Annotated[Optional[T], ...]] = None
are equivalent in runtime prior to Python 3.11, there is no way for us
to distinguish the two. So we cannot detect that in runtime.
See also: https://github.com/python/cpython/issues/90353
2023-09-08 08:20:17 -07:00
Zixuan James Li 5a7b1065e5 api: Rewrite argument type test for clarity.
We refactor HostRequestMock so that it now proper populates the request
body given the post data, assuming that the request is JSON encoded.
2023-09-08 08:20:17 -07:00
Zixuan James Li f4caf9dd79 api: Add new typed_endpoint decorators.
The goal of typed_endpoint is to replicate most features supported by
has_request_variables, and to improve on top of it. There are some
unresolved issues that we don't plan to work on currently. For example,
typed_endpoint does not support ignored_parameters_supported for 400
responses, and it does not run validators on path-only arguments.

Unlike has_request_variables, typed_endpoint supports error handling by
processing validation errors from Pydantic.

Most features supported by has_request_variables are supported by
typed_endpoint in various ways.

To define a function, use a syntax like this with Annotated if there is
any metadata you want to associate with a parameter, do note that
parameters that are not keyword-only are ignored from the request:
```
@typed_endpoint
def view(
    request: HttpRequest,
    user_profile: UserProfile,
    *,
    foo: Annotated[int, ApiParamConfig(path_only=True)],
    bar: Json[int],
    other: Annotated[
        Json[int],
        ApiParamConfig(
            whence="lorem",
            documentation_status=NTENTIONALLY_UNDOCUMENTED
        )
    ] = 10,
) -> HttpResponse:
    ....
```

There are also some shorthands for the commonly used annotated types,
which are encouraged when applicable for better readability and less
typing:
```
WebhookPayload = Annotated[Json[T], ApiParamConfig(argument_type_is_body=True)]
PathOnly = Annotated[T, ApiParamConfig(path_only=True)]
```

Then the view function above can be rewritten as:
```
@typed_endpoint
def view(
    request: HttpRequest,
    user_profile: UserProfile,
    *,
    foo: PathOnly[int],
    bar: Json[int],
    other: Annotated[
        Json[int],
        ApiParamConfig(
            whence="lorem",
            documentation_status=INTENTIONALLY_UNDOCUMENTED
        )
    ] = 10,
) -> HttpResponse:
    ....
```

There are some intentional restrictions:
- A single parameter cannot have more than one ApiParamConfig
- Path-only parameters cannot have default values
- argument_type_is_body is incompatible with whence
- Arguments of name "request", "user_profile", "args", and "kwargs" and
  etc. are ignored by typed_endpoint.
- positional-only arguments are not supported by typed_endpoint. Only
  keyword-only parameters are expected to be parsed from the request.
- Pydantic's strict mode is always enabled, because we don't want to
  coerce input parsed from JSON into other types unnecessarily.
- Using strict mode all the time also means that we should always use
  Json[int] instead of int, because it is only possible for the request
  to have data of type str, and a type annotation of int will always
  reject such data.

typed_endpoint's handling of ignored_parameters_unsupported is mostly
identical to that of has_request_variables.
2023-09-08 08:20:17 -07:00
Anders Kaseorg 0ce6dcb905 mypy: Upgrade mypy from 1.4.1 to 1.5.1.
_default_manager is the same as objects on most of our models. But
when a model class is stored in a variable, the type system doesn’t
know which model the variable is referring to, so it can’t know that
objects even exists (Django doesn’t add it if the user added a custom
manager of a different name). django-stubs used to incorrectly assume
it exists unconditionally, but it no longer does.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-09-07 17:51:42 -07:00
Anders Kaseorg 2cd018ce57 models: Remove duplicate index definition for date_sent.
Commit cf0eb46afc added this to let
Django understand the CREATE INDEX CONCURRENTLY statement that had
been hidden in a RunSQL query in migration 0244.  However, migration
0245 explained that same index to Django in a different way by setting
db_index=True.  Move that to 0244 where the index is actually created,
using SeparateDatabaseAndState.

Also remove the part of the SQL in 0245 that was mirrored by dummy
state_operations, and replace it with real operations.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-09-07 16:44:43 -07:00
Tim Abbott 6c83bbcbdb settings: Disallow everyone group for new setting.
This is important because the "guests" value isn't one that we'd
expect anyone to pick intentionally, and in particular isn't an
available option for the similar/adjacent "email invitations" setting.
2023-09-07 14:21:01 -07:00
Ujjawal Modi 88ec312b21 events: Send invites changes event to non-admin users also.
Earlier whenever a new invitation is created a event was sent
to only admin users. So, if invites by a non-admins user are changed
the invite panel does not live update.

This commit makes changes to also send event to non-admin
user if invites by them are changed.
2023-09-07 14:21:01 -07:00
Ujjawal Modi 5e31a6b1c0 invites: Make it possible for non-admins to revoke multiuse invites.
This commit makes changes to allow non-admins to revoke multiuse
invitations created by them.
2023-09-07 14:21:01 -07:00
Ujjawal Modi ec49c3acc8 invites: Rename `can_invite_others_to_realm` local variables.
This commit rename the existing setting `Who can invite users to this
organization` to `Who can send email invitations to new users` and
also renames all the variables related to this setting that do not
require a change to the API.

This was done for better code readability as a new setting
`Who can create invite links` will be added in future commits.
2023-09-07 14:21:01 -07:00
Ujjawal Modi f67cef8885 invite: Add new setting for "Who can create multiuse invite links".
This commit does the backend changes required for adding a realm
setting based on groups permission model and does the API changes
required for the new setting `Who can create multiuse invite link`.
2023-09-07 14:21:01 -07:00
Ujjawal Modi 9eccb4336e types: Add id_field_name field to GroupPermissionSetting type.
This commit adds id_field_name field to GroupPermissionSetting
type which will be used to store the string formed by concatenation
of setting_name and `_id`.
2023-09-07 14:21:01 -07:00
Tim Abbott 5f8bbfa652 invite: Explicitly mark REALM_OWNER as requiring an admin.
This was already enforced via separate logic that requires an owner to
invite an owner, but it makes the intent of the code a lot more clear
if we don't have this value mysteriously absent.
2023-09-07 14:21:01 -07:00
Ujjawal Modi a0b16e550e invites: Add a function to check if owner or admin is required.
Earlier there was a function to check if owner is
required to create invitations for the role specified
in invite and check for administrator was done
without any function call.

This commit adds a new function to check whether
owner or administrator is required for creating
invitations for the specified role and
refactors the code to use that new function.
2023-09-07 14:21:01 -07:00
Ujjawal Modi 2e59b1f30e tests: Use function to create realm rather than django ORM.
This commit makes changes in backend tests to use
`do_create_realm` function to create realm.
2023-09-07 14:21:01 -07:00
Ujjawal Modi 72b099524d internal_realm: Single transaction for changes while creating realm.
This commit makes the database changes while creating internal_realm
to be done in a single transaction.
This is needed for deferring the foreign key constraints
to the end of transaction.
2023-09-07 14:21:01 -07:00
Anders Kaseorg 48a3588cdb docs: Fix typos caught by ‘typos’.
https://github.com/crate-ci/typos

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-09-06 18:59:05 -07:00
Anders Kaseorg 6c76bad65a middleware: Fix exception logging format on JSON views.
Previously (with ERROR_REPORTING = True), we’d stuff the entire
traceback of the initial exception into the subject line of an error
email, and then also send a separate email for the JSON 500 response.
Instead, log one error with the standard Django format.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-09-06 09:14:49 -07:00
Zixuan James Li 1e1f98edb2 transaction_tests: Remove testing URL.
Rewrite the test so that we don't have a dedicated URL for testing.
dev_update_subgroups is called directly from the tests without using the
test client.
2023-09-06 09:13:02 -07:00
Anders Kaseorg 81bd63cb46 ruff: Fix PIE808 Unnecessary `start` argument in `range`.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-09-01 14:57:01 -07:00
Prakhar Pratyush 9cff7e14c4 push_notifications: Test message content truncated in a simple manner.
'test_get_message_payload_gcm_stream_message' verifies the payload
for notifications generated (for stream messages) due to any of the
push notification triggers, including
'NotificationTriggers.STREAM_PUSH'.

Earlier, 'test_get_message_payload_gcm_stream_notifications' tested
the same thing as 'test_get_message_payload_gcm_stream_message' with
the only difference that it included content that was not truncated.

This commit removes the test
'test_get_message_payload_gcm_stream_notifications' and updates
the test 'test_get_message_payload_gcm_stream_message' to cover
both the cases, i.e., truncated as well as not truncated content.
2023-09-01 10:46:16 -07:00
Prakhar Pratyush 5d8897b909 push_notifications: Remove 'alert' field from the payload for android.
This commit removes the 'alert' field from the payload for
Android via GCM/FCM.

The alert strings generated do not get used at all and have
not been used since at least 2019. On Android, we construct
the notification UI ourselves in the client, and we ignore
the alert string.
2023-09-01 10:46:16 -07:00
Lauryn Menard 91b40a45fe demo-orgs: Add email and password process for demo organization owners.
Creates process for demo organization owners to add an email address
and password to their account.

Uses the same flow as changing an email (via user settings) at the
beginning, but then sends a different email template to the user
for the email confirmation process.

We also encourage users to set their full name field in the modal for
adding an email in a demo organization. We disable the submit button
on the form if either input is empty, email or full name.

When the user clicks the 'confirm and set password' button in the
email sent to confirm the email address sent via the form, their
email is updated via confirm_email_change, but the user is redirected
to the reset password page for their account (instead of the page for
confirming an email change has happened).

Once the user successfully sets a password, then they will be
prompted to log in with their newly configured email and password.
2023-08-31 15:02:16 -07:00
Lauryn Menard a9eb70ac68 demo-orgs: Set owner `email_address_visibility` on account creation.
Since an email address is not required to create a demo organization,
we need a Zulip API email address for the web-app to use until the
owner configures an email for their account.

Here, we set the owner's `email_address_visibility` to "Nobody" when
the owner's account is created so that the Zulip API email field in
their profile is a fake email address string.
2023-08-31 15:02:16 -07:00
Eeshan Garg 5e33ae8adf demo-orgs: Create dev environment demo organization without email.
To make creation of demo organizations feel lightweight for users,
we do not want to require an email address at sign-up. Instead an
empty string will used for the new realm owner's email. Currently
implements that for new demo organizations in the development
environment.

Because the user's email address does not exist, we don't enqueue
any of the welcome emails upon account/realm creation, and we
don't create/send new login emails.

This is a part of #19523.

Co-authored by: Tim Abbott <tabbott@zulip.com>
Co-authored by: Lauryn Menard <lauryn@zulip.com>
2023-08-31 15:02:16 -07:00
Lauryn Menard ebfe9637c8 subscribe-unsubscribe: Improve error response for unexpected users.
Updates the API error response when there is an unknown or
deactivated user in the `principals` parameter for either the
`/api/subscribe` or `/api/unsubscribe` endpoints. We now use
the `access_user_by_email` and `access_user_by_id` code paths,
which return an HTTP response of 400 and a "BAD_REQUEST" code.

Previously, an HTTP response of 403 was returned with a special
"UNAUTHORIZED_PRINCIPAL" code in the error response. This code
was not documented in the API documentation and is removed as
a potential JsonableError code with these changes.

Fixes #26593.
2023-08-31 13:48:39 -07:00
David Rosa 4b8c99b01a widgets: Rename confusing attribute name in `tabbed_sections.py`.
Renames misleading attribute in HTML template using `code-section`
to refer to both language toggles in API docs and app toggles in
help center docs.
2023-08-31 11:55:28 -07:00
Lauryn Menard a5be9b5463 api-docs: Update feature level 205 changelog entry and changes notes.
Updates API changelog entries for feature level 205 for minor
revisions and the addition of help center links. Also, revises
the Changes notes for the stream creation and deletion events
for the same feature level.
2023-08-31 11:54:21 -07:00
Satyam Bansal 52e2298d65 api-docs: Clarify name field in realm_incoming_webhook_bots object. 2023-08-30 15:54:13 -07:00
Satyam Bansal d8998ab040 events: Add display name and event types to realm_incoming_webhook_bots. 2023-08-30 15:54:13 -07:00
Satyam Bansal 2370372705 integrations: Extract integration event types returning function. 2023-08-30 15:54:13 -07:00
Lauryn Menard 38ba97c1ac api-docs: Adds a Changes note for `is_owner` in register response.
Adds a feature level 11 note to the `is_owner` field in the
`POST /register` response and revises the changelog entry
for the same.
2023-08-30 15:50:28 -07:00
Lauryn Menard 5fdf7758d3 api-docs: Fix wrong feature level for /api/get-subscription-status.
Commit 8c39ddfd28 was when the endpoint was added with the wrong
feature level in the Changes note.
2023-08-30 15:50:28 -07:00
Anders Kaseorg 792a44b382 push_notifications: Fix logging.exception misuse.
logging.exception should only be called from an exception handler.
https://docs.python.org/3/library/logging.html#logging.exception

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-08-30 12:45:45 -07:00
Alex Vandiver 7787fe3f49 push_notifications: Send all APNS devices in parallel.
Instead of starting up one event loop for every device send, use
asyncio.gather to send to all of a user's devices at once.
2023-08-30 11:56:52 -07:00
Alex Vandiver 69825cd54c push_notifications: Drop error messages from failure to send.
We handle, and possibly log, these errors ourselves.
2023-08-30 11:56:52 -07:00
Alex Vandiver 733c6da298 push_notifications: Upgrade aioapns. 2023-08-30 11:56:52 -07:00
Alex Vandiver 168e1d5f85 send_email: Provide the realm and string_id, for ease of "if" logic. 2023-08-30 11:54:28 -07:00
Alex Vandiver c2c67a7640 send_email: Broaden type of context dict.
jinja2 can take objects and other data structures.
2023-08-30 11:54:28 -07:00
Akshat 3977514dec fenced_code: Enable code-highlighting without language markers.
The 'startinline' option is utilized in the `CodeHilite` instantiation
to indicate that the provided PHP code snippet should be highlighted
even if it doesn't start with the opening tag or marker of the
associated programming language (which will rarely be the case in
Zulip, since one discusses a section of a file much more often than a
whole file).

Fixes: https://chat.zulip.org/#narrow/stream/137-feedback/topic/php.20syntax.20highlighting.20should.20not.20require.20.60.3C.3Fphp.60.

Signed-off-by: Akshat <akshat25iiit@gmail.com>
2023-08-29 18:16:11 -07:00
Prakhar Pratyush 46446d0e1d test_push_notifications: Add a test for FOLLOWED_TOPIC_PUSH trigger.
This commit adds a test to verify the payload
'get_message_payload_apns' returns when the notification trigger is
'NotificationTriggers.FOLLOWED_TOPIC_PUSH'.
2023-08-29 17:12:21 -07:00
Prakhar Pratyush 6dc3b1a052 push_notifications: Return a common subtitle for wildcard mentions.
This commit updates the 'get_apns_alert_subtitle' function to
return a common subtitle, i.e., "{full_name} mentioned everyone:"
for wildcard mentions.

The triggers for the stream or topic wildcard mentions include:
* NotificationTriggers.TOPIC_WILDCARD_MENTION_IN_FOLLOWED_TOPIC
* NotificationTriggers.STREAM_WILDCARD_MENTION_IN_FOLLOWED_TOPIC
* NotificationTriggers.TOPIC_WILDCARD_MENTION
* NotificationTriggers.STREAM_WILDCARD_MENTION
2023-08-29 17:12:21 -07:00
Prakhar Pratyush e2c9b283f3 get_apns_alert_subtitle: Remove the stale 'user_profile' parameter.
This commit removes the 'user_profile' parameter that wasn't
getting used.

This should have been removed in ce6f6a3.
2023-08-29 17:12:21 -07:00
Danny Su 2414ddd2d4 compose: Verify request payload in Zoom meeting creation test
Verify that Zoom meeting creation logic sends the expected request payload for
configuring host_video and participant video.
2023-08-29 17:09:43 -07:00
Danny Su 66b9c06de6 compose: Add support for Zoom audio call
This PR implements the audio call feature for Zoom. This is done by explicitly
telling Zoom to create a meeting where the host's video and participants' video
are off by default.

Another key change is that when creating a video call, the host's and
participants' video will be on by default. The old code doesn't specify that
setting, so meetings actually start with video being off. This new behavior has
less work for users to do. They don't have to turn on video when joining a call
advertised as "video call". It still respects users' preferences because they
can still configure their own personal setting that overrides the meeting
defaults.

The Zoom API documentation can be found at
https://developers.zoom.us/docs/api/rest/reference/zoom-api/methods/#operation/meetingCreate

Fixes #26549.
2023-08-28 18:32:20 -07:00
David 75a654b9ab
help: Update links to "Edit a message" and "Delete a message".
The series including 0349152f0f split
this page into two smaller pages.
2023-08-28 18:09:41 -07:00
Lauryn Menard 965a25d91b api-docs: Clean up text in various parts of `unread_msgs` object.
Adds a Changes note for when `other_user_id` was added to the `pms`
object.

Changes a few uses of "you" to be "current user" instead.

Clarifies type of direct message (one-on-one or group) and that
messages are unread messages.
2023-08-28 12:33:02 -07:00
Lauryn Menard 0c9fef2444 api-docs: Fix incorrect documentation of subfield in `unread_msgs`.
Fixes the field in both the pms and huddles objects to be correctly
documented as `unread_message_ids`, instead of `message_ids`.

The documentation of the similar field in the stream object of
`unread_msgs` was corrected in commit 27ddb554fb.
2023-08-28 12:33:02 -07:00
neiljp (Neil Pilgrim) 4fc1bac473 api-docs: Fix formatting on realm_user in enter_sends from register. 2023-08-28 10:46:53 -07:00