Commit Graph

17496 Commits

Author SHA1 Message Date
Greg Price ff32d51700 push_notifs: Support APNs token auth, as well as cert auth.
This will make it possible to send notifications to multiple
distinct app IDs over the same connection.
2023-11-16 12:03:30 -08:00
Alex Vandiver d6e809d185 streams: Fix unsorted import. 2023-11-16 19:02:13 +00:00
Alex Vandiver 5e4bb69498 subscription_info: Fix typo caught by codespell. 2023-11-16 18:55:31 +00:00
Sahil Batra dc98136346 streams: Send stream deletion events on unsubscribing users.
This commit adds code to send stream deletion events when
unsubscribing non-admin users from private streams and
when unsubscribing guests from public streams since
non-admins cannot access unsubscribed private streams
and guests cannot access unsubscribed public streams.
2023-11-16 13:25:33 -05:00
Sahil Batra d4fb244d2d CVE-2023-47642: Invalid metadata access for formerly subscribed streams.
It was discovered by the Zulip development team that active users who
had previously been subscribed to a stream incorrectly continued being
able to use the Zulip API to access metadata for that stream. As a
result, users who had been removed from a stream, but still had an
account in the organization, could still view metadata for that
stream (including the stream name, description, settings, and an email
address used to send emails into the stream via the incoming email
integration). This potentially allowed users to see changes to a
stream’s metadata after they had lost access to the stream.

This bug was present in all Zulip releases prior to today's Zulip
Server 7.5.
2023-11-16 13:25:33 -05:00
Sahil Batra 3c8701ee36 streams: Add API endpoint to get stream email.
This commit adds new API endpoint to get stream email which is
used by the web-app as well to get the email when a user tries
to open the stream email modal.

The stream email is returned only to the users who have access
to it. Specifically for private streams only subscribed users
have access to its email. And for public streams, all non-guest
users and only subscribed guests have access to its email.
All users can access email of web-public streams.
2023-11-16 13:25:33 -05:00
Sahil Batra 432001656e streams: Remove "email_address" field from Subscription objects.
This commit removes "email_address" field from Subscription objects
and we would instead a new endpoint in next commit to get email
address for stream with proper access check.

This change also fixes the bug where we would include email address
for the unsubscribed private stream as well when user did not have
permission to send message to the stream, and having email allowed
the unsubscribed user to send message to the stream.

Note that the unsubscribed user can still send message to the stream
if the user had noted down the email before being unsubscribed
and the stream token is not changed after unsubscribing the user.
2023-11-16 13:25:33 -05:00
Mateusz Mandera 1819b85b85 management: Allow changing is_billing_admin using change_user_role. 2023-11-15 18:36:07 -08:00
Anders Kaseorg c8784634e1 openapi: Remove unnecessary cast.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-11-15 17:10:48 -08:00
Anders Kaseorg f4e7a11c35 requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-11-15 15:27:54 -08:00
Anders Kaseorg a688e753de test_helpers: Fix logging in cursor_executemany mock.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-11-15 15:27:54 -08:00
Anders Kaseorg cb9a04d3e3 test_create_video_call: Add missing not None assertions.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-11-15 15:27:54 -08:00
Anders Kaseorg 2c9c105bbb db: Check connection type in wrapper_execute.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-11-15 15:27:54 -08:00
Anders Kaseorg 248ee82db0 migrations: Remove useless null argument for ManyToManyField.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-11-15 15:27:54 -08:00
Anders Kaseorg 9552014047 models: Include explicit app name in ManyToManyField reference strings.
https://github.com/typeddjango/django-stubs/issues/1802

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-11-15 15:27:54 -08:00
Tim Abbott 2e2997bd7d typing: Limit typing notifications in large streams. 2023-11-15 09:42:25 -08:00
Alex Vandiver 091e2f177b linkifiers: Match JS implementation to server implementation.
Since the server-side implementation no longer uses look-ahead
or (more importantly) look-behind, it is possible to exactly implement
in Javascript.  This removes a common class which would prevent local
echo.

This requires reworking the topic linking algorithm, to march the
server's as well.  The tests and behaviour are adjusted in so doing --
previously, the JS implementation would have linked `#foo` with a
`foo` regex on the linkifier, but the server implementation would not
have.
2023-11-14 20:43:39 -08:00
Alex Vandiver 70b20e9d2b markdown: Use \p{White_Space} equivalent for linkifier boundaries.
We do not use \p{White_Space} itself because re2 does not support it.
2023-11-14 20:43:39 -08:00
Alex Vandiver 549dd8a4c4 markdown: Expand prepare_linkifier_pattern to make it more legible. 2023-11-14 20:43:39 -08:00
Sahil Batra 532293c4cf streams: Unset is_web_public and is_realm_public fields on attachments.
This commit adds code to unset is_web_public and is_realm_public fields
on attachments when deactivating a stream as we do not want to allow
spectators to access them after the stream is deactivated.

This commit also adds a comment explaining why we don't use
do_change_stream_permission to set the privacy fields on deactivating
a stream.

Fixes #27634.
2023-11-13 10:26:14 -08:00
Sahil Batra 30e2c14f4a streams: Unset is_realm_public on attachment when unarchiving streams.
We did not unset is_realm_public field on attachements when unarchiving
streams, but we do unset is_web_public field. This commit adds code to
unset the is_realm_public field as well as we make the stream private
while unarchiving it.
2023-11-13 10:26:14 -08:00
Alex Vandiver 1f3a53db2d cache: Drop the huddle cache.
This cache was only used in one place, which is infrequently
called (only when sending messages, or searching explicitly for a list
of users) and the overhead of maintaining the cache is not worth
trying to avoid the well-indexed lookup of the huddle.
2023-11-13 10:01:29 -08:00
Sahil Batra 8f79cec51a users: Pass bogus data for inaccessible users.
We now pass bogus data for inaccessible users when sending
the users data in "realm_users" field of "register" response
or when using endpoints like "GET /users" to get data of
all the users in realm.

We would add a client capability field in future commits
such that new clients would receive data only for accessible
users and they can form the bogus data by themselves.
2023-11-13 08:04:45 -08:00
Sahil Batra aedb9b23b4 models: Pass "realm__can_access_all_users_group" to select_related. 2023-11-13 08:04:45 -08:00
Sahil Batra c82bb3ec76 models: Add can_access_all_users_group setting.
This commit adds new setting for controlling who can access
all users in the realm which would have "Everyone" and
"Members only" option.

Fixes part of #10970.
2023-11-13 08:04:45 -08:00
Sahil Batra f20d427a4d users: Remove unnecessary comment.
This comment can be removed as the code was updated in 2ab71fb67d.
2023-11-13 08:04:45 -08:00
Sahil Batra df01d0725c message_send: Remove select_related call for stream.
We do not access any related field from the stream object,
so there is no need for select_related.
2023-11-13 08:04:45 -08:00
Mateusz Mandera 48db4bf854 counts: Add new mobile_pushes RemoteRealmCount stats.
This requires a bit of complexity to avoid a name collision in
COUNT_STATS with the RemoteInstallationCount stats with the same name.
2023-11-10 16:09:11 -08:00
Mateusz Mandera 8a6d5b4997 counts: Add new Add new mobile_pushes_sent::day LoggingCountStat.
This is a CountStat for tracking how many mobile notifications the
server requested.
1. On a self-hosted server, that means requesting from the push bouncer.
2. On a server that's its own push bouncer, that's just the number
   directly sent.

This number has room for inaccuracy due to incrementing by the number of
user devices on a self-hosted server, as it doesn't account for errors
that may occur in the GCM/APNs low-level sending codepaths on the bouncer.

Also tests that a server that's its own push bouncer correctly
increments its mobile_pushes_sent::day CountStat, by basing it on the
values returned from the send_apple/android_push_notification functions
which tell us the actual number of successfully sent notifications.

Since the return values of send_..._push_notification are now
used in those codepaths, we need to tweak our mocks in some unrelated
tests to set up some return value to avoid errors.
2023-11-10 16:09:11 -08:00
Prakhar Pratyush c597de6a1d topic_mentions: Rename wildcard_mentioned to stream_wildcard_mentioned.
Rename the existing 'wildcard_mentioned' flag to
'stream_wildcard_mentioned'.

The 'wildcard_mentioned' flag is deprecated and exists for
backwards compatibility.

We have two separate flags for stream and topic wildcard mentions,
i.e., 'stream_wildcard_mentioned' and 'topic_wildcard_mentioned',
respectively.

* stream wildcard mentions: `@all`, `@everyone`, and `@stream`
* topic wildcard mentions: `@topic`

The `wildcard_mentioned` flag is included in the events and
API response if either `stream_wildcard_mentioned` or
`topic_wildcard_mentioned` is set.
2023-11-10 11:06:26 -08:00
Prakhar Pratyush 24fa361f40 migration: Clear old data for unused usermessage flags.
In c37871ac3a, we renamed the
two unused and historical bits of the 'flags' bitfield of
the 'UserMessage' table:

* 'summarize_in_home' to 'topic_wildcard_mentioned'
* 'summarize_in_stream' to 'group_mentioned'

This commit clears out the old data for those bits.

Additionally, we are clearing 'force_expand' and 'force_collapse'
unused flags to save future work.
2023-11-10 11:06:26 -08:00
Mateusz Mandera 1312c7ccd7 zilencer: Add mechanism to update RemoteRealm when Realm is changed.
This requires a migration to allow RemoteRealmAuditLog.remote_id to be
NULL, and to add a RemoteRealmAuditLog.remote_realm.
2023-11-08 15:54:22 -08:00
Mateusz Mandera 76e0511481 zilencer: Add new model RemoteRealm and send the data to the bouncer.
Add the new model for recording basic information about Realms on remote
server, to go with the other analytics data. Also adds necessary changes
to the bouncer endpoint and the send_analytics_to_push_bouncer()
function to submit such Realm information.
2023-11-08 15:54:22 -08:00
Tim Abbott 0d525cf644 integrations: Recommend new integration URL tool.
Fixes part of #25976.
2023-11-08 14:19:05 -08:00
David Rosa 0c4ef58acd help: Document new help menu.
Updates help center docs to reflect the new dedicated help menu.

- Adds `keyboard_tip` to document the `?` keyboard shortcut.

Fixes #27384.
2023-11-08 14:17:13 -08:00
Alex Vandiver acb4e9e783 cache_helpers: Show item and DB query counts. 2023-11-08 11:00:00 -08:00
roanster007 dc492867af user_mention: Fix mentions of deactivated users.
Previously, when a deactivated user was mentioned, he wasn't
rendered as a Pill. This is because the dataset for validating mentions
only included active users, which is fixed by removing that filter.

To allow only silent mentions of them, an extra is_active property
added to FullNameInfo class, which is populated from the query,
which tells if user is deactivated. This is used to convert any
mentions of them to silent mentions in the backend markdown.

Fixes #26857
2023-11-08 09:48:31 -08:00
Sahil Batra 617d2d509c users: Update format_user_row to return a TypedDict.
This commit updates format_user_row to return a TypedDict.

This commit is a prep commit for feature of restricting user
access such that code can be easy to read and understand when
we add that feature.
2023-11-08 09:22:26 -08:00
Sahil Batra 2ab71fb67d users: Update user_profile_to_user_row to return a TypedDict.
This commit updates user_profile_to_user_row to return a TypedDict
and also updates the return type of get_realm_user_dicts to be a
TypedDict.

This commit is a prep commit for feature of restricting user
access such that code can be easy to read and understand when
we add that feature.
2023-11-08 09:22:26 -08:00
Sahil Batra deb8431a47 users: Rename get_raw_user_data to get_users_for_api.
This is a prep commit for adding feature of restricting
user access to guests such that we can keep the code
easy to read and understand when that feature is added.
2023-11-08 09:22:26 -08:00
Greg Price 7a0ebc0d18 push_notifs: Log an error if ios_app_id missing on any APNs token. 2023-11-07 16:19:42 -08:00
Greg Price f109e3b598 push_notifs: Backfill ios_app_id on bouncer. 2023-11-07 16:19:42 -08:00
Greg Price be2a9a03d0 push_notifs: Check app ID has a plausible shape at bouncer. 2023-11-07 16:19:42 -08:00
Greg Price 1b2178f558 push_notifs: Require ios_app_id on register at bouncer, too.
We'll need this information in order to properly direct APNs
notifications.  Happily, the Zulip server always sends it when
registering an APNs token; and it appears it always has done so
since the commit:

cddee49e7 Add support infrastructure for push notification bouncer service.

back in 2016.  So there's no compatibility issue from requiring it.
2023-11-07 16:19:42 -08:00
Greg Price 9c2d53bcef push_notifs: Stop dropping ios_app_id on floor at bouncer.
This missing `REQ` call has meant we just drop this parameter:
even though the remote Zulip server passes it (for all APNs tokens),
we never notice and never store it.  Fix that.
2023-11-07 16:19:42 -08:00
Greg Price 9072ddda05 push_notifs: Remove remaining references to ZULIP_IOS_APP_ID setting. 2023-11-07 16:19:42 -08:00
Greg Price a9796ec503 push_notifs: Check app ID has a plausible shape. 2023-11-07 16:19:42 -08:00
Greg Price fb3af7fbcb push_notifs: Make appid required on add_apns_device_token.
We're going to need to use this information, so we shouldn't just
assume a value; the client should tell us the actual value.

Conveniently, the Zulip mobile app does already pass this parameter
and has since forever.  So we can just start requiring it, with no
compatibility constraint.
2023-11-07 16:19:42 -08:00
Greg Price 5d0d35d3f5 push_notifs: Always supply appid to add_apns_device_token in tests.
We already always pass this parameter from the mobile client,
so this makes the tests more realistic already.  And we'll shortly
be making this parameter required.
2023-11-07 16:19:42 -08:00
Greg Price 3893a70888 push_notifs: Cut spurious token_kind arguments in some test data.
These endpoints distinguish the token kinds by having different
URL paths, and don't take any argument by this name.
2023-11-07 16:19:42 -08:00