Commit Graph

17521 Commits

Author SHA1 Message Date
Sahil Batra 650e55fef8 users: Send events only to users who can access the modified user.
This commit adds code to make sure that update events for changing
a user's role, email, etc. are not sent to guests who cannot access
the modified user.
2023-11-21 23:58:45 -08:00
Sahil Batra 6f14d105a7 create_user: Update data in user creation events for guests.
We do not send the original user data in user creation events
to guests if user access is restricted in realm, as they would
receive the information about user if user is subscribed to some
common streams after account creation.
2023-11-21 23:58:45 -08:00
Sahil Batra a23eff20fe users: Restrict read access to users in access_user_by_id.
This commit adds code to update access_user_by_id to raise
error if guest tries to access an inaccessible user.

One notable behavioral change due to this is that we do
not allow guest to mute or unmute a deactivated user if
that user was not involved in DMs.
2023-11-21 23:58:45 -08:00
Satyam Bansal cda7ed7101 integrations: Use correct type for GitHub pull request comment messages. 2023-11-21 21:05:26 -08:00
Satyam Bansal d2589a5bd1 integrations: Send GitHub pull request comment alerts to correct topic.
Pull request comment alerts were previously sent to a topic for an issue,
which resulted in two different topics for the same PR.

Fixes: #26086.

Co-authored-by: Lauryn Menard <lauryn@zulip.com>
2023-11-21 21:05:26 -08:00
Satyam Bansal 7ebf572b8e integrations: Add pull request comment fixture to GitHub Integration.
Updated the repo name and pull request number/title for the new
pull request commit fixture to be the same as the one used for the
other pull request test fixtures (e.g. pull_request__opened) so
that the TOPIC_PR can be used in the subsequent updates.

Co-authored-by: Lauryn Menard <lauryn@zulip.com>
2023-11-21 21:05:26 -08:00
Alex Vandiver 8b0cecc7e4 migrations: Fix revert migration to not lose all preferences.
Renumbering 4 -> 3, and then 3 -> 2 leads to everyone having their
preferences set to 2.  Swap the order, so that we renumber 3 -> 2,
then 4 -> 3.
2023-11-21 17:56:53 -08:00
Alex Vandiver dd954749be zilencer: Log, and drop, duplicated rows from remote servers.
This may happen if there are multiple servers with the same UUID
submitting data (e.g. if they were cloned after initial creation), or
if there is one server, but `./manage.py clear_analytics_tables` was
used to truncate the analytics tables.

In the case of `clear_analytics_tables`, the data submitted likely has
identical historical values with new remote `id` values; preserving
the originally-submitted contemporaneous data is the best option.  For
the case of submissions from multiple servers, there is no completely
sensible outcome, so the best we can do is detect the case and move
on.

Since we have a lock on the RemoteZulipServer, we know that no other
inserts are happening, so counting before and after will return the
true number of rows inserted (which `bulk_create` cannot do in the
face of `ignore_conflicts`[^1]).  We compare this to the expected
number of new inserted rows to detect dropped duplicates.

[^1]: See https://code.djangoproject.com/ticket/30138.
2023-11-21 11:44:55 -08:00
Alex Vandiver b363999d19 analytics: Slew record reporting by up to 10 minutes.
This reduces the giant load spike at 5 minute past the hour, when all
remote servers currently attempt to submit their records.

We do not wish to slew over a full hour, because we want to ensure
that we do not hold the lock when the next hour's analytics runs.  It
is also not necessary to have that much variation; 10 minutes is
picked as an arbitrary "long enough" time to spread requests over.
2023-11-21 10:49:57 -08:00
Alex Vandiver 85cc8b6a20 remote_server: Use analytics logger when reporting analytics. 2023-11-21 10:49:57 -08:00
Aman Agrawal 69d8442ab4 billing: Allow user to switch between billing frequencies. 2023-11-21 10:42:12 -08:00
Alex Vandiver 9bc41ca040 zilencer: Store the last-reported server version when storing analytics.
Servers since 216d2ec1bf (version 2.0.0)
have submitted this, but we have never stored it.
2023-11-20 14:36:27 -08:00
Prakhar Pratyush f7f5131aa8 email_notification: Fix bad rendering of math formulas.
Earlier, for the emails having latex math like
"$$d^* = +\infty$$", the bad rendering led to the math
being included multiple times in the email body.

This was due to displaying KaTeX HTML without the CSS.

This commit fixes the incorrect behavior by replacing
the KaTeX with the raw LaTex source.

Fixes part of #25289.
2023-11-20 10:41:02 -08:00
Aman Agrawal 71ea6e8863 realm_inline_image_preview: Use it to toggle video previews too.
This setting now also works to decide whether to show previews of
uploaded or linked videos.
2023-11-20 08:48:39 -08:00
Mateusz Mandera 357dceb05a typing: Rewrite remote_server_post_analytics to use @typed_endpoint.
The main point is the RealmDataForAnalytics structure, which we can next
re-use for other endpoints that will take it in in their params.
2023-11-17 18:32:49 -08:00
Mateusz Mandera ab633f4557 analytics: Add send_realms_only_to_push_bouncer function.
This is a useful helper using the same API as
send_analytics_to_push_bouncer(), but uploading only realms info. This
is useful to upload realms info without the risk of taking a long time
to process the request due to too much of the *Count analytics data.
2023-11-17 18:32:49 -08:00
Mateusz Mandera 1800b2c797 ldap: Tweak AUTH_LDAP_ADVANCED_REALM_ACCESS_CONTROL behavior.
The original behavior of this setting was to disable LDAP
authentication for any realms not configured to use it. This was an
arbitrary choice, and its only value was to potentially help catch
typos for users who are lazy about testing their configuration.

Since it makes it a very inconvenient to potentially host multiple
organizations with different LDAP configurations, remove that
behavior.
2023-11-17 14:40:26 -08:00
Prakhar Pratyush cc934429fe settings: Add option for followed topics to unread count badge setting.
This commit adds a new option 'DMs, mentions, and followed topics'
to 'desktop_icon_count_display' setting.

The total unread count of DMs, mentions, and followed topics appears
in desktop sidebar and browser tab when this option is configured.

Some existing options are relabeled and renumbered. We finally have:
* All unread messages
* DMs, mentions, and followed topics
* DMs and mentions
* None

Fixes #27503.
2023-11-17 14:07:20 -08:00
Tim Abbott 34f4622091 api_docs: Add missing followed topic visibility policy.
While the server implementation has accepted this value for a few
months as part of building the feature, following topics was not a
fully supported feature of the Zulip server before
3f2ab44f94, just before feature
level 219.

So that's probably the correct level to document as the first feature
level at which we recommend that clients supporting the followed
topics feature process the value.
2023-11-17 13:47:08 -08:00
Lauryn Menard 11cb37c9a4 billing: Add prototype remote billing sessions.
These new models are incomplete and totally untested, but merging this
will provide valuable scaffolding for doing smaller PRs working on
individual gaps, and reveals a clear set of TODOs/refactoring/model
changes needed to support where want to end up.

Co-authored-by: Tim Abbott <tabbott@zulip.com>
2023-11-17 12:58:37 -08:00
David 81f99dde55
gear_menu: Rename "Manage streams" -> "Stream settings".
Fixes #27754.
2023-11-17 08:53:41 -08:00
Tim Abbott c71f62089b api_docs: Clarify found_oldest/newest descriptions. 2023-11-16 18:00:37 -08:00
Mateusz Mandera 13aa24a56a drafts: Update some stale docstrings regarding draft data validation.
These were written before the draft endpoints were converted to use
@typed_endpoint and pydantic-based DraftData(BaseModel) for param
validation. Update them to avoid the confusion of talking about dicts
and dict_validator functions when those are no longer a thing.
2023-11-16 17:29:12 -08:00
Tim Abbott 3cfe4b720c Revert "linkifiers: Match JS implementation to server implementation."
This reverts commit 091e2f177b.

This version of python_to_js_linkifier fails for at least some real
linkifiers. We'll likely re-introduce this after a bit more debugging.
2023-11-16 14:59:48 -08:00
Greg Price 528a76a419 push_notifs: Set APNs topic individually on each notification.
This makes it possible to send notifications to more than one app ID
from the same server: for example, the main Zulip mobile app and the
new Flutter-based app, which has a separate app ID for use through its
beta period so that it can be installed alongside the existing app.
2023-11-16 12:03:30 -08:00
Greg Price ff32d51700 push_notifs: Support APNs token auth, as well as cert auth.
This will make it possible to send notifications to multiple
distinct app IDs over the same connection.
2023-11-16 12:03:30 -08:00
Alex Vandiver d6e809d185 streams: Fix unsorted import. 2023-11-16 19:02:13 +00:00
Alex Vandiver 5e4bb69498 subscription_info: Fix typo caught by codespell. 2023-11-16 18:55:31 +00:00
Sahil Batra dc98136346 streams: Send stream deletion events on unsubscribing users.
This commit adds code to send stream deletion events when
unsubscribing non-admin users from private streams and
when unsubscribing guests from public streams since
non-admins cannot access unsubscribed private streams
and guests cannot access unsubscribed public streams.
2023-11-16 13:25:33 -05:00
Sahil Batra d4fb244d2d CVE-2023-47642: Invalid metadata access for formerly subscribed streams.
It was discovered by the Zulip development team that active users who
had previously been subscribed to a stream incorrectly continued being
able to use the Zulip API to access metadata for that stream. As a
result, users who had been removed from a stream, but still had an
account in the organization, could still view metadata for that
stream (including the stream name, description, settings, and an email
address used to send emails into the stream via the incoming email
integration). This potentially allowed users to see changes to a
stream’s metadata after they had lost access to the stream.

This bug was present in all Zulip releases prior to today's Zulip
Server 7.5.
2023-11-16 13:25:33 -05:00
Sahil Batra 3c8701ee36 streams: Add API endpoint to get stream email.
This commit adds new API endpoint to get stream email which is
used by the web-app as well to get the email when a user tries
to open the stream email modal.

The stream email is returned only to the users who have access
to it. Specifically for private streams only subscribed users
have access to its email. And for public streams, all non-guest
users and only subscribed guests have access to its email.
All users can access email of web-public streams.
2023-11-16 13:25:33 -05:00
Sahil Batra 432001656e streams: Remove "email_address" field from Subscription objects.
This commit removes "email_address" field from Subscription objects
and we would instead a new endpoint in next commit to get email
address for stream with proper access check.

This change also fixes the bug where we would include email address
for the unsubscribed private stream as well when user did not have
permission to send message to the stream, and having email allowed
the unsubscribed user to send message to the stream.

Note that the unsubscribed user can still send message to the stream
if the user had noted down the email before being unsubscribed
and the stream token is not changed after unsubscribing the user.
2023-11-16 13:25:33 -05:00
Mateusz Mandera 1819b85b85 management: Allow changing is_billing_admin using change_user_role. 2023-11-15 18:36:07 -08:00
Anders Kaseorg c8784634e1 openapi: Remove unnecessary cast.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-11-15 17:10:48 -08:00
Anders Kaseorg f4e7a11c35 requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-11-15 15:27:54 -08:00
Anders Kaseorg a688e753de test_helpers: Fix logging in cursor_executemany mock.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-11-15 15:27:54 -08:00
Anders Kaseorg cb9a04d3e3 test_create_video_call: Add missing not None assertions.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-11-15 15:27:54 -08:00
Anders Kaseorg 2c9c105bbb db: Check connection type in wrapper_execute.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-11-15 15:27:54 -08:00
Anders Kaseorg 248ee82db0 migrations: Remove useless null argument for ManyToManyField.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-11-15 15:27:54 -08:00
Anders Kaseorg 9552014047 models: Include explicit app name in ManyToManyField reference strings.
https://github.com/typeddjango/django-stubs/issues/1802

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-11-15 15:27:54 -08:00
Tim Abbott 2e2997bd7d typing: Limit typing notifications in large streams. 2023-11-15 09:42:25 -08:00
Alex Vandiver 091e2f177b linkifiers: Match JS implementation to server implementation.
Since the server-side implementation no longer uses look-ahead
or (more importantly) look-behind, it is possible to exactly implement
in Javascript.  This removes a common class which would prevent local
echo.

This requires reworking the topic linking algorithm, to march the
server's as well.  The tests and behaviour are adjusted in so doing --
previously, the JS implementation would have linked `#foo` with a
`foo` regex on the linkifier, but the server implementation would not
have.
2023-11-14 20:43:39 -08:00
Alex Vandiver 70b20e9d2b markdown: Use \p{White_Space} equivalent for linkifier boundaries.
We do not use \p{White_Space} itself because re2 does not support it.
2023-11-14 20:43:39 -08:00
Alex Vandiver 549dd8a4c4 markdown: Expand prepare_linkifier_pattern to make it more legible. 2023-11-14 20:43:39 -08:00
Sahil Batra 532293c4cf streams: Unset is_web_public and is_realm_public fields on attachments.
This commit adds code to unset is_web_public and is_realm_public fields
on attachments when deactivating a stream as we do not want to allow
spectators to access them after the stream is deactivated.

This commit also adds a comment explaining why we don't use
do_change_stream_permission to set the privacy fields on deactivating
a stream.

Fixes #27634.
2023-11-13 10:26:14 -08:00
Sahil Batra 30e2c14f4a streams: Unset is_realm_public on attachment when unarchiving streams.
We did not unset is_realm_public field on attachements when unarchiving
streams, but we do unset is_web_public field. This commit adds code to
unset the is_realm_public field as well as we make the stream private
while unarchiving it.
2023-11-13 10:26:14 -08:00
Alex Vandiver 1f3a53db2d cache: Drop the huddle cache.
This cache was only used in one place, which is infrequently
called (only when sending messages, or searching explicitly for a list
of users) and the overhead of maintaining the cache is not worth
trying to avoid the well-indexed lookup of the huddle.
2023-11-13 10:01:29 -08:00
Sahil Batra 8f79cec51a users: Pass bogus data for inaccessible users.
We now pass bogus data for inaccessible users when sending
the users data in "realm_users" field of "register" response
or when using endpoints like "GET /users" to get data of
all the users in realm.

We would add a client capability field in future commits
such that new clients would receive data only for accessible
users and they can form the bogus data by themselves.
2023-11-13 08:04:45 -08:00
Sahil Batra aedb9b23b4 models: Pass "realm__can_access_all_users_group" to select_related. 2023-11-13 08:04:45 -08:00
Sahil Batra c82bb3ec76 models: Add can_access_all_users_group setting.
This commit adds new setting for controlling who can access
all users in the realm which would have "Everyone" and
"Members only" option.

Fixes part of #10970.
2023-11-13 08:04:45 -08:00