Commit Graph

17973 Commits

Author SHA1 Message Date
Prakhar Pratyush b9d9676b43 zulip_news: Update the group DM sent initially to admins.
As a part of the zulip news feature, we send an initial
group DM to admins suggesting them to update or set
the 'zulip_update_announcements_stream'.

This commit updates those messages to make it better.
2024-03-27 11:42:00 -07:00
Lauryn Menard c53a96a7b0 i18n-test: Update translated strings in test_email_translation.
Updates the check email translation test for updated email text in
confirm_new_email.html and onboarding_zulip_topics.html for current
translated strings in German.
2024-03-27 11:16:59 -07:00
Alex Vandiver 7de527f3bc soft_deactivation: Use much faster bulk_insert_all_ums.
As noted in the docstring for `bulk_insert_ums`, this is at least one
order of magnitude faster than using `bulk_create`.  This also
includes a `ON CONFLICT DO NOTHING` which allows multiple
soft-reactivations to run at once without failing.  We also adjust the
update of `last_active_message_id` to be safe against races.
2024-03-26 10:52:37 -07:00
Alex Vandiver 7988aad159 user_message: Use INSERT ... ON CONFLICT for historical UM creation.
Rather than use a bulk insert via Django, use the faster
`bulk_insert_all_ums` that we already have.  This also adds a `ON
CONFLICT` clause, to make the insert resilient to race conditions.

There are currently two callsites, with different desired `ON
CONFLICT` behaviours:
 - For `notify_reaction_update`, if the `UserMessage` had already been
   created, we would have done nothing to change it.
 - For `do_update_message_flags`, we would have ensured a specific bit
   was (un)set.

Extend `create_historical_user_messages` and `bulk_insert_all_ums` to
support `ON CONFLICT (...) UPDATE SET flags = ...`.
2024-03-26 10:52:37 -07:00
Alex Vandiver 52e3c8e1b2 user_message: Move create_historical_user_messages to zerver.lib.user_message.
It is not even used in zerver.actions.create_user.
2024-03-26 10:52:37 -07:00
Alex Vandiver a7ec7e924e import: Skip "active" check, and analytics, for crossrealm bots.
The bots do not exist in the user table to look up their active
status, and attempting to import them into the analytics table will
result in duplicate rows.
2024-03-25 10:21:48 -07:00
Aditya Bajaj a20b454ebb
notifications: Shorten string for redacted content.
Replace the long string for organisations that have notification
body/content disabled (settings.PUSH_NOTIFICATION_REDACT_CONTENT
set to true) with "New message".

This allows more of the limited space on the mobile device screen to
be used for additional messages rather than this verbose content.

Fixes #29152
2024-03-22 09:37:51 -07:00
Mateusz Mandera 89a22fb076 register_server: Check write access to secrets file when rotate_key.
Using --rotate-key without write access to the secrets file is currently
quite painful, since you end up rotating your registration's secret with
no local record of it; so effectively you lose your registration and
need help from support. We should just prevent this failure mode.
2024-03-22 09:33:00 -07:00
Alex Vandiver 7d0c12aa3a semgrep: Enforce no bare select_related() calls.
Previously, #26419 addressed the majority of these calls, but did not
prevent more from creeping in.  Remove the one remaining
callsite (after the cleanup from the previous commits), and ban any
future use of the pattern.
2024-03-22 09:30:17 -07:00
Alex Vandiver fd5a091b30 messages: Only check the UserMessage row if necessary.
For the common case of not needing to reference the UserMessage row
later, and for being a stream without private history, the UserMessage
row is irrelevant.  Convert `has_user_message` to a thunk, and defer
loading it unless necessary.
2024-03-22 09:30:17 -07:00
Alex Vandiver f92d43c690 messages: Use overloads to only return a user_message if needed. 2024-03-22 09:30:17 -07:00
Alex Vandiver 6ace34c374 messages: Stop joining through every table.
Calling `.select_related()` with no arguments joins through every
possible table, recursively.  In this case, this currently produces a
query which joins through forty-three tables.

This is rather inefficient, particularly for what is a very common
call which should be very fast.

No callsite depends on having prefetched any joined table on the
object; drop all of the joins.
2024-03-22 09:30:17 -07:00
John Lu a5cf0ec526
refactor: Replace HUDDLE with DIRECT_MESSAGE_GROUP.
Replaced HUDDLE attribute with DIRECT_MESSAGE_GROUP using VS Code search,
part of a general renaming of the object class.

Fixes part of #28640.

Co-authored-by: JohnLu2004 <JohnLu10212004@gmail.com>
2024-03-21 16:39:33 -07:00
Anders Kaseorg 224b484195 test_classes: Skip OpenAPI validation on exceptions.
This gives more helpful error tracebacks from failing tests.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-03-21 16:36:48 -07:00
Alex Vandiver 9d8d2d138b missedmessage_emails: Add Sentry spans to worker thread. 2024-03-21 12:46:13 -07:00
Alex Vandiver 9451d08bb9 worker: Split out worker sampling rate, and add Sentry transactions. 2024-03-21 12:46:13 -07:00
Alex Vandiver 3cbce0c5c7 missedmessage_emails: Clear caches and db query tracking per-loop.
Otherwise, these accumulate and leak memory.
2024-03-21 12:46:13 -07:00
Alex Vandiver 6ad777c86f dev_urls: Make requests to /docs redirect to the index.
This makes it easier to browse to the docs in development mode, and
more closely matches ReadTheDocs production.
2024-03-21 11:27:14 -07:00
Vector73 f758ca596b custom_profile_fields: Add "required" parameter to the profile fields.
Fixes #28512.
2024-03-21 10:48:54 -07:00
Anders Kaseorg a1a341f0ae ruff: Fix UP032 Use f-string instead of `format` call.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-03-21 08:59:17 -07:00
Lauryn Menard 373671080d remote-support: Include billing entity name in internal billing notice.
Adds a line to the top of the internal_billing_notice email with
the billing entity's display name.

Makes sure all internal_billng_notice email subjects also include
the billing entity's display name.

Makes small updates to the notice text for some cases.
2024-03-20 11:58:15 -07:00
Prakhar Pratyush 118a7e8d9d zulip_updates: Send zulip updates based on zulip_update_*_level.
This commit adds a management command that will run regularly
as a cron job to send zulip updates to realms based on their
current and latest zulip_update_announcements_level.

For realms with:
* level = None: Send a group DM to admins notifying them about
this new feature & suggestion to set the stream accordingly.

* level = 0:
  * If stream is still not configured, wait for a week
    before setting their level to latest level. They will
    miss updates until their configure the stream.
  * If stream is configured, send updates.

* level > 0: Send one message/update per level & increase
  the level by 1 till the latest level.

Fixes #28604.
2024-03-20 11:48:06 -07:00
Prakhar Pratyush 336d46001c message_send: Add 'internal_prep_huddle_message' function.
This is a prep commit to extract out the logic to
create message from 'internal_send_huddle_message'
into a separate function 'internal_prep_huddle_message'.

We will use this new function to get the huddle message
without sending it immediately.
2024-03-20 11:48:06 -07:00
Prakhar Pratyush 23fc04577b message_send: Don't create savepoint in do_send_messages.
In general, we never want to use savepoints.

This prep commit adds savepoint=False in do_send_messages
as we don't want to just rollback to this savepoint and
proceed if we encounter any error while sending zulip updates
via cron.
2024-03-20 11:48:06 -07:00
Prakhar Pratyush 6484532830 help: Migrate configure-notification-bot help page to a new URL.
This commit migrates the 'help/configure-notification-bot' help
page to a new URL i.e. 'help/configure-automated-notices'.
2024-03-20 11:48:06 -07:00
Alex Vandiver 6bf9b7a7f7 migrations: Merge duplicated migration heads. 2024-03-19 19:04:32 -04:00
Alex Vandiver 0f0631813f message_edit: Do set differences in QuerySets. 2024-03-19 18:22:24 -04:00
Alex Vandiver c71b7afe9f message_edit: Handle user, not subscription, collections.
Nothing about the subscription is relevant -- we instead use
collections of User objects for readability.
2024-03-19 18:22:24 -04:00
Alex Vandiver f15e006873 message_edit: Perform cross join in SQL, not Python.
The database can do this faster than we can, and it reduces the amount
of information transmitted over the wire.
2024-03-19 18:22:24 -04:00
Alex Vandiver a8795c5d0b message_edit: Pull user_ids_gaining_usermessages higher, rename. 2024-03-19 18:22:24 -04:00
Alex Vandiver 7ce6095003 message_edit: Handle previous subscribers and private-history streams. 2024-03-19 18:22:24 -04:00
Alex Vandiver cf8b9adad4 CVE-2024-27286: Delete dangling UserMessage rows.
This cleans up dangling UserMessage rows for moved messages which were
affected by bugs in one of the previous two commits.
2024-03-19 18:22:24 -04:00
Alex Vandiver 7b1feac06a CVE-2024-27286: Remove UserMessage rows for non-active Subscriptions.
A user who was no longer subscribed to a private stream kept their
UserMessage row for a message sent while they were in it; this is
expected.  However, they _also_ kept that row even if the message was
moved to a different private stream that they were also not subscribed
to.  This violates the invariant that users without subscriptions
never have UserMessage rows.
2024-03-19 18:22:24 -04:00
Alex Vandiver e964536139 CVE-2024-27286: Run usermessage modifications even for change_one.
This `if new_stream is not None` block was improperly indented,
causing it to only run if the propagation mode was not `change_one`.
Since the block controlled creation and deletion of UserMessage rows,
this led to messages being improperly still visible to members of the
old stream if they were being moved from public to private streams.
Clients also failed to receive `delete_message` events, so the
messages remained visible in their feeds until they reloaded the
application.
2024-03-19 18:22:24 -04:00
Alex Vandiver 64df8c7075 send_custom_email: is_realm_admin is a property, not a real column. 2024-03-19 13:19:45 -07:00
Prakhar Pratyush de53e372dd onboarding: Don't show introduce zulip view modals to existing users.
We don't want to show one-time modals introducing 'Inbox' and
'Recent conversations' views to existing users.

When a user views a modal, we mark it as read by storing a row
in the 'OnboardingStep' model. Once marked as read, the user will
no longer see the modal.

This commit adds a migration to create
two rows per user (mark them as read) in OnboardingStep model.
2024-03-19 09:32:59 -07:00
Prakhar Pratyush ad3603c0aa onboarding: Show one-time modal to introduce recent conversations view.
To improve onboarding experience, this commit adds a
one-time modal which introduces the recent conversations view.

Users see this one-time modal on visiting the recent
conversations view.

Fixes #29073.
2024-03-19 09:32:59 -07:00
Prakhar Pratyush 9a7634d527 onboarding: Show one-time modal to introduce inbox view.
To improve onboarding experience, this commit adds
a one-time modal which introduces the inbox view.

Users see this one-time modal on visiting the inbox view.

Fixes part of #29073.
2024-03-19 09:32:59 -07:00
Alex Vandiver c13e3dee24 katex: Replace subprocess call with minimal external service.
Replace a separate call to subprocess, starting `node` from scratch,
with an optional standalone node Express service which performs the
rendering.  In benchmarking, this reduces the overhead of a KaTeX call
from 120ms to 2.8ms.  This is notable because enough calls to KaTeX in
a single message would previously time out the whole message
rendering.

The service is optional because he majority of deployments do not use
enough LaTeX to merit the additional memory usage (60Mb).

Fixes: #17425.
2024-03-15 15:34:12 -07:00
Alex Vandiver c7e92ee2f3 outgoing_http: Support fractional-second timeouts. 2024-03-15 15:34:12 -07:00
Lauryn Menard 2b9220ff58 api-docs: Update feature level 224 changelog and **Changes** notes.
Links to the available message flag table in the feature level 224
changelog entry, as there are relevant **Changes** notes for this
feature level in that part of the API documentation.

Updates the order and formatting of these new and deprecated flags
in the available flags table. Also, adds a link to the topic
wildcard mentions section of the help center documentation.

Makes small clean ups to the changes notes for this feature level,
as well as the changelog entry itself.

The original commit for these feature level 224 API changes was
c597de6a1d.
2024-03-13 16:50:48 -07:00
Mahhheshh 6c0818ff51 email_page: Migrate has_request_variables to typed_endpoint.
Refactor `email_page` view to use `typed_endpoint` decorator instead of
`has_request_variables`.
2024-03-13 16:47:34 -07:00
Mahhheshh 22bd8048b1 parse_client: Migrate has_request_variables to typed_endpoint.
Refactor `parse_client` view to use `typed_endpoint decorator`
instead of `has_request_variables`. This change improves code consistency
and enhances codebase comprehension.
2024-03-12 13:27:21 -07:00
Aman Agrawal 5548a28980 message_fetch: Allow spectators to access all messages view. 2024-03-11 14:13:26 -07:00
Aman Agrawal 2b12c8b4b9 narrow: Fix comment. 2024-03-11 14:13:26 -07:00
Alex Vandiver 6e91e326e9 deferred_work: Reduce batch size due to bad statistics.
PostgreSQL's estimate of the number of usermessage rows for a single
message can be wildly off, due to poor statistics generation.  This
causes this query, with 100-message batch sizes, to incorrectly
estimate millions of matched rows, causing it to perform a full-table
index scan, rather than piecemeal using the `message_id` index.

Reduce the batch size to 50, which is enough to tip in favor of a
rational query plan.
2024-03-11 09:24:59 -07:00
Mahhheshh 073b70a516 csp_violations: Migrate has_request_variables to typed_endpoint.
Refactor `report_csp_violations` view to use `typed_endpoint` decorator
instead of `has_request_variables`. This change improves code
consistency and enhances codebase comprehension.
2024-03-08 11:26:00 -08:00
Alex Vandiver 597704fa5f tests: Clarify test_inaccessible_msg_after_stream_change. 2024-03-06 17:31:16 -08:00
Alex Vandiver 40eb5bf42d tests: Split test_message_edit.py. 2024-03-06 17:31:16 -08:00
Alex Vandiver 90abb2281d tests: Split DeleteMessageTest from test_message_edit.py. 2024-03-06 17:31:16 -08:00