Commit Graph

8188 Commits

Author SHA1 Message Date
Prakhar Pratyush ad3603c0aa onboarding: Show one-time modal to introduce recent conversations view.
To improve onboarding experience, this commit adds a
one-time modal which introduces the recent conversations view.

Users see this one-time modal on visiting the recent
conversations view.

Fixes #29073.
2024-03-19 09:32:59 -07:00
Prakhar Pratyush 9a7634d527 onboarding: Show one-time modal to introduce inbox view.
To improve onboarding experience, this commit adds
a one-time modal which introduces the inbox view.

Users see this one-time modal on visiting the inbox view.

Fixes part of #29073.
2024-03-19 09:32:59 -07:00
Alex Vandiver c13e3dee24 katex: Replace subprocess call with minimal external service.
Replace a separate call to subprocess, starting `node` from scratch,
with an optional standalone node Express service which performs the
rendering.  In benchmarking, this reduces the overhead of a KaTeX call
from 120ms to 2.8ms.  This is notable because enough calls to KaTeX in
a single message would previously time out the whole message
rendering.

The service is optional because he majority of deployments do not use
enough LaTeX to merit the additional memory usage (60Mb).

Fixes: #17425.
2024-03-15 15:34:12 -07:00
Alex Vandiver c7e92ee2f3 outgoing_http: Support fractional-second timeouts. 2024-03-15 15:34:12 -07:00
Aman Agrawal 2b12c8b4b9 narrow: Fix comment. 2024-03-11 14:13:26 -07:00
Mateusz Mandera b572fa93cd context_managers: Open file in write mode in lockfile_nonblocking.
Otherwise this fails if the file doesn't yet exist.
2024-03-06 10:35:02 -08:00
Mateusz Mandera 6593ccd7fc test_helpers: Remove /self-hosted-billing/ from exempt_patterns.
This was added early on and hasn't been removed since, even though we
have tests for the endpoint for a while now.
2024-03-05 11:53:51 -08:00
Mateusz Mandera e39f400f94 remote_billing: Make "plan management" always available.
Just shows a config error page if the bouncer is not enabled. Uses a new
endpoint for this so that it can work nicely for both browser and
desktop app clients.
It's necessary, because the desktop app expects to get a json response
with either an error or billing_access_url to redirect to. Showing a
nice config error page can't be done via the json error mechanism, so
instead we just serve a redirect to the new error page, which the app
will open in the browser in a new window or tab.
2024-03-05 11:53:51 -08:00
Mateusz Mandera 1ede8da46c events: Change format of authentication_methods realm_update_dict event.
Makes the event in line with state_data.realm_authentication_methods.
2024-03-05 11:48:58 -08:00
Mateusz Mandera da9e4e6e54 backends: Implementation of restricting certain backends by plan.
Only affects zulipchat, by being based on the BILLING_ENABLED setting.

The restricted backends in this commit are
- AzureAD - restricted to Standard plan
- SAML - restricted to Plus plan, although it was already practically
  restricted due to requiring server-side configuration to be done by us

This restriction is placed upon **enabling** a backend - so
organizations that already have a backend enabled, will continue to be
able to use it. This allows us to make exceptions and enable a backend
for an org manually via the shell, and to grandfather organizations into
keeping the backend they have been relying on.
2024-03-05 11:48:58 -08:00
David Rosa 3d63ea20b6 help: Add "View stream subscribers" page.
Documents how to view stream subscribers via settings and
the right sidebar.

Fixes #28887.
2024-03-05 10:22:18 -08:00
Mateusz Mandera 634015411a update_analytics_count: Use a correct lock mechanism.
Adds a re-usable lockfile_nonblocking helper to context_managers.

Relying on naive `os.mkdir` is not enough especially now that the
successful operation of this command is necessary for push notifications
to work for many servers.

We can't use `lockfile` context manager from
`zerver.lib.context_managers`, because we want the custom behavior of
failing if the lock can't be acquired, instead of waiting.
That's because if an instance of this gets stuck, we don't want to start
queueing up more processes waiting forever whenever the cronjob runs
again and fail->exit is preferrable instead.
2024-03-05 10:21:14 -08:00
Alex Vandiver 352dbf9387 caches: Only take the realm_id, not the Realm, as a cache key function.
This saves a hit to the database to fetch the Realm of a UserProfile
that we are trying to flush.
2024-03-04 16:35:57 -08:00
Anders Kaseorg 59b0548433 timezone: Only look up canonical time zones from the system.
Legacy time zone aliases were removed from the Debian tzdata package
in tzdata 2023c-8.

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1040997

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-03-01 17:38:08 -08:00
Anders Kaseorg 7b1bb984b3 ruff: Fix RUF022 `__all__` is not sorted.
This is a preview rule, not yet enabled by default.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-03-01 09:30:04 -08:00
Anders Kaseorg 3b114c516c ruff: Fix PLR2044 Line with empty comment.
This is a preview rule, not yet enabled by default.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-03-01 09:30:04 -08:00
Anders Kaseorg 87992b8b29 ruff: Fix PERF403 Use a dictionary comprehension instead of a for-loop.
This is a preview rule, not yet enabled by default.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-03-01 09:30:04 -08:00
Anders Kaseorg d748ec8d52 ruff: Fix PLW0108 Lambda may be unnecessary.
This is a preview rule, not yet enabled by default.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-03-01 09:30:04 -08:00
Anders Kaseorg 865febb307 ruff: Fix RUF021 Parenthesize when chaining `and` and `or` together.
This is a preview rule, not yet enabled by default.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-03-01 09:30:04 -08:00
Anders Kaseorg 570f3dd447 python: Reformat with Ruff formatter.
https://docs.astral.sh/ruff/formatter/

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-02-29 17:07:16 -08:00
Alex Vandiver 4712031528 email_validation: Merge two implementations of a disposable-email check.
Only one of them properly checked for improperly-formed email
addresses, thanks to #22723.
2024-02-27 10:19:23 -08:00
Alex Vandiver 4b512b3409 email_validation: Rename to validate_is_not_disposable.
This clarifies what we are asserting -- namely, that the email is
_not_ disposable.
2024-02-27 10:19:23 -08:00
Alex Vandiver c1aadbe52e events: Do nothing in apply_event for restart events.
These signal that the Tornado process restarted, which in itself is
not notable for apply_events.
2024-02-27 10:18:11 -08:00
Lauryn Menard cf82d3316b push-bouncer: Exclude LoggingCountStats with partial data.
LoggingCountStats with a daily duration and that are directly stored
on the RealmCount table (not via aggregation in process_count_stat),
can be in a state, after the hourly cron job to update analytics
counts, where the logged value will be live-updated later, because
the end time for the stat is still in the future.

As these logging counts are designed to be used on the self-hosted
installation for either debugging or rate limiting, sending these
partial/incomplete counts to the bouncer has low value.
2024-02-26 17:53:12 -08:00
Prakhar Pratyush fe1a20ebb3 settings: Add realm-level setting 'zulip_update_announcements_stream'.
This commit adds a realm-level setting named
'zulip_update_announcements_stream' that configures the
stream to which zulip updates should be posted.

Fixes part of #28604.
2024-02-25 09:33:00 -08:00
David Rosa 5215e3963c help: Clean up beta app installation instructions.
Converts the beta app installation instructions into our standard
tabbed instructions format.

Fixes #28885.
2024-02-24 08:12:34 -08:00
David Rosa 948b427de7 api-docs: Add instructions to download `zuliprc` files.
- Adds instructions for downloading a zuliprc file for a bot or for
  yourself.
- Updates the button label to "Download zuliprc", since that's the
  filename it downloads.

Fixes #28881.
2024-02-23 09:43:57 -08:00
Prakhar Pratyush ee612dafac settings: Rename signup_notifications_stream realm setting.
This commit renames the realm-level setting
'signup_notifications_stream' to 'signup_announcements_stream'.

The new name reflects better what the setting does.
2024-02-21 09:04:23 -08:00
Prakhar Pratyush ab453fbe20 settings: Rename notifications_stream to new_stream_announce..._stream.
This commit renames the realm-level setting 'notifications_stream'
to 'new_stream_announcements_stream'.

The new name reflects better what the setting does.
2024-02-21 09:04:23 -08:00
Prakhar Pratyush 0c83bca81e events: Use 'get_notifications_stream' realm method.
This commit updates the code in 'fetch_initial_state_data'
function to use the 'get_notifications_stream' method
instead of re-writing the logic.
2024-02-21 09:04:23 -08:00
Alex Vandiver b747ea285f topic: Fix history order for topic moves.
5c96f94206 mistakenly appended, rather than prepended, the edit to
the history.  This caused AssertionErrors when attempting to view the
history of moved messages, which check that the `last_edit_time`
matches the timestamp of the first edit in the list.

Fix the ordering, and update the `edit_history` for messages that were
affected.  We limit to only messages edited since the commit was
merged, since that helps bound the affected messages somewhat.
2024-02-20 21:30:32 -08:00
Anders Kaseorg e568064f4e page_params: Set and expect state_data = null for spectators.
Fix a ZodError when spectators load the app, introduced by commit
a4938d3760 (#28971).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-02-19 03:03:58 -05:00
Anders Kaseorg a4938d3760 page_params: Parse page_params and state_data with Zod.
This establishes a runtime check that their types continue to reflect
reality going forward.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-02-17 00:02:38 -08:00
Alex Vandiver 0079688c49 tornado: Drop WebReloadClientError logic.
The widening of the time between when a process is marked for
reload (at Tornado startup) and when it sends reload events makes it
unlikely-to-impossible that a single `/` request will span both of
them, and thus hit the WebReloadClientError corner case.

Remove it, as it is not worth the complication.  The bad behaviour it
is attempting to prevent (of a reload right after opening `/`) was
always still possible  -- if the `/` request completed right before
Tornado restarted -- so it is not clear that it was ever worth the
complication.
2024-02-15 15:42:50 -08:00
Alex Vandiver fc41d6085b tornado: Split server restart events from web client reload events. 2024-02-15 15:42:50 -08:00
Anders Kaseorg bf6978185d home: Send state_data in a separate object.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-02-15 10:22:52 -08:00
Alex Vandiver 23baabba86 message: Merge unnecessary cache_transformer step.
Having a non-identity `cache_transformer` is no different from running
it on every row of the query_function.  Simplify understanding of the
codepath used in caching by merging the pieces of code.
2024-02-14 12:27:03 -08:00
Alex Vandiver 6597c560cb message_cache: Rename "to_dict" functions which deal with bytes. 2024-02-14 12:27:03 -08:00
Alex Vandiver b3e8878252 message_cache: Rename update_to_dict_cache to update_message_cache.
This better describes what it does.
2024-02-14 12:27:03 -08:00
Alex Vandiver 737a751f5c message: Split MessageDict and friends into its own file. 2024-02-14 12:27:03 -08:00
Alex Vandiver 0f9b7f112b message: Move render_markdown into zerver.lib.markdown. 2024-02-14 12:27:03 -08:00
Alex Vandiver 22837fc1b4 message_edit: Carry the QuerySet through as much as possible.
Rather than pass around a list of message objects in-memory, we
instead keep the same constructed QuerySet which includes the later
propagated messages (if any), and use that same query to pick out
affected Attachment objects, rather than limiting to the set of ids.
This is not necessarily a win -- the list of message-ids *may* be very
long, and thus the query may be more concise, easier to send to
PostgreSQL, and faster for PostgreSQL to parse.  However, the list of
ids is almost certainly better-indexed.

After processing the move, the QuerySet must be re-defined as a search
of ids (and possibly a very long list of such), since there is no
other way which is guaranteed to correctly single out the moved
messages.  At this point, it is mostly equivalent to the list of
Message objects, and certainly takes no less memory.
2024-02-14 12:27:03 -08:00
Alex Vandiver 5c96f94206 topic: Use a single SQL statement to propagate message moves.
Rather than use `bulk_update()` to batch-move chunks of messages, use
a single SQL query to move the messages.  This is much more efficient
for large topic moves.  Since the `edit_history` field is not yet
JSON (see #26496) this requires that PostgreSQL cast the current data
into `jsonb`, append the new data (also cast to `jsonb`), and then
re-cast that as text.

For single-message moves, this _increases_ the SQL query count by one,
since we have to re-query for the updated data from the database after
the bulk update.  However, this is overall still a performance
improvement, which improves to 2x or 3x for larger topic moves.  Below
is a table of duration in seconds to run `do_update_message` to move a
topic to a new stream, based on messages in the topic, for before and
after this change:

| Topic size |  Before  |  After  |
| ---------- | -------- | ------- |
| 1          |   0.1036 |  0.0868 |
| 2          |   0.1108 |  0.0925 |
| 5          |   0.1139 |  0.0959 |
| 10         |   0.1218 |  0.0972 |
| 20         |   0.1310 |  0.1098 |
| 50         |   0.1759 |  0.1366 |
| 100        |   0.2307 |  0.1662 |
| 200        |   0.3880 |  0.2229 |
| 500        |   0.7676 |  0.4052 |
| 1000       |   1.3990 |  0.6848 |
| 2000       |   2.9706 |  1.3370 |
| 5000       |   7.5218 |  3.2882 |
| 10000      |  14.0272 |  5.4434 |
2024-02-14 12:27:03 -08:00
Alex Vandiver 822131fef4 message: Add a bulk_access_stream_messages_query method.
This applies access restrictions in SQL, so that individual messages
do not need to be walked one-by-one.  It only functions for stream
messages.

Use of this method significantly speeds up checks if we moved "all
visible messages" in a topic, since we no longer need to walk every
remaining message in the old topic to determine that at least one was
visible to the user.  Similarly, it significantly speeds up merging
into existing topics, since it no longer must walk every message in
the new topic to determine if the user could see at least one.

Finally, it unlocks the ability to bulk-update only messages the user
has access to, in a single query (see subsequent commit).
2024-02-14 12:27:03 -08:00
Alex Vandiver 628be8d433 streams: Remove a lie from the docstring. 2024-02-14 12:27:03 -08:00
Alex Vandiver 06710631ff tests: Reserve "Internal" client, used by email gateway and topic moves. 2024-02-14 12:27:03 -08:00
Alex Vandiver 7f46773ef1 tests: Clear in-memory Client caches before testing query counts.
This makes counts more apples-to-apples comparable when run
back-to-back.
2024-02-14 12:27:03 -08:00
Anders Kaseorg e79572d0d5 page_params: Remove unused first_in_realm.
It’s unused since commit e1843dd1b9
(#5819).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-02-08 10:08:15 -08:00
Anders Kaseorg b59faf540f page_params: Remove unused prompt_for_invites.
It’s unused since commit ebe959f2b0.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-02-08 10:08:15 -08:00
Alya Abbott e9b0c7f2c0 name_restrictions: Reserve additional subdomains. 2024-02-07 12:10:00 -08:00