Commit Graph

8359 Commits

Author SHA1 Message Date
Anders Kaseorg 6e871e7731 ruff: Fix UP036 Version block is outdated for minimum Python version.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-04-01 18:32:52 -07:00
Anders Kaseorg a82a3eb4d7 ruff: Fix UP033 Use `@functools.cache`.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-04-01 18:32:52 -07:00
Aman Agrawal d21f5c9b75 registration: Ask user how they found Zulip. 2024-04-01 12:44:12 -07:00
Prakhar Pratyush a7dc7c0734 message: Add function to remove single newline in triple quoted string.
For multiline strings in triple quotes, a '\n' is included
at the end of each line.

Earlier, to skip '\n' we used to add an escape character '\'
at the end of each line.

This commit adds a function to avoid manually adding '\'.
2024-03-28 09:03:59 -07:00
Prakhar Pratyush b9d9676b43 zulip_news: Update the group DM sent initially to admins.
As a part of the zulip news feature, we send an initial
group DM to admins suggesting them to update or set
the 'zulip_update_announcements_stream'.

This commit updates those messages to make it better.
2024-03-27 11:42:00 -07:00
Alex Vandiver 7de527f3bc soft_deactivation: Use much faster bulk_insert_all_ums.
As noted in the docstring for `bulk_insert_ums`, this is at least one
order of magnitude faster than using `bulk_create`.  This also
includes a `ON CONFLICT DO NOTHING` which allows multiple
soft-reactivations to run at once without failing.  We also adjust the
update of `last_active_message_id` to be safe against races.
2024-03-26 10:52:37 -07:00
Alex Vandiver 7988aad159 user_message: Use INSERT ... ON CONFLICT for historical UM creation.
Rather than use a bulk insert via Django, use the faster
`bulk_insert_all_ums` that we already have.  This also adds a `ON
CONFLICT` clause, to make the insert resilient to race conditions.

There are currently two callsites, with different desired `ON
CONFLICT` behaviours:
 - For `notify_reaction_update`, if the `UserMessage` had already been
   created, we would have done nothing to change it.
 - For `do_update_message_flags`, we would have ensured a specific bit
   was (un)set.

Extend `create_historical_user_messages` and `bulk_insert_all_ums` to
support `ON CONFLICT (...) UPDATE SET flags = ...`.
2024-03-26 10:52:37 -07:00
Alex Vandiver 52e3c8e1b2 user_message: Move create_historical_user_messages to zerver.lib.user_message.
It is not even used in zerver.actions.create_user.
2024-03-26 10:52:37 -07:00
Alex Vandiver a7ec7e924e import: Skip "active" check, and analytics, for crossrealm bots.
The bots do not exist in the user table to look up their active
status, and attempting to import them into the analytics table will
result in duplicate rows.
2024-03-25 10:21:48 -07:00
Aditya Bajaj a20b454ebb
notifications: Shorten string for redacted content.
Replace the long string for organisations that have notification
body/content disabled (settings.PUSH_NOTIFICATION_REDACT_CONTENT
set to true) with "New message".

This allows more of the limited space on the mobile device screen to
be used for additional messages rather than this verbose content.

Fixes #29152
2024-03-22 09:37:51 -07:00
Alex Vandiver fd5a091b30 messages: Only check the UserMessage row if necessary.
For the common case of not needing to reference the UserMessage row
later, and for being a stream without private history, the UserMessage
row is irrelevant.  Convert `has_user_message` to a thunk, and defer
loading it unless necessary.
2024-03-22 09:30:17 -07:00
Alex Vandiver f92d43c690 messages: Use overloads to only return a user_message if needed. 2024-03-22 09:30:17 -07:00
John Lu a5cf0ec526
refactor: Replace HUDDLE with DIRECT_MESSAGE_GROUP.
Replaced HUDDLE attribute with DIRECT_MESSAGE_GROUP using VS Code search,
part of a general renaming of the object class.

Fixes part of #28640.

Co-authored-by: JohnLu2004 <JohnLu10212004@gmail.com>
2024-03-21 16:39:33 -07:00
Anders Kaseorg 224b484195 test_classes: Skip OpenAPI validation on exceptions.
This gives more helpful error tracebacks from failing tests.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-03-21 16:36:48 -07:00
Alex Vandiver 6ad777c86f dev_urls: Make requests to /docs redirect to the index.
This makes it easier to browse to the docs in development mode, and
more closely matches ReadTheDocs production.
2024-03-21 11:27:14 -07:00
Vector73 f758ca596b custom_profile_fields: Add "required" parameter to the profile fields.
Fixes #28512.
2024-03-21 10:48:54 -07:00
Anders Kaseorg a1a341f0ae ruff: Fix UP032 Use f-string instead of `format` call.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-03-21 08:59:17 -07:00
Prakhar Pratyush 118a7e8d9d zulip_updates: Send zulip updates based on zulip_update_*_level.
This commit adds a management command that will run regularly
as a cron job to send zulip updates to realms based on their
current and latest zulip_update_announcements_level.

For realms with:
* level = None: Send a group DM to admins notifying them about
this new feature & suggestion to set the stream accordingly.

* level = 0:
  * If stream is still not configured, wait for a week
    before setting their level to latest level. They will
    miss updates until their configure the stream.
  * If stream is configured, send updates.

* level > 0: Send one message/update per level & increase
  the level by 1 till the latest level.

Fixes #28604.
2024-03-20 11:48:06 -07:00
Prakhar Pratyush 6484532830 help: Migrate configure-notification-bot help page to a new URL.
This commit migrates the 'help/configure-notification-bot' help
page to a new URL i.e. 'help/configure-automated-notices'.
2024-03-20 11:48:06 -07:00
Alex Vandiver f15e006873 message_edit: Perform cross join in SQL, not Python.
The database can do this faster than we can, and it reduces the amount
of information transmitted over the wire.
2024-03-19 18:22:24 -04:00
Alex Vandiver 7ce6095003 message_edit: Handle previous subscribers and private-history streams. 2024-03-19 18:22:24 -04:00
Prakhar Pratyush ad3603c0aa onboarding: Show one-time modal to introduce recent conversations view.
To improve onboarding experience, this commit adds a
one-time modal which introduces the recent conversations view.

Users see this one-time modal on visiting the recent
conversations view.

Fixes #29073.
2024-03-19 09:32:59 -07:00
Prakhar Pratyush 9a7634d527 onboarding: Show one-time modal to introduce inbox view.
To improve onboarding experience, this commit adds
a one-time modal which introduces the inbox view.

Users see this one-time modal on visiting the inbox view.

Fixes part of #29073.
2024-03-19 09:32:59 -07:00
Alex Vandiver c13e3dee24 katex: Replace subprocess call with minimal external service.
Replace a separate call to subprocess, starting `node` from scratch,
with an optional standalone node Express service which performs the
rendering.  In benchmarking, this reduces the overhead of a KaTeX call
from 120ms to 2.8ms.  This is notable because enough calls to KaTeX in
a single message would previously time out the whole message
rendering.

The service is optional because he majority of deployments do not use
enough LaTeX to merit the additional memory usage (60Mb).

Fixes: #17425.
2024-03-15 15:34:12 -07:00
Alex Vandiver c7e92ee2f3 outgoing_http: Support fractional-second timeouts. 2024-03-15 15:34:12 -07:00
Aman Agrawal 2b12c8b4b9 narrow: Fix comment. 2024-03-11 14:13:26 -07:00
Mateusz Mandera b572fa93cd context_managers: Open file in write mode in lockfile_nonblocking.
Otherwise this fails if the file doesn't yet exist.
2024-03-06 10:35:02 -08:00
Mateusz Mandera 6593ccd7fc test_helpers: Remove /self-hosted-billing/ from exempt_patterns.
This was added early on and hasn't been removed since, even though we
have tests for the endpoint for a while now.
2024-03-05 11:53:51 -08:00
Mateusz Mandera e39f400f94 remote_billing: Make "plan management" always available.
Just shows a config error page if the bouncer is not enabled. Uses a new
endpoint for this so that it can work nicely for both browser and
desktop app clients.
It's necessary, because the desktop app expects to get a json response
with either an error or billing_access_url to redirect to. Showing a
nice config error page can't be done via the json error mechanism, so
instead we just serve a redirect to the new error page, which the app
will open in the browser in a new window or tab.
2024-03-05 11:53:51 -08:00
Mateusz Mandera 1ede8da46c events: Change format of authentication_methods realm_update_dict event.
Makes the event in line with state_data.realm_authentication_methods.
2024-03-05 11:48:58 -08:00
Mateusz Mandera da9e4e6e54 backends: Implementation of restricting certain backends by plan.
Only affects zulipchat, by being based on the BILLING_ENABLED setting.

The restricted backends in this commit are
- AzureAD - restricted to Standard plan
- SAML - restricted to Plus plan, although it was already practically
  restricted due to requiring server-side configuration to be done by us

This restriction is placed upon **enabling** a backend - so
organizations that already have a backend enabled, will continue to be
able to use it. This allows us to make exceptions and enable a backend
for an org manually via the shell, and to grandfather organizations into
keeping the backend they have been relying on.
2024-03-05 11:48:58 -08:00
David Rosa 3d63ea20b6 help: Add "View stream subscribers" page.
Documents how to view stream subscribers via settings and
the right sidebar.

Fixes #28887.
2024-03-05 10:22:18 -08:00
Mateusz Mandera 634015411a update_analytics_count: Use a correct lock mechanism.
Adds a re-usable lockfile_nonblocking helper to context_managers.

Relying on naive `os.mkdir` is not enough especially now that the
successful operation of this command is necessary for push notifications
to work for many servers.

We can't use `lockfile` context manager from
`zerver.lib.context_managers`, because we want the custom behavior of
failing if the lock can't be acquired, instead of waiting.
That's because if an instance of this gets stuck, we don't want to start
queueing up more processes waiting forever whenever the cronjob runs
again and fail->exit is preferrable instead.
2024-03-05 10:21:14 -08:00
Alex Vandiver 352dbf9387 caches: Only take the realm_id, not the Realm, as a cache key function.
This saves a hit to the database to fetch the Realm of a UserProfile
that we are trying to flush.
2024-03-04 16:35:57 -08:00
Anders Kaseorg 59b0548433 timezone: Only look up canonical time zones from the system.
Legacy time zone aliases were removed from the Debian tzdata package
in tzdata 2023c-8.

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1040997

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-03-01 17:38:08 -08:00
Anders Kaseorg 7b1bb984b3 ruff: Fix RUF022 `__all__` is not sorted.
This is a preview rule, not yet enabled by default.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-03-01 09:30:04 -08:00
Anders Kaseorg 3b114c516c ruff: Fix PLR2044 Line with empty comment.
This is a preview rule, not yet enabled by default.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-03-01 09:30:04 -08:00
Anders Kaseorg 87992b8b29 ruff: Fix PERF403 Use a dictionary comprehension instead of a for-loop.
This is a preview rule, not yet enabled by default.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-03-01 09:30:04 -08:00
Anders Kaseorg d748ec8d52 ruff: Fix PLW0108 Lambda may be unnecessary.
This is a preview rule, not yet enabled by default.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-03-01 09:30:04 -08:00
Anders Kaseorg 865febb307 ruff: Fix RUF021 Parenthesize when chaining `and` and `or` together.
This is a preview rule, not yet enabled by default.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-03-01 09:30:04 -08:00
Anders Kaseorg 570f3dd447 python: Reformat with Ruff formatter.
https://docs.astral.sh/ruff/formatter/

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-02-29 17:07:16 -08:00
Alex Vandiver 4712031528 email_validation: Merge two implementations of a disposable-email check.
Only one of them properly checked for improperly-formed email
addresses, thanks to #22723.
2024-02-27 10:19:23 -08:00
Alex Vandiver 4b512b3409 email_validation: Rename to validate_is_not_disposable.
This clarifies what we are asserting -- namely, that the email is
_not_ disposable.
2024-02-27 10:19:23 -08:00
Alex Vandiver c1aadbe52e events: Do nothing in apply_event for restart events.
These signal that the Tornado process restarted, which in itself is
not notable for apply_events.
2024-02-27 10:18:11 -08:00
Lauryn Menard cf82d3316b push-bouncer: Exclude LoggingCountStats with partial data.
LoggingCountStats with a daily duration and that are directly stored
on the RealmCount table (not via aggregation in process_count_stat),
can be in a state, after the hourly cron job to update analytics
counts, where the logged value will be live-updated later, because
the end time for the stat is still in the future.

As these logging counts are designed to be used on the self-hosted
installation for either debugging or rate limiting, sending these
partial/incomplete counts to the bouncer has low value.
2024-02-26 17:53:12 -08:00
Prakhar Pratyush fe1a20ebb3 settings: Add realm-level setting 'zulip_update_announcements_stream'.
This commit adds a realm-level setting named
'zulip_update_announcements_stream' that configures the
stream to which zulip updates should be posted.

Fixes part of #28604.
2024-02-25 09:33:00 -08:00
David Rosa 5215e3963c help: Clean up beta app installation instructions.
Converts the beta app installation instructions into our standard
tabbed instructions format.

Fixes #28885.
2024-02-24 08:12:34 -08:00
David Rosa 948b427de7 api-docs: Add instructions to download `zuliprc` files.
- Adds instructions for downloading a zuliprc file for a bot or for
  yourself.
- Updates the button label to "Download zuliprc", since that's the
  filename it downloads.

Fixes #28881.
2024-02-23 09:43:57 -08:00
Prakhar Pratyush ee612dafac settings: Rename signup_notifications_stream realm setting.
This commit renames the realm-level setting
'signup_notifications_stream' to 'signup_announcements_stream'.

The new name reflects better what the setting does.
2024-02-21 09:04:23 -08:00
Prakhar Pratyush ab453fbe20 settings: Rename notifications_stream to new_stream_announce..._stream.
This commit renames the realm-level setting 'notifications_stream'
to 'new_stream_announcements_stream'.

The new name reflects better what the setting does.
2024-02-21 09:04:23 -08:00
Prakhar Pratyush 0c83bca81e events: Use 'get_notifications_stream' realm method.
This commit updates the code in 'fetch_initial_state_data'
function to use the 'get_notifications_stream' method
instead of re-writing the logic.
2024-02-21 09:04:23 -08:00
Alex Vandiver b747ea285f topic: Fix history order for topic moves.
5c96f94206 mistakenly appended, rather than prepended, the edit to
the history.  This caused AssertionErrors when attempting to view the
history of moved messages, which check that the `last_edit_time`
matches the timestamp of the first edit in the list.

Fix the ordering, and update the `edit_history` for messages that were
affected.  We limit to only messages edited since the commit was
merged, since that helps bound the affected messages somewhat.
2024-02-20 21:30:32 -08:00
Anders Kaseorg e568064f4e page_params: Set and expect state_data = null for spectators.
Fix a ZodError when spectators load the app, introduced by commit
a4938d3760 (#28971).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-02-19 03:03:58 -05:00
Anders Kaseorg a4938d3760 page_params: Parse page_params and state_data with Zod.
This establishes a runtime check that their types continue to reflect
reality going forward.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-02-17 00:02:38 -08:00
Alex Vandiver 0079688c49 tornado: Drop WebReloadClientError logic.
The widening of the time between when a process is marked for
reload (at Tornado startup) and when it sends reload events makes it
unlikely-to-impossible that a single `/` request will span both of
them, and thus hit the WebReloadClientError corner case.

Remove it, as it is not worth the complication.  The bad behaviour it
is attempting to prevent (of a reload right after opening `/`) was
always still possible  -- if the `/` request completed right before
Tornado restarted -- so it is not clear that it was ever worth the
complication.
2024-02-15 15:42:50 -08:00
Alex Vandiver fc41d6085b tornado: Split server restart events from web client reload events. 2024-02-15 15:42:50 -08:00
Anders Kaseorg bf6978185d home: Send state_data in a separate object.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-02-15 10:22:52 -08:00
Alex Vandiver 23baabba86 message: Merge unnecessary cache_transformer step.
Having a non-identity `cache_transformer` is no different from running
it on every row of the query_function.  Simplify understanding of the
codepath used in caching by merging the pieces of code.
2024-02-14 12:27:03 -08:00
Alex Vandiver 6597c560cb message_cache: Rename "to_dict" functions which deal with bytes. 2024-02-14 12:27:03 -08:00
Alex Vandiver b3e8878252 message_cache: Rename update_to_dict_cache to update_message_cache.
This better describes what it does.
2024-02-14 12:27:03 -08:00
Alex Vandiver 737a751f5c message: Split MessageDict and friends into its own file. 2024-02-14 12:27:03 -08:00
Alex Vandiver 0f9b7f112b message: Move render_markdown into zerver.lib.markdown. 2024-02-14 12:27:03 -08:00
Alex Vandiver 22837fc1b4 message_edit: Carry the QuerySet through as much as possible.
Rather than pass around a list of message objects in-memory, we
instead keep the same constructed QuerySet which includes the later
propagated messages (if any), and use that same query to pick out
affected Attachment objects, rather than limiting to the set of ids.
This is not necessarily a win -- the list of message-ids *may* be very
long, and thus the query may be more concise, easier to send to
PostgreSQL, and faster for PostgreSQL to parse.  However, the list of
ids is almost certainly better-indexed.

After processing the move, the QuerySet must be re-defined as a search
of ids (and possibly a very long list of such), since there is no
other way which is guaranteed to correctly single out the moved
messages.  At this point, it is mostly equivalent to the list of
Message objects, and certainly takes no less memory.
2024-02-14 12:27:03 -08:00
Alex Vandiver 5c96f94206 topic: Use a single SQL statement to propagate message moves.
Rather than use `bulk_update()` to batch-move chunks of messages, use
a single SQL query to move the messages.  This is much more efficient
for large topic moves.  Since the `edit_history` field is not yet
JSON (see #26496) this requires that PostgreSQL cast the current data
into `jsonb`, append the new data (also cast to `jsonb`), and then
re-cast that as text.

For single-message moves, this _increases_ the SQL query count by one,
since we have to re-query for the updated data from the database after
the bulk update.  However, this is overall still a performance
improvement, which improves to 2x or 3x for larger topic moves.  Below
is a table of duration in seconds to run `do_update_message` to move a
topic to a new stream, based on messages in the topic, for before and
after this change:

| Topic size |  Before  |  After  |
| ---------- | -------- | ------- |
| 1          |   0.1036 |  0.0868 |
| 2          |   0.1108 |  0.0925 |
| 5          |   0.1139 |  0.0959 |
| 10         |   0.1218 |  0.0972 |
| 20         |   0.1310 |  0.1098 |
| 50         |   0.1759 |  0.1366 |
| 100        |   0.2307 |  0.1662 |
| 200        |   0.3880 |  0.2229 |
| 500        |   0.7676 |  0.4052 |
| 1000       |   1.3990 |  0.6848 |
| 2000       |   2.9706 |  1.3370 |
| 5000       |   7.5218 |  3.2882 |
| 10000      |  14.0272 |  5.4434 |
2024-02-14 12:27:03 -08:00
Alex Vandiver 822131fef4 message: Add a bulk_access_stream_messages_query method.
This applies access restrictions in SQL, so that individual messages
do not need to be walked one-by-one.  It only functions for stream
messages.

Use of this method significantly speeds up checks if we moved "all
visible messages" in a topic, since we no longer need to walk every
remaining message in the old topic to determine that at least one was
visible to the user.  Similarly, it significantly speeds up merging
into existing topics, since it no longer must walk every message in
the new topic to determine if the user could see at least one.

Finally, it unlocks the ability to bulk-update only messages the user
has access to, in a single query (see subsequent commit).
2024-02-14 12:27:03 -08:00
Alex Vandiver 628be8d433 streams: Remove a lie from the docstring. 2024-02-14 12:27:03 -08:00
Alex Vandiver 06710631ff tests: Reserve "Internal" client, used by email gateway and topic moves. 2024-02-14 12:27:03 -08:00
Alex Vandiver 7f46773ef1 tests: Clear in-memory Client caches before testing query counts.
This makes counts more apples-to-apples comparable when run
back-to-back.
2024-02-14 12:27:03 -08:00
Anders Kaseorg e79572d0d5 page_params: Remove unused first_in_realm.
It’s unused since commit e1843dd1b9
(#5819).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-02-08 10:08:15 -08:00
Anders Kaseorg b59faf540f page_params: Remove unused prompt_for_invites.
It’s unused since commit ebe959f2b0.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-02-08 10:08:15 -08:00
Alya Abbott e9b0c7f2c0 name_restrictions: Reserve additional subdomains. 2024-02-07 12:10:00 -08:00
Mateusz Mandera 5672595c2a push_notifs: Gracefully handle exception when server cant push.
The problem was that earlier this was just an uncaught JsonableError,
leading to a full traceback getting spammed to the admins.
The prior commit introduced a clear .code for this error on the bouncer
side, meaning the self-hosted server can now detect that and handle it
nicely, by just logging.error about it and also take the opportunity to
adjust the realm.push_notifications_... flags.
2024-02-07 10:36:33 -08:00
Mateusz Mandera 3bda31c48c zilencer: Improve json error when plan doesn't allow push notifs.
This allows the self-hosted server to explicitly test for that exception
and catch and log it nicely.
2024-02-07 10:36:33 -08:00
David Rosa d29cd04387 integrations: Create incoming webhook for GitHub Sponsors.
Creates an incoming webhook integration for Github Sponsors. The
main use case is getting notifications when new sponsors sign up.

Fixes #18320.
2024-02-07 09:52:03 -08:00
Anders Kaseorg 029e765e20 openapi: Validate real requests and responses, not fictional mocks.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-02-05 19:57:21 -05:00
Anders Kaseorg 131b230e2b openapi: Represent OpenAPI parameters with a Parameter class.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-02-05 19:57:21 -05:00
Anders Kaseorg 0dd92d2116 test_classes: Add Content-Type header to empty DELETE/POST bodies.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-02-05 19:57:21 -05:00
Anders Kaseorg a356ec7011 test_classes: Default client_post to application/x-www-form-urlencoded.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-02-05 19:57:21 -05:00
Anders Kaseorg 53e80c41ea ruff: Fix SIM113 Use `enumerate()` for index variable in `for` loop.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-02-02 10:30:45 -08:00
Anders Kaseorg 712917b2c9 ruff: Fix RUF019 Unnecessary key check before dictionary access.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-02-02 10:30:45 -08:00
David Rosa fe0d4db153 help: Improve integrations documentation.
- Renames "Bots and integrations" to "Bots overview" everywhere
  (sidebar, page title, page URL).
- Adds a copy of /api/integrations-overview (symbolic link) as the
  second page in the Bots & integrations section, titled
  "Integrations overview".

Fixes #28758.
2024-02-01 09:45:56 -08:00
David Rosa 1e4f5c6433 integrations: Create incoming webhook for Patreon.
Creates an incoming webhook integration for Patreon. The main
use case is getting notifications when new patrons sign up.

Fixes #18321.

Co-authored-by: Hari Prashant Bhimaraju <haripb01@gmail.com>
Co-authored-by: Sudipto Mondal <sudipto.mondal1997@gmail.com>
2024-01-30 13:13:19 -08:00
Anders Kaseorg 93198a19ed requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-01-29 10:41:54 -08:00
Alya Abbott 10d8d4578e help: Change "All older versions" tab to "All versions".
The instructions actually work on 8.0+ as well, not just older versions.
2024-01-25 18:18:04 -08:00
Alex Vandiver d80b063b61 import: Rewrite "delivered_message" column of scheduled messages.
This also requires shuffling the message import to before the
scheduled messages.

Fixes: #28690.
2024-01-24 13:29:47 -08:00
Alex Vandiver 07c4291749 message: Rewrite personals query to be more performant and accurate.
The previous query suffered from bad corner cases when the user had
received a large number of direct messages but sent very few,
comparatively.  This mean that the first half of the UNION would
retrieve a very large number of UserMessage rows, requiring fetching a
large number of Message rows, merely to throw them away upon
determining that the recipient was the current user.

Instead of merging two queries of "last 1k received" + "last 1k sent",
we instead make better use of the UserMessage rows to find "last 1k
sent or received."  This may change the list of recipients, as large
disparities in sent/received messages may result in pushing the
most-recently-sent users off of the list.  These are likely uncommon
edge cases, however -- and the disparity is the whole reason for the
performance problem.

This also provides more correct answers.  In the case where a user's
1001'th message sent was to person A today, but my most recent message
received was from them yesterday, the previous plan would show the
message I received yesterday message-id as the max, and not the more
recent message I sent today.

While we could theoretically raise the `RECENT_CONVERSATIONS_LIMIT` to
more frequently match the same recipient list as previously, this
increases the cost of the most common cases unreasonably.  With a
1000-message limit, the common cases are slightly faster, and the tail
latencies are very much improved; raising `RECENT_CONVERSATIONS_LIMIT`
would increase the result similarity to the old algorithm, at the cost
of the p50 and p75.

|        |   Old   |   New   |
| ------ | ------- | ------- |
| Mean   | 0.05287 | 0.02520 |
| p50    | 0.00695 | 0.00556 |
| p75    | 0.05592 | 0.03351 |
| p90    | 0.14645 | 0.08026 |
| p95    | 0.20181 | 0.10906 |
| p99    | 0.30691 | 0.16014 |
| p99.9  | 0.57894 | 0.19521 |
| max    | 22.0610 | 0.22184 |

On the whole, however, the much more bounded worst case are worth the
small changes to the resultset.
2024-01-18 09:30:20 -08:00
Mateusz Mandera 80f5963bbc auth: Add a configurable wrapper around authenticate calls. 2024-01-15 12:18:48 -08:00
Prakhar Pratyush b7e56ccbdc lib: Rename *topic local variables to *topic_name.
This is preparatory work towards adding a Topic model.
We plan to use the local variable name as 'topic' for
the Topic model objects.

Currently, we use *topic as the local variable name for
topic names.

We rename local variables of the form *topic to *topic_name
so that we don't need to think about type collisions in
individual code paths where we might want to talk about both
Topic objects and strings for the topic name.
2024-01-15 09:40:43 -08:00
Prakhar Pratyush bc66eaee7d views: Rename *topic local variables to *topic_name.
This is preparatory work towards adding a Topic model.
We plan to use the local variable name as 'topic' for
the Topic model objects.

Currently, we use *topic as the local variable name for
topic names.

We rename local variables of the form *topic to *topic_name
so that we don't need to think about type collisions in
individual code paths where we might want to talk about both
Topic objects and strings for the topic name.
2024-01-15 09:40:43 -08:00
Prakhar Pratyush 1eef052bd1 actions: Rename *topic local variables to *topic_name.
This is preparatory work towards adding a Topic model.
We plan to use the local variable name as 'topic' for
the Topic model objects.

Currently, we use *topic as the local variable name for
topic names.

We rename local variables of the form *topic to *topic_name
so that we don't need to think about type collisions in
individual code paths where we might want to talk about both
Topic objects and strings for the topic name.
2024-01-15 09:40:43 -08:00
Sahil Batra c0c9623ae4 message: Allow system bots to mention group if everyone else can.
We now allow system bots to mention a group if can_mention_group
setting is set to "role:everyone" group and not when it is set
to some other group.
2024-01-10 14:57:21 -08:00
Evgenii 3f06596cf0
dev_ldap_directory: Use f-strings for better readability. 2024-01-09 12:09:09 -08:00
Mateusz Mandera 3ec3ac63f2 zilencer: Have server send realm_uuid to remaining bouncer endpoints.
Requests to these endpoint are about a specified user, and therefore
also have a notion of the RemoteRealm for these requests. Until now
these endpoints weren't getting the realm_uuid value, because it wasn't
used - but now it is needed for updating .last_request_datetime on the
RemoteRealm.
2024-01-05 13:09:09 -08:00
Alex Vandiver 4ab9cd7cf2 markdown: Prevent OverflowError with large time integers.
`<time:1234567890123>` causes a "signed integer is greater than
maximum" exception from dateutil.parser; datetime also cannot handle
it ("year 41091 is out of range") but that is a ValueError which is
already caught.

Catch the OverflowError thrown by dateutil.
2024-01-05 12:01:06 -08:00
Alex Vandiver 75d6f35069 s3: Add a setting for S3 addressing style.
This controls if boto3 attempts to use
`https://bucketname.endpointname/` or `https://endpointname/bucket/`
as its prefix.  See
https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html

Fixes: #28424.
2024-01-05 11:12:18 -08:00
Alex Vandiver 3aea67a8ed s3: Only use get_bucket to get to boto3 clients and resources.
boto3 has two different modalities of making API calls -- through
resources, and through clients.  Resources are a higher-level
abstraction, and thus more generally useful, but some APIs are only
accessible through clients.  It is possible to get to a client object
from a resource, but not vice versa.

Use `get_bucket(...).meta.client` when we need direct access to the
client object for more complex API calls; this lets all of the
configuration for how to access S3 to sit within `get_bucket`.  Client
objects are not bound to only one bucket, but we get to them based on
the bucket we will be interacting with, for clarity.

We removed the cached session object, as it serves no real purpose.
2024-01-05 11:12:18 -08:00
Alex Vandiver 214bd4ed88 s3: Stop caching get_boto_client, which is only ever called once.
e883ab057f started caching the boto client, which we had identified
as slow call.  e883ab057f went further, calling
`get_boto_client().generate_presigned_url()` once and caching that
result.

This makes the inner cache on the client useless.  Remove it.
2024-01-05 11:12:18 -08:00
Alex Vandiver bd38e6cb69 send_email: Distinct emails means distinct, case-insensitively. 2024-01-04 10:46:53 -08:00
Alex Vandiver 8d9ead0f6d send_custom_email: Order by delivery_email if necessary.
If we `.distinct("delivery_email")` then we must also
`.order_by("delivery_email")`; adc987dc43 added the `.order_by`
call, which broke the newsletter codepath, since it did not contain
the `delivery_email` in the ordering fields.

Add a flag to distinct on emails in `send_custom_email`.
2024-01-04 10:46:53 -08:00
Anders Kaseorg c343d7c30e models: Move query_for_ids to zerver.lib.query_helpers.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-12-16 22:08:44 -08:00
Anders Kaseorg 33d140c8dc models: Extract zerver.models.alert_words.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-12-16 22:08:44 -08:00
Anders Kaseorg 1f1b2f9a68 models: Extract zerver.models.bots.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-12-16 22:08:44 -08:00
Anders Kaseorg 27c0b507af models: Extract zerver.models.custom_profile_fields.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-12-16 22:08:44 -08:00
Anders Kaseorg c9c819e1d7 models: Extract zerver.models.scheduled_jobs.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-12-16 22:08:44 -08:00
Anders Kaseorg cff0b78771 models: Move some functions to zerver.lib.attachments.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-12-16 22:08:44 -08:00
Anders Kaseorg b15999c799 models: Extract zerver.models.messages.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-12-16 22:08:44 -08:00
Anders Kaseorg bac027962f models: Extract zerver.models.clients.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-12-16 22:08:44 -08:00
Anders Kaseorg 4aa2d76bea models: Extract zerver.models.streams.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-12-16 22:08:44 -08:00
Anders Kaseorg 51f1dc257d models: Extract zerver.models.recipients.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-12-16 22:08:44 -08:00
Anders Kaseorg 3c11fd9466 models: Move some functions to zerver.lib.display_recipient.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-12-16 22:08:44 -08:00
Anders Kaseorg 33610c8782 models: Extract zerver.models.realm_playgrounds.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-12-16 22:08:44 -08:00
Anders Kaseorg 21ab3858a7 models: Extract zerver.models.linkifiers.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-12-16 22:08:44 -08:00
Anders Kaseorg 67fb485797 models: Extract zerver.models.realm_emoji.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-12-16 22:08:44 -08:00
Anders Kaseorg cd96193768 models: Extract zerver.models.realms.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-12-16 22:08:44 -08:00
Anders Kaseorg 7001a0dfc0 models: Extract zerver.models.groups.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-12-16 22:08:44 -08:00
Anders Kaseorg 45bb8d2580 models: Extract zerver.models.users.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-12-16 22:08:44 -08:00
Anders Kaseorg 37a9c4501f models: Extract zerver.models.constants.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-12-16 22:08:44 -08:00
Anders Kaseorg e601d0ae7c models: Rename zerver/models.py to zerver/models/__init__.py.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-12-16 22:08:44 -08:00
Alex Vandiver f1597afb2b send_email: Send emails to remote servers from BILLING_SUPPORT_EMAIL. 2023-12-15 13:15:27 -08:00
Alex Vandiver 8102519242 send_email: Remove `List-Unsubscribe-Post` from remote-server emails.
For remote servers, we cannot advertise `List-Unsubscribe=One-Click`,
which is specified in RFC 8058[^1] to mean that the `List-Unsubscribe`
URL supports a POST request with no arguments to unsubscribe.  Because
we show an interstitial and confirmation page, as this is not just a
mailing list which is disabled if you click the link, it does not
support the mail system performing the unsubscribe for the user.

Remove the inaccurate header for remote servers.

[^1]: https://datatracker.ietf.org/doc/html/rfc8058
2023-12-15 09:12:25 -08:00
Alex Vandiver 0f2cc383f6 send_email: Support add_context for remote servers.
612f2c73d6 started passing add_context to
`send_custom_server_email`, but did not make it make use of it.

Also add the `hostname` as a built-in value, since that is most likely
the most useful property.
2023-12-15 08:37:02 -08:00
Tim Abbott 63e5712837 push_notifications: Improve logging when not contacting bouncer. 2023-12-15 08:15:36 -08:00
Tim Abbott 53f40133eb remote_server: Handle JsonableErrors and connection failures better.
This fixes the exception case on the initial
`/api/v1/remotes/server/analytics/status` case.  Other exceptions from
`send_to_push_bouncer` are allowed to escape.

Co-authored-by: Alex Vandiver <alexmv@zulip.com>
2023-12-14 15:08:52 -08:00
Sahil Batra 633ec698f5 realm: Enfore length restriction on jitsi_server_url at API level.
Previously, passing a url longer than 200 characters for
jitsi_server_url caused a low-level failure at DB level. This
commit adds this restriction at API level.

Fixes part of #27355.
2023-12-14 12:11:59 -08:00
Anders Kaseorg 77a6f44455 message_send: Add read_by_sender API parameter.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-12-14 08:16:31 -08:00
Anders Kaseorg d893ff5ba8 digest: Exclude bots with sender.is_bot, not sent_by_human.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-12-14 08:16:31 -08:00
Alex Vandiver 2de1a23721 send_custom_email: Provide an unsubscribe link for remote servers. 2023-12-13 22:28:20 -08:00
Alex Vandiver 791d66fe28 send_custom_email: Split out the sending to remote servers. 2023-12-13 22:28:20 -08:00
Alex Vandiver 62e6b10ecd send_email: Pull admins_only back into send_custom_email filtering.
There is no reason for this to be an explicit argument to
send_custom_email, when the rest of the filtering already happens
upstream.
2023-12-13 22:28:20 -08:00
Alex Vandiver 1229e38a0d send_custom_email: Stop lying about the type of options.
Saying `**options: str` is a lie, since it contains bools.  We pluck
out the two bools that we need properly typed because we will be
pushing them into function calls, and type them explicitly as bools.
2023-12-13 22:28:20 -08:00
Sahil Batra f0d429d0f9 remote_server: Handle data of non existent realms.
This commit adds code to make sure that the push
notification does not crash on receiving data for
a non-existent realm.
2023-12-13 13:13:12 -08:00
Sayam Samal c61b353b4b gear_menu_popover: Update the rocket icon used for plan and pricing.
Fixes #28163.
2023-12-13 07:59:13 -08:00
Alex Vandiver 2b37a35f71 queue: Only NAK the events if the channel is still open.
If the exception was because the channel closed, attempting to NAK the
events will just raise another error, and is pointless, as the server
already marked the pending events as NAK'd.
2023-12-12 09:20:29 -08:00
Mateusz Mandera c1988a14a7 zilencer: Return can_push info at the push/notify endpoint.
This provides the remote server this information to refresh it on its
Realm attributes whenever it sends a push notification.

Fixes #27483.
2023-12-12 08:06:12 -08:00
Mateusz Mandera 3bcfb9c005 push_notifs: Order device args to send_notifications_to_bouncer by id.
This ensures determinism in these tests doing mock_send.assert_called
with - avoids producing test flakes due to a different order of
retrieval of these objects from the database.
2023-12-11 18:36:23 -08:00
Mateusz Mandera 2916a601b3 push_notifs: Don't send request to bouncer if no devices found. 2023-12-11 18:36:23 -08:00
Mateusz Mandera b82ea179ac zilencer: Have push/notify endpoint return registrations to delete.
- The server sends the list of registrations it believes to have with
  the bouncer.
- The bouncer includes in the response the registrations that it doesn't
  actually have and therefore the server should delete.
2023-12-11 18:36:23 -08:00
Prakhar Pratyush dd8a33f03e import_realm: Create audit log with user count data.
This commit creates a RealmAuditlog entry with a new event_type
'RealmAuditLog.REALM_IMPORTED' after the realm is reactivated.

It contains user count data (using realm_user_count_by_role)
stored in extra_data.

This helps to have an accurate user count data for the billing
system if someone tries to signup just after doing an import.
2023-12-11 15:03:24 -08:00
Prakhar Pratyush c1daabd3c0 remote_server: Rename to 'send_server_data_to_push_bouncer'.
This commit renames 'send_analytics_to_push_bouncer'
to 'send_server_data_to_push_bouncer'.
2023-12-11 14:07:39 -08:00
Tim Abbott 5c1a5a816f remote_server: Rename register_realm_with_push_bouncer.
We plan to have this potentially happen more than once for a given
realm.
2023-12-11 14:07:39 -08:00
Tim Abbott 4fe02be825 remote_server: Rename maybe_enqueue_audit_log_upload.
This is a rename of the previous
enqueue_register_realm_with_push_bouncer_if_needed but is clearer
about the fact that this will also upload audit logs if available.
2023-12-11 14:07:39 -08:00
Tim Abbott b9af6c7962 remote_server: Queue deferred work after commit. 2023-12-11 14:07:39 -08:00
Prakhar Pratyush d763fae9d0 remote_server: Eliminate separate realms-only code path.
Given that most of the use cases for realms-only code path would
really like to upload audit logs too, and the others would likely
produce a better user experience if they upoaded audit logs, we
should just have a single main code path here i.e.
'send_analytics_to_push_bouncer'.

We still only upload usage statistics according to documented
option, and only from the analytics cron job.

The error handling takes place in 'send_analytics_to_push_bouncer'
itself.
2023-12-11 14:07:39 -08:00
Tim Abbott 1100d3be51 sessions: Make delete_realm_user_sessions not quadratic.
This is exactly the scenario that Set objects exist for.
2023-12-11 14:07:39 -08:00
Aman Agrawal b4e4ca14d5 models: Store `is_system_bot_realm` information for `RemoteRealm`.
This will help us filter out system bot realm and control
feature access to it.
2023-12-11 13:23:49 -08:00
Aman Agrawal ac8d5a5f0b remote_billing_page: Show error page for registration mismatch.
When a self-hosted Zulip server does a data export and then import
process into a different hosting environment (i.e. not sharing the
RemoteZulipServer with the original, we'll have various things that
fail where we look up the RemoteRealm by UUID and find it but the
RemoteZulipServer it is associated with is the wrong one.

Right now, we ask user to contact support via an error page but
might develop UI to help user do the migration directly.
2023-12-10 19:33:48 -08:00
Sahil Batra 198568522a message: Do not include details of inaccessible users in message data.
This commit adds code to not include original details of senders like
name, email and avatar url in the message objects sent through events
and in the response of endpoint used to fetch messages.

This is the last major commit for the project to add support for
limiting guest access to an entire organization.

Fixes #10970.
2023-12-09 17:23:16 -08:00
Sahil Batra 72aa4b256d message: Do not allow guest to mention inaccessible users. 2023-12-09 16:59:38 -08:00
Tim Abbott ac441b77fa users: Add get_inaccessible_user_ids helper. 2023-12-09 16:59:38 -08:00
Anders Kaseorg f86becfc94 remote_server: Send API feature level along with Zulip version.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-12-09 12:01:22 -08:00
Anders Kaseorg 0400614a48 remote_server: Serialize analytics requests with Pydantic.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-12-09 12:01:22 -08:00
David Rosa 91ddcd4e1d help: Tweak stream/group settings instructions for logged in users.
Adds gear menu instructions for how to navigate to "All streams"
and "All groups".
2023-12-08 23:36:24 -08:00
David Rosa 9bb90b7358 help: Update gear menu icons.
- Documents new gear menu icons.
2023-12-08 23:36:24 -08:00
Aman Agrawal c5bfdaaeeb push_notifications: Return after receiving an error. 2023-12-08 12:25:46 -08:00
Sahil Batra 380ff91c0e users: Do not pass unknown users data for webapp.
This commit sets the client capability value to not pass
unknown users data in the webapp and also does some changes
to avoid errors while loading the web-app home page.

This commit only does some basic webapp changes to not show
inaccessible users in sidebar and we would need need more
changes to make the web-app work as expected which will be
done in further commits.
2023-12-07 19:34:07 -08:00
Lauryn Menard a018f2611b push-notifications: Update payload for realm name and event string.
Adds `user.realm.string_id` as the realm name to the base payload
for notifications. Uses this realm name in the body of the alert
in the `apns_data`.

Changes the event string from "test-by-device-token" to "test".

Fixes #28075.
2023-12-07 14:14:37 -08:00
Tim Abbott d29d132e8a push_notifications: Set more reasonable timeouts. 2023-12-07 13:45:57 -08:00
Tim Abbott 19ac558d5f push_notifications: Fix handling of 500s from bouncer.
The comments explain the context, but we shouldn't mark our access to
push notifications as disabled incorrectly here.
2023-12-07 13:45:57 -08:00
Sahil Batra 17d25284a2 help: Update documentation to use new "#groups" UI. 2023-12-07 06:35:38 -08:00
Prakhar Pratyush bbfcb2dcb3 banner: Make banner about automatic follow/unmute topics one-time only.
Fixes: #27847.
2023-12-06 18:19:20 -08:00
Prakhar Pratyush 83bd9955e3 events: Add 'onboarding_steps' event deprecating 'hotspots'.
Earlier, the event sent when an onboarding step (hotspot till now)
is marked as read generated an event with type='hotspots' and
'hotspots' named array in it.

This commit renames the type to 'onboarding_steps' and the array
to 'onboarding_steps' to reflect the fact that it'll also contain
data for elements other than hotspots.
2023-12-06 18:19:20 -08:00
Prakhar Pratyush dde3d72100 onboarding_steps: Add 'OneTimeNotice' dataclass.
This commit adds a 'OneTimeNotice' dataclass to
support one time banner and similar UI elements.
2023-12-06 18:19:20 -08:00
Prakhar Pratyush df379b5e86 hotspots: Add 'type' field to objects in 'hotspots' array.
This commit adds a 'type' field to the objects
in 'hotspots' array sent in 'hotspots' events.

We have explicitly added this field as we eventually
plan to have two type of onboarding steps, 'hotspots'
and 'one_time_notice'.

This will help clients to easily identify them.
2023-12-06 18:19:20 -08:00
Prakhar Pratyush ac8af3d6de urls: Add a new endpoint for hotspot and deprecate the old one.
This commit adds a new endpoint 'users/me/onboarding_steps'
deprecating the older 'users/me/hotspots' to mark hotspot as read.

We also renamed the view `mark_hotspot_as_read` to
`mark_onboarding_step_as_read`.

Reason: Our plan is to make this endpoint flexible to support
other types of UI elements not just restricted to hotspots.
2023-12-06 18:19:20 -08:00
Prakhar Pratyush 62bfc20ebc models: Rename 'UserHotspot' model to 'OnboardingStep'.
This commit renames the 'UserHotspot' model to 'OnboardingStep'.

Also, it renames the 'hotspot' field in that model
to 'onboarding_step'.
2023-12-06 18:19:20 -08:00
Prakhar Pratyush 32a5c422e9 migration: Make 'rename_indexes_constraints' a lib function.
This prep commit moves the 'rename_indexes_constraints'
function to 'lib/migrate' as we're going to re-use it for
the 'UserHotspot' to 'OnboardingStep' table rename operation.

In general, this function would be helpful in migrations
involving table rename operations, subject to the caution
mentioned in the function via comments.
2023-12-06 18:19:20 -08:00
Prakhar Pratyush 777398fc82 test_helpers: Maintain alphabetical order of models in 'use_db_models'. 2023-12-06 18:19:20 -08:00
Alex Vandiver 8d5573b395 narrow: Prevent contradicting DM and stream narrows.
These confused `ok_to_include_history` and caused exceptions looking
for the "flags" column.
2023-12-06 16:06:15 -08:00
Sahil Batra 6c3fbcc383 push_notifications: Do not include details of inaccessible users.
This commit adds code to include original name, email and avatar
for inaccessible users which can happen when a user sends message
to an unsubscribed stream.
2023-12-06 00:09:53 -08:00
Sahil Batra 965869d3f8 register: Add client capability to not receive unknown users data.
This commit adds a new client capability to decide whether the
client needs unknown users data or not.
2023-12-06 00:09:53 -08:00
Sayam Samal bcc58edfda hotspots: Fix misplaced onboarding hotspots.
We rename "intro_gear" to "intro_personal" because after the menu
was split into help menu, main menu and personal menu, the "Settings"
option now resides inside the personal menu.

Fixes #27878.
2023-12-05 17:37:47 -08:00
Anders Kaseorg 223b626256 python: Use urlsplit instead of urlparse.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-12-05 13:03:07 -08:00
Anders Kaseorg 3853fa875a python: Consistently use from…import for urllib.parse.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-12-05 13:03:07 -08:00
Anders Kaseorg 8a7916f21a python: Consistently use from…import for datetime.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-12-05 12:01:18 -08:00
Mateusz Mandera 8b55d60f9e populate_db: Create RemoteZulipServer with proper details from settings.
This creates a valid registration, for two reasons:
1. Avoid the need to run "manage.py register_server" in dev env to
   register, when wanting to to test stuff with
   `PUSH_NOTIFICATION_BOUNCER_URL = "http://localhost:9991"`.
2. Avoid breaking RemoteRealm syncing, due to duplicate registrations
   (first set of registrations that gets set up with the dummy
   RemoteZulipServer in populate_db, and the second that gets set up via
   the regular syncing mechanism with the new RemoteZulipServer created
   during register_server).
2023-12-05 11:34:57 -08:00
Tim Abbott fbd8ed1425 import_realm: Move push bouncer ping after realm reactivation. 2023-12-04 07:08:39 -08:00
Tim Abbott f6c7eaf1e5 models: Add push_notifications_enabled & corresponding end_timestamp.
Add two fields to Realm model:
*push_notifications_enabled
*push_notifications_enabled_end_timestamp

Co-authored-by: Prakhar Pratyush <prakhar@zulip.com>
2023-12-04 07:08:39 -08:00
Prakhar Pratyush 6aa911a9b2 remote_server_post_analytics: Return remote realms data in response.
This is a prep commit to return, for each remote realm, the 'uuid',
'can_push', and 'expected_end_timestamp'.

This data will be used in 'initialize_push_notifications'.
2023-12-04 07:08:39 -08:00
Mateusz Mandera ec7245d4e1 remote_billing: Add redirect flow for users with expired session.
Implements a nice redirect flow to give a good UX for users attempting
to access a remote billing page with an expired RemoteRealm session e.g.
/realm/some-uuid/sponsorship - perhaps through their browser
history or just their session expired while they were doing things in
this billing system.

The logic has a few pieces:
1. get_remote_realm_from_session, if the user doesn't have a
   identity_dict will raise RemoteBillingAuthenticationError.
2. If the user has an identity_dict, but it's expired, then
   get_identity_dict_from_session inside of get_remote_realm_from_session
   will raise RemoteBillingIdentityExpiredError.
3. The decorator authenticated_remote_realm_management_endpoint
   catches that exception and uses some general logic, described in more
   detail in the comments in the code, to figure out the right URL to
   redirect them to. Something like:
   https://theirserver.example.com/self-hosted-billing/?next_page=...
   where the next_page param is determined based on parsing request.path
   to see what kind of endpoint they're trying to access.
4. The remote_server_billing_entry endpoint is tweaked to also send
   its uri scheme to the bouncer, so that the bouncer can know whether
   to do the redirect on http or https.
2023-12-03 10:39:56 -08:00
Mateusz Mandera 7f33d6f0ea zilencer: Tie RemotePushDeviceToken to RemoteRealm at registration.
This consists of the following pieces:
1. Makes servers using the bouncer send realm_uuid in requests for token
   registration. (Sidenote: realm_uuid is already sent in the "send
   notification" codepath as of
   48db4bf854)
2. This allows the bouncer to tie RemotePushDeviceToken to the
   RemoteRealm with matching realm_uuid at registration time.
3. Introduce handling of some potential weird edge cases around the
   realm_uuid and RemoteRealm objects in get_remote_realm_helper.
2023-12-03 09:51:45 -08:00
Mateusz Mandera c9b0602320 tests: Create default RemoteRealms in populate_db.
This default setup will be more realistic, matching the ordinary
conditions for a modern server.
Especially needed as we add bouncer code that will expect to have
RemoteRealm entries for realm_uuid values for which it receives
requests.
2023-12-03 09:51:45 -08:00
Mateusz Mandera a67dd6dc1f realms: Call send_realms_only_to_push_bouncer at realm creation/import. 2023-12-03 08:49:58 -08:00
Anders Kaseorg 1efc6efd23 requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-12-01 18:52:00 -08:00
Alex Vandiver 7f96bed17b stream_traffic: Use the realm_id to get a much better-indexed query.
This reduces the query time by an order of magnitude, since it is able
to switch from a raw `stream_id` index to an index over all of
`realm_id, property, end_time`.
2023-11-30 12:32:30 -08:00
Alya Abbott 7b71ea3314 help: Add tab for Zulip 8.0+ features. 2023-11-29 23:44:00 -08:00
Tim Abbott 610338d192 billing: Add BILLING_SCHEDULE_ prefix to values. 2023-11-29 23:32:56 -08:00
Tim Abbott 5d6b635efe billing: Use better variable names for plan tiers.
The existing values didn't have our standard type-prefixing naming
scheme.

Add some extra unused placeholder values while we're at it.
2023-11-29 23:32:56 -08:00
David Rosa af3956e1a9 help: Document upgrade flow, billing, and sponsorship requests.
Documents how to upgrade to a paid plan, manage billing, and
apply for sponsorship.

Fixes #27946.
2023-11-29 21:50:44 -08:00
Mateusz Mandera 9b1a495e2c zilencer: Sync name and authentication_methods on RemoteRealm. 2023-11-29 15:54:38 -08:00
Tim Abbott 4ef6b7cc44 notifications: Rename apns_enabled and gcm_enabled.
The new names are a lot clearer when thinking about self-hosted
systems that might indirectly use these via a bouncer.
2023-11-29 15:06:46 -08:00
Tim Abbott b71c5746ab notifications: Rename push_notifications_enabled for clarity.
This doesn't actually check if push notifications are working, just
whether there is configuration for them.
2023-11-29 15:06:46 -08:00
Tim Abbott 7db15176f3 push bouncer: Submit basic metadata unconditionally.
These metadata are essentially all publicily available anyway, and
making uploading them unconditional will simplify some things.

The documentation is not quite accurate in that it claims the server
will upload some metadata that is not actually uploaded yet (but will
by soon). This seems harmless.
2023-11-29 14:45:53 -08:00
Prakhar Pratyush d8cf12eaaa send_email: Improve configurability for outgoing email sender name.
Currently, the sender names for outgoing emails sent by Zulip
are hardcoded. It should be configurable for self-hosted systems.

This commit makes the 'Zulip' part a variable in the following
email sender names: 'Zulip Account Security', 'Zulip Digest',
and 'Zulip Notifications' by introducing a settings variable
'SERVICE_NAME' with the default value as f"{EXTERNAL_HOST} Zulip".

Fixes: #23857
2023-11-29 14:20:01 -08:00
Prakhar Pratyush 78a75ab28c digest: Fix the `_` used for an unused variable.
This prep commit replaces `_` with `ignored` to represent
an unused variable.

In later commits, we are going to use `_` for translation,
which leads to a lint error.
2023-11-29 14:20:01 -08:00
Sahil Batra a6fa875c23 models: Refactor get_fake_email_domain to take realm.host as arg.
This commit updates get_fake_email_domain to accept realm.host as
argument instead of the Realm object since we only use realm.host
to get the fake email domain.

This is a preparatory commit for the limited guest feature as we
would be sending the fake email of the message sender in message
event object to a guest user who cannot access the sender and
there we would need to compute the fake email.
2023-11-29 12:01:37 -08:00
Alex Vandiver 82c08dd153 python: Prevent bare timedelta(), which defaults to days. 2023-11-28 15:10:39 -08:00
Alex Vandiver 8c8dbb3d66 markdown: Stop attempting to expand/collapse re2 regex.
549dd8a4c4 changed the regex that we build to contain whitespace for
readability, and strip that back out before returning it.
Unfortunately, this also serves to strip out whitespace in the source
linkifier, causing it to not match expected strings.

Revert 549dd8a4c4.

Fixes: #27854.
2023-11-28 15:07:23 -08:00
Mateusz Mandera 02d5740f0f remote_realm: Add syncing of org_type. 2023-11-28 14:41:16 -08:00
roanster007 8bd92fe801 bots: Fix muting of cross realm bots.
Previously, we weren't able to mute the cross realm bots. This was
because, for muting the users, we access only those profiles which are
in realm, excluding the cross realm system bots.

This is fixed by replacing the access_user_by_id method with a new
method access_user_by_id_including_cross_realm for this specific test.

Fixes #27823
2023-11-27 16:16:23 -08:00
Aman Agrawal a96619639c events_register: Set home view for spectators as recent view. 2023-11-27 09:41:52 -08:00