Commit Graph

8119 Commits

Author SHA1 Message Date
Anders Kaseorg 2e4785f51c requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 93198a19ed)
2024-02-16 12:17:34 -08:00
Mateusz Mandera d977dfe6e9 push_notifs: Gracefully handle exception when server cant push.
The problem was that earlier this was just an uncaught JsonableError,
leading to a full traceback getting spammed to the admins.
The prior commit introduced a clear .code for this error on the bouncer
side, meaning the self-hosted server can now detect that and handle it
nicely, by just logging.error about it and also take the opportunity to
adjust the realm.push_notifications_... flags.

(cherry picked from commit e8018a7285)
2024-02-16 11:26:05 -08:00
Mateusz Mandera 0a1905ea8d zilencer: Improve json error when plan doesn't allow push notifs.
This allows the self-hosted server to explicitly test for that exception
and catch and log it nicely.

(cherry picked from commit 5b03932d5c)
2024-02-16 11:26:05 -08:00
Alex Vandiver fcef73fbbf import: Rewrite "delivered_message" column of scheduled messages.
This also requires shuffling the message import to before the
scheduled messages.

Fixes: #28690.
(cherry picked from commit c974b421e3)
2024-01-24 16:38:46 -08:00
Mateusz Mandera 4df748d151 auth: Add a configurable wrapper around authenticate calls.
(cherry picked from commit 3c29a68cd1)
2024-01-18 15:58:02 -08:00
Mateusz Mandera 0cbbb98cc2 zilencer: Have server send realm_uuid to remaining bouncer endpoints.
Requests to these endpoint are about a specified user, and therefore
also have a notion of the RemoteRealm for these requests. Until now
these endpoints weren't getting the realm_uuid value, because it wasn't
used - but now it is needed for updating .last_request_datetime on the
RemoteRealm.

(cherry picked from commit e2291b7656)
2024-01-15 16:51:09 -08:00
Sahil Batra 363d98a203 message: Allow system bots to mention group if everyone else can.
We now allow system bots to mention a group if can_mention_group
setting is set to "role:everyone" group and not when it is set
to some other group.

(cherry picked from commit 76b2e6de0b)
2024-01-15 12:02:53 -08:00
Alex Vandiver da49f1bac1 s3: Add a setting for S3 addressing style.
This controls if boto3 attempts to use
`https://bucketname.endpointname/` or `https://endpointname/bucket/`
as its prefix.  See
https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html

Fixes: #28424.
(cherry picked from commit c1a26380a9)
2024-01-15 12:02:53 -08:00
Alex Vandiver 5192ae0fd2 s3: Only use get_bucket to get to boto3 clients and resources.
boto3 has two different modalities of making API calls -- through
resources, and through clients.  Resources are a higher-level
abstraction, and thus more generally useful, but some APIs are only
accessible through clients.  It is possible to get to a client object
from a resource, but not vice versa.

Use `get_bucket(...).meta.client` when we need direct access to the
client object for more complex API calls; this lets all of the
configuration for how to access S3 to sit within `get_bucket`.  Client
objects are not bound to only one bucket, but we get to them based on
the bucket we will be interacting with, for clarity.

We removed the cached session object, as it serves no real purpose.

(cherry picked from commit 617c35398a)
2024-01-15 12:02:53 -08:00
Alex Vandiver c5394d9ca0 s3: Stop caching get_boto_client, which is only ever called once.
e883ab057f started caching the boto client, which we had identified
as slow call.  e883ab057f went further, calling
`get_boto_client().generate_presigned_url()` once and caching that
result.

This makes the inner cache on the client useless.  Remove it.

(cherry picked from commit f8c432bd53)
2024-01-15 12:02:53 -08:00
Anders Kaseorg cf14f27602 models: Move query_for_ids to zerver.lib.query_helpers.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-01-05 10:32:54 -05:00
Anders Kaseorg 25592eba22 models: Extract zerver.models.alert_words.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-01-05 10:32:54 -05:00
Anders Kaseorg 7781591131 models: Extract zerver.models.bots.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-01-05 10:32:54 -05:00
Anders Kaseorg df111bc35a models: Extract zerver.models.custom_profile_fields.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-01-05 10:32:54 -05:00
Anders Kaseorg d07d773809 models: Extract zerver.models.scheduled_jobs.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-01-05 10:32:54 -05:00
Anders Kaseorg 1fbdcf3fdd models: Move some functions to zerver.lib.attachments.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-01-05 10:32:54 -05:00
Anders Kaseorg 488fec7300 models: Extract zerver.models.messages.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-01-05 10:32:54 -05:00
Anders Kaseorg 26811c5049 models: Extract zerver.models.clients.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-01-05 10:32:54 -05:00
Anders Kaseorg 776a0eeae8 models: Extract zerver.models.streams.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-01-05 10:32:54 -05:00
Anders Kaseorg 56a09f426b models: Extract zerver.models.recipients.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-01-05 10:32:54 -05:00
Anders Kaseorg 0c8d6dab65 models: Move some functions to zerver.lib.display_recipient.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-01-05 10:32:54 -05:00
Anders Kaseorg f77648f279 models: Extract zerver.models.realm_playgrounds.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-01-05 10:32:54 -05:00
Anders Kaseorg 968b7e6c1d models: Extract zerver.models.linkifiers.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-01-05 10:32:54 -05:00
Anders Kaseorg 8855d8b3bc models: Extract zerver.models.realm_emoji.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-01-05 10:32:54 -05:00
Anders Kaseorg 086df4a81e models: Extract zerver.models.realms.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-01-05 10:32:54 -05:00
Anders Kaseorg a8b5a99281 models: Extract zerver.models.groups.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-01-05 10:32:54 -05:00
Anders Kaseorg ee85ac5433 models: Extract zerver.models.users.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-01-05 10:32:54 -05:00
Anders Kaseorg c1c9db59cd models: Extract zerver.models.constants.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-01-05 10:32:54 -05:00
Anders Kaseorg f78d8e13c1 models: Rename zerver/models.py to zerver/models/__init__.py.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-01-05 10:32:54 -05:00
Alex Vandiver 8102519242 send_email: Remove `List-Unsubscribe-Post` from remote-server emails.
For remote servers, we cannot advertise `List-Unsubscribe=One-Click`,
which is specified in RFC 8058[^1] to mean that the `List-Unsubscribe`
URL supports a POST request with no arguments to unsubscribe.  Because
we show an interstitial and confirmation page, as this is not just a
mailing list which is disabled if you click the link, it does not
support the mail system performing the unsubscribe for the user.

Remove the inaccurate header for remote servers.

[^1]: https://datatracker.ietf.org/doc/html/rfc8058
2023-12-15 09:12:25 -08:00
Alex Vandiver 0f2cc383f6 send_email: Support add_context for remote servers.
612f2c73d6 started passing add_context to
`send_custom_server_email`, but did not make it make use of it.

Also add the `hostname` as a built-in value, since that is most likely
the most useful property.
2023-12-15 08:37:02 -08:00
Tim Abbott 63e5712837 push_notifications: Improve logging when not contacting bouncer. 2023-12-15 08:15:36 -08:00
Tim Abbott 53f40133eb remote_server: Handle JsonableErrors and connection failures better.
This fixes the exception case on the initial
`/api/v1/remotes/server/analytics/status` case.  Other exceptions from
`send_to_push_bouncer` are allowed to escape.

Co-authored-by: Alex Vandiver <alexmv@zulip.com>
2023-12-14 15:08:52 -08:00
Sahil Batra 633ec698f5 realm: Enfore length restriction on jitsi_server_url at API level.
Previously, passing a url longer than 200 characters for
jitsi_server_url caused a low-level failure at DB level. This
commit adds this restriction at API level.

Fixes part of #27355.
2023-12-14 12:11:59 -08:00
Anders Kaseorg 77a6f44455 message_send: Add read_by_sender API parameter.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-12-14 08:16:31 -08:00
Anders Kaseorg d893ff5ba8 digest: Exclude bots with sender.is_bot, not sent_by_human.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-12-14 08:16:31 -08:00
Alex Vandiver 2de1a23721 send_custom_email: Provide an unsubscribe link for remote servers. 2023-12-13 22:28:20 -08:00
Alex Vandiver 791d66fe28 send_custom_email: Split out the sending to remote servers. 2023-12-13 22:28:20 -08:00
Alex Vandiver 62e6b10ecd send_email: Pull admins_only back into send_custom_email filtering.
There is no reason for this to be an explicit argument to
send_custom_email, when the rest of the filtering already happens
upstream.
2023-12-13 22:28:20 -08:00
Alex Vandiver 1229e38a0d send_custom_email: Stop lying about the type of options.
Saying `**options: str` is a lie, since it contains bools.  We pluck
out the two bools that we need properly typed because we will be
pushing them into function calls, and type them explicitly as bools.
2023-12-13 22:28:20 -08:00
Sahil Batra f0d429d0f9 remote_server: Handle data of non existent realms.
This commit adds code to make sure that the push
notification does not crash on receiving data for
a non-existent realm.
2023-12-13 13:13:12 -08:00
Sayam Samal c61b353b4b gear_menu_popover: Update the rocket icon used for plan and pricing.
Fixes #28163.
2023-12-13 07:59:13 -08:00
Alex Vandiver 2b37a35f71 queue: Only NAK the events if the channel is still open.
If the exception was because the channel closed, attempting to NAK the
events will just raise another error, and is pointless, as the server
already marked the pending events as NAK'd.
2023-12-12 09:20:29 -08:00
Mateusz Mandera c1988a14a7 zilencer: Return can_push info at the push/notify endpoint.
This provides the remote server this information to refresh it on its
Realm attributes whenever it sends a push notification.

Fixes #27483.
2023-12-12 08:06:12 -08:00
Mateusz Mandera 3bcfb9c005 push_notifs: Order device args to send_notifications_to_bouncer by id.
This ensures determinism in these tests doing mock_send.assert_called
with - avoids producing test flakes due to a different order of
retrieval of these objects from the database.
2023-12-11 18:36:23 -08:00
Mateusz Mandera 2916a601b3 push_notifs: Don't send request to bouncer if no devices found. 2023-12-11 18:36:23 -08:00
Mateusz Mandera b82ea179ac zilencer: Have push/notify endpoint return registrations to delete.
- The server sends the list of registrations it believes to have with
  the bouncer.
- The bouncer includes in the response the registrations that it doesn't
  actually have and therefore the server should delete.
2023-12-11 18:36:23 -08:00
Prakhar Pratyush dd8a33f03e import_realm: Create audit log with user count data.
This commit creates a RealmAuditlog entry with a new event_type
'RealmAuditLog.REALM_IMPORTED' after the realm is reactivated.

It contains user count data (using realm_user_count_by_role)
stored in extra_data.

This helps to have an accurate user count data for the billing
system if someone tries to signup just after doing an import.
2023-12-11 15:03:24 -08:00
Prakhar Pratyush c1daabd3c0 remote_server: Rename to 'send_server_data_to_push_bouncer'.
This commit renames 'send_analytics_to_push_bouncer'
to 'send_server_data_to_push_bouncer'.
2023-12-11 14:07:39 -08:00
Tim Abbott 5c1a5a816f remote_server: Rename register_realm_with_push_bouncer.
We plan to have this potentially happen more than once for a given
realm.
2023-12-11 14:07:39 -08:00