Commit Graph

8659 Commits

Author SHA1 Message Date
Alex Vandiver e125ad823d exports: Add a separate bucket for realm exports.
This allows finer-grained access control and auditing.  The links
generated also expire after one week, and the suggested configuration
is that the underlying data does as well.

Co-authored-by: Prakhar Pratyush <prakhar@zulip.com>
2024-09-20 15:43:49 -07:00
Alex Vandiver c1e8ecd08f uploads: Cache boto client in the module and be writable.
The `get_signed_upload_url` code is called for every S3 file serve
request, and is thus in the hot path.  The boto3 client caching
optimization is thus potentially useful as a performance optimization.
2024-09-20 15:43:49 -07:00
Alex Vandiver 1a7b3ef7ed upload: Use get_export_tarball_url in upload_export_tarball. 2024-09-20 15:43:49 -07:00
Alex Vandiver 4cf835d9dd upload: Remove common cache from get_export_tarball_url.
This is not called in the hot path like get_avatar_url is.
2024-09-20 15:43:49 -07:00
Alex Vandiver a5bf452202 upload: Realm is not Optional in upload_export_tarball.
af4eb8c0d5 marked the base class and local backend as non-Optional,
but left the S3 backend as Optional for some reason.

Remove it.
2024-09-20 15:43:49 -07:00
Alex Vandiver 91ac5c3c8b export: Log before the compression step, which can be slow. 2024-09-20 15:43:49 -07:00
Tim Abbott 51d0dfb504 docs: Tweak some documentation around send_event. 2024-09-20 15:28:18 -07:00
Prakhar Pratyush e763d065a3 django_api: Rename 'send_event' to 'send_event_rollback_unsafe'.
This commit renames the 'send_event' function to
'send_event_rollback_unsafe' to reflect the fact that it doesn't
wait for the db transaction (within which it gets called, if any)
to commit and sends event irrespective of commit or rollback.

In most of the cases we don't want to send event in the case of
rollbacks, so the caller should be aware that calling the function
directly is rollback unsafe.
2024-09-20 15:20:18 -07:00
Mateusz Mandera 18fbb5d146 retention: Limit number of ids passed to db in delete messages query.
If do_delete_messages (and friends) are called for a massive number of
messages, the giant list of message ids is passed to Postgres even
though chunk_size makes all but the first chunk_size of message ids
useless.
2024-09-20 09:31:21 -07:00
Mateusz Mandera ed7c330548 retention: Rename run_archiving_in_chunks to run_archiving. 2024-09-20 09:31:21 -07:00
Sahil Batra c53563d0e7 user_groups: Handle deactivated groups in webapp. 2024-09-18 13:41:13 -07:00
Sahil Batra 5c790aac72 user_groups: Rename allow_deactivated in 'GET /user_groups'.
This commit renames "allow_deactivated" parameter in
"GET /user_groups" endpoint to "include_deactivated_groups", so
that we can have consistent naming here and for client capability
used for deciding whether to send deactivated groups in register
response and how to handle the related events.
2024-09-18 13:41:13 -07:00
Sahil Batra fb63c47ea6 user_groups: Add client capability to handle deactivated groups.
This commit adds a client capability to not receive data about
deactivated groups.
2024-09-18 13:41:13 -07:00
Sahil Batra 7a6135371e settings: Handle guests separately for group-based settings.
This commit adds code to handle guests separately for group
based settings, where guest will only have permission if
that particular setting can be set to "role:everyone" group
even if the guest user is part of the group which is used
for that setting. This is to make sure that guests do not
get permissions for actions that we generally do not want
guests to have.

Currently the guests do not have permission for most of them
except for "Who can delete any message", where guest could
delete a message if the setting was set to a user defined
group with guest being its member. But this commit still
update the code to use the new function for all the settings
as we want to have a consistent pattern of how to check whether
a user has permission for group-based settings.
2024-09-18 11:51:11 -07:00
Alex Vandiver 9a1f78db22 thumbnail: Support checking for images from streaming sources.
We may not always have trivial access to all of the bytes of the
uploaded file -- for instance, if the file was uploaded previously, or
by some other process.  Downloading the entire image in order to check
its headers is an inefficient use of time and bandwidth.

Adjust `maybe_thumbnail` and dependencies to potentially take a
`pyvips.Source` which supports streaming data from S3 or disk.  This
allows making the ImageAttachment row, if deemed appropriate, based on
only a few KB of data, and not the entire image.
2024-09-17 12:51:30 -07:00
Alex Vandiver ef21dd9b99 thumbnail: Set a stable ordering on ImageAttachment rows for locking.
Failure to have a stable ordering can lead to deadlocks.
2024-09-17 09:14:52 -07:00
Johan Nilsson bf9cbe30a7 integrations: Add support for release events to GitLab integration. 2024-09-16 09:26:20 -07:00
sujal shah 614caf111e user_groups: Add `creator` and `date_created` field in user groups.
This commit introduced 'creator' and 'date_created'
fields in user groups, allowing users to view who
created the groups and when.

Both fields can be null for groups without creator data.
2024-09-13 18:44:58 -07:00
Vector73 0cd51f2fea settings: Remove `delete_own_message_policy` realm property.
Removed `delete_own_message_policy` realm property as the permission
to delete own messages is controlled by `can_delete_own_message_group`
setting.
2024-09-12 09:36:02 -07:00
Vector73 28c7a04734 settings: Add `can_delete_own_message_group` realm setting.
Added `can_delete_message_group` realm setting to replace
`delete_own_message_policy` property.
2024-09-12 09:36:02 -07:00
Sahil Batra 688c5ad0af groups: Allow excluding deactivated groups in 'GET /user_groups' response. 2024-09-11 09:43:56 -07:00
Sahil Batra fef2925ff0 mention: Handle mentioning deactivated user groups.
If someone tries to mention a deactivated user group,
then it is automatically converted to a silent mention
like we do for deactivated users.
2024-09-11 09:43:56 -07:00
Sahil Batra c1d6da0a5c user_groups: Update handling of deactivated groups.
We only allow updating name of a deactivated group, and not
allow updating description, members, subgroups and any setting
of a deactivated user group.

Deactivated user groups cannot be a a subgroup of any group
or used as a setting for a group.
2024-09-11 09:43:56 -07:00
Sahil Batra e1cfe61452 user_groups: Add API support for deactivating user groups. 2024-09-11 09:43:56 -07:00
Sahil Batra bef7cfe00f user_groups: Add deactivated field to NamedUserGroup model. 2024-09-11 09:43:56 -07:00
Sahil Batra f0c0eef8dc groups: Lock user group rows while using them for settings.
This is important to make sure that we handle cases when there
are two parallel requests - one for using a group for a setting
and one for deactivating the same group. This makes sure that
atleast one of the above task fails.
2024-09-11 09:43:56 -07:00
Mateusz Mandera a36f906d1a presence: Add history_limit_days param to the API.
This param allows clients to specify how much presence history they want
to fetch. Previously, the server always returned 14 days of history.
With the recent migration of the presence API to the much more efficient
system relying on incremental fetches via the last_update_id param added
in #29999, we can now afford to provide much more history to clients
that request it - as all that historical data will only be fetched once.

There are three endpoints involved:
- `/register` - this is the main useful endpoint for this, used by API
clients to fetch initial data and register an events queue. Clients can
pass the `presence_history_limit_days` param here.
- `/users/me/presence` - this endpoint is currently used by clients to
update their presence status and fetch incremental data, making the new
functionality not particularly useful here. However, we still add the
new `history_limit_days` param here, in case in the future clients
transition to using this also for the initial presence data fetch.
- `/` - used when opening the webapp. Naturally, params aren't passed
here, so the server just assumes a value from
`settings.PRESENCE_HISTORY_LIMIT_DAYS_FOR_WEB_APP` and returns
information about this default value in page_params.
2024-09-10 13:15:35 -07:00
Prakhar Pratyush e8349aa5d7 custom_email: Add manage_preferences block to the plaintext version.
Earlier, the content of the "manage_preferences" block that includes
the unsubscribe_link, personal settings link, etc was missing in the
plaintext version of the custom emails.

This commit updates the logic to include the manage_preferences block
content in the plaintext version.
2024-09-10 09:36:56 -07:00
Alex Vandiver 8bacdbc895 thumbnail: Put the original dimensions on spinner images.
This lets us reserve the right amount of space in the message feed
immediately.
2024-09-09 15:59:02 -07:00
Alex Vandiver 903bfb31e6 upload: Provide the frontend with the less-modified filename. 2024-09-09 12:40:17 -07:00
Alex Vandiver b4764f49df upload: Download files with their original names.
Fixes: #29491.
2024-09-09 12:40:17 -07:00
Alex Vandiver 933e3cb375 upload: Explicitly return a bool and the Attachment object. 2024-09-09 12:40:17 -07:00
Alex Vandiver 0b0a7fa251 attachments: Tighten type of validate_attachment_request_for_spectator_access. 2024-09-09 12:40:17 -07:00
Alex Vandiver ca72e756eb upload: Rename "upload_image_to_s3"; it is not only for images. 2024-09-09 12:40:17 -07:00
Lauryn Menard d431a5aad6 audit-log: Move user group event types to AuditLogEventType enum.
Event types moved: USER_GROUP_CREATED, USER_GROUP_DELETED
USER_GROUP_DIRECT_USER_MEMBERSHIP_ADDED,
USER_GROUP_DIRECT_USER_MEMBERSHIP_REMOVED,
USER_GROUP_DIRECT_SUBGROUP_MEMBERSHIP_ADDED,
USER_GROUP_DIRECT_SUBGROUP_MEMBERSHIP_REMOVED,
USER_GROUP_DIRECT_SUPERGROUP_MEMBERSHIP_ADDED,
USER_GROUP_DIRECT_SUPERGROUP_MEMBERSHIP_REMOVED,
USER_GROUP_NAME_CHANGED, USER_GROUP_DESCRIPTION_CHANGED,
USER_GROUP_GROUP_BASED_SETTING_CHANGED
2024-09-09 11:50:13 -07:00
Lauryn Menard df1e9093a9 audit-log: Move stream/channel event types to AuditLogEventType enum.
Renamed event types below in the enum class to use channel instead of
stream.

Event types moved: STREAM_CREATED, STREAM_DEACTIVATED, STREAM_NAME_CHANGED
STREAM_REACTIVATED, STREAM_MESSAGE_RETENTION_DAYS_CHANGED
STREAM_PROPERTY_CHANGED, STREAM_GROUP_BASED_SETTING_CHANGED
2024-09-09 11:50:13 -07:00
Lauryn Menard 10d161638e audit-log: Move subscription event types to AuditLogEventType enum.
Event types moved: SUBSCRIPTION_CREATED, SUBSCRIPTION_ACTIVATED,
SUBSCRIPTION_DEACTIVATED, SUBSCRIPTION_PROPERTY_CHANGED.
2024-09-09 11:50:13 -07:00
Lauryn Menard 56c8cbde1e audit-log: Move realm event types to AuditLogEventType enum.
Event types moved: REALM_CREATED, REALM_DEFAULT_USER_SETTINGS_CHANGED
REALM_ORG_TYPE_CHANGED, REALM_DOMAIN_ADDED, REALM_DOMAIN_CHANGED
REALM_DOMAIN_REMOVED, REALM_PLAYGROUND_ADDED, REALM_PLAYGROUND_REMOVED
REALM_LINKIFIER_ADDED, REALM_LINKIFIER_CHANGED, REALM_LINKIFIER_REMOVED
REALM_EMOJI_ADDED, REALM_EMOJI_REMOVED, REALM_LINKIFIERS_REORDERED
REALM_IMPORTED
2024-09-09 11:50:13 -07:00
Lauryn Menard d2c32f23db audit-log: Move realm event types to AuditLogEventType enum.
Event types moved: REALM_DEACTIVATED, REALM_REACTIVATED, REALM_SCRUBBED
REALM_PLAN_TYPE_CHANGED, REALM_LOGO_CHANGED, REALM_EXPORTED
REALM_PROPERTY_CHANGED, REALM_ICON_SOURCE_CHANGED, REALM_DISCOUNT_CHANGED
REALM_SPONSORSHIP_APPROVED, REALM_BILLING_MODALITY_CHANGED
REALM_REACTIVATION_EMAIL_SENT, REALM_SPONSORSHIP_PENDING_STATUS_CHANGED
REALM_SUBDOMAIN_CHANGED
2024-09-09 11:50:13 -07:00
Lauryn Menard caeeaf3c3f audit-log: Move user event types to AuditLogEventType enum.
Event types moved: USER_SOFT_ACTIVATED, USER_SOFT_DEACTIVATED
USER_PASSWORD_CHANGED, USER_AVATAR_SOURCE_CHANGED, USER_FULL_NAME_CHANGED
USER_EMAIL_CHANGED, USER_TERMS_OF_SERVICE_VERSION_CHANGED
USER_API_KEY_CHANGED, USER_BOT_OWNER_CHANGED,
USER_DEFAULT_SENDING_STREAM_CHANGED, USER_DEFAULT_REGISTER_STREAM_CHANGED
USER_DEFAULT_ALL_PUBLIC_STREAMS_CHANGED, USER_SETTING_CHANGED
USER_DIGEST_EMAIL_CREATED
2024-09-09 11:50:13 -07:00
Lauryn Menard e5daa3470f audit-log: Move user event types to AuditLogEventType enum.
Event types moved: USER_CREATED, USER_ACTIVATED, USER_DEACTIVATED
USER_REACTIVATED, USER_ROLE_CHANGED, USER_DELETED
USER_DELETED_PRESERVING_MESSAGES
2024-09-09 11:50:13 -07:00
joseph b0a20d2cae attachments: Return a list of removed attachments while editing.
Currently, we want to ask users if they would like to delete their
attachments after they have removed the attachments while editing. These
changes are preparatory changes on the backend to return a list of removed
attachments after the user has removed attachments while editing.

Fixes part of #25525.
2024-09-06 11:40:51 -07:00
Kenneth Rodrigues 1d580f143e endpoints: Modify comments mentioning has_request_variables and REQ. 2024-09-05 16:02:12 -07:00
Kenneth Rodrigues dc32396180 endpoints: Remove the has_request_variables decorator.
All endpoints have been migrated to the typed_endpoint decorator,
therefore the has_request_variables decorator and the REQ function are
no longer needed and have been removed.
2024-09-05 16:02:12 -07:00
Anders Kaseorg e3abd09e67 thumbnail: Fix corrupted email notifications due to HTML5 entities.
BeautifulSoup with formatter="html5" unnecessarily escapes many
characters with HTML5-specific entities that cannot be correctly
parsed by lxml during generation of email notifications.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-09-05 16:00:45 -07:00
Alex Vandiver cec0942b4b users: Allow zero-width-joiners in user names.
Partially fixes: #20128.
2024-09-03 16:34:55 -07:00
Anders Kaseorg 91ade25ba3 python: Simplify with str.removeprefix, str.removesuffix.
These are available in Python ≥ 3.9.
https://docs.python.org/3/library/stdtypes.html#str.removeprefix

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-09-03 12:30:16 -07:00
Alya Abbott d7761f7163 help: Add a guide on moving to Zulip. 2024-09-01 16:16:04 -07:00
Alya Abbott 7df0db142b help: Document how to invite users in imported organizations. 2024-09-01 16:16:04 -07:00
bedo 3da91e951c auth: Migrate to @typed_endpoint.
Since this is the last has_request_variables endpoint outside tests,
more test_openapi code needs to be deleted in this transition.
2024-08-30 16:56:14 -07:00