Commit Graph

19126 Commits

Author SHA1 Message Date
Alex Vandiver 579cf4ada7 upload: Make local-file save_attachment_contents chunk-at-a-time.
This means it does not attempt to hold large files entirely in memory
when writing them to a new location on disk.
2024-09-26 12:01:11 -07:00
Alex Vandiver 2dc737335e upload: Switch from BinaryIO to IO[bytes].
This is slightly more generally-compatible.
2024-09-26 12:01:11 -07:00
Alex Vandiver 638c579c56 tusd: Set metadata correctly in S3.
The Content-Type, Content-Disposition, StorageClass, and general
metadata are not set according to our patterns by tusd; copy the file
to itself to update those properties.
2024-09-26 12:00:43 -07:00
Alex Vandiver 287850d08d tusd: Remove non-ASCII characters from path-ids. 2024-09-26 12:00:43 -07:00
Alex Vandiver 84280ed7c2 upload: When serving s3 download URLs, send real filename.
Setting `ResponseContentDisposition=attachment` means that we override
the stored `ContentDisposition`, which includes a filename.  This
means that using the "Download" link on servers with S3 storage
produced a file named the sanitized version we stored.

Explicitly build a `ContentDisposition` to tell S3 to return, which
includes both `attachment` as well as the filename (if we have it
locally).
2024-09-26 12:00:43 -07:00
Tim Abbott 9b67164270 email_notifications: Fix emoji being giant in Outlook.
Apparently, Outlook ignores height/width CSS rules, but does support
the attribute on the image element itself, so specify that instead.

I don't think there are likely to be image tag implementations that
don't support the attribute, given that's the only thing that works in
Outlook.
2024-09-26 11:58:12 -07:00
Anders Kaseorg a9fb8dccae requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-09-24 19:27:21 -07:00
Anders Kaseorg f0f048de69 corporate: Import corporate.lib.stripe lazily.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-09-24 18:18:26 -07:00
Anders Kaseorg 88782f2917 integrations: Lazily load webhook integrations.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-09-24 18:17:52 -07:00
Anders Kaseorg 6c442273ee test_urls: Remove dead URLResolutionTest.
This test was written back when Django accepted view function names as
strings that might be wrong; that’s not possible in Django ≥ 1.10.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-09-24 18:17:52 -07:00
Anders Kaseorg 184c0203f3 upload: Lazily import boto3.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-09-24 16:38:37 -07:00
Vector73 9e4e85e140 saved_snippets: Add backend for saved snippets.
Part of #31227.
2024-09-24 15:27:58 -07:00
Tim Abbott 90a4b4934a text_fixtures: Fix buggy skip-checks placement. 2024-09-24 15:00:46 -07:00
Prakhar Pratyush 65f465562f export_realm: Remove the 'react on consent message' approach.
For exporting full with consent:

* Earlier, a message advertising users to react with thumbs up
  was sent and later used to determine the users who consented.

* Now, we no longer need to send such a message. This commit
  updates the logic to use `allow_private_data_export` user-setting
  to determine users who consented.

Fixes part of #31201.
2024-09-24 14:32:42 -07:00
Prakhar Pratyush f67d7549de message: Add test coverage for 'set_visibility_policy_possible'.
This commit adds test coverage for the
'if user_profile.realm != message.get_realm():' block in
'set_visibility_policy_possible' function.

Currently, 'test_command_with_consented_message_id' adds the
test coverage for this code block. But we plan to remove that
test in the next commit as a part of removing 'react on consent
message' approach.

So, this commit explicitly adds code for the test coverage of
the concerned code block.
2024-09-24 14:32:42 -07:00
Tim Abbott 9d74abee0b tools: Skip checks in recursive management calls. 2024-09-24 14:26:34 -07:00
Aditya a534e9fdb6 integrations: Remove ignore_pull_requests from Travis CI webhook.
Removes the ignore_pull_requests URL parameter from the Travis CI
integration because its functionality is now fully covered by the
standard incoming webhook event filtering framework.

Fixes #30934.
2024-09-24 11:04:49 -07:00
tnmkr ddecba4e1c custom_profile_fields: Add "editable_by_user" setting.
This new property allows organization administrators to specify whether
users can modify the custom profile field value on their own account.
This property is configurable for individual fields.

By default, existing and newly created fields have this property set to
true, that is, they allow users to edit the value of the fields.

Fixes part of #22883.

Co-Authored-By: Ujjawal Modi <umodi2003@gmail.com>
2024-09-23 18:09:38 -07:00
Alex Vandiver ce0df00e44 export: Notify all realm admins on realm export. 2024-09-23 10:02:43 -07:00
Alex Vandiver 7afe6800f7 export: Use relative paths, include more data. 2024-09-23 10:02:43 -07:00
Aman Agrawal cbb73bc82f accounts_accept_terms: Check if imported user wants marketing emails. 2024-09-20 18:17:53 -07:00
Alex Vandiver e125ad823d exports: Add a separate bucket for realm exports.
This allows finer-grained access control and auditing.  The links
generated also expire after one week, and the suggested configuration
is that the underlying data does as well.

Co-authored-by: Prakhar Pratyush <prakhar@zulip.com>
2024-09-20 15:43:49 -07:00
Alex Vandiver c1e8ecd08f uploads: Cache boto client in the module and be writable.
The `get_signed_upload_url` code is called for every S3 file serve
request, and is thus in the hot path.  The boto3 client caching
optimization is thus potentially useful as a performance optimization.
2024-09-20 15:43:49 -07:00
Alex Vandiver 1a7b3ef7ed upload: Use get_export_tarball_url in upload_export_tarball. 2024-09-20 15:43:49 -07:00
Alex Vandiver 4cf835d9dd upload: Remove common cache from get_export_tarball_url.
This is not called in the hot path like get_avatar_url is.
2024-09-20 15:43:49 -07:00
Alex Vandiver a5bf452202 upload: Realm is not Optional in upload_export_tarball.
af4eb8c0d5 marked the base class and local backend as non-Optional,
but left the S3 backend as Optional for some reason.

Remove it.
2024-09-20 15:43:49 -07:00
Alex Vandiver 91ac5c3c8b export: Log before the compression step, which can be slow. 2024-09-20 15:43:49 -07:00
Tim Abbott 51d0dfb504 docs: Tweak some documentation around send_event. 2024-09-20 15:28:18 -07:00
Prakhar Pratyush e763d065a3 django_api: Rename 'send_event' to 'send_event_rollback_unsafe'.
This commit renames the 'send_event' function to
'send_event_rollback_unsafe' to reflect the fact that it doesn't
wait for the db transaction (within which it gets called, if any)
to commit and sends event irrespective of commit or rollback.

In most of the cases we don't want to send event in the case of
rollbacks, so the caller should be aware that calling the function
directly is rollback unsafe.
2024-09-20 15:20:18 -07:00
Alex Vandiver 86d5ee5bdf tus: Document new API level for TUS endpoints. 2024-09-20 12:16:13 -07:00
Lauryn Menard 4bd4534450 billing: Enforce manual license management for guest role changes.
Adds a check for changing an existing guest user's role before
calling do_update_user in the case that a realm has a current
paid plan with manual license management.
2024-09-20 12:02:39 -07:00
Lauryn Menard 7861c1ba63 billing: Enforce manual billing renewal licenses for new users.
In addition to checking for available licenses in the current
billing period when adding or inviting new non-guest users, for
manual billing, we also verify that the number of licenses set
for the next billing period will be enough when adding/inviting
new users.

Realms that are exempt from license number checks do not have
this restriction applied.

Admins are notified via group direct message when a user fails
to register due to this restriction.
2024-09-20 12:02:39 -07:00
Mateusz Mandera 18fbb5d146 retention: Limit number of ids passed to db in delete messages query.
If do_delete_messages (and friends) are called for a massive number of
messages, the giant list of message ids is passed to Postgres even
though chunk_size makes all but the first chunk_size of message ids
useless.
2024-09-20 09:31:21 -07:00
Mateusz Mandera ed7c330548 retention: Rename run_archiving_in_chunks to run_archiving. 2024-09-20 09:31:21 -07:00
Prakhar Pratyush 86194efe7b realm_export: Add a new endpoint to fetch private data export consents.
This commit adds a new endpoint `export/realm/consents` to
fetch the consents of users for their private data exports.

Fixes part of #31201.
2024-09-20 09:10:32 -07:00
Alex Vandiver e43eaaebf0 openapi: Update upload-file example response to add filename. 2024-09-20 09:07:21 -07:00
Alex Vandiver 88acfbca8c docs: Adjust for 100MB MAX_FILE_UPLOAD_SIZE. 2024-09-20 09:05:11 -07:00
Mateusz Mandera 1d7d3fae61 signup: Mirror dummy user should be registered with role from invite.
Aside of what's generally explained in the code comment, this is
motivated by the specific situation of import of Slack Connect channels.
These channels contain users who are "external collaborators" and
limited to a single channel in Slack. We don't have more sophisticated
handling of their import, which would map this concept 1-to-1 in Zulip -
but we create them as inactive dummy users, meaning they have to go
through signup before their account is usable.

The issue is that their imported UserProfile.role is set to Member and
when they register, the UserProfile gets reactivated with that role
unchanged. However, if e.g. the user is signing up after they received
an invitation from the admin, they should get the role that was
configured on the invite. In particular important if the user is meant
to still be "limited" and thus the admin invites them as a guest - they
definitely don't want the user to get a full Member account because of
this weird interaction between import and registration.
2024-09-19 15:26:27 -07:00
Alex Vandiver 818c30372f upload: Use tusd for resumable, larger uploads.
Currently, it handles two hook types: 'pre-create' (to verify that the
user is authenticated and the file size is within the limit) and
'pre-finish' (which creates an attachment row).

No secret is shared between Django and tusd for authentication of the
hooks endpoints, because none is necessary -- tusd forwards the
end-user's credentials, and the hook checks them like it would any
end-user request.  An end-user gaining access to the endpoint would be
able to do no more harm than via tusd or the normal file upload API.

Regardless, the previous commit has restricted access to the endpoint
at the nginx layer.

Co-authored-by: Brijmohan Siyag <brijsiyag@gmail.com>
2024-09-19 11:37:29 -07:00
Sahil Batra 5f3a8334be user_groups: Do not allow deleting user groups. 2024-09-18 13:41:13 -07:00
Sahil Batra c53563d0e7 user_groups: Handle deactivated groups in webapp. 2024-09-18 13:41:13 -07:00
Sahil Batra 5c790aac72 user_groups: Rename allow_deactivated in 'GET /user_groups'.
This commit renames "allow_deactivated" parameter in
"GET /user_groups" endpoint to "include_deactivated_groups", so
that we can have consistent naming here and for client capability
used for deciding whether to send deactivated groups in register
response and how to handle the related events.
2024-09-18 13:41:13 -07:00
Sahil Batra fb63c47ea6 user_groups: Add client capability to handle deactivated groups.
This commit adds a client capability to not receive data about
deactivated groups.
2024-09-18 13:41:13 -07:00
Sahil Batra aa123b38b4 tests: Test can_manage_group setting for deactivating groups.
There were no tests for testing group deactivation permission
based on can_manage_group setting.
2024-09-18 13:41:13 -07:00
Sahil Batra 7a6135371e settings: Handle guests separately for group-based settings.
This commit adds code to handle guests separately for group
based settings, where guest will only have permission if
that particular setting can be set to "role:everyone" group
even if the guest user is part of the group which is used
for that setting. This is to make sure that guests do not
get permissions for actions that we generally do not want
guests to have.

Currently the guests do not have permission for most of them
except for "Who can delete any message", where guest could
delete a message if the setting was set to a user defined
group with guest being its member. But this commit still
update the code to use the new function for all the settings
as we want to have a consistent pattern of how to check whether
a user has permission for group-based settings.
2024-09-18 11:51:11 -07:00
Alex Vandiver 9a1f78db22 thumbnail: Support checking for images from streaming sources.
We may not always have trivial access to all of the bytes of the
uploaded file -- for instance, if the file was uploaded previously, or
by some other process.  Downloading the entire image in order to check
its headers is an inefficient use of time and bandwidth.

Adjust `maybe_thumbnail` and dependencies to potentially take a
`pyvips.Source` which supports streaming data from S3 or disk.  This
allows making the ImageAttachment row, if deemed appropriate, based on
only a few KB of data, and not the entire image.
2024-09-17 12:51:30 -07:00
Alex Vandiver ef21dd9b99 thumbnail: Set a stable ordering on ImageAttachment rows for locking.
Failure to have a stable ordering can lead to deadlocks.
2024-09-17 09:14:52 -07:00
Prakhar Pratyush 764083d31b settings: Add 'allow_private_data_export' user setting.
This commit adds a user-setting to allow users to decide
whether to let administrators export their private data.

Fixes part of #31201.
2024-09-16 15:48:40 -07:00
Lauryn Menard bc8993f159 user-groups: Add helper for date_created comparison. 2024-09-16 11:42:55 -07:00
Johan Nilsson bf9cbe30a7 integrations: Add support for release events to GitLab integration. 2024-09-16 09:26:20 -07:00
sujal shah 614caf111e user_groups: Add `creator` and `date_created` field in user groups.
This commit introduced 'creator' and 'date_created'
fields in user groups, allowing users to view who
created the groups and when.

Both fields can be null for groups without creator data.
2024-09-13 18:44:58 -07:00
Tim Abbott cb53c60563 models: Clean up CommonMessagePolicyEnum.
This is no longer used outside an old migration.
2024-09-12 09:36:02 -07:00
Vector73 0cd51f2fea settings: Remove `delete_own_message_policy` realm property.
Removed `delete_own_message_policy` realm property as the permission
to delete own messages is controlled by `can_delete_own_message_group`
setting.
2024-09-12 09:36:02 -07:00
Vector73 28c7a04734 settings: Add `can_delete_own_message_group` realm setting.
Added `can_delete_message_group` realm setting to replace
`delete_own_message_policy` property.
2024-09-12 09:36:02 -07:00
Tim Abbott 2585d209c0 user_groups: Rename misleadingly named view function. 2024-09-11 09:43:56 -07:00
Sahil Batra 688c5ad0af groups: Allow excluding deactivated groups in 'GET /user_groups' response. 2024-09-11 09:43:56 -07:00
Sahil Batra fef2925ff0 mention: Handle mentioning deactivated user groups.
If someone tries to mention a deactivated user group,
then it is automatically converted to a silent mention
like we do for deactivated users.
2024-09-11 09:43:56 -07:00
Sahil Batra c1d6da0a5c user_groups: Update handling of deactivated groups.
We only allow updating name of a deactivated group, and not
allow updating description, members, subgroups and any setting
of a deactivated user group.

Deactivated user groups cannot be a a subgroup of any group
or used as a setting for a group.
2024-09-11 09:43:56 -07:00
Sahil Batra e1cfe61452 user_groups: Add API support for deactivating user groups. 2024-09-11 09:43:56 -07:00
Sahil Batra bef7cfe00f user_groups: Add deactivated field to NamedUserGroup model. 2024-09-11 09:43:56 -07:00
Sahil Batra f0c0eef8dc groups: Lock user group rows while using them for settings.
This is important to make sure that we handle cases when there
are two parallel requests - one for using a group for a setting
and one for deactivating the same group. This makes sure that
atleast one of the above task fails.
2024-09-11 09:43:56 -07:00
Mateusz Mandera b572b1536a test_retention: Delete irrelevant comment.
This comment is clearly out of place and at this point doesn't match any
of the classes in a useful way.
2024-09-10 17:07:47 -07:00
Mateusz Mandera ce5439d87f retention: Remove outdated compat block for delete_message events.
As noted in the comment in that block, this is no longer useful.
2024-09-10 17:07:47 -07:00
Tim Abbott ad890890f6 lint: Fix several duplicate word typos. 2024-09-10 15:59:00 -07:00
Ikko Eltociear Ashimine 0507f65da5
realm_settings: Fix duplicate word typos in comments. 2024-09-10 15:55:14 -07:00
joseph 65893292b5 channel_subscribe: Use IDs instead of emails when processing results.
As part of our todo in the code, we want to use the unique user IDs
instead of emails when processing the results of subscribing users to a
channel. These changes apply those changes and streamlines the use of IDs.
2024-09-10 15:37:38 -07:00
Mateusz Mandera a36f906d1a presence: Add history_limit_days param to the API.
This param allows clients to specify how much presence history they want
to fetch. Previously, the server always returned 14 days of history.
With the recent migration of the presence API to the much more efficient
system relying on incremental fetches via the last_update_id param added
in #29999, we can now afford to provide much more history to clients
that request it - as all that historical data will only be fetched once.

There are three endpoints involved:
- `/register` - this is the main useful endpoint for this, used by API
clients to fetch initial data and register an events queue. Clients can
pass the `presence_history_limit_days` param here.
- `/users/me/presence` - this endpoint is currently used by clients to
update their presence status and fetch incremental data, making the new
functionality not particularly useful here. However, we still add the
new `history_limit_days` param here, in case in the future clients
transition to using this also for the initial presence data fetch.
- `/` - used when opening the webapp. Naturally, params aren't passed
here, so the server just assumes a value from
`settings.PRESENCE_HISTORY_LIMIT_DAYS_FOR_WEB_APP` and returns
information about this default value in page_params.
2024-09-10 13:15:35 -07:00
Mateusz Mandera 6ce096c0ff zerver: Rename some message_type variables to recipient_type. 2024-09-10 12:50:28 -07:00
Prakhar Pratyush e8349aa5d7 custom_email: Add manage_preferences block to the plaintext version.
Earlier, the content of the "manage_preferences" block that includes
the unsubscribe_link, personal settings link, etc was missing in the
plaintext version of the custom emails.

This commit updates the logic to include the manage_preferences block
content in the plaintext version.
2024-09-10 09:36:56 -07:00
Prakhar Pratyush e967226f8c remote_server_email: Add 'unsubscribe link' in the mail footer.
Previously, the emails sent to the remote servers had the
'unsubscribe link' only present in the 'List-Unsubscribe' header.

Not all email clients expose that header.

So, this commit adds the link in the footer too.
2024-09-10 09:36:56 -07:00
Vector73 eacf54d52d test_realm: Fix `test_invalid_integer_attribute_values`.
Fixes test that checks for error when invalid value is given for a property
in realm. Currently, only properties with type int are checked, leaving
properties having optional int type. This commit fixes that.
2024-09-10 09:32:33 -07:00
Alex Vandiver 8bacdbc895 thumbnail: Put the original dimensions on spinner images.
This lets us reserve the right amount of space in the message feed
immediately.
2024-09-09 15:59:02 -07:00
Tim Abbott f757f5f5f8 api_docs: Document feature level 286. 2024-09-09 15:59:02 -07:00
Alex Vandiver 56ca594abf upload: Preserve content-type from upload for local-file storage.
Now that we store the content-type in the database, use that value
(if we have it, since we did not backfill) when serving content back
to the client.  This means the file backend has parity with the S3
backend.
2024-09-09 12:40:17 -07:00
Alex Vandiver d7ebe7296d test_upload: Switch to SimpleUploadedFile. 2024-09-09 12:40:17 -07:00
Alex Vandiver 485e235719 test_upload: Remove an incorrect comment.
S3 uploads are now (after 04cf68b45e) handled like local uploads
were -- no redirects are used.
2024-09-09 12:40:17 -07:00
Alex Vandiver 903bfb31e6 upload: Provide the frontend with the less-modified filename. 2024-09-09 12:40:17 -07:00
Alex Vandiver b4764f49df upload: Download files with their original names.
Fixes: #29491.
2024-09-09 12:40:17 -07:00
Alex Vandiver 933e3cb375 upload: Explicitly return a bool and the Attachment object. 2024-09-09 12:40:17 -07:00
Alex Vandiver 0b0a7fa251 attachments: Tighten type of validate_attachment_request_for_spectator_access. 2024-09-09 12:40:17 -07:00
Alex Vandiver ca72e756eb upload: Rename "upload_image_to_s3"; it is not only for images. 2024-09-09 12:40:17 -07:00
Alex Vandiver c8c480baef mattermost: Handle duplicate dm-groups.
Observed in the wild, cause unknown.

Partially fixes: #24131.

Co-authored-by: Mateusz Mandera <mateusz.mandera@zulip.com>
2024-09-09 12:05:28 -07:00
Alex Vandiver 989d425023 users: Soften assumptions that all bots have owners.
Imported Slack bots currently do not have owners (#23145).  Soften the
deactivation codepath to allow them to be successfully deactivated
despite this.

Co-authored-by: Mateusz Mandera <mateusz.mandera@zulip.com>
2024-09-09 11:55:46 -07:00
Mateusz Mandera 7ebeffbcdc test_bots: Remove misleading comment.
This comment looks like an ancient leftover from early days (moved here
in a test_bots extraction in 123b4c1877 in
2017). Whatever its history, this comment and test name don't make sense
anymore. The response here is an error, not a silent success.
2024-09-09 11:55:46 -07:00
Lauryn Menard d610e09397 audit-logs: Use AuditLogEventType enum name for string method.
Reorders audit log string methods to have the following pattern:
"event_type event_time (id): modified_object". And the event type
is the name for the AuditLogEventType enum.
2024-09-09 11:50:13 -07:00
Lauryn Menard 00ecd4c8f0 audit-log: Move remote realm event types to AuditLogEventType enum.
Event types moved: REMOTE_REALM_VALUE_UPDATED,
REMOTE_PLAN_TRANSFERRED_SERVER_TO_REALM, REMOTE_REALM_LOCALLY_DELETED
REMOTE_REALM_LOCALLY_DELETED_RESTORED
2024-09-09 11:50:13 -07:00
Lauryn Menard aad93e149e audit-log: Move remote server event types to AuditLogEventType enum.
Event types moved: REMOTE_SERVER_DEACTIVATED, REMOTE_SERVER_REACTIVATED
REMOTE_SERVER_PLAN_TYPE_CHANGED, REMOTE_SERVER_DISCOUNT_CHANGED
REMOTE_SERVER_SPONSORSHIP_APPROVED, REMOTE_SERVER_BILLING_MODALITY_CHANGED
REMOTE_SERVER_SPONSORSHIP_PENDING_STATUS_CHANGED, REMOTE_SERVER_CREATED
2024-09-09 11:50:13 -07:00
Lauryn Menard d431a5aad6 audit-log: Move user group event types to AuditLogEventType enum.
Event types moved: USER_GROUP_CREATED, USER_GROUP_DELETED
USER_GROUP_DIRECT_USER_MEMBERSHIP_ADDED,
USER_GROUP_DIRECT_USER_MEMBERSHIP_REMOVED,
USER_GROUP_DIRECT_SUBGROUP_MEMBERSHIP_ADDED,
USER_GROUP_DIRECT_SUBGROUP_MEMBERSHIP_REMOVED,
USER_GROUP_DIRECT_SUPERGROUP_MEMBERSHIP_ADDED,
USER_GROUP_DIRECT_SUPERGROUP_MEMBERSHIP_REMOVED,
USER_GROUP_NAME_CHANGED, USER_GROUP_DESCRIPTION_CHANGED,
USER_GROUP_GROUP_BASED_SETTING_CHANGED
2024-09-09 11:50:13 -07:00
Lauryn Menard df1e9093a9 audit-log: Move stream/channel event types to AuditLogEventType enum.
Renamed event types below in the enum class to use channel instead of
stream.

Event types moved: STREAM_CREATED, STREAM_DEACTIVATED, STREAM_NAME_CHANGED
STREAM_REACTIVATED, STREAM_MESSAGE_RETENTION_DAYS_CHANGED
STREAM_PROPERTY_CHANGED, STREAM_GROUP_BASED_SETTING_CHANGED
2024-09-09 11:50:13 -07:00
Lauryn Menard f7e07e91a2 audit-log: Move customer event types to AuditLogEventType enum.
Event types moved: CUSTOMER_CREATED, CUSTOMER_PLAN_CREATED
CUSTOMER_SWITCHED_FROM_MONTHLY_TO_ANNUAL_PLAN,
CUSTOMER_SWITCHED_FROM_ANNUAL_TO_MONTHLY_PLAN,
CUSTOMER_PROPERTY_CHANGED, CUSTOMER_PLAN_PROPERTY_CHANGED
2024-09-09 11:50:13 -07:00
Lauryn Menard bf4a650999 audit-log: Move stripe event types to AuditLogEventType enum.
Event types moved: STRIPE_CUSTOMER_CREATED, STRIPE_CARD_CHANGED
STRIPE_PLAN_CHANGED, STRIPE_PLAN_QUANTITY_RESET
2024-09-09 11:50:13 -07:00
Lauryn Menard fc6bc6df60 audit-log: Move user muted event types to AuditLogEventType enum.
Event types moved: USER_MUTED, USER_UNMUTED
2024-09-09 11:50:13 -07:00
Lauryn Menard 10d161638e audit-log: Move subscription event types to AuditLogEventType enum.
Event types moved: SUBSCRIPTION_CREATED, SUBSCRIPTION_ACTIVATED,
SUBSCRIPTION_DEACTIVATED, SUBSCRIPTION_PROPERTY_CHANGED.
2024-09-09 11:50:13 -07:00
Lauryn Menard 56c8cbde1e audit-log: Move realm event types to AuditLogEventType enum.
Event types moved: REALM_CREATED, REALM_DEFAULT_USER_SETTINGS_CHANGED
REALM_ORG_TYPE_CHANGED, REALM_DOMAIN_ADDED, REALM_DOMAIN_CHANGED
REALM_DOMAIN_REMOVED, REALM_PLAYGROUND_ADDED, REALM_PLAYGROUND_REMOVED
REALM_LINKIFIER_ADDED, REALM_LINKIFIER_CHANGED, REALM_LINKIFIER_REMOVED
REALM_EMOJI_ADDED, REALM_EMOJI_REMOVED, REALM_LINKIFIERS_REORDERED
REALM_IMPORTED
2024-09-09 11:50:13 -07:00
Lauryn Menard d2c32f23db audit-log: Move realm event types to AuditLogEventType enum.
Event types moved: REALM_DEACTIVATED, REALM_REACTIVATED, REALM_SCRUBBED
REALM_PLAN_TYPE_CHANGED, REALM_LOGO_CHANGED, REALM_EXPORTED
REALM_PROPERTY_CHANGED, REALM_ICON_SOURCE_CHANGED, REALM_DISCOUNT_CHANGED
REALM_SPONSORSHIP_APPROVED, REALM_BILLING_MODALITY_CHANGED
REALM_REACTIVATION_EMAIL_SENT, REALM_SPONSORSHIP_PENDING_STATUS_CHANGED
REALM_SUBDOMAIN_CHANGED
2024-09-09 11:50:13 -07:00
Lauryn Menard caeeaf3c3f audit-log: Move user event types to AuditLogEventType enum.
Event types moved: USER_SOFT_ACTIVATED, USER_SOFT_DEACTIVATED
USER_PASSWORD_CHANGED, USER_AVATAR_SOURCE_CHANGED, USER_FULL_NAME_CHANGED
USER_EMAIL_CHANGED, USER_TERMS_OF_SERVICE_VERSION_CHANGED
USER_API_KEY_CHANGED, USER_BOT_OWNER_CHANGED,
USER_DEFAULT_SENDING_STREAM_CHANGED, USER_DEFAULT_REGISTER_STREAM_CHANGED
USER_DEFAULT_ALL_PUBLIC_STREAMS_CHANGED, USER_SETTING_CHANGED
USER_DIGEST_EMAIL_CREATED
2024-09-09 11:50:13 -07:00
Lauryn Menard e5daa3470f audit-log: Move user event types to AuditLogEventType enum.
Event types moved: USER_CREATED, USER_ACTIVATED, USER_DEACTIVATED
USER_REACTIVATED, USER_ROLE_CHANGED, USER_DELETED
USER_DELETED_PRESERVING_MESSAGES
2024-09-09 11:50:13 -07:00
joseph b0a20d2cae attachments: Return a list of removed attachments while editing.
Currently, we want to ask users if they would like to delete their
attachments after they have removed the attachments while editing. These
changes are preparatory changes on the backend to return a list of removed
attachments after the user has removed attachments while editing.

Fixes part of #25525.
2024-09-06 11:40:51 -07:00
Lauryn Menard 4b954a42ef billing: Create LicenseLedger entry for guest user role changes.
As guest users are charged at a different rate on Zulip Cloud, it
makes sense to track these changes in our LicenseLedger table so
that we can have an accurate record of the use of licenses that
a realm has purchased.
2024-09-05 16:11:36 -07:00
Kenneth Rodrigues 1d580f143e endpoints: Modify comments mentioning has_request_variables and REQ. 2024-09-05 16:02:12 -07:00
Kenneth Rodrigues dc32396180 endpoints: Remove the has_request_variables decorator.
All endpoints have been migrated to the typed_endpoint decorator,
therefore the has_request_variables decorator and the REQ function are
no longer needed and have been removed.
2024-09-05 16:02:12 -07:00
Anders Kaseorg e3abd09e67 thumbnail: Fix corrupted email notifications due to HTML5 entities.
BeautifulSoup with formatter="html5" unnecessarily escapes many
characters with HTML5-specific entities that cannot be correctly
parsed by lxml during generation of email notifications.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-09-05 16:00:45 -07:00
Alex Vandiver cec0942b4b users: Allow zero-width-joiners in user names.
Partially fixes: #20128.
2024-09-03 16:34:55 -07:00
Tim Abbott ca406288d9 webhooks: Fix strange joining code in greenhous webhook. 2024-09-03 12:31:36 -07:00
Anders Kaseorg 91ade25ba3 python: Simplify with str.removeprefix, str.removesuffix.
These are available in Python ≥ 3.9.
https://docs.python.org/3/library/stdtypes.html#str.removeprefix

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-09-03 12:30:16 -07:00
Prakhar Pratyush a2ff4f52c4 actions: Use 'durable=True' in outermost transaction.atomic block.
We used 'savepoint=False' in #31169 which was prior to our discussion
in CZO to use 'durable=True' instead.

This commit makes changes to use 'durable=True' in the outermost
transaction.atomic block.
2024-09-03 09:17:38 -07:00
Alya Abbott d7761f7163 help: Add a guide on moving to Zulip. 2024-09-01 16:16:04 -07:00
Alya Abbott 7df0db142b help: Document how to invite users in imported organizations. 2024-09-01 16:16:04 -07:00
bedo 3da91e951c auth: Migrate to @typed_endpoint.
Since this is the last has_request_variables endpoint outside tests,
more test_openapi code needs to be deleted in this transition.
2024-08-30 16:56:14 -07:00
Prakhar Pratyush 656c65c74c edit_history: Remove 'prev_rendered_content_version' field.
This commit removes the 'prev_rendered_content_version'
field from:

* the 'edit_history' object within message objects in the
API response of `GET /messages`, `GET /messages/{message_id}`
and `POST /zulip-outgoing-webhook`.
* the 'update_message' event type

as it is an internal server implementation detail not used
by any client.

Note: The field is still stored in the 'edit_history' column
of the 'Message' table as it will be helpful when making
major changes to the markup rendering process.
2024-08-29 15:37:12 -07:00
Alex Vandiver 0c07c6531c thumbnail: Enqueue thumbnails when we render a spinner.
Thumbnails are usually enqueued in the worker when the image is
uploaded.  However, for images which were uploaded before the
existence of the thumbnailing worker, and whose metadata was
backfilled (see previous commit) this leaves a permanent spinner,
since nothing triggers the thumbnail worker for them.

Enqueue a thumbnail worker for every spinner which we render into
Markdown.  This ensures that _something_ is attempting to resolve the
spinner which the user sees.  In the case of freshly-uploaded images
which are still in the queue, this results in a duplicate entry in the
thumbnailing queue -- this is harmless, since the worker determines
that all of the thumbnails we need have already been generated, and it
does no further work.  However, in the case of historical uploads, it
properly kicks off the thumbnailing process and results in a
subsequent message update to include the freshly-generated thumbnail.

While specifically useful for backfilled uploads, this is also
generally a good safety step for a good user experience, as it also
prevents dropped events in the queue from unknown causes from leaving
perpetual spinners in the message feed.

Because `get_user_upload_previews` is potentially called twice for
every message with spinners (see 6f20c15ae9), we add an additional
flag to `get_user_upload_previews` to suppress a _second_ event from
being enqueued for every spinner generated.
2024-08-29 12:11:51 -07:00
Alex Vandiver df91cdf333 thumbnail: Backfill ImageAttachment rows.
We previously used the file extension to determine if we should
attempt to inline an image.  After b42863be4b, we rely on the
existence of ImageAttachment rows to determine if something is an
image which can be viewed inline.  This means that messages
containing files uploaded before that commit, when (re-)rendered, will
be judged as not having inline'able images.

Backfill all of the ImageAttachment rows for image-like file
extensions.  We are careful to only download the bytes that we need in
the image headers, to minimize bandwidth from S3 in the event that the
S3 backend is in use.  We do _not_ produce thumbnails for the images
during this migration; see the subsequent commit.

Because this migration will be backported to 9.x, it is marked as only
depending on the last migration in `9.x`, with a subsequent merge
migration into the tip of `main`.
2024-08-29 12:11:51 -07:00
Alex Vandiver 704423787b do_delete_old_unclaimed_attachments: Cap deletions at batch size.
Since each loop may add more than one file to the `storage_paths`
list, this may result in more than 1000 files being sent to
delete_message_attachments.  Since the S3 backend only supports 1000
elements being deleted at once, we must partition the list into chunks
which are no more than 1000 elements long.
2024-08-28 13:57:57 -07:00
Lauryn Menard 8ace23ffa9 docs: Update help center links for stream to channel rename.
Fixes the URLRedirects for "help/about-streams-and-topics" and
"help/streams-and-topics" to go to "/help/introduction-to-topics"
since "help/channels-and-topics" has been redirected to that URL.
2024-08-27 13:30:28 -07:00
Kenneth Rodrigues 00a54099a2 analytics: Migrate to @typed_endpoint. 2024-08-26 08:57:49 -07:00
Lauryn Menard 7af87c7e14 realm-settings: Get max invites for realm plan type helper.
There are a few places where we want to set the max invites for a
realm to the default for a realm's plan type, so this creates a
helper function that can be used consistently to get that default
value.
2024-08-26 08:29:50 -07:00
Anders Kaseorg 8c0b2d14aa mypy: Remove use of ValuesQuerySet and QuerySetAny.
This was made unnecessary in django-stubs 5.0.1 and mypy 1.10.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-08-24 17:30:41 -07:00
Tim Abbott f4d98b03d5 migrations: Squash zerver migrations.
This has the impact of making rebuilding the database in a Zulip
development environment, or initializing a new production database,
dramatically faster.

This was generated by merging the output of `manage.py makemigrations`
with an empty migration.

Tested using `tools/rebuild-test-database` before and after this
change, and comparing the output of `pg_dump -d zulip_test` using
Git's diff comparison algorithm. Differences in that SQL dump include:

- The actual generated table contents, due to timestamps and the like;
  this is expected and unrelated to schema.

- Orders of fields within tables, which is not significant in SQL.

- IDs assigned to tables in the ContentType table, which is expected
  and not a problem with how that Django table is designed.

- Names of generated indexes and constraints; modern Django seems to
  abbreviate long names differently for these, and it's not obviously
  possible to configure those used by the `db_index` property. If
  necessarily, likely this can be converged via a migration filled
  with `IF EXISTS` rename operations like the one done in
  zerver/migrations/0246_message_date_sent_finalize_part2.py.

- Names of the ~3 sequences related to renamed tables:
  usertopic/mutedtopic, botconfigdata/botuserconfigdata,
  realmdomain/realmalias. Probably there's no action required here,
  but we could do rename operations if desired.
2024-08-23 17:15:35 -07:00
Tim Abbott 029ceb2a48 migrations: Optimize initial zerver migration.
Generated using manage.py squashmigrations, with minimal manual
surgery to replace the original migration.

This should in theory produce the exact same database state as
previously.
2024-08-23 17:15:35 -07:00
Tim Abbott 7080c94245 migrations: Reorder initial zerver migration.
This has no functional changes, but it helps the squashmigrations tool
realize some squashing opportunities to not have models declared
before other models that they will gain a foreign key to.
2024-08-23 17:15:35 -07:00
Tim Abbott afecb2eca6 migrations: Add missing elidable tags on RunPython/RunSQL steps.
This helps the squashmigrations tool know that it can squash these.
2024-08-23 17:15:35 -07:00
Tim Abbott 5188cf3d8e migrations: Remove RunPython block from initial migration.
Since it's the initial migration, this can't have any useful effect.

I'm pretty sure the backstory is we did a manual squash of migrations
during the process of open-sourcing Zulip, and incorrectly didn't
remove this code.
2024-08-23 17:15:35 -07:00
Tim Abbott 547377a295 zerver: Remove now-unnecessary dependency in migration 0383.
It's not possible to directly upgrade from pre-5.3 versions to main,
since they do not have any supported OSes in common. Thus, this
dependency on the confirmation model, which risks creating a circular
dependency if we squash migrations, can be removed.
2024-08-23 17:15:35 -07:00
Alex Vandiver 73e5364838 import: Use inspection to determine sequence names. 2024-08-23 17:15:35 -07:00
Tim Abbott 31623911d1 test_message_fetch: Avoid assumptions about fixture data set. 2024-08-23 17:15:35 -07:00
Lauryn Menard d708c3c039 support: Add ability to update max daily invitations for realm.
Adds some validation for changing the realm's max invites via the
support view so that it is not set below the default max for the
realm's plan type, and so that if it's currently set to the default
max it's not reset to that same value.
2024-08-23 16:08:30 -07:00
afeefuddin f013eb1037 message_fetch: Convert module to TypeScript. 2024-08-23 13:34:13 -07:00
roanster007 c6a06d4684 direct_message_group: Add new `group_size` field.
This commit adds a new `group_size` field to the `DirectMessageGroup`
model, and backfills its value to each of the existing direct message
groups.

Fixes part of #25713
2024-08-23 11:09:41 -07:00
Prakhar Pratyush 3ea6ba97c0 users: Send event on commit in do_change_is_billing_admin.
Earlier, we were using 'send_event' in 'do_change_is_billing_admin'
which can lead to a situation, if any db operation is added after
the 'send_event' in future, where we enqueue events but the action
function fails at a later stage.

Events should not be sent until we know we're not rolling back.

Fixes part of #30489.
2024-08-23 09:10:56 -07:00
Prakhar Pratyush 2b6c5260a5 users: Send event on commit in do_update_outgoing_webhook_service.
Earlier, we were using 'send_event' in
'do_update_outgoing_webhook_service' which can lead to a
situation, if any db operation is added after the 'send_event'
in future, where we enqueue events but the action function fails
at a later stage.

Events should not be sent until we know we're not rolling back.

Fixes part of #30489.
2024-08-23 09:10:56 -07:00
Prakhar Pratyush 30edd1a3b0 users: Add transaction.atomic decorator to do_update_bot_config_data.
The db operations in do_update_bot_config_data should be atomic.
2024-08-23 09:10:56 -07:00
Prakhar Pratyush 8f2727fcea realm_settings: Send event on commit in do_set_push_notif...timestamp.
Earlier, we were using 'send_event' in
'do_set_push_notifications_enabled_end_timestamp' which
can lead to a situation, if any db operation is added after the
'send_event' in future, where we enqueue events but the action
function fails at a later stage.

Events should not be sent until we know we're not rolling back.

Fixes part of #30489.
2024-08-23 09:10:56 -07:00
Prakhar Pratyush 5af6c04a9e realm_settings: Update do_set_realm_stream to send event on commit.
Earlier, we were using 'send_event' in 'do_set_realm_stream' which
can lead to a situation, if any db operation is added after the
'send_event' in future, where we enqueue events but the action
function fails at a later stage.

Events should not be sent until we know we're not rolling back.

Fixes part of #30489.
2024-08-23 09:10:56 -07:00
Prakhar Pratyush c0dc005d05 realm_settings: Send event on commit in do_set...authentication_methods.
Earlier, we were using 'send_event' in
'do_set_realm_authentication_methods' which can lead to a situation
where we enqueue events but there's an error at a later stage in
the codepath using this function.

Events should not be sent until we know we're not rolling back.

Fixes part of #30489.
2024-08-23 09:10:56 -07:00
Prakhar Pratyush 63064e926b realm_settings: Update do_set...default_setting to send event on commit.
Earlier, we were using 'send_event' in do_set_realm_user_default_setting
which can lead to a situation, if any db operation is added after
the 'send_event' in future, where we enqueue events but the action
function fails at a later stage.

Events should not be sent until we know we're not rolling back.

Fixes part of #30489.
2024-08-23 09:10:56 -07:00
Prakhar Pratyush 8ddaa949fc user_settings: Send event on commit in do_regenerate_api_key.
Earlier, we were using 'send_event' in 'do_regenerate_api_key'
which can lead to a situation, if any db operation is added after
the 'send_event' in future, where we enqueue events but the action
function fails at a later stage.

Events should not be sent until we know we're not rolling back.

Fixes part of #30489.
2024-08-23 09:10:56 -07:00
Prakhar Pratyush 7e03569495 user_settings: Send event on commit in do_change_full_name.
Earlier, we were using 'send_event' in 'do_change_full_name'
which can lead to a situation, if any db operation is added after
the 'send_event' in future, where we enqueue events but the action
function fails at a later stage.

Events should not be sent until we know we're not rolling back.

Fixes part of #30489.
2024-08-23 09:10:56 -07:00
Mateusz Mandera 9cad9644e7 import_util: Fix file extensions of emoji files downloaded from Slack.
The Slack API when returning the emoji records, returns the record for
its thumbsup_all emoji with the url ending with .png, even though the
file is a gif.

For that reason, we have to make that code correct file extensions based
on the response content-type. Emojis are the smallest set of images to
download, so for simplicity of implementation, we remove the
parallelization of the downloads in favor of just processing them
serially.
2024-08-21 16:30:19 -07:00
Mateusz Mandera 669e0a3e47 import: Fix export->import of emojis from Slack.
Ideally this would besplit up into two commits, but it's hard to split
into self-contained, atomic chunks now that this segment of the
import/export system is generally kind of broken after thumbnailing
system changes.

1. 3rd party export converters don't make .original image files.
Insteadthey provide a single file, which the import should treat as if
it's .original.

2. 3rd party converters create all the records with is_animated=False.
That's an issue, because without setting that correctly on the
RealmEmoji objects, Zulip doesn't know that it should use the "still"
thumbnail when the emoji is being used in a user's status. Which leads
to incorrectly displaying the user status with the distracting
animation.
2024-08-21 16:30:19 -07:00
Mateusz Mandera 5476340b52 import: Export and import .original emoji files correctly.
The export tool was only exporting the already-thumbnailed emoji file,
omitting the original one. Now we make sure to export the .original file
too, like we do for avatars, and make the import tool process it
directly, to thumbnail it directly and generate a still in the case of
animated emojis.

Otherwise, the imported realm wouldn't have the <emoji>.png.original
file that we generally expect to have accessible, and stills for
animated emojis were completely missing.
2024-08-21 16:30:19 -07:00
Tim Abbott f1d828cb7d test_openapi: Remove now-unused blocks.
As we remove uses of the has_request_variables system in favor of
typed_endpoints, parts of this file have lost test coverage.
2024-08-21 11:33:42 -07:00
Tim Abbott 9348b87d73 topic: Remove now-unused REQ_topic. 2024-08-21 11:27:54 -07:00
Kenneth Rodrigues 2483e600a2 message_send: Convert to typed endpoint.
Convert `message_send.py` use `typed endpoint`.

Disable `message_send` endpoint `to` parameter in the `openapi`
`validate_json_schema` check, because it is a special case where the
content type of the parameter is application/json but the
parameter may or may not be JSON encoded since previously we also
accepted a raw string and some ad-hoc bot might still depend on sending
a raw string.

Remove unused validators from `validator.py`.
2024-08-21 11:13:00 -07:00
Prakhar Pratyush 8b3d5a9455 custom_profile_fields: Update transaction to be durable.
This commit updates the db transaction to be durable for
do_update_user_custom_profile_data_if_changed to avoid
addition of any outer atomic block.

While adding any outer atomic block this will raise a runtime error
and we can replace the durable argument with 'savepoint=False'
otherwise we'll have to manually track down the action functions
getting called in that outer atomic block and set the savepoint=False
otherwise it'll lead to creation of savepoints which we don't want.

We can't set savepoint=False before hand to the outermost action
function because it leads to rollback of transaction in tests when
an error is raised in action function.
2024-08-21 10:51:23 -07:00
Prakhar Pratyush e729c51485 custom_profile_fields: Update remove...value to send event on commit.
Earlier, we were using 'send_event' in
check_remove_custom_profile_field_value which can lead to a
situation where we enqueue events but the function fails at a
later stage.

Events should not be sent until we know we're not rolling back.

Fixes part of #30489.
2024-08-21 10:51:23 -07:00
Prakhar Pratyush bfe7eb6b2c do_change_full_name: Noop if value isn't really changing.
In do_change_full_name, we shouldn't create RealmAuditLog
entry and sends events if the value isn't changing.
2024-08-21 10:51:23 -07:00
Mateusz Mandera 7e1f468f04 saml: Fix exception when syncing missing value to custom profile field.
There was a bug here that would trigger an exception inside
`sync_user_profile_custom_fields`, causing it to get logged with
logging.warning, when an attribute configured for SAML custom profile
field sync was missing from a SAMLResponse or had an empty value.
`sync_user_profile_custom_fields` expects valid values, and None is not
valid.

We could consider a slightly different behavior here instead - when an
attribute is sent with no value in the SAMLResponse, that means the attr
has no value in the IdP's user directory - so perhaps a better behavior
would be to also remove the custom profile field value in Zulip. However
there are two issues with that:

1. It's not necessarily the best behavior, because an organization might
want the "user doesn't have this attribute set at the IdP level" state
to just mean that the user should be free to set the value manually in
Zulip if they wish. And having that value get reset on every login would
then be an issue. The implementation in this commit is consistent with
this philosophy.

2. There's some implementation difficulty - upstream
`self.get_attr(...)`, which we use for reading the attr value from the
SAMLResponse, doesn't distinguish between an attribute being sent with
no value and the attribute not being sent at all - in both cases it
returns None. So we'd need some extra work here with parsing the
SAMLResponse properly, to be able to know when the custom profile field
should get cleared.
2024-08-20 13:57:30 -07:00
Mateusz Mandera 833dce8a13 saml: Add support for syncing user role.
Replace the SOCIAL_AUTH_SYNC_CUSTOM_ATTRS_DICT with
SOCIAL_AUTH_SYNC_ATTRS_DICT, designed to support also regular user attrs
like role or full name (in the future).

Custom attributes can stay configured as they were and will get merged
into SOCIAL_AUTH_SYNC_ATTRS_DICT in computed_settings, or can be
specified in SOCIAL_AUTH_SYNC_ATTRS_DICT directly with "custom__"
prefix.

The role sync is plumbed through to user creation, so users can
immediately be created with their intended role as provided by the IdP
when they're creating their account, even when doing this flow without
an invitiation.
2024-08-20 11:53:24 -07:00
Mateusz Mandera 8c1a1ea8db scim: Extract ROLE_TYPE_TO_NAME dict to UserProfile.
This allows these mappings to used in other APIs. Specifically, we want
to use this for syncing role during SAML auth.
2024-08-20 11:53:24 -07:00
Mateusz Mandera 76b41e433a maybe_send_to_registration: Enforce kwargs.
Most of these should be kwargs, to make sure we're not accidentally
getting the order wrong when calling it.
2024-08-20 11:53:24 -07:00
Kenneth Rodrigues 86bf0d3ea3 decorators: Migrate to typed_endpoint.
Modify `semgrep-py.yml` to treat arguments after `*args` as
keyword-only arguments.
2024-08-20 10:15:30 -07:00
Kenneth Rodrigues 50712bfa81 scheduled_messages: Migrate to typed_endpoint.
Migrate `scheduled_message.py` to typed_endpoint.

Perform Json parsing in the endpoint itself instead of
in `recipient_parsing.py`.
2024-08-20 10:03:22 -07:00
Ujjawal Modi 5d613ce98d user_groups: Add support to update can_manage_group setting.
This commit adds API support to update can_manage_group setting
of a user group.
2024-08-20 09:28:05 -07:00
Ujjawal Modi b0b36e884c user_groups: Add support to set can_manage_group during creation.
This commit adds API support to set can_manage_group while
creating a user group.
2024-08-20 09:28:05 -07:00
Ujjawal Modi 2505813359 user_groups: Add can_manage_group field to UserGroup objects.
This commit make changes in code to include can_manage_group
field to UserGroup objects passed with response of various endpoints
including "/register" endpoint and also in the group object
send with user group creation event.
2024-08-20 09:28:05 -07:00
Ujjawal Modi 423d5c42f6 user_groups: Update code to check whether user can edit a user group.
Earlier there was only a realm level setting for configuring
who can edit user groups. A new group level setting is also added
for configuring who can manage that particular group.
Now, a user group can be edited by a user if it is allowed from
realm level setting or group level setting.

This commit make changes to also use group level setting
in determining whether a group can be edited by user or not.

Also, updated tests to use api_post and api_delete helpers instead
of using client_post and client_delete helpers with different users
being logged in.
2024-08-20 09:28:05 -07:00
Ujjawal Modi 03220ba456 user_groups: Add can_manage_group field in database.
This commit adds a new group level setting can_manage_group
for configuring who can manage a group. This commit only adds
the field in database and make changes to automatically create
single user groups corresponsing to acting user
which will be the default value for this setting.

Fixes part of #25928.
2024-08-20 09:28:05 -07:00
Ujjawal Modi 9e699dfc85 tests: Separate test of group edit setting for creation and deletion.
Earlier there was a single backend test for testing group edit policy
for creating and deleting user group.This commit make changes in the test
and now there are two separate tests for testing group edit policy for
creating and deleting user groups.

This was done because in future commits we will be adding a
realm level setting for configuring who can create user groups.

Also, updated tests to use api_post and api_delete helpers instead
of using client_post and client_delete helpers with different users
being logged in.
2024-08-20 09:28:05 -07:00
Ujjawal Modi bb0b6900df user_groups: Add a decorator to check group creation permission.
Earlier there was a single decorator function to check whether
user can create and edit user groups. This commit adds a new
decorator function to check whether user has permissions to
create user groups.

This was done because in future commits we will be adding a
realm level setting for configuring who can create user groups.
2024-08-20 09:28:05 -07:00
Sahil Batra 3f800002be user_groups: Allow not passing old_setting_api_value. 2024-08-20 09:28:05 -07:00
evykassirer b1198785be message_edit: Convert module to typescript. 2024-08-19 16:33:04 -07:00
Sahil Batra 3dc82eed57 user_groups: Refactor code to compute user group objects.
This commit refactors code in user_groups_in_realm_serialized
such that we do not prefetch "can_mention_group__direct_members"
and "can_mention_group__direct_subgroups" using prefetch_related
and instead fetch members and subgroups for all groups in separate
queries and then use that data to find the members and subgroups
of the group used for that setting.

This change helps us in avoiding two prefetch queries for each
setting when we add more group settings.
2024-08-19 15:41:55 -07:00
Prakhar Pratyush 2406a3feea streams: Update do_change...retention_days to send event on commit.
Earlier, we were using 'send_event' in
do_change_stream_message_retention_days which can lead to a situation
where we enqueue events but the function fails at a later stage.

Events should not be sent until we know we're not rolling back.

Fixes part of #30489.
2024-08-19 10:14:49 -07:00
Prakhar Pratyush a9af1c0ce2 streams: Update do_change_stream_description to send event on commit.
Earlier, we were using 'send_event' in do_change_stream_description
which can lead to a situation where we enqueue events but the
function fails at a later stage.

Events should not be sent until we know we're not rolling back.

Fixes part of #30489.
2024-08-19 10:14:49 -07:00
Prakhar Pratyush e474ab7ee0 streams: Add transaction.atomic decorator to 'do_rename_stream'.
This commit updates the 'do_rename_stream' action function
to make the db operations atomic.

Also, send_event is replaced with send_event_on_commit.
2024-08-19 10:14:49 -07:00
Prakhar Pratyush 69dec9dbc6 streams: Update 'do_change_stream_post_policy' to send event on commit.
Earlier, we were using 'send_event' in do_change_stream_post_policy
which can lead to a situation where we enqueue events but the
function fails at a later stage.

Events should not be sent until we know we're not rolling back.

Fixes part of #30489.
2024-08-19 10:14:49 -07:00
Prakhar Pratyush 41d947b89d streams: Make db operations in do_change_subscription_property atomic.
This commit adds a transaction.atomic decorator to the
'do_change_subscription_property' function to make
the db operations in the action function atomic.

Also, send_event is changed to send_event_on_commit.
2024-08-19 10:14:49 -07:00
Prakhar Pratyush 5d0b3d1a6d muted_users: Update 'do_unmute_user' to send event on commit.
Earlier, we were using 'send_event' in 'do_unmute_user'
which can lead to a situation where we enqueue events but the
function fails at a later stage.

Events should not be sent until we know we're not rolling back.

Fixes part of #30489.
2024-08-19 10:14:49 -07:00
Prakhar Pratyush 3b9a21443a muted_users: Update 'do_mute_user' to send event on commit.
Earlier, we were using 'send_event' in 'do_mute_user' which
can lead to a situation where we enqueue events but the
function fails at a later stage.

Events should not be sent until we know we're not rolling back.

Fixes part of #30489.
2024-08-19 10:14:49 -07:00
Prakhar Pratyush 9026e6ecc1 message_flags: Update 'do_update_message_flags' to send event on commit.
Earlier, we were using 'send_event' in do_update_message_flags
which can lead to a situation where we enqueue events but the
function fails at a later stage.

Events should not be sent until we know we're not rolling back.

Fixes part of #30489.
2024-08-19 10:14:49 -07:00
Prakhar Pratyush 32a4a112b1 message_flags: Update do_mark_muted_user...read to send event on commit.
Earlier, we were using 'send_event' in
'do_mark_muted_user_messages_as_read' which can lead to a
situation where we enqueue events but the function fails at a
later stage.

Events should not be sent until we know we're not rolling back.

Fixes part of #30489.
2024-08-19 10:14:49 -07:00
Prakhar Pratyush 62eb850423 message_flags: Update transactions to be durable in do_mark_all_as_read.
In 'do_mark_all_as_read', the transactions which mark the messages
as read in batches should be marked as durable to avoid addition
of any outer atomic block as we support marking a few batches
(not all messages) as read in the case of a timeout.
2024-08-19 10:14:49 -07:00
Prakhar Pratyush 64beea2765 message_flags: Update 'do_mark_stream...read' to send event on commit.
Earlier, we were using 'send_event' in do_mark_stream_messages_as_read
codepath which can lead to a situation where we enqueue events but the
function fails at a later stage.

Events should not be sent until we know we're not rolling back.

Fixes part of #30489.
2024-08-19 10:14:49 -07:00
Prakhar Pratyush ed512f06bb message_edit: Queue event on commit in do_update_message codepath.
'do_update_message' is within a db transaction, this commit
updates the 'do_clear_mobile_push_notifications_for_ids' function
used in 'do_update_message' to queue event on commit.

Events should not be sent until we know we're not rolling back,
otherwise it can lead to a situation where we enqueue events but
the function fails at a later stage.
2024-08-19 10:14:49 -07:00
Tim Abbott a3806b4165 migrations: Fix quoting bug in tutorial_status migration.
Fixes #31381.
2024-08-17 22:17:21 -07:00
bedo 00c9f36434 create stream: Bulk conversion of principals to UserProfiles.
Previously, this logic did the database queries to look up UserProfile
objects in a loop.

Fixes #21820.

Significantly improves Stream creation time and also unsusbcribing users.
Tested stream creation with 10k stream subscribers:

- before: 127 seconds ~2 mins
- after: 17 seconds ~0.3 min

Add a test case for user unsubscribing themself.
2024-08-16 17:46:19 -07:00
Alex Vandiver 854ec40def home: Add rel=dns-prefetch to Gravatar and any Zulip avatar host.
This hints to the browser that it should start DNS lookups for the
host, since it is likely to be necessary.  It is a softer form than
`rel-preconnect`, which may be unnecessary in these cases, if the
client has the resources cached already.
2024-08-16 14:33:25 -07:00
Lauryn Menard cabe0b2e05 api-docs: Update topics array description in get-stream-topics response. 2024-08-15 18:27:43 -07:00
Shubham Padia 2b3a41be58 user_card_popover: Show popover on all .messagebox user mention.
Show user card popover for scheduled messages overlay, compose box
preview, message edit preview, message edit history.

`.messagebox` was chosen as the selector since that was the nearest
parent class that was common for all of the above.

`@all` does not have a popover and that's why it will have the same
pointer as its parent element. We also introduce a new class called
`.user-mention-all` for managing css rules specific to that mention.
2024-08-15 16:28:08 -07:00
Prakhar Pratyush 52a9846cdf user_profile: Remove 'tutorial_status' field.
The 'tutorial_status' field on 'UserProfile' model is
no longer used to show onboarding tutorial.

This commit removes the 'tutorial_status' field,
'POST users/me/tutorial_status' endpoint, and
'needs_tutorial' parameter in 'page_params'.

Fixes part of zulip#30043.
2024-08-15 13:31:40 -07:00
Prakhar Pratyush ee806c49b9 ui_init: Use OnboardingStep for narrowing on first web app load.
We plan to remove the 'tutorial_status' field from UserProfile
table as it is no longer used to show tutorial.

The field is also used to narrow a new user in DM with
welcome bot on the first load.

This prep commit updates the logic to use a new OnboardingStep
for the narrowing behaviour on the first load. This will help
in removing the 'tutorial_status' field.
2024-08-15 13:31:40 -07:00
PieterCK d974c10efc integration-docs: Update Open Collective for new doc format.
Other than reformatting documentation for Open Collective, this
commit also moves it to the "Financial" category from "Communications".
This is because Open Collective is mainly a fundrising + legal status +
money management platform, as stated in https://opencollective.com/.

Part of #29592.
2024-08-15 13:16:14 -07:00
PieterCK 00f9892d8d integration-docs: Update Mention for new doc format.
Part of #29592.
2024-08-15 13:16:14 -07:00
PieterCK 07fda86ff9 integration-docs: Update Netlify for new doc format.
Besides reformatting the Netlify doc, this commit also updates the
instructions to match some UI changes in Netlify. The "Outgoing Webhook"
menu is now called "HTTP Post request".

Part of #29592.
2024-08-15 13:16:14 -07:00
PieterCK 9494620022 integration: Delete Opbeat integration.
Opbeat was acquired by Elastic in 2017[1] and maintained until Elastic
APM went live[2]. Now that Elastic APM is online, it looks like Opbeat
repositories have been marked as archived/unmaintained[3].

[1]: https://www.elastic.co/blog/welcome-opbeat-to-the-elastic-family
[2]:
https://www.elastic.co/observability/application-performance-monitoring
[3]: https://github.com/opbeat
2024-08-15 13:16:14 -07:00
Vector73 80969a62e6 settings: Add `can_delete_any_message_group` setting.
Fixes #30717.
2024-08-15 12:03:49 -07:00
Prakhar Pratyush 19d56f77b5 rocketchat: Fix "OSError: [Errno 36] File name too long" error.
Earlier, we were replacing too long attachment name with random uuid
when the character count of the file name was greater than 255.

This results in "OSError: [Errno 36] File name too long" error in
few cases when the file name has less than 255 characters but more
than 255 bytes (file name with Non-ASCII characters).

This commit updates the code to check the file name's byte size
instead of characters count.
2024-08-14 18:18:31 -07:00
Mateusz Mandera bf7fc5701f management: Add change_auth_backends command.
A utility command to enable or disable certain authentication backends
for a realm from the command line. Can be helpful e.g. if the
administrator accidentally disables some auth methods in the UI leaving
themselves with none remaining that they could actually use to log back
into the organization.

Example usage:
```
(zulip-py3-venv) vagrant@c32c137f59a0:/srv/zulip$ ./manage.py change_auth_backends -r zulip --show
Current authentication backends for the realm:
Enabled backends:
  Dev
  Email
  GitHub
  GitLab
  Google
  Apple
  SAML
  OpenID Connect
(zulip-py3-venv) vagrant@c32c137f59a0:/srv/zulip$ ./manage.py change_auth_backends -r zulip --disable GitHub
Disabling GitHub backend for realm Zulip Dev
Updated authentication backends for the realm:
Enabled backends:
  Dev
  Email
  GitLab
  Google
  Apple
  SAML
  OpenID Connect
Disabled backends:
  GitHub
Done!
(zulip-py3-venv) vagrant@c32c137f59a0:/srv/zulip$ ./manage.py change_auth_backends -r zulip --enable GitHub
Enabling GitHub backend for realm Zulip Dev
Updated authentication backends for the realm:
Enabled backends:
  Dev
  Email
  GitHub
  GitLab
  Google
  Apple
  SAML
  OpenID Connect
Done!
```
2024-08-14 14:04:46 -07:00
Alex Vandiver 2bfdccd065 narrow: Do not error when given an empty DM recipient list. 2024-08-14 10:39:40 -07:00
Vector73 fa408a969e api-docs: Handle multiple examples in `responses`. 2024-08-14 09:37:15 -07:00
Tim Abbott 07c927ae88 migrations: Fix Nagios bot cleanup crash.
We didn't correctly check for the possibility that these users don't
exist.
2024-08-13 23:30:32 -07:00
Shubham Padia 8b96aa277a css: Use classname instead of `div` selector for .blocks.
Having the `div` selector there affects the performance. See
https://chat.zulip.org/#narrow/stream/6-frontend/topic/CSS.20selector.20performance/near/1845719
for more details.
2024-08-13 22:36:35 -07:00
Alex Vandiver 16dffeb941 migrations: Fix missing Literal() calls in 0564. 2024-08-13 08:04:13 -07:00
Prakhar Pratyush e7f2a0958a custom_profile_fields: Send event on commit in notify_realm...fields.
Earlier, we were using 'send_event' in
'notify_realm_custom_profile_fields' which can lead to a situation,
if any db operation is added after the 'send_event' in the action
functions using it, where we enqueue event but the action function
fails at a later stage.

Events should not be sent until we know we're not rolling back.

Fixes part of #30489.
2024-08-12 12:16:14 -07:00
Prakhar Pratyush bae6188d92 user_groups: Add transaction.atomic decorator to add_user_group view.
The database operations in 'access_user_group_for_setting' and
'check_add_user_group' used in 'add_user_group' view should be
collectively atomic.

This commit adds transaction.atomic decorator for that purpose.
2024-08-12 12:16:14 -07:00
Prakhar Pratyush 427c59ec68 user_groups: Update delete_user_group codepath to send event on commit.
Earlier, we were using 'send_event' in 'delete_user_grou' codepath
which can lead to a situation, if any db operation is added after
the 'send_event' in future, where we enqueue events but the
function fails at a later stage.

Events should not be sent until we know we're not rolling back.

Fixes part of #30489.
2024-08-12 12:16:14 -07:00
Prakhar Pratyush 66ff646e99 user_groups: Update edit_user_group codepath to send event on commit.
Earlier, we were using 'send_event' in 'edit_user_group' codepath
which can lead to a situation where we enqueue events but the
function fails at a later stage.

Events should not be sent until we know we're not rolling back.

Fixes part of #30489.
2024-08-12 12:16:14 -07:00
Prakhar Pratyush 69a2775e9b video_calls: Update do_set_zoom_token to send event on commit.
Earlier, we were using 'send_event' in 'do_set_zoom_token' which
can lead to a situation, if any db operation is added after the
'send_event' in future, where we enqueue events but the action
function fails at a later stage.

Events should not be sent until we know we're not rolling back.

Fixes part of #30489.
2024-08-12 12:16:14 -07:00
Prakhar Pratyush 0fddda232d drafts: Update do_delete_draft to send event on commit.
Earlier, we were using 'send_event' in 'do_delete_draft' which
can lead to a situation, if any db operation is added after the
'send_event' in future, where we enqueue events but the action
function fails at a later stage.

Events should not be sent until we know we're not rolling back.

Fixes part of #30489.
2024-08-12 12:16:14 -07:00
Prakhar Pratyush 096647fbd5 drafts: Update do_edit_draft to send event on commit.
Earlier, we were using 'send_event' in 'do_edit_draft' which
can lead to a situation, if any db operation is added after the
'send_event' in future, where we enqueue events but the action
function fails at a later stage.

Events should not be sent until we know we're not rolling back.

Fixes part of #30489.
2024-08-12 12:16:14 -07:00