Commit Graph

18967 Commits

Author SHA1 Message Date
Anders Kaseorg 850baaa368 migrations: Fix 0511 to backfill in SQL rather than Python loops.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-09-24 23:37:03 -07:00
Anders Kaseorg 184c0203f3 upload: Lazily import boto3.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-09-24 16:38:37 -07:00
Vector73 9e4e85e140 saved_snippets: Add backend for saved snippets.
Part of #31227.
2024-09-24 15:27:58 -07:00
Tim Abbott 90a4b4934a text_fixtures: Fix buggy skip-checks placement. 2024-09-24 15:00:46 -07:00
Prakhar Pratyush 65f465562f export_realm: Remove the 'react on consent message' approach.
For exporting full with consent:

* Earlier, a message advertising users to react with thumbs up
  was sent and later used to determine the users who consented.

* Now, we no longer need to send such a message. This commit
  updates the logic to use `allow_private_data_export` user-setting
  to determine users who consented.

Fixes part of #31201.
2024-09-24 14:32:42 -07:00
Prakhar Pratyush f67d7549de message: Add test coverage for 'set_visibility_policy_possible'.
This commit adds test coverage for the
'if user_profile.realm != message.get_realm():' block in
'set_visibility_policy_possible' function.

Currently, 'test_command_with_consented_message_id' adds the
test coverage for this code block. But we plan to remove that
test in the next commit as a part of removing 'react on consent
message' approach.

So, this commit explicitly adds code for the test coverage of
the concerned code block.
2024-09-24 14:32:42 -07:00
Tim Abbott 9d74abee0b tools: Skip checks in recursive management calls. 2024-09-24 14:26:34 -07:00
Aditya a534e9fdb6 integrations: Remove ignore_pull_requests from Travis CI webhook.
Removes the ignore_pull_requests URL parameter from the Travis CI
integration because its functionality is now fully covered by the
standard incoming webhook event filtering framework.

Fixes #30934.
2024-09-24 11:04:49 -07:00
tnmkr ddecba4e1c custom_profile_fields: Add "editable_by_user" setting.
This new property allows organization administrators to specify whether
users can modify the custom profile field value on their own account.
This property is configurable for individual fields.

By default, existing and newly created fields have this property set to
true, that is, they allow users to edit the value of the fields.

Fixes part of #22883.

Co-Authored-By: Ujjawal Modi <umodi2003@gmail.com>
2024-09-23 18:09:38 -07:00
Alex Vandiver ce0df00e44 export: Notify all realm admins on realm export. 2024-09-23 10:02:43 -07:00
Alex Vandiver 7afe6800f7 export: Use relative paths, include more data. 2024-09-23 10:02:43 -07:00
Aman Agrawal cbb73bc82f accounts_accept_terms: Check if imported user wants marketing emails. 2024-09-20 18:17:53 -07:00
Alex Vandiver e125ad823d exports: Add a separate bucket for realm exports.
This allows finer-grained access control and auditing.  The links
generated also expire after one week, and the suggested configuration
is that the underlying data does as well.

Co-authored-by: Prakhar Pratyush <prakhar@zulip.com>
2024-09-20 15:43:49 -07:00
Alex Vandiver c1e8ecd08f uploads: Cache boto client in the module and be writable.
The `get_signed_upload_url` code is called for every S3 file serve
request, and is thus in the hot path.  The boto3 client caching
optimization is thus potentially useful as a performance optimization.
2024-09-20 15:43:49 -07:00
Alex Vandiver 1a7b3ef7ed upload: Use get_export_tarball_url in upload_export_tarball. 2024-09-20 15:43:49 -07:00
Alex Vandiver 4cf835d9dd upload: Remove common cache from get_export_tarball_url.
This is not called in the hot path like get_avatar_url is.
2024-09-20 15:43:49 -07:00
Alex Vandiver a5bf452202 upload: Realm is not Optional in upload_export_tarball.
af4eb8c0d5 marked the base class and local backend as non-Optional,
but left the S3 backend as Optional for some reason.

Remove it.
2024-09-20 15:43:49 -07:00
Alex Vandiver 91ac5c3c8b export: Log before the compression step, which can be slow. 2024-09-20 15:43:49 -07:00
Tim Abbott 51d0dfb504 docs: Tweak some documentation around send_event. 2024-09-20 15:28:18 -07:00
Prakhar Pratyush e763d065a3 django_api: Rename 'send_event' to 'send_event_rollback_unsafe'.
This commit renames the 'send_event' function to
'send_event_rollback_unsafe' to reflect the fact that it doesn't
wait for the db transaction (within which it gets called, if any)
to commit and sends event irrespective of commit or rollback.

In most of the cases we don't want to send event in the case of
rollbacks, so the caller should be aware that calling the function
directly is rollback unsafe.
2024-09-20 15:20:18 -07:00
Alex Vandiver 86d5ee5bdf tus: Document new API level for TUS endpoints. 2024-09-20 12:16:13 -07:00
Lauryn Menard 4bd4534450 billing: Enforce manual license management for guest role changes.
Adds a check for changing an existing guest user's role before
calling do_update_user in the case that a realm has a current
paid plan with manual license management.
2024-09-20 12:02:39 -07:00
Lauryn Menard 7861c1ba63 billing: Enforce manual billing renewal licenses for new users.
In addition to checking for available licenses in the current
billing period when adding or inviting new non-guest users, for
manual billing, we also verify that the number of licenses set
for the next billing period will be enough when adding/inviting
new users.

Realms that are exempt from license number checks do not have
this restriction applied.

Admins are notified via group direct message when a user fails
to register due to this restriction.
2024-09-20 12:02:39 -07:00
Mateusz Mandera 18fbb5d146 retention: Limit number of ids passed to db in delete messages query.
If do_delete_messages (and friends) are called for a massive number of
messages, the giant list of message ids is passed to Postgres even
though chunk_size makes all but the first chunk_size of message ids
useless.
2024-09-20 09:31:21 -07:00
Mateusz Mandera ed7c330548 retention: Rename run_archiving_in_chunks to run_archiving. 2024-09-20 09:31:21 -07:00
Prakhar Pratyush 86194efe7b realm_export: Add a new endpoint to fetch private data export consents.
This commit adds a new endpoint `export/realm/consents` to
fetch the consents of users for their private data exports.

Fixes part of #31201.
2024-09-20 09:10:32 -07:00
Alex Vandiver e43eaaebf0 openapi: Update upload-file example response to add filename. 2024-09-20 09:07:21 -07:00
Alex Vandiver 88acfbca8c docs: Adjust for 100MB MAX_FILE_UPLOAD_SIZE. 2024-09-20 09:05:11 -07:00
Mateusz Mandera 1d7d3fae61 signup: Mirror dummy user should be registered with role from invite.
Aside of what's generally explained in the code comment, this is
motivated by the specific situation of import of Slack Connect channels.
These channels contain users who are "external collaborators" and
limited to a single channel in Slack. We don't have more sophisticated
handling of their import, which would map this concept 1-to-1 in Zulip -
but we create them as inactive dummy users, meaning they have to go
through signup before their account is usable.

The issue is that their imported UserProfile.role is set to Member and
when they register, the UserProfile gets reactivated with that role
unchanged. However, if e.g. the user is signing up after they received
an invitation from the admin, they should get the role that was
configured on the invite. In particular important if the user is meant
to still be "limited" and thus the admin invites them as a guest - they
definitely don't want the user to get a full Member account because of
this weird interaction between import and registration.
2024-09-19 15:26:27 -07:00
Alex Vandiver 818c30372f upload: Use tusd for resumable, larger uploads.
Currently, it handles two hook types: 'pre-create' (to verify that the
user is authenticated and the file size is within the limit) and
'pre-finish' (which creates an attachment row).

No secret is shared between Django and tusd for authentication of the
hooks endpoints, because none is necessary -- tusd forwards the
end-user's credentials, and the hook checks them like it would any
end-user request.  An end-user gaining access to the endpoint would be
able to do no more harm than via tusd or the normal file upload API.

Regardless, the previous commit has restricted access to the endpoint
at the nginx layer.

Co-authored-by: Brijmohan Siyag <brijsiyag@gmail.com>
2024-09-19 11:37:29 -07:00
Sahil Batra 5f3a8334be user_groups: Do not allow deleting user groups. 2024-09-18 13:41:13 -07:00
Sahil Batra c53563d0e7 user_groups: Handle deactivated groups in webapp. 2024-09-18 13:41:13 -07:00
Sahil Batra 5c790aac72 user_groups: Rename allow_deactivated in 'GET /user_groups'.
This commit renames "allow_deactivated" parameter in
"GET /user_groups" endpoint to "include_deactivated_groups", so
that we can have consistent naming here and for client capability
used for deciding whether to send deactivated groups in register
response and how to handle the related events.
2024-09-18 13:41:13 -07:00
Sahil Batra fb63c47ea6 user_groups: Add client capability to handle deactivated groups.
This commit adds a client capability to not receive data about
deactivated groups.
2024-09-18 13:41:13 -07:00
Sahil Batra aa123b38b4 tests: Test can_manage_group setting for deactivating groups.
There were no tests for testing group deactivation permission
based on can_manage_group setting.
2024-09-18 13:41:13 -07:00
Sahil Batra 7a6135371e settings: Handle guests separately for group-based settings.
This commit adds code to handle guests separately for group
based settings, where guest will only have permission if
that particular setting can be set to "role:everyone" group
even if the guest user is part of the group which is used
for that setting. This is to make sure that guests do not
get permissions for actions that we generally do not want
guests to have.

Currently the guests do not have permission for most of them
except for "Who can delete any message", where guest could
delete a message if the setting was set to a user defined
group with guest being its member. But this commit still
update the code to use the new function for all the settings
as we want to have a consistent pattern of how to check whether
a user has permission for group-based settings.
2024-09-18 11:51:11 -07:00
Alex Vandiver 9a1f78db22 thumbnail: Support checking for images from streaming sources.
We may not always have trivial access to all of the bytes of the
uploaded file -- for instance, if the file was uploaded previously, or
by some other process.  Downloading the entire image in order to check
its headers is an inefficient use of time and bandwidth.

Adjust `maybe_thumbnail` and dependencies to potentially take a
`pyvips.Source` which supports streaming data from S3 or disk.  This
allows making the ImageAttachment row, if deemed appropriate, based on
only a few KB of data, and not the entire image.
2024-09-17 12:51:30 -07:00
Alex Vandiver ef21dd9b99 thumbnail: Set a stable ordering on ImageAttachment rows for locking.
Failure to have a stable ordering can lead to deadlocks.
2024-09-17 09:14:52 -07:00
Prakhar Pratyush 764083d31b settings: Add 'allow_private_data_export' user setting.
This commit adds a user-setting to allow users to decide
whether to let administrators export their private data.

Fixes part of #31201.
2024-09-16 15:48:40 -07:00
Lauryn Menard bc8993f159 user-groups: Add helper for date_created comparison. 2024-09-16 11:42:55 -07:00
Johan Nilsson bf9cbe30a7 integrations: Add support for release events to GitLab integration. 2024-09-16 09:26:20 -07:00
sujal shah 614caf111e user_groups: Add `creator` and `date_created` field in user groups.
This commit introduced 'creator' and 'date_created'
fields in user groups, allowing users to view who
created the groups and when.

Both fields can be null for groups without creator data.
2024-09-13 18:44:58 -07:00
Tim Abbott cb53c60563 models: Clean up CommonMessagePolicyEnum.
This is no longer used outside an old migration.
2024-09-12 09:36:02 -07:00
Vector73 0cd51f2fea settings: Remove `delete_own_message_policy` realm property.
Removed `delete_own_message_policy` realm property as the permission
to delete own messages is controlled by `can_delete_own_message_group`
setting.
2024-09-12 09:36:02 -07:00
Vector73 28c7a04734 settings: Add `can_delete_own_message_group` realm setting.
Added `can_delete_message_group` realm setting to replace
`delete_own_message_policy` property.
2024-09-12 09:36:02 -07:00
Tim Abbott 2585d209c0 user_groups: Rename misleadingly named view function. 2024-09-11 09:43:56 -07:00
Sahil Batra 688c5ad0af groups: Allow excluding deactivated groups in 'GET /user_groups' response. 2024-09-11 09:43:56 -07:00
Sahil Batra fef2925ff0 mention: Handle mentioning deactivated user groups.
If someone tries to mention a deactivated user group,
then it is automatically converted to a silent mention
like we do for deactivated users.
2024-09-11 09:43:56 -07:00
Sahil Batra c1d6da0a5c user_groups: Update handling of deactivated groups.
We only allow updating name of a deactivated group, and not
allow updating description, members, subgroups and any setting
of a deactivated user group.

Deactivated user groups cannot be a a subgroup of any group
or used as a setting for a group.
2024-09-11 09:43:56 -07:00
Sahil Batra e1cfe61452 user_groups: Add API support for deactivating user groups. 2024-09-11 09:43:56 -07:00