Commit Graph

18856 Commits

Author SHA1 Message Date
Tim Abbott afecb2eca6 migrations: Add missing elidable tags on RunPython/RunSQL steps.
This helps the squashmigrations tool know that it can squash these.
2024-08-23 17:15:35 -07:00
Tim Abbott 5188cf3d8e migrations: Remove RunPython block from initial migration.
Since it's the initial migration, this can't have any useful effect.

I'm pretty sure the backstory is we did a manual squash of migrations
during the process of open-sourcing Zulip, and incorrectly didn't
remove this code.
2024-08-23 17:15:35 -07:00
Tim Abbott 547377a295 zerver: Remove now-unnecessary dependency in migration 0383.
It's not possible to directly upgrade from pre-5.3 versions to main,
since they do not have any supported OSes in common. Thus, this
dependency on the confirmation model, which risks creating a circular
dependency if we squash migrations, can be removed.
2024-08-23 17:15:35 -07:00
Alex Vandiver 73e5364838 import: Use inspection to determine sequence names. 2024-08-23 17:15:35 -07:00
Tim Abbott 31623911d1 test_message_fetch: Avoid assumptions about fixture data set. 2024-08-23 17:15:35 -07:00
Lauryn Menard d708c3c039 support: Add ability to update max daily invitations for realm.
Adds some validation for changing the realm's max invites via the
support view so that it is not set below the default max for the
realm's plan type, and so that if it's currently set to the default
max it's not reset to that same value.
2024-08-23 16:08:30 -07:00
afeefuddin f013eb1037 message_fetch: Convert module to TypeScript. 2024-08-23 13:34:13 -07:00
roanster007 c6a06d4684 direct_message_group: Add new `group_size` field.
This commit adds a new `group_size` field to the `DirectMessageGroup`
model, and backfills its value to each of the existing direct message
groups.

Fixes part of #25713
2024-08-23 11:09:41 -07:00
Prakhar Pratyush 3ea6ba97c0 users: Send event on commit in do_change_is_billing_admin.
Earlier, we were using 'send_event' in 'do_change_is_billing_admin'
which can lead to a situation, if any db operation is added after
the 'send_event' in future, where we enqueue events but the action
function fails at a later stage.

Events should not be sent until we know we're not rolling back.

Fixes part of #30489.
2024-08-23 09:10:56 -07:00
Prakhar Pratyush 2b6c5260a5 users: Send event on commit in do_update_outgoing_webhook_service.
Earlier, we were using 'send_event' in
'do_update_outgoing_webhook_service' which can lead to a
situation, if any db operation is added after the 'send_event'
in future, where we enqueue events but the action function fails
at a later stage.

Events should not be sent until we know we're not rolling back.

Fixes part of #30489.
2024-08-23 09:10:56 -07:00
Prakhar Pratyush 30edd1a3b0 users: Add transaction.atomic decorator to do_update_bot_config_data.
The db operations in do_update_bot_config_data should be atomic.
2024-08-23 09:10:56 -07:00
Prakhar Pratyush 8f2727fcea realm_settings: Send event on commit in do_set_push_notif...timestamp.
Earlier, we were using 'send_event' in
'do_set_push_notifications_enabled_end_timestamp' which
can lead to a situation, if any db operation is added after the
'send_event' in future, where we enqueue events but the action
function fails at a later stage.

Events should not be sent until we know we're not rolling back.

Fixes part of #30489.
2024-08-23 09:10:56 -07:00
Prakhar Pratyush 5af6c04a9e realm_settings: Update do_set_realm_stream to send event on commit.
Earlier, we were using 'send_event' in 'do_set_realm_stream' which
can lead to a situation, if any db operation is added after the
'send_event' in future, where we enqueue events but the action
function fails at a later stage.

Events should not be sent until we know we're not rolling back.

Fixes part of #30489.
2024-08-23 09:10:56 -07:00
Prakhar Pratyush c0dc005d05 realm_settings: Send event on commit in do_set...authentication_methods.
Earlier, we were using 'send_event' in
'do_set_realm_authentication_methods' which can lead to a situation
where we enqueue events but there's an error at a later stage in
the codepath using this function.

Events should not be sent until we know we're not rolling back.

Fixes part of #30489.
2024-08-23 09:10:56 -07:00
Prakhar Pratyush 63064e926b realm_settings: Update do_set...default_setting to send event on commit.
Earlier, we were using 'send_event' in do_set_realm_user_default_setting
which can lead to a situation, if any db operation is added after
the 'send_event' in future, where we enqueue events but the action
function fails at a later stage.

Events should not be sent until we know we're not rolling back.

Fixes part of #30489.
2024-08-23 09:10:56 -07:00
Prakhar Pratyush 8ddaa949fc user_settings: Send event on commit in do_regenerate_api_key.
Earlier, we were using 'send_event' in 'do_regenerate_api_key'
which can lead to a situation, if any db operation is added after
the 'send_event' in future, where we enqueue events but the action
function fails at a later stage.

Events should not be sent until we know we're not rolling back.

Fixes part of #30489.
2024-08-23 09:10:56 -07:00
Prakhar Pratyush 7e03569495 user_settings: Send event on commit in do_change_full_name.
Earlier, we were using 'send_event' in 'do_change_full_name'
which can lead to a situation, if any db operation is added after
the 'send_event' in future, where we enqueue events but the action
function fails at a later stage.

Events should not be sent until we know we're not rolling back.

Fixes part of #30489.
2024-08-23 09:10:56 -07:00
Mateusz Mandera 9cad9644e7 import_util: Fix file extensions of emoji files downloaded from Slack.
The Slack API when returning the emoji records, returns the record for
its thumbsup_all emoji with the url ending with .png, even though the
file is a gif.

For that reason, we have to make that code correct file extensions based
on the response content-type. Emojis are the smallest set of images to
download, so for simplicity of implementation, we remove the
parallelization of the downloads in favor of just processing them
serially.
2024-08-21 16:30:19 -07:00
Mateusz Mandera 669e0a3e47 import: Fix export->import of emojis from Slack.
Ideally this would besplit up into two commits, but it's hard to split
into self-contained, atomic chunks now that this segment of the
import/export system is generally kind of broken after thumbnailing
system changes.

1. 3rd party export converters don't make .original image files.
Insteadthey provide a single file, which the import should treat as if
it's .original.

2. 3rd party converters create all the records with is_animated=False.
That's an issue, because without setting that correctly on the
RealmEmoji objects, Zulip doesn't know that it should use the "still"
thumbnail when the emoji is being used in a user's status. Which leads
to incorrectly displaying the user status with the distracting
animation.
2024-08-21 16:30:19 -07:00
Mateusz Mandera 5476340b52 import: Export and import .original emoji files correctly.
The export tool was only exporting the already-thumbnailed emoji file,
omitting the original one. Now we make sure to export the .original file
too, like we do for avatars, and make the import tool process it
directly, to thumbnail it directly and generate a still in the case of
animated emojis.

Otherwise, the imported realm wouldn't have the <emoji>.png.original
file that we generally expect to have accessible, and stills for
animated emojis were completely missing.
2024-08-21 16:30:19 -07:00
Tim Abbott f1d828cb7d test_openapi: Remove now-unused blocks.
As we remove uses of the has_request_variables system in favor of
typed_endpoints, parts of this file have lost test coverage.
2024-08-21 11:33:42 -07:00
Tim Abbott 9348b87d73 topic: Remove now-unused REQ_topic. 2024-08-21 11:27:54 -07:00
Kenneth Rodrigues 2483e600a2 message_send: Convert to typed endpoint.
Convert `message_send.py` use `typed endpoint`.

Disable `message_send` endpoint `to` parameter in the `openapi`
`validate_json_schema` check, because it is a special case where the
content type of the parameter is application/json but the
parameter may or may not be JSON encoded since previously we also
accepted a raw string and some ad-hoc bot might still depend on sending
a raw string.

Remove unused validators from `validator.py`.
2024-08-21 11:13:00 -07:00
Prakhar Pratyush 8b3d5a9455 custom_profile_fields: Update transaction to be durable.
This commit updates the db transaction to be durable for
do_update_user_custom_profile_data_if_changed to avoid
addition of any outer atomic block.

While adding any outer atomic block this will raise a runtime error
and we can replace the durable argument with 'savepoint=False'
otherwise we'll have to manually track down the action functions
getting called in that outer atomic block and set the savepoint=False
otherwise it'll lead to creation of savepoints which we don't want.

We can't set savepoint=False before hand to the outermost action
function because it leads to rollback of transaction in tests when
an error is raised in action function.
2024-08-21 10:51:23 -07:00
Prakhar Pratyush e729c51485 custom_profile_fields: Update remove...value to send event on commit.
Earlier, we were using 'send_event' in
check_remove_custom_profile_field_value which can lead to a
situation where we enqueue events but the function fails at a
later stage.

Events should not be sent until we know we're not rolling back.

Fixes part of #30489.
2024-08-21 10:51:23 -07:00
Prakhar Pratyush bfe7eb6b2c do_change_full_name: Noop if value isn't really changing.
In do_change_full_name, we shouldn't create RealmAuditLog
entry and sends events if the value isn't changing.
2024-08-21 10:51:23 -07:00
Mateusz Mandera 7e1f468f04 saml: Fix exception when syncing missing value to custom profile field.
There was a bug here that would trigger an exception inside
`sync_user_profile_custom_fields`, causing it to get logged with
logging.warning, when an attribute configured for SAML custom profile
field sync was missing from a SAMLResponse or had an empty value.
`sync_user_profile_custom_fields` expects valid values, and None is not
valid.

We could consider a slightly different behavior here instead - when an
attribute is sent with no value in the SAMLResponse, that means the attr
has no value in the IdP's user directory - so perhaps a better behavior
would be to also remove the custom profile field value in Zulip. However
there are two issues with that:

1. It's not necessarily the best behavior, because an organization might
want the "user doesn't have this attribute set at the IdP level" state
to just mean that the user should be free to set the value manually in
Zulip if they wish. And having that value get reset on every login would
then be an issue. The implementation in this commit is consistent with
this philosophy.

2. There's some implementation difficulty - upstream
`self.get_attr(...)`, which we use for reading the attr value from the
SAMLResponse, doesn't distinguish between an attribute being sent with
no value and the attribute not being sent at all - in both cases it
returns None. So we'd need some extra work here with parsing the
SAMLResponse properly, to be able to know when the custom profile field
should get cleared.
2024-08-20 13:57:30 -07:00
Mateusz Mandera 833dce8a13 saml: Add support for syncing user role.
Replace the SOCIAL_AUTH_SYNC_CUSTOM_ATTRS_DICT with
SOCIAL_AUTH_SYNC_ATTRS_DICT, designed to support also regular user attrs
like role or full name (in the future).

Custom attributes can stay configured as they were and will get merged
into SOCIAL_AUTH_SYNC_ATTRS_DICT in computed_settings, or can be
specified in SOCIAL_AUTH_SYNC_ATTRS_DICT directly with "custom__"
prefix.

The role sync is plumbed through to user creation, so users can
immediately be created with their intended role as provided by the IdP
when they're creating their account, even when doing this flow without
an invitiation.
2024-08-20 11:53:24 -07:00
Mateusz Mandera 8c1a1ea8db scim: Extract ROLE_TYPE_TO_NAME dict to UserProfile.
This allows these mappings to used in other APIs. Specifically, we want
to use this for syncing role during SAML auth.
2024-08-20 11:53:24 -07:00
Mateusz Mandera 76b41e433a maybe_send_to_registration: Enforce kwargs.
Most of these should be kwargs, to make sure we're not accidentally
getting the order wrong when calling it.
2024-08-20 11:53:24 -07:00
Kenneth Rodrigues 86bf0d3ea3 decorators: Migrate to typed_endpoint.
Modify `semgrep-py.yml` to treat arguments after `*args` as
keyword-only arguments.
2024-08-20 10:15:30 -07:00
Kenneth Rodrigues 50712bfa81 scheduled_messages: Migrate to typed_endpoint.
Migrate `scheduled_message.py` to typed_endpoint.

Perform Json parsing in the endpoint itself instead of
in `recipient_parsing.py`.
2024-08-20 10:03:22 -07:00
Ujjawal Modi 5d613ce98d user_groups: Add support to update can_manage_group setting.
This commit adds API support to update can_manage_group setting
of a user group.
2024-08-20 09:28:05 -07:00
Ujjawal Modi b0b36e884c user_groups: Add support to set can_manage_group during creation.
This commit adds API support to set can_manage_group while
creating a user group.
2024-08-20 09:28:05 -07:00
Ujjawal Modi 2505813359 user_groups: Add can_manage_group field to UserGroup objects.
This commit make changes in code to include can_manage_group
field to UserGroup objects passed with response of various endpoints
including "/register" endpoint and also in the group object
send with user group creation event.
2024-08-20 09:28:05 -07:00
Ujjawal Modi 423d5c42f6 user_groups: Update code to check whether user can edit a user group.
Earlier there was only a realm level setting for configuring
who can edit user groups. A new group level setting is also added
for configuring who can manage that particular group.
Now, a user group can be edited by a user if it is allowed from
realm level setting or group level setting.

This commit make changes to also use group level setting
in determining whether a group can be edited by user or not.

Also, updated tests to use api_post and api_delete helpers instead
of using client_post and client_delete helpers with different users
being logged in.
2024-08-20 09:28:05 -07:00
Ujjawal Modi 03220ba456 user_groups: Add can_manage_group field in database.
This commit adds a new group level setting can_manage_group
for configuring who can manage a group. This commit only adds
the field in database and make changes to automatically create
single user groups corresponsing to acting user
which will be the default value for this setting.

Fixes part of #25928.
2024-08-20 09:28:05 -07:00
Ujjawal Modi 9e699dfc85 tests: Separate test of group edit setting for creation and deletion.
Earlier there was a single backend test for testing group edit policy
for creating and deleting user group.This commit make changes in the test
and now there are two separate tests for testing group edit policy for
creating and deleting user groups.

This was done because in future commits we will be adding a
realm level setting for configuring who can create user groups.

Also, updated tests to use api_post and api_delete helpers instead
of using client_post and client_delete helpers with different users
being logged in.
2024-08-20 09:28:05 -07:00
Ujjawal Modi bb0b6900df user_groups: Add a decorator to check group creation permission.
Earlier there was a single decorator function to check whether
user can create and edit user groups. This commit adds a new
decorator function to check whether user has permissions to
create user groups.

This was done because in future commits we will be adding a
realm level setting for configuring who can create user groups.
2024-08-20 09:28:05 -07:00
Sahil Batra 3f800002be user_groups: Allow not passing old_setting_api_value. 2024-08-20 09:28:05 -07:00
evykassirer b1198785be message_edit: Convert module to typescript. 2024-08-19 16:33:04 -07:00
Sahil Batra 3dc82eed57 user_groups: Refactor code to compute user group objects.
This commit refactors code in user_groups_in_realm_serialized
such that we do not prefetch "can_mention_group__direct_members"
and "can_mention_group__direct_subgroups" using prefetch_related
and instead fetch members and subgroups for all groups in separate
queries and then use that data to find the members and subgroups
of the group used for that setting.

This change helps us in avoiding two prefetch queries for each
setting when we add more group settings.
2024-08-19 15:41:55 -07:00
Prakhar Pratyush 2406a3feea streams: Update do_change...retention_days to send event on commit.
Earlier, we were using 'send_event' in
do_change_stream_message_retention_days which can lead to a situation
where we enqueue events but the function fails at a later stage.

Events should not be sent until we know we're not rolling back.

Fixes part of #30489.
2024-08-19 10:14:49 -07:00
Prakhar Pratyush a9af1c0ce2 streams: Update do_change_stream_description to send event on commit.
Earlier, we were using 'send_event' in do_change_stream_description
which can lead to a situation where we enqueue events but the
function fails at a later stage.

Events should not be sent until we know we're not rolling back.

Fixes part of #30489.
2024-08-19 10:14:49 -07:00
Prakhar Pratyush e474ab7ee0 streams: Add transaction.atomic decorator to 'do_rename_stream'.
This commit updates the 'do_rename_stream' action function
to make the db operations atomic.

Also, send_event is replaced with send_event_on_commit.
2024-08-19 10:14:49 -07:00
Prakhar Pratyush 69dec9dbc6 streams: Update 'do_change_stream_post_policy' to send event on commit.
Earlier, we were using 'send_event' in do_change_stream_post_policy
which can lead to a situation where we enqueue events but the
function fails at a later stage.

Events should not be sent until we know we're not rolling back.

Fixes part of #30489.
2024-08-19 10:14:49 -07:00
Prakhar Pratyush 41d947b89d streams: Make db operations in do_change_subscription_property atomic.
This commit adds a transaction.atomic decorator to the
'do_change_subscription_property' function to make
the db operations in the action function atomic.

Also, send_event is changed to send_event_on_commit.
2024-08-19 10:14:49 -07:00
Prakhar Pratyush 5d0b3d1a6d muted_users: Update 'do_unmute_user' to send event on commit.
Earlier, we were using 'send_event' in 'do_unmute_user'
which can lead to a situation where we enqueue events but the
function fails at a later stage.

Events should not be sent until we know we're not rolling back.

Fixes part of #30489.
2024-08-19 10:14:49 -07:00
Prakhar Pratyush 3b9a21443a muted_users: Update 'do_mute_user' to send event on commit.
Earlier, we were using 'send_event' in 'do_mute_user' which
can lead to a situation where we enqueue events but the
function fails at a later stage.

Events should not be sent until we know we're not rolling back.

Fixes part of #30489.
2024-08-19 10:14:49 -07:00
Prakhar Pratyush 9026e6ecc1 message_flags: Update 'do_update_message_flags' to send event on commit.
Earlier, we were using 'send_event' in do_update_message_flags
which can lead to a situation where we enqueue events but the
function fails at a later stage.

Events should not be sent until we know we're not rolling back.

Fixes part of #30489.
2024-08-19 10:14:49 -07:00