Commit Graph

19057 Commits

Author SHA1 Message Date
Tim Abbott 7080c94245 migrations: Reorder initial zerver migration.
This has no functional changes, but it helps the squashmigrations tool
realize some squashing opportunities to not have models declared
before other models that they will gain a foreign key to.
2024-08-23 17:15:35 -07:00
Tim Abbott afecb2eca6 migrations: Add missing elidable tags on RunPython/RunSQL steps.
This helps the squashmigrations tool know that it can squash these.
2024-08-23 17:15:35 -07:00
Tim Abbott 5188cf3d8e migrations: Remove RunPython block from initial migration.
Since it's the initial migration, this can't have any useful effect.

I'm pretty sure the backstory is we did a manual squash of migrations
during the process of open-sourcing Zulip, and incorrectly didn't
remove this code.
2024-08-23 17:15:35 -07:00
Tim Abbott 547377a295 zerver: Remove now-unnecessary dependency in migration 0383.
It's not possible to directly upgrade from pre-5.3 versions to main,
since they do not have any supported OSes in common. Thus, this
dependency on the confirmation model, which risks creating a circular
dependency if we squash migrations, can be removed.
2024-08-23 17:15:35 -07:00
Alex Vandiver 73e5364838 import: Use inspection to determine sequence names. 2024-08-23 17:15:35 -07:00
Tim Abbott 31623911d1 test_message_fetch: Avoid assumptions about fixture data set. 2024-08-23 17:15:35 -07:00
Lauryn Menard d708c3c039 support: Add ability to update max daily invitations for realm.
Adds some validation for changing the realm's max invites via the
support view so that it is not set below the default max for the
realm's plan type, and so that if it's currently set to the default
max it's not reset to that same value.
2024-08-23 16:08:30 -07:00
afeefuddin f013eb1037 message_fetch: Convert module to TypeScript. 2024-08-23 13:34:13 -07:00
roanster007 c6a06d4684 direct_message_group: Add new `group_size` field.
This commit adds a new `group_size` field to the `DirectMessageGroup`
model, and backfills its value to each of the existing direct message
groups.

Fixes part of #25713
2024-08-23 11:09:41 -07:00
Prakhar Pratyush 3ea6ba97c0 users: Send event on commit in do_change_is_billing_admin.
Earlier, we were using 'send_event' in 'do_change_is_billing_admin'
which can lead to a situation, if any db operation is added after
the 'send_event' in future, where we enqueue events but the action
function fails at a later stage.

Events should not be sent until we know we're not rolling back.

Fixes part of #30489.
2024-08-23 09:10:56 -07:00
Prakhar Pratyush 2b6c5260a5 users: Send event on commit in do_update_outgoing_webhook_service.
Earlier, we were using 'send_event' in
'do_update_outgoing_webhook_service' which can lead to a
situation, if any db operation is added after the 'send_event'
in future, where we enqueue events but the action function fails
at a later stage.

Events should not be sent until we know we're not rolling back.

Fixes part of #30489.
2024-08-23 09:10:56 -07:00
Prakhar Pratyush 30edd1a3b0 users: Add transaction.atomic decorator to do_update_bot_config_data.
The db operations in do_update_bot_config_data should be atomic.
2024-08-23 09:10:56 -07:00
Prakhar Pratyush 8f2727fcea realm_settings: Send event on commit in do_set_push_notif...timestamp.
Earlier, we were using 'send_event' in
'do_set_push_notifications_enabled_end_timestamp' which
can lead to a situation, if any db operation is added after the
'send_event' in future, where we enqueue events but the action
function fails at a later stage.

Events should not be sent until we know we're not rolling back.

Fixes part of #30489.
2024-08-23 09:10:56 -07:00
Prakhar Pratyush 5af6c04a9e realm_settings: Update do_set_realm_stream to send event on commit.
Earlier, we were using 'send_event' in 'do_set_realm_stream' which
can lead to a situation, if any db operation is added after the
'send_event' in future, where we enqueue events but the action
function fails at a later stage.

Events should not be sent until we know we're not rolling back.

Fixes part of #30489.
2024-08-23 09:10:56 -07:00
Prakhar Pratyush c0dc005d05 realm_settings: Send event on commit in do_set...authentication_methods.
Earlier, we were using 'send_event' in
'do_set_realm_authentication_methods' which can lead to a situation
where we enqueue events but there's an error at a later stage in
the codepath using this function.

Events should not be sent until we know we're not rolling back.

Fixes part of #30489.
2024-08-23 09:10:56 -07:00
Prakhar Pratyush 63064e926b realm_settings: Update do_set...default_setting to send event on commit.
Earlier, we were using 'send_event' in do_set_realm_user_default_setting
which can lead to a situation, if any db operation is added after
the 'send_event' in future, where we enqueue events but the action
function fails at a later stage.

Events should not be sent until we know we're not rolling back.

Fixes part of #30489.
2024-08-23 09:10:56 -07:00
Prakhar Pratyush 8ddaa949fc user_settings: Send event on commit in do_regenerate_api_key.
Earlier, we were using 'send_event' in 'do_regenerate_api_key'
which can lead to a situation, if any db operation is added after
the 'send_event' in future, where we enqueue events but the action
function fails at a later stage.

Events should not be sent until we know we're not rolling back.

Fixes part of #30489.
2024-08-23 09:10:56 -07:00
Prakhar Pratyush 7e03569495 user_settings: Send event on commit in do_change_full_name.
Earlier, we were using 'send_event' in 'do_change_full_name'
which can lead to a situation, if any db operation is added after
the 'send_event' in future, where we enqueue events but the action
function fails at a later stage.

Events should not be sent until we know we're not rolling back.

Fixes part of #30489.
2024-08-23 09:10:56 -07:00
Mateusz Mandera 9cad9644e7 import_util: Fix file extensions of emoji files downloaded from Slack.
The Slack API when returning the emoji records, returns the record for
its thumbsup_all emoji with the url ending with .png, even though the
file is a gif.

For that reason, we have to make that code correct file extensions based
on the response content-type. Emojis are the smallest set of images to
download, so for simplicity of implementation, we remove the
parallelization of the downloads in favor of just processing them
serially.
2024-08-21 16:30:19 -07:00
Mateusz Mandera 669e0a3e47 import: Fix export->import of emojis from Slack.
Ideally this would besplit up into two commits, but it's hard to split
into self-contained, atomic chunks now that this segment of the
import/export system is generally kind of broken after thumbnailing
system changes.

1. 3rd party export converters don't make .original image files.
Insteadthey provide a single file, which the import should treat as if
it's .original.

2. 3rd party converters create all the records with is_animated=False.
That's an issue, because without setting that correctly on the
RealmEmoji objects, Zulip doesn't know that it should use the "still"
thumbnail when the emoji is being used in a user's status. Which leads
to incorrectly displaying the user status with the distracting
animation.
2024-08-21 16:30:19 -07:00
Mateusz Mandera 5476340b52 import: Export and import .original emoji files correctly.
The export tool was only exporting the already-thumbnailed emoji file,
omitting the original one. Now we make sure to export the .original file
too, like we do for avatars, and make the import tool process it
directly, to thumbnail it directly and generate a still in the case of
animated emojis.

Otherwise, the imported realm wouldn't have the <emoji>.png.original
file that we generally expect to have accessible, and stills for
animated emojis were completely missing.
2024-08-21 16:30:19 -07:00
Tim Abbott f1d828cb7d test_openapi: Remove now-unused blocks.
As we remove uses of the has_request_variables system in favor of
typed_endpoints, parts of this file have lost test coverage.
2024-08-21 11:33:42 -07:00
Tim Abbott 9348b87d73 topic: Remove now-unused REQ_topic. 2024-08-21 11:27:54 -07:00
Kenneth Rodrigues 2483e600a2 message_send: Convert to typed endpoint.
Convert `message_send.py` use `typed endpoint`.

Disable `message_send` endpoint `to` parameter in the `openapi`
`validate_json_schema` check, because it is a special case where the
content type of the parameter is application/json but the
parameter may or may not be JSON encoded since previously we also
accepted a raw string and some ad-hoc bot might still depend on sending
a raw string.

Remove unused validators from `validator.py`.
2024-08-21 11:13:00 -07:00
Prakhar Pratyush 8b3d5a9455 custom_profile_fields: Update transaction to be durable.
This commit updates the db transaction to be durable for
do_update_user_custom_profile_data_if_changed to avoid
addition of any outer atomic block.

While adding any outer atomic block this will raise a runtime error
and we can replace the durable argument with 'savepoint=False'
otherwise we'll have to manually track down the action functions
getting called in that outer atomic block and set the savepoint=False
otherwise it'll lead to creation of savepoints which we don't want.

We can't set savepoint=False before hand to the outermost action
function because it leads to rollback of transaction in tests when
an error is raised in action function.
2024-08-21 10:51:23 -07:00
Prakhar Pratyush e729c51485 custom_profile_fields: Update remove...value to send event on commit.
Earlier, we were using 'send_event' in
check_remove_custom_profile_field_value which can lead to a
situation where we enqueue events but the function fails at a
later stage.

Events should not be sent until we know we're not rolling back.

Fixes part of #30489.
2024-08-21 10:51:23 -07:00
Prakhar Pratyush bfe7eb6b2c do_change_full_name: Noop if value isn't really changing.
In do_change_full_name, we shouldn't create RealmAuditLog
entry and sends events if the value isn't changing.
2024-08-21 10:51:23 -07:00
Mateusz Mandera 7e1f468f04 saml: Fix exception when syncing missing value to custom profile field.
There was a bug here that would trigger an exception inside
`sync_user_profile_custom_fields`, causing it to get logged with
logging.warning, when an attribute configured for SAML custom profile
field sync was missing from a SAMLResponse or had an empty value.
`sync_user_profile_custom_fields` expects valid values, and None is not
valid.

We could consider a slightly different behavior here instead - when an
attribute is sent with no value in the SAMLResponse, that means the attr
has no value in the IdP's user directory - so perhaps a better behavior
would be to also remove the custom profile field value in Zulip. However
there are two issues with that:

1. It's not necessarily the best behavior, because an organization might
want the "user doesn't have this attribute set at the IdP level" state
to just mean that the user should be free to set the value manually in
Zulip if they wish. And having that value get reset on every login would
then be an issue. The implementation in this commit is consistent with
this philosophy.

2. There's some implementation difficulty - upstream
`self.get_attr(...)`, which we use for reading the attr value from the
SAMLResponse, doesn't distinguish between an attribute being sent with
no value and the attribute not being sent at all - in both cases it
returns None. So we'd need some extra work here with parsing the
SAMLResponse properly, to be able to know when the custom profile field
should get cleared.
2024-08-20 13:57:30 -07:00
Mateusz Mandera 833dce8a13 saml: Add support for syncing user role.
Replace the SOCIAL_AUTH_SYNC_CUSTOM_ATTRS_DICT with
SOCIAL_AUTH_SYNC_ATTRS_DICT, designed to support also regular user attrs
like role or full name (in the future).

Custom attributes can stay configured as they were and will get merged
into SOCIAL_AUTH_SYNC_ATTRS_DICT in computed_settings, or can be
specified in SOCIAL_AUTH_SYNC_ATTRS_DICT directly with "custom__"
prefix.

The role sync is plumbed through to user creation, so users can
immediately be created with their intended role as provided by the IdP
when they're creating their account, even when doing this flow without
an invitiation.
2024-08-20 11:53:24 -07:00
Mateusz Mandera 8c1a1ea8db scim: Extract ROLE_TYPE_TO_NAME dict to UserProfile.
This allows these mappings to used in other APIs. Specifically, we want
to use this for syncing role during SAML auth.
2024-08-20 11:53:24 -07:00
Mateusz Mandera 76b41e433a maybe_send_to_registration: Enforce kwargs.
Most of these should be kwargs, to make sure we're not accidentally
getting the order wrong when calling it.
2024-08-20 11:53:24 -07:00
Kenneth Rodrigues 86bf0d3ea3 decorators: Migrate to typed_endpoint.
Modify `semgrep-py.yml` to treat arguments after `*args` as
keyword-only arguments.
2024-08-20 10:15:30 -07:00
Kenneth Rodrigues 50712bfa81 scheduled_messages: Migrate to typed_endpoint.
Migrate `scheduled_message.py` to typed_endpoint.

Perform Json parsing in the endpoint itself instead of
in `recipient_parsing.py`.
2024-08-20 10:03:22 -07:00
Ujjawal Modi 5d613ce98d user_groups: Add support to update can_manage_group setting.
This commit adds API support to update can_manage_group setting
of a user group.
2024-08-20 09:28:05 -07:00
Ujjawal Modi b0b36e884c user_groups: Add support to set can_manage_group during creation.
This commit adds API support to set can_manage_group while
creating a user group.
2024-08-20 09:28:05 -07:00
Ujjawal Modi 2505813359 user_groups: Add can_manage_group field to UserGroup objects.
This commit make changes in code to include can_manage_group
field to UserGroup objects passed with response of various endpoints
including "/register" endpoint and also in the group object
send with user group creation event.
2024-08-20 09:28:05 -07:00
Ujjawal Modi 423d5c42f6 user_groups: Update code to check whether user can edit a user group.
Earlier there was only a realm level setting for configuring
who can edit user groups. A new group level setting is also added
for configuring who can manage that particular group.
Now, a user group can be edited by a user if it is allowed from
realm level setting or group level setting.

This commit make changes to also use group level setting
in determining whether a group can be edited by user or not.

Also, updated tests to use api_post and api_delete helpers instead
of using client_post and client_delete helpers with different users
being logged in.
2024-08-20 09:28:05 -07:00
Ujjawal Modi 03220ba456 user_groups: Add can_manage_group field in database.
This commit adds a new group level setting can_manage_group
for configuring who can manage a group. This commit only adds
the field in database and make changes to automatically create
single user groups corresponsing to acting user
which will be the default value for this setting.

Fixes part of #25928.
2024-08-20 09:28:05 -07:00
Ujjawal Modi 9e699dfc85 tests: Separate test of group edit setting for creation and deletion.
Earlier there was a single backend test for testing group edit policy
for creating and deleting user group.This commit make changes in the test
and now there are two separate tests for testing group edit policy for
creating and deleting user groups.

This was done because in future commits we will be adding a
realm level setting for configuring who can create user groups.

Also, updated tests to use api_post and api_delete helpers instead
of using client_post and client_delete helpers with different users
being logged in.
2024-08-20 09:28:05 -07:00
Ujjawal Modi bb0b6900df user_groups: Add a decorator to check group creation permission.
Earlier there was a single decorator function to check whether
user can create and edit user groups. This commit adds a new
decorator function to check whether user has permissions to
create user groups.

This was done because in future commits we will be adding a
realm level setting for configuring who can create user groups.
2024-08-20 09:28:05 -07:00
Sahil Batra 3f800002be user_groups: Allow not passing old_setting_api_value. 2024-08-20 09:28:05 -07:00
evykassirer b1198785be message_edit: Convert module to typescript. 2024-08-19 16:33:04 -07:00
Sahil Batra 3dc82eed57 user_groups: Refactor code to compute user group objects.
This commit refactors code in user_groups_in_realm_serialized
such that we do not prefetch "can_mention_group__direct_members"
and "can_mention_group__direct_subgroups" using prefetch_related
and instead fetch members and subgroups for all groups in separate
queries and then use that data to find the members and subgroups
of the group used for that setting.

This change helps us in avoiding two prefetch queries for each
setting when we add more group settings.
2024-08-19 15:41:55 -07:00
Prakhar Pratyush 2406a3feea streams: Update do_change...retention_days to send event on commit.
Earlier, we were using 'send_event' in
do_change_stream_message_retention_days which can lead to a situation
where we enqueue events but the function fails at a later stage.

Events should not be sent until we know we're not rolling back.

Fixes part of #30489.
2024-08-19 10:14:49 -07:00
Prakhar Pratyush a9af1c0ce2 streams: Update do_change_stream_description to send event on commit.
Earlier, we were using 'send_event' in do_change_stream_description
which can lead to a situation where we enqueue events but the
function fails at a later stage.

Events should not be sent until we know we're not rolling back.

Fixes part of #30489.
2024-08-19 10:14:49 -07:00
Prakhar Pratyush e474ab7ee0 streams: Add transaction.atomic decorator to 'do_rename_stream'.
This commit updates the 'do_rename_stream' action function
to make the db operations atomic.

Also, send_event is replaced with send_event_on_commit.
2024-08-19 10:14:49 -07:00
Prakhar Pratyush 69dec9dbc6 streams: Update 'do_change_stream_post_policy' to send event on commit.
Earlier, we were using 'send_event' in do_change_stream_post_policy
which can lead to a situation where we enqueue events but the
function fails at a later stage.

Events should not be sent until we know we're not rolling back.

Fixes part of #30489.
2024-08-19 10:14:49 -07:00
Prakhar Pratyush 41d947b89d streams: Make db operations in do_change_subscription_property atomic.
This commit adds a transaction.atomic decorator to the
'do_change_subscription_property' function to make
the db operations in the action function atomic.

Also, send_event is changed to send_event_on_commit.
2024-08-19 10:14:49 -07:00
Prakhar Pratyush 5d0b3d1a6d muted_users: Update 'do_unmute_user' to send event on commit.
Earlier, we were using 'send_event' in 'do_unmute_user'
which can lead to a situation where we enqueue events but the
function fails at a later stage.

Events should not be sent until we know we're not rolling back.

Fixes part of #30489.
2024-08-19 10:14:49 -07:00
Prakhar Pratyush 3b9a21443a muted_users: Update 'do_mute_user' to send event on commit.
Earlier, we were using 'send_event' in 'do_mute_user' which
can lead to a situation where we enqueue events but the
function fails at a later stage.

Events should not be sent until we know we're not rolling back.

Fixes part of #30489.
2024-08-19 10:14:49 -07:00
Prakhar Pratyush 9026e6ecc1 message_flags: Update 'do_update_message_flags' to send event on commit.
Earlier, we were using 'send_event' in do_update_message_flags
which can lead to a situation where we enqueue events but the
function fails at a later stage.

Events should not be sent until we know we're not rolling back.

Fixes part of #30489.
2024-08-19 10:14:49 -07:00
Prakhar Pratyush 32a4a112b1 message_flags: Update do_mark_muted_user...read to send event on commit.
Earlier, we were using 'send_event' in
'do_mark_muted_user_messages_as_read' which can lead to a
situation where we enqueue events but the function fails at a
later stage.

Events should not be sent until we know we're not rolling back.

Fixes part of #30489.
2024-08-19 10:14:49 -07:00
Prakhar Pratyush 62eb850423 message_flags: Update transactions to be durable in do_mark_all_as_read.
In 'do_mark_all_as_read', the transactions which mark the messages
as read in batches should be marked as durable to avoid addition
of any outer atomic block as we support marking a few batches
(not all messages) as read in the case of a timeout.
2024-08-19 10:14:49 -07:00
Prakhar Pratyush 64beea2765 message_flags: Update 'do_mark_stream...read' to send event on commit.
Earlier, we were using 'send_event' in do_mark_stream_messages_as_read
codepath which can lead to a situation where we enqueue events but the
function fails at a later stage.

Events should not be sent until we know we're not rolling back.

Fixes part of #30489.
2024-08-19 10:14:49 -07:00
Prakhar Pratyush ed512f06bb message_edit: Queue event on commit in do_update_message codepath.
'do_update_message' is within a db transaction, this commit
updates the 'do_clear_mobile_push_notifications_for_ids' function
used in 'do_update_message' to queue event on commit.

Events should not be sent until we know we're not rolling back,
otherwise it can lead to a situation where we enqueue events but
the function fails at a later stage.
2024-08-19 10:14:49 -07:00
Tim Abbott a3806b4165 migrations: Fix quoting bug in tutorial_status migration.
Fixes #31381.
2024-08-17 22:17:21 -07:00
bedo 00c9f36434 create stream: Bulk conversion of principals to UserProfiles.
Previously, this logic did the database queries to look up UserProfile
objects in a loop.

Fixes #21820.

Significantly improves Stream creation time and also unsusbcribing users.
Tested stream creation with 10k stream subscribers:

- before: 127 seconds ~2 mins
- after: 17 seconds ~0.3 min

Add a test case for user unsubscribing themself.
2024-08-16 17:46:19 -07:00
Alex Vandiver 854ec40def home: Add rel=dns-prefetch to Gravatar and any Zulip avatar host.
This hints to the browser that it should start DNS lookups for the
host, since it is likely to be necessary.  It is a softer form than
`rel-preconnect`, which may be unnecessary in these cases, if the
client has the resources cached already.
2024-08-16 14:33:25 -07:00
Lauryn Menard cabe0b2e05 api-docs: Update topics array description in get-stream-topics response. 2024-08-15 18:27:43 -07:00
Shubham Padia 2b3a41be58 user_card_popover: Show popover on all .messagebox user mention.
Show user card popover for scheduled messages overlay, compose box
preview, message edit preview, message edit history.

`.messagebox` was chosen as the selector since that was the nearest
parent class that was common for all of the above.

`@all` does not have a popover and that's why it will have the same
pointer as its parent element. We also introduce a new class called
`.user-mention-all` for managing css rules specific to that mention.
2024-08-15 16:28:08 -07:00
Prakhar Pratyush 52a9846cdf user_profile: Remove 'tutorial_status' field.
The 'tutorial_status' field on 'UserProfile' model is
no longer used to show onboarding tutorial.

This commit removes the 'tutorial_status' field,
'POST users/me/tutorial_status' endpoint, and
'needs_tutorial' parameter in 'page_params'.

Fixes part of zulip#30043.
2024-08-15 13:31:40 -07:00
Prakhar Pratyush ee806c49b9 ui_init: Use OnboardingStep for narrowing on first web app load.
We plan to remove the 'tutorial_status' field from UserProfile
table as it is no longer used to show tutorial.

The field is also used to narrow a new user in DM with
welcome bot on the first load.

This prep commit updates the logic to use a new OnboardingStep
for the narrowing behaviour on the first load. This will help
in removing the 'tutorial_status' field.
2024-08-15 13:31:40 -07:00
PieterCK d974c10efc integration-docs: Update Open Collective for new doc format.
Other than reformatting documentation for Open Collective, this
commit also moves it to the "Financial" category from "Communications".
This is because Open Collective is mainly a fundrising + legal status +
money management platform, as stated in https://opencollective.com/.

Part of #29592.
2024-08-15 13:16:14 -07:00
PieterCK 00f9892d8d integration-docs: Update Mention for new doc format.
Part of #29592.
2024-08-15 13:16:14 -07:00
PieterCK 07fda86ff9 integration-docs: Update Netlify for new doc format.
Besides reformatting the Netlify doc, this commit also updates the
instructions to match some UI changes in Netlify. The "Outgoing Webhook"
menu is now called "HTTP Post request".

Part of #29592.
2024-08-15 13:16:14 -07:00
PieterCK 9494620022 integration: Delete Opbeat integration.
Opbeat was acquired by Elastic in 2017[1] and maintained until Elastic
APM went live[2]. Now that Elastic APM is online, it looks like Opbeat
repositories have been marked as archived/unmaintained[3].

[1]: https://www.elastic.co/blog/welcome-opbeat-to-the-elastic-family
[2]:
https://www.elastic.co/observability/application-performance-monitoring
[3]: https://github.com/opbeat
2024-08-15 13:16:14 -07:00
Vector73 80969a62e6 settings: Add `can_delete_any_message_group` setting.
Fixes #30717.
2024-08-15 12:03:49 -07:00
Prakhar Pratyush 19d56f77b5 rocketchat: Fix "OSError: [Errno 36] File name too long" error.
Earlier, we were replacing too long attachment name with random uuid
when the character count of the file name was greater than 255.

This results in "OSError: [Errno 36] File name too long" error in
few cases when the file name has less than 255 characters but more
than 255 bytes (file name with Non-ASCII characters).

This commit updates the code to check the file name's byte size
instead of characters count.
2024-08-14 18:18:31 -07:00
Mateusz Mandera bf7fc5701f management: Add change_auth_backends command.
A utility command to enable or disable certain authentication backends
for a realm from the command line. Can be helpful e.g. if the
administrator accidentally disables some auth methods in the UI leaving
themselves with none remaining that they could actually use to log back
into the organization.

Example usage:
```
(zulip-py3-venv) vagrant@c32c137f59a0:/srv/zulip$ ./manage.py change_auth_backends -r zulip --show
Current authentication backends for the realm:
Enabled backends:
  Dev
  Email
  GitHub
  GitLab
  Google
  Apple
  SAML
  OpenID Connect
(zulip-py3-venv) vagrant@c32c137f59a0:/srv/zulip$ ./manage.py change_auth_backends -r zulip --disable GitHub
Disabling GitHub backend for realm Zulip Dev
Updated authentication backends for the realm:
Enabled backends:
  Dev
  Email
  GitLab
  Google
  Apple
  SAML
  OpenID Connect
Disabled backends:
  GitHub
Done!
(zulip-py3-venv) vagrant@c32c137f59a0:/srv/zulip$ ./manage.py change_auth_backends -r zulip --enable GitHub
Enabling GitHub backend for realm Zulip Dev
Updated authentication backends for the realm:
Enabled backends:
  Dev
  Email
  GitHub
  GitLab
  Google
  Apple
  SAML
  OpenID Connect
Done!
```
2024-08-14 14:04:46 -07:00
Alex Vandiver 2bfdccd065 narrow: Do not error when given an empty DM recipient list. 2024-08-14 10:39:40 -07:00
Vector73 fa408a969e api-docs: Handle multiple examples in `responses`. 2024-08-14 09:37:15 -07:00
Tim Abbott 07c927ae88 migrations: Fix Nagios bot cleanup crash.
We didn't correctly check for the possibility that these users don't
exist.
2024-08-13 23:30:32 -07:00
Shubham Padia 8b96aa277a css: Use classname instead of `div` selector for .blocks.
Having the `div` selector there affects the performance. See
https://chat.zulip.org/#narrow/stream/6-frontend/topic/CSS.20selector.20performance/near/1845719
for more details.
2024-08-13 22:36:35 -07:00
Alex Vandiver 16dffeb941 migrations: Fix missing Literal() calls in 0564. 2024-08-13 08:04:13 -07:00
Prakhar Pratyush e7f2a0958a custom_profile_fields: Send event on commit in notify_realm...fields.
Earlier, we were using 'send_event' in
'notify_realm_custom_profile_fields' which can lead to a situation,
if any db operation is added after the 'send_event' in the action
functions using it, where we enqueue event but the action function
fails at a later stage.

Events should not be sent until we know we're not rolling back.

Fixes part of #30489.
2024-08-12 12:16:14 -07:00
Prakhar Pratyush bae6188d92 user_groups: Add transaction.atomic decorator to add_user_group view.
The database operations in 'access_user_group_for_setting' and
'check_add_user_group' used in 'add_user_group' view should be
collectively atomic.

This commit adds transaction.atomic decorator for that purpose.
2024-08-12 12:16:14 -07:00
Prakhar Pratyush 427c59ec68 user_groups: Update delete_user_group codepath to send event on commit.
Earlier, we were using 'send_event' in 'delete_user_grou' codepath
which can lead to a situation, if any db operation is added after
the 'send_event' in future, where we enqueue events but the
function fails at a later stage.

Events should not be sent until we know we're not rolling back.

Fixes part of #30489.
2024-08-12 12:16:14 -07:00
Prakhar Pratyush 66ff646e99 user_groups: Update edit_user_group codepath to send event on commit.
Earlier, we were using 'send_event' in 'edit_user_group' codepath
which can lead to a situation where we enqueue events but the
function fails at a later stage.

Events should not be sent until we know we're not rolling back.

Fixes part of #30489.
2024-08-12 12:16:14 -07:00
Prakhar Pratyush 69a2775e9b video_calls: Update do_set_zoom_token to send event on commit.
Earlier, we were using 'send_event' in 'do_set_zoom_token' which
can lead to a situation, if any db operation is added after the
'send_event' in future, where we enqueue events but the action
function fails at a later stage.

Events should not be sent until we know we're not rolling back.

Fixes part of #30489.
2024-08-12 12:16:14 -07:00
Prakhar Pratyush 0fddda232d drafts: Update do_delete_draft to send event on commit.
Earlier, we were using 'send_event' in 'do_delete_draft' which
can lead to a situation, if any db operation is added after the
'send_event' in future, where we enqueue events but the action
function fails at a later stage.

Events should not be sent until we know we're not rolling back.

Fixes part of #30489.
2024-08-12 12:16:14 -07:00
Prakhar Pratyush 096647fbd5 drafts: Update do_edit_draft to send event on commit.
Earlier, we were using 'send_event' in 'do_edit_draft' which
can lead to a situation, if any db operation is added after the
'send_event' in future, where we enqueue events but the action
function fails at a later stage.

Events should not be sent until we know we're not rolling back.

Fixes part of #30489.
2024-08-12 12:16:14 -07:00
Prakhar Pratyush ed7c399644 drafts: Update do_create_drafts to send event on commit.
Earlier, we were using 'send_event' in 'do_create_drafts' which
can lead to a situation, if any db operation is added after the
'send_event' in future, where we enqueue events but the action
function fails at a later stage.

Events should not be sent until we know we're not rolling back.

Fixes part of #30489.
2024-08-12 12:16:14 -07:00
Prakhar Pratyush 475a25e739 scheduled_messages: Send event on commit in delete_scheduled_message.
Earlier, we were using 'send_event' in 'delete_scheduled_messages'
which can lead to a situation, if any db operation is added after
the 'send_event' in future, where we enqueue events but the action
function fails at a later stage.

Events should not be sent until we know we're not rolling back.

Fixes part of #30489.
2024-08-12 12:16:14 -07:00
Prakhar Pratyush 500fb3d804 scheduled_messages: Send event on commit in edit_scheduled_message.
Earlier, we were using 'send_event' in 'edit_scheduled_messages'
which can lead to a situation, if any db operation is added after
the 'send_event' in future, where we enqueue events but the action
function fails at a later stage.

Events should not be sent until we know we're not rolling back.

Fixes part of #30489.
2024-08-12 12:16:14 -07:00
Prakhar Pratyush b0390ce1ee scheduled_messages: Send event on commit in try_deliver_..._message.
Earlier, we were using 'send_event' in
'try_deliver_one_scheduled_messages' which can lead to a situation
where we enqueue events but the action function fails at a later stage.

Events should not be sent until we know we're not rolling back.

Fixes part of #30489.
2024-08-12 12:16:14 -07:00
Prakhar Pratyush 567615a484 onboarding_steps: Send event on commit in mark onboarding_step as read.
Earlier, we were using 'send_event' in 'do_mark_onboarding_step_as_read'
which can lead to a situation, if any db operation is added after the
'send_event' in future, where we enqueue events but the action
function fails at a later stage.

Events should not be sent until we know we're not rolling back.

Fixes part of #30489.
2024-08-12 12:16:14 -07:00
Alex Vandiver b287efce43 migrations: Purge database of old nagios messages.
This deletes them directly, rather than move them into archival,
because that would be slow, and bloat the archival table in a way
which might interfere with other deletions.
2024-08-12 12:02:04 -07:00
Alex Vandiver 1f045bc794 migrations: Allow zulipinternal bots to delete their own messages. 2024-08-12 12:02:04 -07:00
PieterCK 68c87dbdf8 integrations: Update comment related notifications in Jira integration.
Previously, comment related notifications only displayed the issue
title as a plain string. This commit reformats the issue title to
include a link back to the Jira issue.
2024-08-12 11:45:09 -07:00
PieterCK c952e7ae86 integrations : Update Jira doc.
Adjusted the Jira documentation for recent changes in their UI
when setting up webhooks, reformatted the note about compatible
Jira version, and added a link to Jira's official webhook guide.
2024-08-12 11:45:09 -07:00
adnan-td e95b51098e integration-docs: Update Papertrail the new doc format.
Part of #29592.
2024-08-12 11:43:16 -07:00
adnan-td 164217f7a3 integration-docs: Update PagerDuty to the new doc format.
Part of #29592.
2024-08-12 11:43:16 -07:00
adnan-td 6e08798c02 integration-docs: Update Patreon to the new doc format.
Part of #29592.
2024-08-12 11:43:16 -07:00
adnan-td 83094860b1 integration-docs: Update Pingdom to the new doc format.
Replaced the image to show **Add Integration** with text
instructions.

Part of #29592.
2024-08-12 11:43:16 -07:00
Lauryn Menard 481fb8e518 help: Rename and redirect allow-image-link-previews article.
Note that the link in zulip_update_announcements.py is not updated
so that the content in the source code reflects what users actually
received in the update announcement message.
2024-08-12 11:22:37 -07:00
Lauryn Menard d54f747f05 api-docs: Update realm_export event to be clearer about export type.
Clarifies that these events are sent for public data exports of a
realm.
2024-08-12 10:12:08 -07:00
Lauryn Menard 80707d97d9 api-docs: Revise descriptions of RealmExport schema fields. 2024-08-12 10:12:08 -07:00
Vector73 b0bd13207d api_docs: Document "/export/realm" endpoints. 2024-08-12 10:12:08 -07:00
Mateusz Mandera 114f13e0ee auth: Fix re-enabling of SAML/AzureAD in organization settings UI.
This bug was introduced in da9e4e6e54.
validate validate_plan_for_authentication_methods is already called
inside validate_authentication_methods_dict_from_api, conditionally on
settings.BILLING_ENABLED. This additional, redundant call runs
regardless of BILLING_ENABLED, and thus prevents a self-hosted server
from enabling certain backends in the organization settings UI.

The impact of this is limited - in order to encounter this bug, a
self-hosted server would have to first disable the backend in the UI, as
self-hosted realms are created with all backend flags enabled. A backend
doesn't show up in the org settings UI until it is first enabled in
AUTHENTICATION_BACKENDS in settings.py - that's why this is a rare
state. A sequence of steps like this has to be followed to reproduce:
1. Add the backend to AUTHENTICATION_BACKENDS in settings.py.
2. Disable the backend in the org settings UI.
3. Now try to re-enable it, which fails due to the bug.
2024-08-05 21:14:43 -07:00
Mateusz Mandera 3f472ec664 test_auth_backends: Remove copied-and-pasted comments.
These makes no sense and were copied from the block at the top of the
test.
2024-08-05 21:14:43 -07:00
Sahil Batra 6919448917 settings: Remove create_web_public_stream_policy setting.
This commit removes create_web_public_stream_policy setting
since web-public channel creation permissions are now
handled by group-based setting.

We still pass "realm_create_web_public_stream_policy" in
"/register" response though for older clients with its
value being set depending on the value of group based
setting. If we cannot set its value to an appropriate enum
corresponding to the group setting, then we set it to
"Admins and moderators" considering that server will not
allow the users without permissions to create web-public
channels but the client can make sure that UI is
available to the users who have permission.
2024-08-01 22:49:33 -07:00
Sahil Batra 09102c5780 streams: Check permission to create web-public streams based on group setting. 2024-08-01 22:49:33 -07:00
Sahil Batra a15bc5e264 realm: Add new group setting for web-public stream creation. 2024-08-01 22:49:33 -07:00
Anders Kaseorg c03839f42f mypy: Reenable explicit-override for models.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-08-01 17:39:04 -07:00
Alex Vandiver 6f20c15ae9 thumbnail: Resolve a race condition when rendering messages.
Messages are rendered outside of a transaction, for performance
reasons, and then sent inside of one.  This opens thumbnailing up to a
race where the thumbnails have not yet been written when the message
is rendered, but the message has not been sent when thumbnailing
completes, causing `rewrite_thumbnailed_images` to be a no-op and the
message being left with a spinner which never resolves.

Explicitly lock and use he ImageAttachment data inside the
message-sending transaction, to rewrite the message content with the
latest information about the existing thumbnails.

Despite the thumbnailing worker taking a lock on Message rows to
update them, this does not lead to deadlocks -- the INSERT of the
Message rows happens in a transaction, ensuring that either the
message rending blocks the thumbnailing until the Message row is
created, or that the `rewrite_thumbnailed_images` and Message INSERT
waits until thumbnailing is complete (and updated no Message rows).
2024-08-01 16:48:16 -07:00
Anders Kaseorg ea60bc3ff8 ruff: Fix PLR1714 Consider merging multiple comparisons.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-08-01 16:46:58 -07:00
Alex Vandiver ab1e858eb9 invites: Use the realm on the PreregistrationUser, to save a join. 2024-08-01 16:44:18 -07:00
Alex Vandiver 2bc118892a migrations: Add missing dependency.
This migration references the "confirmation" app for the first time,
which means we must have migrated at least part of it by this point.

Set the migration to depend on the latest "confirmation" migration at
the time of this migration.
2024-08-01 16:44:18 -07:00
bedo 1ab4723d59 user_groups: Migrate to @typed_endpoint.
Migrate the following endpoints from @has_request_variables
to @typed_endpoint:

- get_user_group()
- delete_user_group()
- update_user_group_backend()
- update_subgroups_of_user_group()
- get_is_user_group_member()
- get_user_group_members()
- get_subgroups_of_user_group()

With tweaks from tabbott to avoid calling thunks unnecessarily.
2024-08-01 09:56:06 -07:00
Alex Vandiver d70935ae04 clients: Drop "desktop app" substring check.
The last desktop client to use that was 0.5.8, released 2017-04-12.
There are no clients with such old versions still observed in the
wild.
2024-08-01 09:23:16 -07:00
roanster007 7b3e163d55 refactor: Rename `huddle` to `direct_message_group` in non api files.
This commit completes rename of "huddle" to "direct_message_group"
in all the non API files.

Part of #28640
2024-07-31 23:25:56 -07:00
Prakhar Pratyush 04347bb315 schedule_messages: Update do_schedule_messages to send event on commit.
Earlier, we were using 'send_event' in 'do_schedule_messages' which
can lead to a situation, if any db operation is added after the
'send_event' in future, where we enqueue events but the action
function fails at a later stage.

Events should not be sent until we know we're not rolling back.

Fixes part of #30489.
2024-07-31 22:33:52 -07:00
Prakhar Pratyush 7302d533fe user_status: Update do_update_user_status to send event on commit.
Earlier, we were using 'send_event' in 'do_update_user_status' which
can lead to a situation, if any db operation is added after the
'send_event' in future, where we enqueue events but the action
function fails at a later stage.

Events should not be sent until we know we're not rolling back.

Fixes part of #30489.
2024-07-31 22:33:52 -07:00
Prakhar Pratyush dc27711399 alert_words: Update remove_alert_word codepath to send event on commit.
Earlier, we were immediately enqueueing event in
'do_remove_alert_words' which can lead to a situation, if any
db operation is added after enqueueing event in future, where the
action function fails at a later stage.

Events should not be sent until we know we're not rolling back.

Fixes part of #30489.
2024-07-31 22:33:52 -07:00
Prakhar Pratyush ba80266b7e alert_words: Update add_alert_words codepath to send event on commit.
Earlier, we were using 'send_event' in 'do_add_alert_words' which
can lead to a situation, if any db operation is added after the
'send_event' in future, where we enqueue events but the action
function fails at a later stage.

Events should not be sent until we know we're not rolling back.

Fixes part of #30489.
2024-07-31 22:33:52 -07:00
Kenneth Rodrigues 454f2f5b0c video_calls: Migrate to typed_endpoint. 2024-07-31 17:10:06 -07:00
Kenneth Rodrigues 97f15d8811 events: Migrate to typed_endpoint.
Migrate `event_register.py` and `tornado` to typed_endpoint.
Modified the tests to work with the migrated endpoints.
2024-07-31 17:10:06 -07:00
Kenneth Rodrigues 0f692436ca user_settings: Migrate to typed_endpoint.
Migrate `user_settings.py` to `typed_endpoint`.
Fix the error messages for the tests.
Migrate required validators.
2024-07-31 17:10:06 -07:00
Mateusz Mandera aaca394813 presence: Remove the queue worker. 2024-07-31 16:46:42 -07:00
Anders Kaseorg 8f1a97f168 users: Reduce date_joined precision to minutes.
Fixes #31188, removing a lot of useless high-entropy data from the
/register response.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-31 12:53:09 -07:00
lumpleme b5c63cfb85 integrations: Prevent duplicate GitHub pull request review messages.
GitHub sends two almost identical payloads when a pull
request is reviewed, which results in two duplicative
notification messages.

The payloads have different "action" value, with one
having the "submitted" action, whereas the other is
"edited" and has an empty "changes" dict.

We now ignore the payload with the "edited" action type
and an empty "changes" dict.

Fixes #26145.

Co-authored-by: Pieter CK <pieterceka123@gmail.com>
2024-07-31 10:42:41 -07:00
adnan-td 76df435dab integration-docs: Update Semaphore to the new doc format.
Part of #29592.

Co-authored-by: Lauryn Menard <lauryn@zulip.com>
2024-07-31 10:16:54 -07:00
adnan-td bdbe3adf1b integration-docs: Update Sentry to the new doc format.
Part of #29592.
2024-07-31 10:16:54 -07:00
Tim Abbott e7cc9090fd api docs: Clarify event structure when moving channels.
The previous language would most naturally be read as suggesting that
moving topics between channels would have these fields.
2024-07-31 09:22:44 -07:00
Mateusz Mandera a0971934d9 thumbnail: Fix typo in comment. 2024-07-30 00:17:59 +02:00
Anders Kaseorg 3a17946c85 zulip_update_announcements: Fix December 2024 typo.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-26 10:20:27 -07:00
Alya Abbott 7a1f5765b7 updates: Add update message (level=8). 2024-07-25 13:19:25 -07:00
Alex Vandiver c726d2ec01 thumbnail: Do not Camo old thumbor URLs; serve images directly.
Providing a signed Camo URL for arbitrary URLs opened the server up to
being an open redirector.  Return 403 if the URL is not a user upload,
and the backend image if it is.  Since we do not have ImageAttachment
rows for uploads at a time we wrote `/thumbnail?` URLs, return the
full-size content.
2024-07-24 16:04:34 -07:00
Alex Vandiver e3a238fc89 thumbnail: Remove unused thumbnail sizes.
47683144ff switched the web client to prefer the 840x560 size, as the
mobile apps prefer; remove the now-unused 300x200 size.  No client was
using the generated `.jpg` formats, as all clients support `.webp`, so
remove the unused `.jpg` thumbnail as well.
2024-07-24 09:57:20 -07:00
Alex Vandiver e4a8304f57 thumbnail: Store the post-orientation-transformation dimensions.
Modern browsers respect the EXIF orientation information of images,
applying rotation and/or mirroring as specified in those tags.  The
the `width="..."` and `height="..."` tags are to size the image
_after_ applying those orientation transformations.

The `.width` and `.height` properties of libvips' images are _before_
any transformations are applied.  Since we intend to use these to hint
to rendering clients the size that the image should be _rendered at_,
change to storing (and providing to clients) the dimensions of the
rendered image, not the stored bytes.
2024-07-24 09:56:42 -07:00
Alex Vandiver e7ac62aad7 tests: Add a test that the thumb is actually the expected size. 2024-07-24 09:56:42 -07:00
Alex Vandiver d824988bc4 tests: Remove THUMBNAIL_IMAGES overrides, since the setting was dropped.
These were omitted from b42863be4b.
2024-07-24 09:53:14 -07:00
Mateusz Mandera 8f461b3c03 email_mirror: Change default topic name if subject ends up empty.
If the email subject is something like `Fwd:`, it gets stripped to an
empty string, activating the "(no topic)" override. This however leads
to failure if the organization enables the setting forcing every message
to have a topic. Such emails should still go through, so we should just
change the topic value used.
2024-07-22 22:31:45 -07:00
Alex Vandiver 2ea0cc0005 thumbnail: Add a data-original-dimensions attribute.
This allows clients to potentially lay out the thumbnails more
intelligently, or to provide a better "progressive-load" experience
when enlarging the thumbnail.
2024-07-22 22:41:10 -04:00
Alex Vandiver 65828b20e9 thumbnail: Factor out a dataclass for markdown image metadata. 2024-07-22 22:41:10 -04:00
roanster007 66a96bee71 settings: Add setting to control how animated images are played.
Previously animated images were automatically played in the
message feed of the web app.

Now that we have still thumbnails available for them, we can add a new
personal setting, "web_animate_image_previews", which controls how the
animated images would be played in the web app message feed -- always
played, on hover, or only in the image viewer.

Fixes #31016.
2024-07-22 14:53:31 -07:00
Prakhar Pratyush 6838a7302d test_import_export: Add test coverage for OnboardingUserMessage. 2024-07-22 10:26:33 -07:00
Prakhar Pratyush f48e87cd3c fetch_initial_state: Avoid doing one db query per announcement stream.
In 'fetch_initial_state_data' we were doing one database query
per announcement stream.

This commit updates the logic to prefetch those streams using
select_related hence avoiding the extra db queries.

Fixes #28909.
2024-07-22 10:21:22 -07:00
Alex Vandiver 3ac14632d8 thumbnail: Disable libvips cache.
The libvips cache is 100MB, 100 operations, or 100 files, whichever is
less.  A single Django process or worker is extremely unlikely to ever
see the same image twice, much less within those timeframes.

Disable the cache, since it is mostly useless memory usage for our use
case.
2024-07-22 10:19:33 -07:00
Alex Vandiver b42863be4b markdown: Show thumbnails for uploaded images.
Fixes: #16210.
2024-07-21 18:41:59 -07:00
Alex Vandiver 71406ac767 thumbnail: Factor frames into account for IMAGE_BOMB_TOTAL_PIXELS. 2024-07-21 18:41:59 -07:00
Alex Vandiver aacf28f7e3 test_classes: Extract a thumbnailing output format helper. 2024-07-21 18:41:59 -07:00
Alex Vandiver 94ff443c00 test_classes: Merge verbose assertEquals into ZulipTestCase. 2024-07-21 18:41:59 -07:00
Niloth P e1d35b1e72 integration-docs: Update Opsgenie for new doc format.
Part of zulip#29592.
2024-07-21 18:09:13 -07:00
Niloth P d972d76725 integration-docs: Update Pivotal for new doc format.
Part of zulip#29592.
2024-07-21 18:09:13 -07:00
Niloth P d571410ba3 integration-docs: Update Rundeck for new doc format.
Part of zulip#29592.
2024-07-21 18:09:13 -07:00
Niloth P 650a1384ca integration-docs: Update RhodeCode for new doc format.
Part of zulip#29592.
2024-07-21 18:09:13 -07:00
Niloth P 8fe7f4e409 integration-docs: Update Review Board for new doc format.
Part of zulip#29592.
2024-07-21 18:09:13 -07:00
Niloth P ec018780b5 integration-docs: Update Raygun for new doc format.
Part of zulip#29592.
2024-07-21 18:09:13 -07:00
Niloth P 335f9b8119 integration-docs: Update Radarr for new doc format.
Part of zulip#29592.
2024-07-21 18:09:13 -07:00
Mateusz Mandera f50104d475 import_realm: Update incorrect comment about emoji dir in exported data.
The emoji dir is present in the data from our export tool. This was
added in 468afe4840.

This comment hasn't been updated since
c4b886d8ae, so probably we just forgot to
refresh it when custom emoji export was added.
2024-07-21 13:16:49 -07:00
Kenneth Rodrigues 8b489f4b96 streams: Convert to typed_enpoint. 2024-07-21 12:57:50 -07:00
Kenneth Rodrigues 6815cded83 zerver: Migrate some files to typed_endpoint.
Migrates `invite.py`, `registration.py` and
`email_mirror.py` to use `typed_endpoint`.
2024-07-20 15:46:48 -07:00
Niloth P 16abd82fa5 integrations: Remove Solano integration. 2024-07-20 15:44:17 -07:00
Sahil Batra c6e9173297 exceptions: Add new error class to be used for invalid parameter values.
This would help us in avoiding adding translation everytime we use
this error for a new pair of parameters.
2024-07-19 09:24:54 -07:00
Anders Kaseorg d574200423 tests: Consume streaming responses.
Fixes warnings like “ResourceWarning: unclosed file <_io.FileIO
name='/srv/zulip/var/044e5d44-87aa-4c43-abbb-28a144fa6654/test-backend/run_1238680/worker_0/test_uploads/files/thumbnail/2/1e/jmUuDhQC8WlaSRCuc0zQyx7D/img.tif/100x75.webp'
mode='rb' closefd=True>” with warnings enabled.

deque(…, 0) is an efficient way to consume an iterator documented at
https://docs.python.org/3/library/itertools.html#itertools-recipes
under consume.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-19 09:20:56 -07:00
Anders Kaseorg c7281b406f test_delete_unclaimed_attachments: Close file in make_attachment.
Fixes “ResourceWarning: unclosed file <_io.BufferedReader
name='/srv/zulip/zerver/tests/images/text.txt'>” with warnings enabled.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-19 09:20:56 -07:00
Tim Abbott de35f44624 zulip_updates: Mention default for send navigation.
Apparently I forgot to merge this before -beta1.
2024-07-18 14:06:04 -07:00
Alex Vandiver 4351cc5914 thumbnail: Move get_image_thumbnail_path and split_thumbnail_path. 2024-07-18 13:50:28 -07:00
Alex Vandiver d474600056 tests: Remove a nonsensical no-op vimeo test.
Embeds are disabled in this test, and it tests that it does not embed
anything.
2024-07-18 13:50:28 -07:00
Alex Vandiver 424f3dd503 tests: Break up and reorganize test_markdown.py. 2024-07-18 13:50:28 -07:00
Alex Vandiver ecdf5713c2 tests: Factor out ZulipVerboseEqualTest class, and use more. 2024-07-18 13:50:28 -07:00
Alex Vandiver 08191d3f69 tests: Default ENABLE_FILE_LINKS to False.
Test configuration should mirror default production as much as possible.
2024-07-18 13:50:28 -07:00
Alex Vandiver e02e9c9181 tests: INLINE_URL_EMBED_PREVIEW=False is the default in tests. 2024-07-18 13:50:28 -07:00
Alex Vandiver 060754980c tests: INLINE_IMAGE_PREVIEW=True is already the default. 2024-07-18 13:50:28 -07:00
Alya Abbott d775b6ffff updates: Add update message (level=7). 2024-07-18 12:38:26 -07:00
Tim Abbott f37082c4ca models: Switch font size default to non-compact.
This is the intended longer-term default, and it's polished enough for
testing.
2024-07-18 12:37:06 -07:00
Niloth P cbe2677a2a integration-docs: Include suggestion for HTTPS hint in Stripe.
With some wording tweaks by tabbott to make it more obviously only
shown in certain environments.
2024-07-18 12:28:08 -07:00
Niloth P ed885ec806 integration-docs: Format keyboard shortcuts in Trello doc. 2024-07-18 12:24:45 -07:00
Niloth P 525f1e28d4 integration-docs: Fix typo in Splunk doc hint. 2024-07-18 12:24:36 -07:00
Sahil Batra c1c4c95ce7 settings: Require clients to pass dense_mode value when needed.
Previously, if someone changed the font-size or line height
settings to some value other than the legacy values, we set
dense_mode to False if it was True. This commit changes the
code to require clients to pass dense_mode as False in such
cases and raise an error otherwise.
2024-07-18 11:36:53 -07:00
Sahil Batra 0ea5daece8 user_settings: Flush cache after commiting changes to DB.
We should flush the cache after commiting changes to DB to
make sure that the cache does not contain stale data when
updating multiple settings in a single request.
2024-07-18 11:36:53 -07:00
Anders Kaseorg 97753fe3b0 emoji_names: Rebuild with CLDR 45, emoji-data 15.1.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-17 22:49:52 -07:00
Mateusz Mandera 4a93149435 settings: Rework how push notifications service is configured.
Instead of the PUSH_NOTIFICATIONS_BOUNCER_URL and
SUBMIT_USAGE_STATISTICS settings, we want servers to configure
individual ZULIP_SERVICE_* settings, while maintaining backward
compatibility with the old settings. Thus, if all the new
ZULIP_SERVICE_* are at their default False value, but the legacy
settings are activated, they need to be translated in computed_settings
to the modern way.
2024-07-17 17:14:06 -07:00
Anders Kaseorg 722842a0aa rocketchat: Remove unnecessary SHA-1 hashing of direct message groups.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-17 15:56:00 -07:00
Anders Kaseorg 541699a1c6 mattermost: Remove unnecessary MD5 hashing of direct message groups.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-17 15:56:00 -07:00
Anders Kaseorg 90cc88c88e mattermost: Split get_recipient_id_from_receiver_name to 3 functions.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-17 15:56:00 -07:00
Anders Kaseorg 27b0618704 data_import: Fix IdMapper typing.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-17 15:56:00 -07:00
Anders Kaseorg 1fd3f983a5 data_import: Remove int detection from IdMapper.
This seems to have been used only for HipChat.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-17 15:56:00 -07:00
Alex Vandiver ab6b6639e6 migrations: Derive emoji content-type from the bytes. 2024-07-17 11:39:04 -07:00
Sahil Batra cf14d8f611 users: Change bot owner only if needed.
This is a small optimization to avoid DB queries if the bot owner
is not changed.

This also helps in avoiding showing the error message in UI if
anything other than the owner is updated for a deactivated bot.
Ideally if the bot owner is not changed, the bot_owner_id field
should not be passed in the request, but we would handle that
later given this is anyways a small nice optimization and we
follow this pattern at other places as well in the API.
2024-07-17 10:07:51 -07:00
Lauryn Menard b9c33ce5f1 api-docs: Add changes section in construct a narrow article.
Adds a header to the section of changes notes about new and
deprecated narrow filters, so that it's easier to directly
link to the relevant content in the construct a narrow article
from the main API changelog and endpoint documentation.
2024-07-16 23:40:45 -07:00
Vector73 7a80fcf042 events: Send `delete_message` event to user who deleted the message.
Fixes #29826.

Co-authored-by: Mukul Goyal <goyal.mukul7689@gmail.com>
Co-authored-by: Aman Agrawal <amanagr@zulip.com>
2024-07-16 23:29:02 -07:00
Niloth P 56aa72af90 integration-docs: Update Trello for new doc format.
Part of zulip#29592.
2024-07-16 17:02:55 -07:00
Niloth P c8781b4de7 integration-docs: Update TeamCity for new doc format.
Part of zulip#29592.
2024-07-16 17:02:55 -07:00
Alex Vandiver 9983dc761f migrations: Further fix emoji re-thumbnailing.
90701978d9 was an incomplete fix.
2024-07-16 15:38:22 -07:00
Niloth P f663544353 integration-docs: Update Slack incoming-webhook for new doc format.
Part of zulip#29592.
2024-07-16 14:55:56 -07:00
Niloth P 32177a9a1c integration-docs: Update Sonarqube for new doc format.
Part of zulip#29592.
2024-07-16 14:55:56 -07:00
Niloth P a546fa719a integration-docs: Update Sonarr for new doc format.
Part of zulip#29592.
2024-07-16 14:55:56 -07:00
Niloth P 1ddf92a98c integration-docs: Update Splunk for new doc format.
Part of zulip#29592.
2024-07-16 14:55:56 -07:00
Niloth P 8549a6dc8a integration-docs: Update Statuspage for new doc format.
Part of zulip#29592.
2024-07-16 14:55:56 -07:00
Niloth P c0ab89ec52 integration-docs: Update Stripe for new doc format.
Part of zulip#29592.
2024-07-16 14:55:56 -07:00
Alex Vandiver e8a049ea57 migrations: Handle missing emoji/images/still/ directory. 2024-07-16 14:55:45 -07:00
Niloth P a17a54b077 integration-docs: Update Zabbix for new doc format.
Part of zulip#29592.
2024-07-16 14:55:27 -07:00
Niloth P c271cdb113 integration-docs: Update WordPress for new doc format.
Part of zulip#29592.

Co-authored-by: Lauryn Menard <lauryn@zulip.com>
2024-07-16 14:55:02 -07:00
Lauryn Menard 3e100d872a integrations: Remove HookPress events from WordPress webhook.
The HookPress plugin has been unavailable since July 2019, so it
doesn't make sense to continue supporting these events in the
WordPress webhook integration.
2024-07-16 14:55:02 -07:00
Niloth P 1e327cc054 integration-docs: Update Wekan for new doc format.
Part of zulip#29592.
2024-07-16 14:55:02 -07:00
Anders Kaseorg dd0d482d76 narrow: Migrate legacy SQLAlchemy select syntax.
Fixes “sqlalchemy.exc.RemovedIn20Warning: Deprecated API features
detected! These feature(s) are not compatible with SQLAlchemy 2.0.”
with warnings enabled.

https://docs.sqlalchemy.org/en/14/changelog/migration_14.html#change-5284

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-16 14:50:30 -07:00
Sahil Batra a0efc0ee76 settings: Refetch realm and user profile objects.
This is a temporary solution to make sure stale objects from
cache are not used when previous request updated multiple settings.
2024-07-16 14:04:19 -07:00
Sahil Batra 83c4c27b18 settings: Do not allow invalid combinations for information density settings. 2024-07-16 14:04:19 -07:00
Sahil Batra 17541ddacb settings: Toggle dense mode if font and line height values are changed. 2024-07-16 14:04:19 -07:00
Anders Kaseorg ad9837911e runtornado: Remove explicit AsyncIOMainLoop installation.
Tornado ≥ 5 uses AsyncIOLoop and AsyncIOMainLoop automatically when
appropriate.  Fixes “DeprecationWarning: make_current is deprecated;
start the event loop first” with warnings enabled.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-16 14:01:32 -07:00
Prakhar Pratyush df7ed437c2 compose: Show banner to explain interleaved view messages fading.
In an interleaved view when composing a message we fade messages
which the user is not replying to, to reduce the chance they send
a message to a recipient they didn't intend to.
Also, it reduces the visual/cognitive processing required
to figure out where their message is going to go.

But, it's not necessarily clear to users that what the
fading means, so this commit adds a one-time compose banner
to explain what's going on the first time this comes up.

Fixes part of #29076.
2024-07-16 13:52:30 -07:00
Prakhar Pratyush 35380b095f compose: Show banner to explain non interleaved view messages fading.
In a non interleaved view when composing a message to another
conversation we fade messages which the user is not replying to,
to reduce the chance they send a message to a recipient they didn't
intend to. Also, it reduces the visual/cognitive processing required
to figure out where their message is going to go.

But, it's not necessarily clear to users that what the
fading means, so this commit adds a one-time compose banner
to explain what's going on the first time this comes up.

Fixes part of #29076.
2024-07-16 13:52:29 -07:00
Alex Vandiver 556b92810b thumbnail: Advertize the thumbnail formats at client registration. 2024-07-16 13:22:15 -07:00
Alex Vandiver 6c624805ce upload: Return the closest-rendered thumbnail. 2024-07-16 13:22:15 -07:00
Alex Vandiver d121a80b78 upload: Serve thumbnailed images. 2024-07-16 13:22:15 -07:00
Alex Vandiver 2e38f426f4 upload: Generate thumbnails when images are uploaded.
A new table is created to track which path_id attachments are images,
and for those their metadata, and which thumbnails have been created.
Using path_id as the effective primary key lets us ignore if the
attachment is archived or not, saving some foreign key messes.

A new worker is added to observe events when rows are added to this
table, and to generate and store thumbnails for those images in
differing sizes and formats.
2024-07-16 13:22:15 -07:00
Alex Vandiver 7aa5bb233d tests: Clarify tests in test_thumbnail.py. 2024-07-16 13:22:15 -07:00
Alex Vandiver 63dbf67120 tests: Ensure we start with a local storage backend. 2024-07-16 13:22:15 -07:00
Alex Vandiver 229dcd0218 upload: Clean up empty directories in local storage. 2024-07-16 13:22:15 -07:00
Alex Vandiver 5f25fae0fa upload: Batch deleting old attachments.
delete_message_attachments() is faster than calling
delete_message_attachment() one-by-one.
2024-07-16 13:22:15 -07:00
Anders Kaseorg 8843f9f62a tests: Remove deprecated SHA1PasswordHasher.
SHA1PasswordHasher will be removed in Django 5.1.  MD5PasswordHasher
will remain for the purpose of speeding up tests.

Followup to commit ac5161f439 (#29620).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-16 13:06:31 -07:00
Alex Vandiver 90701978d9 migrations: Fix emoji re-thumbnailing for imports.
Imports (including conversions from Slack, Hipchat, etc) do not write
the `.original` of the file (nor the still versions, for animations),
only the thumbnailed (and possibly animated) version.  Looking for
`.original` would thus fail, and overwrite the (working) emoji with
the "?" and disable it.

In the event that we do not find the `.original`, fall back to the
bytes from the non-`.original`, to handle these import cases.
2024-07-16 12:57:49 -07:00
Alex Vandiver 1745d87f2e migrations: Add enough information to logs to reverse bad emoji "fixes." 2024-07-16 12:57:49 -07:00
Niloth P 6afed1432c integration-docs: Update Travis CI for new doc format.
Part of zulip#29592.
2024-07-16 09:40:45 -07:00
Niloth P 5db83f87e2 integration-docs: Update Transifex for new doc format.
Part of zulip#29592.
2024-07-16 09:40:45 -07:00
Niloth P 1f4310f893 integration-docs: Update Thinkst for new doc format.
Part of zulip#29592.
2024-07-16 09:40:45 -07:00
Niloth P 0f7b1ba401 integration-docs: Update Taiga for new doc format.
Part of zulip#29592.
2024-07-16 09:40:45 -07:00
Prakhar Pratyush 0de9f7f5cf import_realm: Add support to import 'OnboardingUserMessage' table. 2024-07-16 09:36:02 -07:00
Prakhar Pratyush 762b1c0f5b import_realm: Improve assert statements verifying table name.
This commit improves the assert statements to verify
that the table name is not "usermessage' instead of
verifying that table name doesn't include a substring
"usermessage".

This prep commit will help to avoid assertion error when
importing "onboardingusermessage" table.
2024-07-16 09:36:02 -07:00
Prakhar Pratyush 7d379e00b0 export: Fix 'OnboardingUserMessage' table not being exported.
Earlier, the export tool was logging a warning:
"??? NO DATA EXPORTED FOR TABLE zerver_onboardingusermessage!!!"

This bug was due to not configuring a Config object for
'OnboardingUserMessage' in 'get_realm_config()'.

This commit fixes the bug to export the table properly.
2024-07-16 09:36:02 -07:00
Kenneth Rodrigues 246df3c884 realm: Migrate smaller files to typed_endpoint.
Migrate `realm_domains.py`, `realm_emoji.py`,
`realm_linkifiers.py`, `realm_logo.py`, `realm_playgrounds.py`
to typed_endpoint.
2024-07-15 16:49:32 -07:00
Kenneth Rodrigues ba79d759f1 realm: Migrate to typed_endpoint. 2024-07-15 16:49:32 -07:00
Kenneth Rodrigues a7da24a36f validators: Use cleaner syntax for AfterValidator.
Created a function that returns an `AfterValidator` for `check_int_in`
and `check_string_in` instead of having to use a
`lambda` wraper everytime.
2024-07-15 16:49:32 -07:00
Niloth P 0ec4b0285e integration-docs: Update Zendesk for new doc format.
Part of zulip #29592.

Co-authored-by: Lauryn Menard <lauryn@zulip.com>
2024-07-15 16:42:22 -07:00
Niloth P 2c2f530fe0 integration-docs: Update Zapier for new doc format.
Part of zulip #29592.

Co-authored-by: Lauryn Menard <lauryn@zulip.com>
2024-07-15 16:42:22 -07:00
Mateusz Mandera d89ab2694f register_server: Call crudini with --inplace when rotating secret key.
This is needed for the command to work in docker-zulip, where the zulip
user may not have write permissions to the parent directory of the
secrets file. By default, crudini creates a temporary file, which makes
the command fail under those conditions.
With --inplace, the secret file gets written to directly without
creation of temporary files.
2024-07-15 13:24:14 -07:00
Vector73 d21ee6fa23 api: Deprecate uri and add url parameter in "/user_uploads" endpoint. 2024-07-14 22:32:36 -07:00
Vector73 a07ebba860 zerver: Replace uri with url in local variables and comments. 2024-07-14 22:30:28 -07:00
Anders Kaseorg fe71eaa8f0 ruff: Fix FURB148 `enumerate` index or value is unused.
This is a preview rule, not yet enabled by default.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-14 13:54:32 -07:00
Anders Kaseorg 80d3e60df8 ruff: Fix FURB180 Use of `metaclass=abc.ABCMeta`.
This is a preview rule, not yet enabled by default.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-14 13:53:40 -07:00
Anders Kaseorg e3a191b99b ruff: Fix FURB154 Use of repeated consecutive `global`, `nonlocal`.
This is a preview rule, not yet enabled by default.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-14 13:53:18 -07:00
Anders Kaseorg 6412c2d630 ruff: Fix FURB142 Use of set.add() in a for loop.
This is a preview rule, not yet enabled by default.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-14 13:52:59 -07:00
Anders Kaseorg 1e9b6445a9 ruff: Fix PLR6104 Use `+=` to perform an augmented assignment directly.
This is a preview rule, not yet enabled by default.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-14 13:49:51 -07:00
Anders Kaseorg b96feb34f6 ruff: Fix SIM117 Use a single `with` statement with multiple contexts.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-14 13:48:32 -07:00
Tim Abbott b0f144327d narrow: Update new function for modern Python typing syntax.
I apparently needed to be more careful rebasing the typing syntax
modernization work.
2024-07-13 23:06:47 -07:00
Anders Kaseorg 48202389b8 ruff: Bump target-version from py38 to py310.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-13 22:28:22 -07:00
Anders Kaseorg 3f29bc42b1 ruff: Fix B905 `zip()` without an explicit `strict=` parameter.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-13 22:28:22 -07:00
Anders Kaseorg 1464009fae ruff: Fix UP038 Use `X | Y` in `isinstance` call instead of `(X, Y)`.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-13 22:28:22 -07:00
Anders Kaseorg 0fa5e7f629 ruff: Fix UP035 Import from `collections.abc`, `typing` instead.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-13 22:28:22 -07:00
Anders Kaseorg 531b34cb4c ruff: Fix UP007 Use `X | Y` for type annotations.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-13 22:28:22 -07:00
Anders Kaseorg e08a24e47f ruff: Fix UP006 Use `list` instead of `List` for type annotation.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-13 22:28:22 -07:00
Anders Kaseorg c2214b3904 test_typed_endpoint: Inline pointless type variable.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-13 22:28:22 -07:00
Anders Kaseorg f04e7fc625 test_openapi: Make get_standardized_argument_type understand X | Y.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-13 22:28:22 -07:00
Anders Kaseorg a2fad74a28 typed_endpoint: Make is_optional understand T | None.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-13 22:28:22 -07:00
Aman Agrawal 70be9e8c51 narrow: Implement rendering of `with` narrow operators.
Adds server and web app support for processing the new `with`
search operator.

Fixes part of #21505.

Co-authored-by: roanster007 <rohan.gudimetla07@gmail.com>
Co-authored-by: Tim Abbott <tabbott@zulip.com>
2024-07-13 07:18:24 -07:00
bedo bfd54e27b1 custom_profile_fields: Bulk fetch of UserProfile.
bulk fetch query of UserPfrofile against which
user_ids are validated, instead of looping
over user_ids and fetchingeach UserPfrofile resulting
in O(n) queries.
2024-07-13 07:05:13 -07:00
Alya Abbott bd04a30bbc updates: Add update message (level=6). 2024-07-12 16:40:41 -07:00
Alex Vandiver 4df9b03451 migrations: Re-thumbnail and rename existing emoji. 2024-07-12 13:26:47 -07:00