Commit Graph

8282 Commits

Author SHA1 Message Date
Mateusz Mandera 7ebeffbcdc test_bots: Remove misleading comment.
This comment looks like an ancient leftover from early days (moved here
in a test_bots extraction in 123b4c1877 in
2017). Whatever its history, this comment and test name don't make sense
anymore. The response here is an error, not a silent success.
2024-09-09 11:55:46 -07:00
Lauryn Menard d610e09397 audit-logs: Use AuditLogEventType enum name for string method.
Reorders audit log string methods to have the following pattern:
"event_type event_time (id): modified_object". And the event type
is the name for the AuditLogEventType enum.
2024-09-09 11:50:13 -07:00
Lauryn Menard 00ecd4c8f0 audit-log: Move remote realm event types to AuditLogEventType enum.
Event types moved: REMOTE_REALM_VALUE_UPDATED,
REMOTE_PLAN_TRANSFERRED_SERVER_TO_REALM, REMOTE_REALM_LOCALLY_DELETED
REMOTE_REALM_LOCALLY_DELETED_RESTORED
2024-09-09 11:50:13 -07:00
Lauryn Menard aad93e149e audit-log: Move remote server event types to AuditLogEventType enum.
Event types moved: REMOTE_SERVER_DEACTIVATED, REMOTE_SERVER_REACTIVATED
REMOTE_SERVER_PLAN_TYPE_CHANGED, REMOTE_SERVER_DISCOUNT_CHANGED
REMOTE_SERVER_SPONSORSHIP_APPROVED, REMOTE_SERVER_BILLING_MODALITY_CHANGED
REMOTE_SERVER_SPONSORSHIP_PENDING_STATUS_CHANGED, REMOTE_SERVER_CREATED
2024-09-09 11:50:13 -07:00
Lauryn Menard d431a5aad6 audit-log: Move user group event types to AuditLogEventType enum.
Event types moved: USER_GROUP_CREATED, USER_GROUP_DELETED
USER_GROUP_DIRECT_USER_MEMBERSHIP_ADDED,
USER_GROUP_DIRECT_USER_MEMBERSHIP_REMOVED,
USER_GROUP_DIRECT_SUBGROUP_MEMBERSHIP_ADDED,
USER_GROUP_DIRECT_SUBGROUP_MEMBERSHIP_REMOVED,
USER_GROUP_DIRECT_SUPERGROUP_MEMBERSHIP_ADDED,
USER_GROUP_DIRECT_SUPERGROUP_MEMBERSHIP_REMOVED,
USER_GROUP_NAME_CHANGED, USER_GROUP_DESCRIPTION_CHANGED,
USER_GROUP_GROUP_BASED_SETTING_CHANGED
2024-09-09 11:50:13 -07:00
Lauryn Menard df1e9093a9 audit-log: Move stream/channel event types to AuditLogEventType enum.
Renamed event types below in the enum class to use channel instead of
stream.

Event types moved: STREAM_CREATED, STREAM_DEACTIVATED, STREAM_NAME_CHANGED
STREAM_REACTIVATED, STREAM_MESSAGE_RETENTION_DAYS_CHANGED
STREAM_PROPERTY_CHANGED, STREAM_GROUP_BASED_SETTING_CHANGED
2024-09-09 11:50:13 -07:00
Lauryn Menard fc6bc6df60 audit-log: Move user muted event types to AuditLogEventType enum.
Event types moved: USER_MUTED, USER_UNMUTED
2024-09-09 11:50:13 -07:00
Lauryn Menard 10d161638e audit-log: Move subscription event types to AuditLogEventType enum.
Event types moved: SUBSCRIPTION_CREATED, SUBSCRIPTION_ACTIVATED,
SUBSCRIPTION_DEACTIVATED, SUBSCRIPTION_PROPERTY_CHANGED.
2024-09-09 11:50:13 -07:00
Lauryn Menard 56c8cbde1e audit-log: Move realm event types to AuditLogEventType enum.
Event types moved: REALM_CREATED, REALM_DEFAULT_USER_SETTINGS_CHANGED
REALM_ORG_TYPE_CHANGED, REALM_DOMAIN_ADDED, REALM_DOMAIN_CHANGED
REALM_DOMAIN_REMOVED, REALM_PLAYGROUND_ADDED, REALM_PLAYGROUND_REMOVED
REALM_LINKIFIER_ADDED, REALM_LINKIFIER_CHANGED, REALM_LINKIFIER_REMOVED
REALM_EMOJI_ADDED, REALM_EMOJI_REMOVED, REALM_LINKIFIERS_REORDERED
REALM_IMPORTED
2024-09-09 11:50:13 -07:00
Lauryn Menard d2c32f23db audit-log: Move realm event types to AuditLogEventType enum.
Event types moved: REALM_DEACTIVATED, REALM_REACTIVATED, REALM_SCRUBBED
REALM_PLAN_TYPE_CHANGED, REALM_LOGO_CHANGED, REALM_EXPORTED
REALM_PROPERTY_CHANGED, REALM_ICON_SOURCE_CHANGED, REALM_DISCOUNT_CHANGED
REALM_SPONSORSHIP_APPROVED, REALM_BILLING_MODALITY_CHANGED
REALM_REACTIVATION_EMAIL_SENT, REALM_SPONSORSHIP_PENDING_STATUS_CHANGED
REALM_SUBDOMAIN_CHANGED
2024-09-09 11:50:13 -07:00
Lauryn Menard caeeaf3c3f audit-log: Move user event types to AuditLogEventType enum.
Event types moved: USER_SOFT_ACTIVATED, USER_SOFT_DEACTIVATED
USER_PASSWORD_CHANGED, USER_AVATAR_SOURCE_CHANGED, USER_FULL_NAME_CHANGED
USER_EMAIL_CHANGED, USER_TERMS_OF_SERVICE_VERSION_CHANGED
USER_API_KEY_CHANGED, USER_BOT_OWNER_CHANGED,
USER_DEFAULT_SENDING_STREAM_CHANGED, USER_DEFAULT_REGISTER_STREAM_CHANGED
USER_DEFAULT_ALL_PUBLIC_STREAMS_CHANGED, USER_SETTING_CHANGED
USER_DIGEST_EMAIL_CREATED
2024-09-09 11:50:13 -07:00
Lauryn Menard e5daa3470f audit-log: Move user event types to AuditLogEventType enum.
Event types moved: USER_CREATED, USER_ACTIVATED, USER_DEACTIVATED
USER_REACTIVATED, USER_ROLE_CHANGED, USER_DELETED
USER_DELETED_PRESERVING_MESSAGES
2024-09-09 11:50:13 -07:00
joseph b0a20d2cae attachments: Return a list of removed attachments while editing.
Currently, we want to ask users if they would like to delete their
attachments after they have removed the attachments while editing. These
changes are preparatory changes on the backend to return a list of removed
attachments after the user has removed attachments while editing.

Fixes part of #25525.
2024-09-06 11:40:51 -07:00
Kenneth Rodrigues 1d580f143e endpoints: Modify comments mentioning has_request_variables and REQ. 2024-09-05 16:02:12 -07:00
Kenneth Rodrigues dc32396180 endpoints: Remove the has_request_variables decorator.
All endpoints have been migrated to the typed_endpoint decorator,
therefore the has_request_variables decorator and the REQ function are
no longer needed and have been removed.
2024-09-05 16:02:12 -07:00
Anders Kaseorg e3abd09e67 thumbnail: Fix corrupted email notifications due to HTML5 entities.
BeautifulSoup with formatter="html5" unnecessarily escapes many
characters with HTML5-specific entities that cannot be correctly
parsed by lxml during generation of email notifications.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-09-05 16:00:45 -07:00
Alex Vandiver cec0942b4b users: Allow zero-width-joiners in user names.
Partially fixes: #20128.
2024-09-03 16:34:55 -07:00
Anders Kaseorg 91ade25ba3 python: Simplify with str.removeprefix, str.removesuffix.
These are available in Python ≥ 3.9.
https://docs.python.org/3/library/stdtypes.html#str.removeprefix

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-09-03 12:30:16 -07:00
bedo 3da91e951c auth: Migrate to @typed_endpoint.
Since this is the last has_request_variables endpoint outside tests,
more test_openapi code needs to be deleted in this transition.
2024-08-30 16:56:14 -07:00
Prakhar Pratyush 656c65c74c edit_history: Remove 'prev_rendered_content_version' field.
This commit removes the 'prev_rendered_content_version'
field from:

* the 'edit_history' object within message objects in the
API response of `GET /messages`, `GET /messages/{message_id}`
and `POST /zulip-outgoing-webhook`.
* the 'update_message' event type

as it is an internal server implementation detail not used
by any client.

Note: The field is still stored in the 'edit_history' column
of the 'Message' table as it will be helpful when making
major changes to the markup rendering process.
2024-08-29 15:37:12 -07:00
Alex Vandiver 0c07c6531c thumbnail: Enqueue thumbnails when we render a spinner.
Thumbnails are usually enqueued in the worker when the image is
uploaded.  However, for images which were uploaded before the
existence of the thumbnailing worker, and whose metadata was
backfilled (see previous commit) this leaves a permanent spinner,
since nothing triggers the thumbnail worker for them.

Enqueue a thumbnail worker for every spinner which we render into
Markdown.  This ensures that _something_ is attempting to resolve the
spinner which the user sees.  In the case of freshly-uploaded images
which are still in the queue, this results in a duplicate entry in the
thumbnailing queue -- this is harmless, since the worker determines
that all of the thumbnails we need have already been generated, and it
does no further work.  However, in the case of historical uploads, it
properly kicks off the thumbnailing process and results in a
subsequent message update to include the freshly-generated thumbnail.

While specifically useful for backfilled uploads, this is also
generally a good safety step for a good user experience, as it also
prevents dropped events in the queue from unknown causes from leaving
perpetual spinners in the message feed.

Because `get_user_upload_previews` is potentially called twice for
every message with spinners (see 6f20c15ae9), we add an additional
flag to `get_user_upload_previews` to suppress a _second_ event from
being enqueued for every spinner generated.
2024-08-29 12:11:51 -07:00
Alex Vandiver 704423787b do_delete_old_unclaimed_attachments: Cap deletions at batch size.
Since each loop may add more than one file to the `storage_paths`
list, this may result in more than 1000 files being sent to
delete_message_attachments.  Since the S3 backend only supports 1000
elements being deleted at once, we must partition the list into chunks
which are no more than 1000 elements long.
2024-08-28 13:57:57 -07:00
Tim Abbott 31623911d1 test_message_fetch: Avoid assumptions about fixture data set. 2024-08-23 17:15:35 -07:00
Lauryn Menard d708c3c039 support: Add ability to update max daily invitations for realm.
Adds some validation for changing the realm's max invites via the
support view so that it is not set below the default max for the
realm's plan type, and so that if it's currently set to the default
max it's not reset to that same value.
2024-08-23 16:08:30 -07:00
roanster007 c6a06d4684 direct_message_group: Add new `group_size` field.
This commit adds a new `group_size` field to the `DirectMessageGroup`
model, and backfills its value to each of the existing direct message
groups.

Fixes part of #25713
2024-08-23 11:09:41 -07:00
Prakhar Pratyush 8ddaa949fc user_settings: Send event on commit in do_regenerate_api_key.
Earlier, we were using 'send_event' in 'do_regenerate_api_key'
which can lead to a situation, if any db operation is added after
the 'send_event' in future, where we enqueue events but the action
function fails at a later stage.

Events should not be sent until we know we're not rolling back.

Fixes part of #30489.
2024-08-23 09:10:56 -07:00
Mateusz Mandera 5476340b52 import: Export and import .original emoji files correctly.
The export tool was only exporting the already-thumbnailed emoji file,
omitting the original one. Now we make sure to export the .original file
too, like we do for avatars, and make the import tool process it
directly, to thumbnail it directly and generate a still in the case of
animated emojis.

Otherwise, the imported realm wouldn't have the <emoji>.png.original
file that we generally expect to have accessible, and stills for
animated emojis were completely missing.
2024-08-21 16:30:19 -07:00
Tim Abbott f1d828cb7d test_openapi: Remove now-unused blocks.
As we remove uses of the has_request_variables system in favor of
typed_endpoints, parts of this file have lost test coverage.
2024-08-21 11:33:42 -07:00
Kenneth Rodrigues 2483e600a2 message_send: Convert to typed endpoint.
Convert `message_send.py` use `typed endpoint`.

Disable `message_send` endpoint `to` parameter in the `openapi`
`validate_json_schema` check, because it is a special case where the
content type of the parameter is application/json but the
parameter may or may not be JSON encoded since previously we also
accepted a raw string and some ad-hoc bot might still depend on sending
a raw string.

Remove unused validators from `validator.py`.
2024-08-21 11:13:00 -07:00
Prakhar Pratyush bfe7eb6b2c do_change_full_name: Noop if value isn't really changing.
In do_change_full_name, we shouldn't create RealmAuditLog
entry and sends events if the value isn't changing.
2024-08-21 10:51:23 -07:00
Mateusz Mandera 7e1f468f04 saml: Fix exception when syncing missing value to custom profile field.
There was a bug here that would trigger an exception inside
`sync_user_profile_custom_fields`, causing it to get logged with
logging.warning, when an attribute configured for SAML custom profile
field sync was missing from a SAMLResponse or had an empty value.
`sync_user_profile_custom_fields` expects valid values, and None is not
valid.

We could consider a slightly different behavior here instead - when an
attribute is sent with no value in the SAMLResponse, that means the attr
has no value in the IdP's user directory - so perhaps a better behavior
would be to also remove the custom profile field value in Zulip. However
there are two issues with that:

1. It's not necessarily the best behavior, because an organization might
want the "user doesn't have this attribute set at the IdP level" state
to just mean that the user should be free to set the value manually in
Zulip if they wish. And having that value get reset on every login would
then be an issue. The implementation in this commit is consistent with
this philosophy.

2. There's some implementation difficulty - upstream
`self.get_attr(...)`, which we use for reading the attr value from the
SAMLResponse, doesn't distinguish between an attribute being sent with
no value and the attribute not being sent at all - in both cases it
returns None. So we'd need some extra work here with parsing the
SAMLResponse properly, to be able to know when the custom profile field
should get cleared.
2024-08-20 13:57:30 -07:00
Mateusz Mandera 833dce8a13 saml: Add support for syncing user role.
Replace the SOCIAL_AUTH_SYNC_CUSTOM_ATTRS_DICT with
SOCIAL_AUTH_SYNC_ATTRS_DICT, designed to support also regular user attrs
like role or full name (in the future).

Custom attributes can stay configured as they were and will get merged
into SOCIAL_AUTH_SYNC_ATTRS_DICT in computed_settings, or can be
specified in SOCIAL_AUTH_SYNC_ATTRS_DICT directly with "custom__"
prefix.

The role sync is plumbed through to user creation, so users can
immediately be created with their intended role as provided by the IdP
when they're creating their account, even when doing this flow without
an invitiation.
2024-08-20 11:53:24 -07:00
Mateusz Mandera 8c1a1ea8db scim: Extract ROLE_TYPE_TO_NAME dict to UserProfile.
This allows these mappings to used in other APIs. Specifically, we want
to use this for syncing role during SAML auth.
2024-08-20 11:53:24 -07:00
Kenneth Rodrigues 86bf0d3ea3 decorators: Migrate to typed_endpoint.
Modify `semgrep-py.yml` to treat arguments after `*args` as
keyword-only arguments.
2024-08-20 10:15:30 -07:00
Kenneth Rodrigues 50712bfa81 scheduled_messages: Migrate to typed_endpoint.
Migrate `scheduled_message.py` to typed_endpoint.

Perform Json parsing in the endpoint itself instead of
in `recipient_parsing.py`.
2024-08-20 10:03:22 -07:00
Ujjawal Modi 5d613ce98d user_groups: Add support to update can_manage_group setting.
This commit adds API support to update can_manage_group setting
of a user group.
2024-08-20 09:28:05 -07:00
Ujjawal Modi b0b36e884c user_groups: Add support to set can_manage_group during creation.
This commit adds API support to set can_manage_group while
creating a user group.
2024-08-20 09:28:05 -07:00
Ujjawal Modi 2505813359 user_groups: Add can_manage_group field to UserGroup objects.
This commit make changes in code to include can_manage_group
field to UserGroup objects passed with response of various endpoints
including "/register" endpoint and also in the group object
send with user group creation event.
2024-08-20 09:28:05 -07:00
Ujjawal Modi 423d5c42f6 user_groups: Update code to check whether user can edit a user group.
Earlier there was only a realm level setting for configuring
who can edit user groups. A new group level setting is also added
for configuring who can manage that particular group.
Now, a user group can be edited by a user if it is allowed from
realm level setting or group level setting.

This commit make changes to also use group level setting
in determining whether a group can be edited by user or not.

Also, updated tests to use api_post and api_delete helpers instead
of using client_post and client_delete helpers with different users
being logged in.
2024-08-20 09:28:05 -07:00
Ujjawal Modi 03220ba456 user_groups: Add can_manage_group field in database.
This commit adds a new group level setting can_manage_group
for configuring who can manage a group. This commit only adds
the field in database and make changes to automatically create
single user groups corresponsing to acting user
which will be the default value for this setting.

Fixes part of #25928.
2024-08-20 09:28:05 -07:00
Ujjawal Modi 9e699dfc85 tests: Separate test of group edit setting for creation and deletion.
Earlier there was a single backend test for testing group edit policy
for creating and deleting user group.This commit make changes in the test
and now there are two separate tests for testing group edit policy for
creating and deleting user groups.

This was done because in future commits we will be adding a
realm level setting for configuring who can create user groups.

Also, updated tests to use api_post and api_delete helpers instead
of using client_post and client_delete helpers with different users
being logged in.
2024-08-20 09:28:05 -07:00
Sahil Batra 3dc82eed57 user_groups: Refactor code to compute user group objects.
This commit refactors code in user_groups_in_realm_serialized
such that we do not prefetch "can_mention_group__direct_members"
and "can_mention_group__direct_subgroups" using prefetch_related
and instead fetch members and subgroups for all groups in separate
queries and then use that data to find the members and subgroups
of the group used for that setting.

This change helps us in avoiding two prefetch queries for each
setting when we add more group settings.
2024-08-19 15:41:55 -07:00
Prakhar Pratyush 9026e6ecc1 message_flags: Update 'do_update_message_flags' to send event on commit.
Earlier, we were using 'send_event' in do_update_message_flags
which can lead to a situation where we enqueue events but the
function fails at a later stage.

Events should not be sent until we know we're not rolling back.

Fixes part of #30489.
2024-08-19 10:14:49 -07:00
Prakhar Pratyush ed512f06bb message_edit: Queue event on commit in do_update_message codepath.
'do_update_message' is within a db transaction, this commit
updates the 'do_clear_mobile_push_notifications_for_ids' function
used in 'do_update_message' to queue event on commit.

Events should not be sent until we know we're not rolling back,
otherwise it can lead to a situation where we enqueue events but
the function fails at a later stage.
2024-08-19 10:14:49 -07:00
bedo 00c9f36434 create stream: Bulk conversion of principals to UserProfiles.
Previously, this logic did the database queries to look up UserProfile
objects in a loop.

Fixes #21820.

Significantly improves Stream creation time and also unsusbcribing users.
Tested stream creation with 10k stream subscribers:

- before: 127 seconds ~2 mins
- after: 17 seconds ~0.3 min

Add a test case for user unsubscribing themself.
2024-08-16 17:46:19 -07:00
Shubham Padia 2b3a41be58 user_card_popover: Show popover on all .messagebox user mention.
Show user card popover for scheduled messages overlay, compose box
preview, message edit preview, message edit history.

`.messagebox` was chosen as the selector since that was the nearest
parent class that was common for all of the above.

`@all` does not have a popover and that's why it will have the same
pointer as its parent element. We also introduce a new class called
`.user-mention-all` for managing css rules specific to that mention.
2024-08-15 16:28:08 -07:00
Prakhar Pratyush 52a9846cdf user_profile: Remove 'tutorial_status' field.
The 'tutorial_status' field on 'UserProfile' model is
no longer used to show onboarding tutorial.

This commit removes the 'tutorial_status' field,
'POST users/me/tutorial_status' endpoint, and
'needs_tutorial' parameter in 'page_params'.

Fixes part of zulip#30043.
2024-08-15 13:31:40 -07:00
Prakhar Pratyush ee806c49b9 ui_init: Use OnboardingStep for narrowing on first web app load.
We plan to remove the 'tutorial_status' field from UserProfile
table as it is no longer used to show tutorial.

The field is also used to narrow a new user in DM with
welcome bot on the first load.

This prep commit updates the logic to use a new OnboardingStep
for the narrowing behaviour on the first load. This will help
in removing the 'tutorial_status' field.
2024-08-15 13:31:40 -07:00
Vector73 80969a62e6 settings: Add `can_delete_any_message_group` setting.
Fixes #30717.
2024-08-15 12:03:49 -07:00
Alex Vandiver 2bfdccd065 narrow: Do not error when given an empty DM recipient list. 2024-08-14 10:39:40 -07:00
Vector73 fa408a969e api-docs: Handle multiple examples in `responses`. 2024-08-14 09:37:15 -07:00
Shubham Padia 8b96aa277a css: Use classname instead of `div` selector for .blocks.
Having the `div` selector there affects the performance. See
https://chat.zulip.org/#narrow/stream/6-frontend/topic/CSS.20selector.20performance/near/1845719
for more details.
2024-08-13 22:36:35 -07:00
Prakhar Pratyush e7f2a0958a custom_profile_fields: Send event on commit in notify_realm...fields.
Earlier, we were using 'send_event' in
'notify_realm_custom_profile_fields' which can lead to a situation,
if any db operation is added after the 'send_event' in the action
functions using it, where we enqueue event but the action function
fails at a later stage.

Events should not be sent until we know we're not rolling back.

Fixes part of #30489.
2024-08-12 12:16:14 -07:00
Vector73 b0bd13207d api_docs: Document "/export/realm" endpoints. 2024-08-12 10:12:08 -07:00
Mateusz Mandera 114f13e0ee auth: Fix re-enabling of SAML/AzureAD in organization settings UI.
This bug was introduced in da9e4e6e54.
validate validate_plan_for_authentication_methods is already called
inside validate_authentication_methods_dict_from_api, conditionally on
settings.BILLING_ENABLED. This additional, redundant call runs
regardless of BILLING_ENABLED, and thus prevents a self-hosted server
from enabling certain backends in the organization settings UI.

The impact of this is limited - in order to encounter this bug, a
self-hosted server would have to first disable the backend in the UI, as
self-hosted realms are created with all backend flags enabled. A backend
doesn't show up in the org settings UI until it is first enabled in
AUTHENTICATION_BACKENDS in settings.py - that's why this is a rare
state. A sequence of steps like this has to be followed to reproduce:
1. Add the backend to AUTHENTICATION_BACKENDS in settings.py.
2. Disable the backend in the org settings UI.
3. Now try to re-enable it, which fails due to the bug.
2024-08-05 21:14:43 -07:00
Mateusz Mandera 3f472ec664 test_auth_backends: Remove copied-and-pasted comments.
These makes no sense and were copied from the block at the top of the
test.
2024-08-05 21:14:43 -07:00
Sahil Batra 6919448917 settings: Remove create_web_public_stream_policy setting.
This commit removes create_web_public_stream_policy setting
since web-public channel creation permissions are now
handled by group-based setting.

We still pass "realm_create_web_public_stream_policy" in
"/register" response though for older clients with its
value being set depending on the value of group based
setting. If we cannot set its value to an appropriate enum
corresponding to the group setting, then we set it to
"Admins and moderators" considering that server will not
allow the users without permissions to create web-public
channels but the client can make sure that UI is
available to the users who have permission.
2024-08-01 22:49:33 -07:00
Sahil Batra 09102c5780 streams: Check permission to create web-public streams based on group setting. 2024-08-01 22:49:33 -07:00
Sahil Batra a15bc5e264 realm: Add new group setting for web-public stream creation. 2024-08-01 22:49:33 -07:00
Alex Vandiver 6f20c15ae9 thumbnail: Resolve a race condition when rendering messages.
Messages are rendered outside of a transaction, for performance
reasons, and then sent inside of one.  This opens thumbnailing up to a
race where the thumbnails have not yet been written when the message
is rendered, but the message has not been sent when thumbnailing
completes, causing `rewrite_thumbnailed_images` to be a no-op and the
message being left with a spinner which never resolves.

Explicitly lock and use he ImageAttachment data inside the
message-sending transaction, to rewrite the message content with the
latest information about the existing thumbnails.

Despite the thumbnailing worker taking a lock on Message rows to
update them, this does not lead to deadlocks -- the INSERT of the
Message rows happens in a transaction, ensuring that either the
message rending blocks the thumbnailing until the Message row is
created, or that the `rewrite_thumbnailed_images` and Message INSERT
waits until thumbnailing is complete (and updated no Message rows).
2024-08-01 16:48:16 -07:00
bedo 1ab4723d59 user_groups: Migrate to @typed_endpoint.
Migrate the following endpoints from @has_request_variables
to @typed_endpoint:

- get_user_group()
- delete_user_group()
- update_user_group_backend()
- update_subgroups_of_user_group()
- get_is_user_group_member()
- get_user_group_members()
- get_subgroups_of_user_group()

With tweaks from tabbott to avoid calling thunks unnecessarily.
2024-08-01 09:56:06 -07:00
roanster007 7b3e163d55 refactor: Rename `huddle` to `direct_message_group` in non api files.
This commit completes rename of "huddle" to "direct_message_group"
in all the non API files.

Part of #28640
2024-07-31 23:25:56 -07:00
Prakhar Pratyush 7302d533fe user_status: Update do_update_user_status to send event on commit.
Earlier, we were using 'send_event' in 'do_update_user_status' which
can lead to a situation, if any db operation is added after the
'send_event' in future, where we enqueue events but the action
function fails at a later stage.

Events should not be sent until we know we're not rolling back.

Fixes part of #30489.
2024-07-31 22:33:52 -07:00
Kenneth Rodrigues 97f15d8811 events: Migrate to typed_endpoint.
Migrate `event_register.py` and `tornado` to typed_endpoint.
Modified the tests to work with the migrated endpoints.
2024-07-31 17:10:06 -07:00
Kenneth Rodrigues 0f692436ca user_settings: Migrate to typed_endpoint.
Migrate `user_settings.py` to `typed_endpoint`.
Fix the error messages for the tests.
Migrate required validators.
2024-07-31 17:10:06 -07:00
Mateusz Mandera aaca394813 presence: Remove the queue worker. 2024-07-31 16:46:42 -07:00
Anders Kaseorg 8f1a97f168 users: Reduce date_joined precision to minutes.
Fixes #31188, removing a lot of useless high-entropy data from the
/register response.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-31 12:53:09 -07:00
Alex Vandiver c726d2ec01 thumbnail: Do not Camo old thumbor URLs; serve images directly.
Providing a signed Camo URL for arbitrary URLs opened the server up to
being an open redirector.  Return 403 if the URL is not a user upload,
and the backend image if it is.  Since we do not have ImageAttachment
rows for uploads at a time we wrote `/thumbnail?` URLs, return the
full-size content.
2024-07-24 16:04:34 -07:00
Alex Vandiver e4a8304f57 thumbnail: Store the post-orientation-transformation dimensions.
Modern browsers respect the EXIF orientation information of images,
applying rotation and/or mirroring as specified in those tags.  The
the `width="..."` and `height="..."` tags are to size the image
_after_ applying those orientation transformations.

The `.width` and `.height` properties of libvips' images are _before_
any transformations are applied.  Since we intend to use these to hint
to rendering clients the size that the image should be _rendered at_,
change to storing (and providing to clients) the dimensions of the
rendered image, not the stored bytes.
2024-07-24 09:56:42 -07:00
Alex Vandiver e7ac62aad7 tests: Add a test that the thumb is actually the expected size. 2024-07-24 09:56:42 -07:00
Alex Vandiver d824988bc4 tests: Remove THUMBNAIL_IMAGES overrides, since the setting was dropped.
These were omitted from b42863be4b.
2024-07-24 09:53:14 -07:00
Mateusz Mandera 8f461b3c03 email_mirror: Change default topic name if subject ends up empty.
If the email subject is something like `Fwd:`, it gets stripped to an
empty string, activating the "(no topic)" override. This however leads
to failure if the organization enables the setting forcing every message
to have a topic. Such emails should still go through, so we should just
change the topic value used.
2024-07-22 22:31:45 -07:00
Alex Vandiver 2ea0cc0005 thumbnail: Add a data-original-dimensions attribute.
This allows clients to potentially lay out the thumbnails more
intelligently, or to provide a better "progressive-load" experience
when enlarging the thumbnail.
2024-07-22 22:41:10 -04:00
roanster007 66a96bee71 settings: Add setting to control how animated images are played.
Previously animated images were automatically played in the
message feed of the web app.

Now that we have still thumbnails available for them, we can add a new
personal setting, "web_animate_image_previews", which controls how the
animated images would be played in the web app message feed -- always
played, on hover, or only in the image viewer.

Fixes #31016.
2024-07-22 14:53:31 -07:00
Prakhar Pratyush 6838a7302d test_import_export: Add test coverage for OnboardingUserMessage. 2024-07-22 10:26:33 -07:00
Prakhar Pratyush f48e87cd3c fetch_initial_state: Avoid doing one db query per announcement stream.
In 'fetch_initial_state_data' we were doing one database query
per announcement stream.

This commit updates the logic to prefetch those streams using
select_related hence avoiding the extra db queries.

Fixes #28909.
2024-07-22 10:21:22 -07:00
Alex Vandiver b42863be4b markdown: Show thumbnails for uploaded images.
Fixes: #16210.
2024-07-21 18:41:59 -07:00
Alex Vandiver 71406ac767 thumbnail: Factor frames into account for IMAGE_BOMB_TOTAL_PIXELS. 2024-07-21 18:41:59 -07:00
Alex Vandiver aacf28f7e3 test_classes: Extract a thumbnailing output format helper. 2024-07-21 18:41:59 -07:00
Alex Vandiver 94ff443c00 test_classes: Merge verbose assertEquals into ZulipTestCase. 2024-07-21 18:41:59 -07:00
Kenneth Rodrigues 8b489f4b96 streams: Convert to typed_enpoint. 2024-07-21 12:57:50 -07:00
Kenneth Rodrigues 6815cded83 zerver: Migrate some files to typed_endpoint.
Migrates `invite.py`, `registration.py` and
`email_mirror.py` to use `typed_endpoint`.
2024-07-20 15:46:48 -07:00
Sahil Batra c6e9173297 exceptions: Add new error class to be used for invalid parameter values.
This would help us in avoiding adding translation everytime we use
this error for a new pair of parameters.
2024-07-19 09:24:54 -07:00
Anders Kaseorg d574200423 tests: Consume streaming responses.
Fixes warnings like “ResourceWarning: unclosed file <_io.FileIO
name='/srv/zulip/var/044e5d44-87aa-4c43-abbb-28a144fa6654/test-backend/run_1238680/worker_0/test_uploads/files/thumbnail/2/1e/jmUuDhQC8WlaSRCuc0zQyx7D/img.tif/100x75.webp'
mode='rb' closefd=True>” with warnings enabled.

deque(…, 0) is an efficient way to consume an iterator documented at
https://docs.python.org/3/library/itertools.html#itertools-recipes
under consume.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-19 09:20:56 -07:00
Anders Kaseorg c7281b406f test_delete_unclaimed_attachments: Close file in make_attachment.
Fixes “ResourceWarning: unclosed file <_io.BufferedReader
name='/srv/zulip/zerver/tests/images/text.txt'>” with warnings enabled.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-19 09:20:56 -07:00
Alex Vandiver 4351cc5914 thumbnail: Move get_image_thumbnail_path and split_thumbnail_path. 2024-07-18 13:50:28 -07:00
Alex Vandiver d474600056 tests: Remove a nonsensical no-op vimeo test.
Embeds are disabled in this test, and it tests that it does not embed
anything.
2024-07-18 13:50:28 -07:00
Alex Vandiver 424f3dd503 tests: Break up and reorganize test_markdown.py. 2024-07-18 13:50:28 -07:00
Alex Vandiver ecdf5713c2 tests: Factor out ZulipVerboseEqualTest class, and use more. 2024-07-18 13:50:28 -07:00
Alex Vandiver 08191d3f69 tests: Default ENABLE_FILE_LINKS to False.
Test configuration should mirror default production as much as possible.
2024-07-18 13:50:28 -07:00
Alex Vandiver e02e9c9181 tests: INLINE_URL_EMBED_PREVIEW=False is the default in tests. 2024-07-18 13:50:28 -07:00
Alex Vandiver 060754980c tests: INLINE_IMAGE_PREVIEW=True is already the default. 2024-07-18 13:50:28 -07:00
Tim Abbott f37082c4ca models: Switch font size default to non-compact.
This is the intended longer-term default, and it's polished enough for
testing.
2024-07-18 12:37:06 -07:00
Sahil Batra c1c4c95ce7 settings: Require clients to pass dense_mode value when needed.
Previously, if someone changed the font-size or line height
settings to some value other than the legacy values, we set
dense_mode to False if it was True. This commit changes the
code to require clients to pass dense_mode as False in such
cases and raise an error otherwise.
2024-07-18 11:36:53 -07:00
Anders Kaseorg 97753fe3b0 emoji_names: Rebuild with CLDR 45, emoji-data 15.1.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-17 22:49:52 -07:00
Mateusz Mandera 4a93149435 settings: Rework how push notifications service is configured.
Instead of the PUSH_NOTIFICATIONS_BOUNCER_URL and
SUBMIT_USAGE_STATISTICS settings, we want servers to configure
individual ZULIP_SERVICE_* settings, while maintaining backward
compatibility with the old settings. Thus, if all the new
ZULIP_SERVICE_* are at their default False value, but the legacy
settings are activated, they need to be translated in computed_settings
to the modern way.
2024-07-17 17:14:06 -07:00
Anders Kaseorg 722842a0aa rocketchat: Remove unnecessary SHA-1 hashing of direct message groups.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-17 15:56:00 -07:00
Anders Kaseorg 541699a1c6 mattermost: Remove unnecessary MD5 hashing of direct message groups.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-17 15:56:00 -07:00
Anders Kaseorg 27b0618704 data_import: Fix IdMapper typing.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-17 15:56:00 -07:00
Vector73 7a80fcf042 events: Send `delete_message` event to user who deleted the message.
Fixes #29826.

Co-authored-by: Mukul Goyal <goyal.mukul7689@gmail.com>
Co-authored-by: Aman Agrawal <amanagr@zulip.com>
2024-07-16 23:29:02 -07:00
Sahil Batra 83c4c27b18 settings: Do not allow invalid combinations for information density settings. 2024-07-16 14:04:19 -07:00
Sahil Batra 17541ddacb settings: Toggle dense mode if font and line height values are changed. 2024-07-16 14:04:19 -07:00
Prakhar Pratyush df7ed437c2 compose: Show banner to explain interleaved view messages fading.
In an interleaved view when composing a message we fade messages
which the user is not replying to, to reduce the chance they send
a message to a recipient they didn't intend to.
Also, it reduces the visual/cognitive processing required
to figure out where their message is going to go.

But, it's not necessarily clear to users that what the
fading means, so this commit adds a one-time compose banner
to explain what's going on the first time this comes up.

Fixes part of #29076.
2024-07-16 13:52:30 -07:00
Prakhar Pratyush 35380b095f compose: Show banner to explain non interleaved view messages fading.
In a non interleaved view when composing a message to another
conversation we fade messages which the user is not replying to,
to reduce the chance they send a message to a recipient they didn't
intend to. Also, it reduces the visual/cognitive processing required
to figure out where their message is going to go.

But, it's not necessarily clear to users that what the
fading means, so this commit adds a one-time compose banner
to explain what's going on the first time this comes up.

Fixes part of #29076.
2024-07-16 13:52:29 -07:00
Alex Vandiver 556b92810b thumbnail: Advertize the thumbnail formats at client registration. 2024-07-16 13:22:15 -07:00
Alex Vandiver 6c624805ce upload: Return the closest-rendered thumbnail. 2024-07-16 13:22:15 -07:00
Alex Vandiver d121a80b78 upload: Serve thumbnailed images. 2024-07-16 13:22:15 -07:00
Alex Vandiver 2e38f426f4 upload: Generate thumbnails when images are uploaded.
A new table is created to track which path_id attachments are images,
and for those their metadata, and which thumbnails have been created.
Using path_id as the effective primary key lets us ignore if the
attachment is archived or not, saving some foreign key messes.

A new worker is added to observe events when rows are added to this
table, and to generate and store thumbnails for those images in
differing sizes and formats.
2024-07-16 13:22:15 -07:00
Alex Vandiver 7aa5bb233d tests: Clarify tests in test_thumbnail.py. 2024-07-16 13:22:15 -07:00
Alex Vandiver 5f25fae0fa upload: Batch deleting old attachments.
delete_message_attachments() is faster than calling
delete_message_attachment() one-by-one.
2024-07-16 13:22:15 -07:00
Anders Kaseorg 8843f9f62a tests: Remove deprecated SHA1PasswordHasher.
SHA1PasswordHasher will be removed in Django 5.1.  MD5PasswordHasher
will remain for the purpose of speeding up tests.

Followup to commit ac5161f439 (#29620).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-16 13:06:31 -07:00
Kenneth Rodrigues ba79d759f1 realm: Migrate to typed_endpoint. 2024-07-15 16:49:32 -07:00
Kenneth Rodrigues a7da24a36f validators: Use cleaner syntax for AfterValidator.
Created a function that returns an `AfterValidator` for `check_int_in`
and `check_string_in` instead of having to use a
`lambda` wraper everytime.
2024-07-15 16:49:32 -07:00
Vector73 d21ee6fa23 api: Deprecate uri and add url parameter in "/user_uploads" endpoint. 2024-07-14 22:32:36 -07:00
Anders Kaseorg fe71eaa8f0 ruff: Fix FURB148 `enumerate` index or value is unused.
This is a preview rule, not yet enabled by default.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-14 13:54:32 -07:00
Anders Kaseorg e3a191b99b ruff: Fix FURB154 Use of repeated consecutive `global`, `nonlocal`.
This is a preview rule, not yet enabled by default.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-14 13:53:18 -07:00
Anders Kaseorg 6412c2d630 ruff: Fix FURB142 Use of set.add() in a for loop.
This is a preview rule, not yet enabled by default.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-14 13:52:59 -07:00
Anders Kaseorg 1e9b6445a9 ruff: Fix PLR6104 Use `+=` to perform an augmented assignment directly.
This is a preview rule, not yet enabled by default.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-14 13:49:51 -07:00
Anders Kaseorg b96feb34f6 ruff: Fix SIM117 Use a single `with` statement with multiple contexts.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-14 13:48:32 -07:00
Anders Kaseorg 48202389b8 ruff: Bump target-version from py38 to py310.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-13 22:28:22 -07:00
Anders Kaseorg 3f29bc42b1 ruff: Fix B905 `zip()` without an explicit `strict=` parameter.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-13 22:28:22 -07:00
Anders Kaseorg 0fa5e7f629 ruff: Fix UP035 Import from `collections.abc`, `typing` instead.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-13 22:28:22 -07:00
Anders Kaseorg 531b34cb4c ruff: Fix UP007 Use `X | Y` for type annotations.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-13 22:28:22 -07:00
Anders Kaseorg e08a24e47f ruff: Fix UP006 Use `list` instead of `List` for type annotation.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-13 22:28:22 -07:00
Anders Kaseorg c2214b3904 test_typed_endpoint: Inline pointless type variable.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-13 22:28:22 -07:00
Anders Kaseorg f04e7fc625 test_openapi: Make get_standardized_argument_type understand X | Y.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-13 22:28:22 -07:00
Aman Agrawal 70be9e8c51 narrow: Implement rendering of `with` narrow operators.
Adds server and web app support for processing the new `with`
search operator.

Fixes part of #21505.

Co-authored-by: roanster007 <rohan.gudimetla07@gmail.com>
Co-authored-by: Tim Abbott <tabbott@zulip.com>
2024-07-13 07:18:24 -07:00
bedo bfd54e27b1 custom_profile_fields: Bulk fetch of UserProfile.
bulk fetch query of UserPfrofile against which
user_ids are validated, instead of looping
over user_ids and fetchingeach UserPfrofile resulting
in O(n) queries.
2024-07-13 07:05:13 -07:00
Alex Vandiver 0442e95276 emoji: Use a non-predictable filename.
We use a truncated SHA256 of the id and a server-side secret to make
emoji have non-guessable filenames, while also making collisions
unlikely.

We also adjust the Slack import to use the same SHA-based name,
instead of taking the same name as it had in Slack.
2024-07-12 13:26:47 -07:00
Alex Vandiver 544d3df057 thumbnail: Stop applying MAX_EMOJI_GIF_FILE_SIZE_BYTES before resizing.
b14a33c659 attempted to make the 128k limit apply _after_ resizing,
but left this check, which examines the pre-resized image size.
2024-07-12 13:26:47 -07:00
Alex Vandiver 54f2fabac0 thumbnail: Still emoji are always pngs. 2024-07-12 13:26:47 -07:00
Alex Vandiver 2b3da0e70f fixup! thumbnail: Fix MAX_EMOJI_GIF_FILE_SIZE_BYTES check to be post-resize. 2024-07-12 13:26:47 -07:00
Alex Vandiver f6b99171ce emoji: Derive the file extension from a limited set of content-types.
We thumbnail and serve emoji with the same format as they were
uploaded.  However, we preserved the original extension, which might
mismatch with the provided content-type.

Limit the content-type to a subset which is both (a) an image format
we can thumbnail, and (b) a media format which is widely-enough
supported that we are willing to provide it to all browsers.  This
prevents uploading a `.tiff` emoji, for instance.

Based on this limited content-type, we then reverse to find the
reasonable extension to use when storing it.  This is particularly
important because the local file storage uses the file extension to
choose what content-type to re-serve the emoji as.

This does nothing for existing emoji, which may have odd or missing
file extensions.
2024-07-12 13:26:47 -07:00
Alex Vandiver 62a0611ddb emoji: Pass down content-type, rather than guessing from extension. 2024-07-12 13:26:47 -07:00
Alex Vandiver fa28e3aa0f tests: Split up test_upload.EmojiTest into test_thumbnail. 2024-07-12 13:26:47 -07:00
Sahil Batra ffb7744974 settings: Allow settings to be set to system groups only in production.
We use the already existing server level setting to only allow
settings to be set to system groups, not a named user defined
group as well, in production. But we allow to settings to be set
to any named or anonymous user group in tests and development server.

"can_mention_group" setting can be set to user defined groups
because some of the realms already do that in production.

The existing server level setting is also renamed to make it clear
that both user defined groups and anonymous groups are not allowed
if that setting is set to False.

This commit also changes the error message to be consistent for the
case when a setting cannot be set to user defined groups as per
server level and setting and when a particular setting cannot be set
to user defined groups due to the configuration of that particular
setting. For this we add a new class SystemGroupRequiredError in
exceptions.py so that we need not re-write the error message in
multiple places.
2024-07-12 09:51:49 -07:00
Niklas Fiekas af3a2500df
email_mirror: Also strip "AW:" from subject.
Besides "RE:" and "FWD:", "AW:" (from German "Antwort" for answer) is
another common prefix in email subjects. Let email_mirror automatically
remove it as well.
2024-07-11 11:16:24 -07:00
Alex Vandiver 382cb5bb13 thumbnail: Lock down which formats we parse. 2024-07-11 07:31:39 -07:00
Alex Vandiver 4bc563128e thumbnail: Use a consistent set of supported image types. 2024-07-11 07:31:39 -07:00
Alex Vandiver 5ae34dc42b slack: Store the content-type of realm icons. 2024-07-11 07:31:39 -07:00
Alex Vandiver ff90e5355f upload: Pass down content-type of realm icon/logo to backend.
This saves having to try to re-derive it from the file extension,
which may be ".original" in some cases.
2024-07-11 07:31:39 -07:00
Vector73 a43d8159a9 exceptions: Update DM permission error string. 2024-07-10 18:43:50 -07:00
Prakhar Pratyush 9c614531fb test_subs: Use 'common_subscribe_to_streams' helper function.
In 'test_subs' we were making POST request to add
subscrption using 'self.api_post'.

This commit updates the code to use the test helper
function 'common_subscribe_to_streams' instead.

This prep commit will also help us to avoid adding
'transaction.atomic' context manager to these API calls
individually in the case of error response in the next commit.
2024-07-10 10:50:37 -07:00
Alya Abbott b6e2a1b50d onboarding: Update Welcome Bot's responses to commands. 2024-07-09 13:08:02 -07:00
codewithnick b926ae0904 onboarding: Modify message that welcome bot does not understand.
Previously the bot sent bot commands whenever an undefined message
was sent by the user. This commit intends to fix the problem so that
the bot will only respond to the first message it does not understand
and not reply to any future undefined messages.

Fixes part of #30049.
2024-07-09 13:08:02 -07:00
codewithnick 4b79a77427 onboarding: Change default reply message of welcome bot.
The reply given by welcome bot when it does not understand a message,
sent by the user is changed, so that the bot is more clear to the user.

Fixes part of #30049.
2024-07-09 13:08:02 -07:00
Vector73 121043b6f2 settings: Remove private_message_policy setting.
Removes `private_message_policy` field from both the API and
the backend.
2024-07-08 19:34:17 -07:00
Vector73 6098c2cebe settings: Add two realm settings to restrict direct messages.
Fixes #24467.
2024-07-08 19:34:17 -07:00
Vector73 318d3e3cca user_groups: Add `is_any_user_in_group` function in `user_groups.py`. 2024-07-08 19:34:17 -07:00
Aman Agrawal 9fc6b93347 user_settings: Add option to configure channel click behavior.
This commit does not implement the setting's behavior, just the API
change and settings UI.
2024-07-08 13:56:13 -07:00
Prakhar Pratyush 5bb66e6c99 onboarding: Update initial direct message content.
This commit updates the Welcome Bot's initial
direct message content.

We inform about the tracked onboarding messages
via direct message only if it exists.

Fixes #30051.
2024-07-08 10:32:19 -07:00
Alya Abbott 721c44df10 portico: Update quotes on home page. 2024-07-07 22:44:04 -07:00
roanster007 02d0566dc5 refactor: Rename `Huddle` Django model class to `DirectMessageGroup`.
This commit renames the "Huddle" Django model class to
"DirectMessageGroup", while maintaining the same table --
"zerver_huddle".

Fixes part of #28640.
2024-07-07 21:31:30 -07:00
Alex Vandiver 79f858b4b8 upload: Pass bytes to create_attachment.
This will be used to analyze the bytes for image metadata.
2024-07-07 14:40:07 -07:00
Alex Vandiver f97a30f240 upload: Reorder arguments to parallel upload_message_attachment. 2024-07-07 14:40:07 -07:00
Alex Vandiver f52a93bc14 upload: Stop requiring callers pass in the file size.
This can be calculated because we have the contents.
2024-07-07 14:40:07 -07:00
Alex Vandiver 0a296b2a6e upload: Start storing content-type for new uploads. 2024-07-07 14:40:07 -07:00
Alex Vandiver e29a455b2d avatars: Encode version into the filename.
Hash the salt, user-id, and now avatar version into the filename.
This allows the URL contents to be immutable, and thus to be marked as
immutable and cacheable.  Since avatars are served unauthenticated,
hashing with a server-side salt makes the current and past avatars not
enumerable.

This requires plumbing the current (or future) avatar version through
various parts of the upload process.

Since this already requires a full migration of current avatars, also
take the opportunity to fix the missing `.png` on S3 uploads (#12852).

We switch from SHA-1 to SHA-256, but truncate it such that avatar URL
data does not substantially increase in size.

Fixes: #12852.
2024-07-07 14:40:07 -07:00
Pedro Almeida ddfc2d230f resolve_topic: Prevent incorrect notification during message move.
This commit fixes the bug where the "topic unresolved" notification
is wrongly triggered when moving a message between a resolved and
unresolved topic, except for when the topics have the same name.

To resolve this issue, the commit ensures that resolved/unresolved
notifications are not sent if a message has been moved to a new
topic. This is achieved by comparing the names of the old and new
topics without considering the "resolved prefix".

The commit also accounts for the scenario where `new_topic_name`
has been truncated, indicating that it was resolved and the name
had to change to accommodate the "resolved prefix".

This solution does not try to specially handle the possible case that
a stream has two topics with the same name, even if one is resolved
and another unresolved.

Fixes #29007.
2024-07-07 11:38:14 -07:00
Pedro Almeida f4ca8025da move_stream: Trigger only move notification.
This commit updates the code so that only the moved notification is
triggered when moving a message between a resolved and unresolved
topic in different streams or when moving a topic itself.

This change takes place even when both stream change and resolve
or unresolve a topic takes place in the same API request, as we
now consider it only a move operation.

This fixes a case where a message is moved between topics that
have the same name, but one resolved and another unresolved and
in different streams. Previously a resolved or unresolved
notification would also be sent. Now, this will not happen,
ensuring only the move operation is notified.

Fixes part of #29007.
2024-07-07 11:38:14 -07:00
Shashank Singh 4cce94b667 invites: Add option to receive notification on accepted invitations.
Previously, when a referrer's invitation to Zulip was accepted,
they got a notification from notification-bot indicating
their invitation has been accepted.

This commit adds an option for referrer to decide
whether he wants to receive the direct notification
from the notification-bot.

Fixes: #20398
2024-07-05 17:14:45 -07:00
Prakhar Pratyush ed605328fb onboarding: Mark a few onboarding messages as starred.
To improve onboarding experience following onboarding
messages are marked as starred:
* First message in each onboarding topic.
* Initial DM sent by Welcome bot

Note: The onboarding topic messages needs to be tracked
in 'OnboardingUserMessage' model to get starred.

Fixes #29298.
2024-07-05 15:39:32 -07:00
Prakhar Pratyush 3c5dc73f50 create_user: Mark only tracked onboarding topic messages as unread.
We give the user some messages in their feed, so that they can
learn how to use the home view in a realistic way.

For realms having older onboarding messages, we mark the very
most recent messages as unread.

This commit updates the logic to ONLY mark the tracked onboarding
messages (if present) i.e. messages tracked in 'OnboardingUserMessage'
as unread.

Fixes part of #29298.
2024-07-05 15:39:32 -07:00
Prakhar Pratyush fb836a4f0a onboarding: Add 'OnboardingUserMessage' model.
This prep commit adds a new OnboardingUserMessage model
that will be used to mark the new onboarding messages
for new users as unread and the first message of each
onboarding topic as starred.

This table won't include the old onboarding messages.
2024-07-05 15:39:32 -07:00
Prakhar Pratyush 485a8a1421 add_new_user_history: Add missing historical flags.
This commit adds the missing historical flag to recent
messages added in a new user's feed.

Reason:
* User didn't receive when they were sent, so semantically
the 'historical' flag should be present.
* It helps to avoid the "You (un)subscribed to.." bookend
while reading older messages.
The bookend appears whenever the historical flag flips
between adjacent messages. Earlier, the bookend was visible
between 'recent messages' and 'older messages'. This makes
sure that the bookend is visible only at the moment new
message is sent by the user after account creation.
2024-07-05 15:39:32 -07:00
Prakhar Pratyush f5a0755f16 create_user: Rename constants used in 'add_new_user_history'.
This commit renames the constants:
MAX_NUM_ONBOARDING_MESSAGES to MAX_NUM_RECENT_MESSAGES
MAX_NUM_ONBOARDING_UNREAD_MESSAGES to MAX_NUM_RECENT_UNREAD_MESSAGES
ONBOARDING_RECENT_TIMEDELTA to RECENT_MESSAGES_TIMEDELTA

The term 'onboarding' is preffered to be used for the
new messages sent during realm creation or new user creation.
These constants are related to already present recent messages.
2024-07-05 15:39:32 -07:00
roanster007 52692a6448 refactor: Rename `huddle` to `direct_message_group` in non API.
This commit performs a sweep on the first batch of non API
files to rename "huddle" to "direct_message_group`.

It also renames variables and methods of type -
"huddle_message" to "group_direct_message".

This is a part of #28640
2024-07-04 07:56:31 -07:00
Vector73 ff0cc79fd5 api_docs: Add test to check links of pages in the sidebar. 2024-07-01 16:42:11 -07:00
Sayam Samal 073b116c55 zcommand: Rename night->dark and day->light in the color scheme backend.
As a follow up for f49a11c810, this
commit standardizes the naming of the day and night themes to light
and dark, respectively in the backend. This makes the backend
consistent with the naming used in the frontend and UI.

This also solves a regression introduced in
f49a11c810, where the frontend was sending
"/light" and "/dark" commands to the backend, but the backend was
expecting "/day" and "/night" commands.
2024-06-30 22:45:26 -07:00
Prakhar Pratyush fb5b721421 onboarding_step: Don't show 'visibility_policy_banner' to new users.
Earlier, a one-time 'visibility_policy_banner' was displayed to
existing as well as new users to inform them about the new
"follow/unmute topics" feature.

It makes sense to educate only the existing Zulip users about
the new feature using this banner. New users don't need to know
about following topics right away.

This commit makes changes to NOT show the banner to new users.

This will also help to avoid banner overload in the new user
experience.

Fixes #30615.
2024-06-28 10:41:48 -07:00
Anders Kaseorg b115d44b6a requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-06-27 15:31:43 -07:00
Lauryn Menard 673a01ea0c realm-deactivation: Send email to owners as part of deactivation.
Creates a new "realm_deactivated" email that can be sent to realm
owners as part of `do_deactivate_realm`, via a boolean flag,
`email_owners`.

This flag is set to `False` when `do_deactivate_realm` is used for
realm exports or changing a realm's subdomain, so that the active
organization owners are not emailed in those cases.

This flag is optional for the `deactivate_realm` management command,
but as there is no active user passed in that case, then the email
is sent without referencing who deactivated the realm.

It is passed as `True` for the support analytics view, but the email
that is generated does not include information about the support
admin user who completed the request for organization deactivation.

When an active organization owner deactivates the organization, then
the flag is `True` and an email is sent to them as well as any other
active organization owners, with a slight variation in the email text
for those two cases.

Adds specific tests for when `email_owners` is passed as `True`. All
existing tests for other functionality of `do_deactivate_user` pass
the flag as `False`.

Adds `localize` from django.util.formats as a jinja env filter so
that the dates in these emails are internationlized for the owner's
default language setting in the "realm_deactivated" email templates.

Fixes #24685.
2024-06-26 16:48:18 -07:00
Alex Vandiver 2eaf098c5d upload: Content-type is always defined. 2024-06-26 16:43:11 -07:00
Alex Vandiver 17fb23746f upload: Move methods into zerver.lib.upload from .base. 2024-06-26 16:43:11 -07:00
Alex Vandiver 5cd10ce51d s3: Allow setting a CloudFront URL prefix for avatar and emoji images. 2024-06-26 16:43:11 -07:00
Alex Vandiver 08b24484d1 upload: Remove redundant acting_user_profile argument.
This argument, effectively added in 9eb47f108c, was never actually
used.
2024-06-26 16:43:11 -07:00
Alex Vandiver fb929ca218 thumbnailing: Remove unnecessary third return value from resize_emoji. 2024-06-26 16:43:09 -07:00
Alex Vandiver 0070b5da78 tests: Switch from PIL to pyvips. 2024-06-26 16:42:59 -07:00
Alex Vandiver b14a33c659 thumbnailing: Switch to libvips, from PIL/pillow.
This is done in as much of a drop-in fashion as possible.  Note that
libvips does not support animated PNGs[^1], and as such this
conversion removes support for them as emoji; however, libvips
includes support for webp images, which future commits will take
advantage of.

This removes the MAX_EMOJI_GIF_SIZE limit, since that existed to work
around bugs in Pillow.  MAX_EMOJI_GIF_FILE_SIZE_BYTES is fixed to
actually be 128KiB (not 128MiB, as it actually was), and is counted
_after_ resizing, since the point is to limit the amount of data
transfer to clients.

[^1]: https://github.com/libvips/libvips/discussions/2000
2024-06-26 16:42:57 -07:00
Alex Vandiver 9fb03cb2c7 upload: Factor out common avatar logic. 2024-06-26 16:38:01 -07:00
Alex Vandiver d92993c972 upload: Factor out common emoji logic. 2024-06-26 16:38:01 -07:00
Sahil Batra b44d30add7 events: Optimize computing can_create_streams and related fields.
Currently, for computing fields like can_create_public_streams
and can_create_private_steams fields, is_user_in_group is called
to check whether the user is part of the group which has the
permission. This means that there will be one DB query for each
field.

To optimize this, we now first fetch all the groups that the
user is member of, including the anonymous groups which are
used for settings, such that we can then just check whether
the user is part of the group which has the permission meaning
we would need only one query to compute all the fields.

This would be helpful when settings for other similar fields
will also be migrated to groups framework.
2024-06-24 10:00:08 -07:00
Mateusz Mandera 9f24b30e78 test_push_notifs: Adjust .filter() in test_analytics_api.
The test asserts entries about the zephyr realm anyway. The reason the
filter hasn't been limiting the query to zephyr is that we might
simultaneously want to ensure no other realms received changes - but
that doesn't seem quite right, given that the test doesn't dilligently
set up the initial conditions for all realms to have control over what
exactly happens with them. That makes this logic pretty fragile since if
some new (potentially unrelated) changes to Realm/RemoteRealm initial
state make it so some updates to other realms occur during the early
analytics upload calls in the test, the remote_audit_logs asserts here
will break in a very annoying to debug way.

If we want this query to be general, without limiting to the zephyr
realm, we could alter the test a bit to set up initial conditions
precisely.
2024-06-23 07:44:11 -07:00
Mateusz Mandera f8616fa013 analytics: Send ZULIP_MERGE_BASE to the bouncer. 2024-06-23 07:44:11 -07:00
Anders Kaseorg f7eecb0e03 sentry: Untangle from page_params.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-06-22 18:53:56 -07:00
Alex Vandiver 0153d6dbcd thumbnailing: Move resizing functions into zerver.lib.thumbnail. 2024-06-20 23:06:08 -04:00
Sahil Batra 6164a41864 realm: Remove create_private_stream_policy setting.
This commit removes create_private_stream_policy setting as
we now use new group based setting.

The "/register" response includes realm_create_private_stream_policy
field to return a value representing superset of users who have the
permission to create private channels, as older clients still expect
this field.
2024-06-20 15:48:14 -07:00
Sahil Batra 0bae59fa4b streams: Check permission to create channels based on group setting.
This commit updates code to use new group based setting when
checking permission to create private channels.
2024-06-20 15:48:14 -07:00
Sahil Batra 27171d103e realm: Add new group setting for who can create private channels.
This commit adds new group based setting to control who can create
private channels.
2024-06-20 15:48:14 -07:00
Sahil Batra 428d4284f9 users: Update has_permission to accept realm as argument.
This commit helps in using the realm object which has the
prefetched group settings so that we can avoid extra queries
when calculating fields like can_create_public_streams.
2024-06-20 15:48:14 -07:00
Kenneth Rodrigues 15accfc983 narrow: Use NarrowParameter instead of dictionary.
Earlier we were using the type `OptionalNarrowListT` for all functions
that required "narrow" as a parameter.
This commit changes all the functions accepting a "narrow"
to use a list of the new `NarrowParameter`
instead of `OptionalNarrowListT` which is a list of dicts.
2024-06-20 12:22:37 -07:00
Kenneth Rodrigues 4e05381897 message_fetch: Migrate to typed_endpoint.
Removed the old `narrow_parameter` as we have shifted to
the new `NarrowParameter` Pydantic Object.
This new object provides better error messages for data validation,
hence changed the error messages in `test_message_fetch`.
2024-06-20 12:22:37 -07:00
Kenneth Rodrigues f791d1e1c0 test_message_fetch: Add support for list of error messages. 2024-06-20 12:22:37 -07:00
Kenneth Rodrigues a865977bd5 get_messages_backend: Pass required parameters.
Earlier `num_before` and `num_after` wasn't being passed while
testing.
Moving to typed_endpoint requires all the "required" parameters
to be passed to the function explicitly in tests.
2024-06-20 12:22:37 -07:00
Kenneth Rodrigues 387fba7156 custom_profile: Convert to typed_endpoint.
Convert `custom_profile_fields.py` to use `typed_endpoint`.

Use `TypedDict` from `typing_extensions` instead of `typing`,
to support Pydantic's type checking.
2024-06-19 18:29:28 -07:00
Shubham Padia 9c3e03ba83 custom_profile_fields: Remove reduntant test code block.
We're updating all the fields later in `test_update` rendering
the block in question redundant.
2024-06-18 09:41:56 -07:00
Shubham Padia cf75007ed3 custom_profile_fields: Extract `display_in_profile_summary` tests.
The current `test_update` function is too long, we're extracting out
the tests for `display_in_profile_summary` related changes to make
it shorter.
2024-06-18 09:41:56 -07:00
Shubham Padia 8eeed08e80 custom_profile_fields: Extract `field_data` tests from `test_update`.
The current `test_update` function is too long, we're extracting out
the tests for `field_data` related changes to make it shorter.
2024-06-18 09:41:56 -07:00
Mateusz Mandera 4917e01ffb push_notifications: Migrate to FCM HTTP v1 API.
The legacy API we use via python-gcm is deprecated and about to be
disabled.

Fixes #29768.
2024-06-17 18:26:26 -07:00
Mateusz Mandera 496aa2a0f1 test_push_notifs: Remove gcm_client mock in test_connection_error.
This wasn't doing anything. The test simulates failure to connect to the
bouncer, so the codepaths inside the bouncer don't even get triggered.
2024-06-17 18:23:00 -07:00