Since each loop may add more than one file to the `storage_paths`
list, this may result in more than 1000 files being sent to
delete_message_attachments. Since the S3 backend only supports 1000
elements being deleted at once, we must partition the list into chunks
which are no more than 1000 elements long.
Adds some validation for changing the realm's max invites via the
support view so that it is not set below the default max for the
realm's plan type, and so that if it's currently set to the default
max it's not reset to that same value.
This commit adds a new `group_size` field to the `DirectMessageGroup`
model, and backfills its value to each of the existing direct message
groups.
Fixes part of #25713
Earlier, we were using 'send_event' in 'do_regenerate_api_key'
which can lead to a situation, if any db operation is added after
the 'send_event' in future, where we enqueue events but the action
function fails at a later stage.
Events should not be sent until we know we're not rolling back.
Fixes part of #30489.
The export tool was only exporting the already-thumbnailed emoji file,
omitting the original one. Now we make sure to export the .original file
too, like we do for avatars, and make the import tool process it
directly, to thumbnail it directly and generate a still in the case of
animated emojis.
Otherwise, the imported realm wouldn't have the <emoji>.png.original
file that we generally expect to have accessible, and stills for
animated emojis were completely missing.
Convert `message_send.py` use `typed endpoint`.
Disable `message_send` endpoint `to` parameter in the `openapi`
`validate_json_schema` check, because it is a special case where the
content type of the parameter is application/json but the
parameter may or may not be JSON encoded since previously we also
accepted a raw string and some ad-hoc bot might still depend on sending
a raw string.
Remove unused validators from `validator.py`.
There was a bug here that would trigger an exception inside
`sync_user_profile_custom_fields`, causing it to get logged with
logging.warning, when an attribute configured for SAML custom profile
field sync was missing from a SAMLResponse or had an empty value.
`sync_user_profile_custom_fields` expects valid values, and None is not
valid.
We could consider a slightly different behavior here instead - when an
attribute is sent with no value in the SAMLResponse, that means the attr
has no value in the IdP's user directory - so perhaps a better behavior
would be to also remove the custom profile field value in Zulip. However
there are two issues with that:
1. It's not necessarily the best behavior, because an organization might
want the "user doesn't have this attribute set at the IdP level" state
to just mean that the user should be free to set the value manually in
Zulip if they wish. And having that value get reset on every login would
then be an issue. The implementation in this commit is consistent with
this philosophy.
2. There's some implementation difficulty - upstream
`self.get_attr(...)`, which we use for reading the attr value from the
SAMLResponse, doesn't distinguish between an attribute being sent with
no value and the attribute not being sent at all - in both cases it
returns None. So we'd need some extra work here with parsing the
SAMLResponse properly, to be able to know when the custom profile field
should get cleared.
Replace the SOCIAL_AUTH_SYNC_CUSTOM_ATTRS_DICT with
SOCIAL_AUTH_SYNC_ATTRS_DICT, designed to support also regular user attrs
like role or full name (in the future).
Custom attributes can stay configured as they were and will get merged
into SOCIAL_AUTH_SYNC_ATTRS_DICT in computed_settings, or can be
specified in SOCIAL_AUTH_SYNC_ATTRS_DICT directly with "custom__"
prefix.
The role sync is plumbed through to user creation, so users can
immediately be created with their intended role as provided by the IdP
when they're creating their account, even when doing this flow without
an invitiation.
This commit make changes in code to include can_manage_group
field to UserGroup objects passed with response of various endpoints
including "/register" endpoint and also in the group object
send with user group creation event.
Earlier there was only a realm level setting for configuring
who can edit user groups. A new group level setting is also added
for configuring who can manage that particular group.
Now, a user group can be edited by a user if it is allowed from
realm level setting or group level setting.
This commit make changes to also use group level setting
in determining whether a group can be edited by user or not.
Also, updated tests to use api_post and api_delete helpers instead
of using client_post and client_delete helpers with different users
being logged in.
This commit adds a new group level setting can_manage_group
for configuring who can manage a group. This commit only adds
the field in database and make changes to automatically create
single user groups corresponsing to acting user
which will be the default value for this setting.
Fixes part of #25928.
Earlier there was a single backend test for testing group edit policy
for creating and deleting user group.This commit make changes in the test
and now there are two separate tests for testing group edit policy for
creating and deleting user groups.
This was done because in future commits we will be adding a
realm level setting for configuring who can create user groups.
Also, updated tests to use api_post and api_delete helpers instead
of using client_post and client_delete helpers with different users
being logged in.
This commit refactors code in user_groups_in_realm_serialized
such that we do not prefetch "can_mention_group__direct_members"
and "can_mention_group__direct_subgroups" using prefetch_related
and instead fetch members and subgroups for all groups in separate
queries and then use that data to find the members and subgroups
of the group used for that setting.
This change helps us in avoiding two prefetch queries for each
setting when we add more group settings.
Earlier, we were using 'send_event' in do_update_message_flags
which can lead to a situation where we enqueue events but the
function fails at a later stage.
Events should not be sent until we know we're not rolling back.
Fixes part of #30489.
'do_update_message' is within a db transaction, this commit
updates the 'do_clear_mobile_push_notifications_for_ids' function
used in 'do_update_message' to queue event on commit.
Events should not be sent until we know we're not rolling back,
otherwise it can lead to a situation where we enqueue events but
the function fails at a later stage.
Previously, this logic did the database queries to look up UserProfile
objects in a loop.
Fixes#21820.
Significantly improves Stream creation time and also unsusbcribing users.
Tested stream creation with 10k stream subscribers:
- before: 127 seconds ~2 mins
- after: 17 seconds ~0.3 min
Add a test case for user unsubscribing themself.
Show user card popover for scheduled messages overlay, compose box
preview, message edit preview, message edit history.
`.messagebox` was chosen as the selector since that was the nearest
parent class that was common for all of the above.
`@all` does not have a popover and that's why it will have the same
pointer as its parent element. We also introduce a new class called
`.user-mention-all` for managing css rules specific to that mention.
The 'tutorial_status' field on 'UserProfile' model is
no longer used to show onboarding tutorial.
This commit removes the 'tutorial_status' field,
'POST users/me/tutorial_status' endpoint, and
'needs_tutorial' parameter in 'page_params'.
Fixes part of zulip#30043.
We plan to remove the 'tutorial_status' field from UserProfile
table as it is no longer used to show tutorial.
The field is also used to narrow a new user in DM with
welcome bot on the first load.
This prep commit updates the logic to use a new OnboardingStep
for the narrowing behaviour on the first load. This will help
in removing the 'tutorial_status' field.
Earlier, we were using 'send_event' in
'notify_realm_custom_profile_fields' which can lead to a situation,
if any db operation is added after the 'send_event' in the action
functions using it, where we enqueue event but the action function
fails at a later stage.
Events should not be sent until we know we're not rolling back.
Fixes part of #30489.
This bug was introduced in da9e4e6e54.
validate validate_plan_for_authentication_methods is already called
inside validate_authentication_methods_dict_from_api, conditionally on
settings.BILLING_ENABLED. This additional, redundant call runs
regardless of BILLING_ENABLED, and thus prevents a self-hosted server
from enabling certain backends in the organization settings UI.
The impact of this is limited - in order to encounter this bug, a
self-hosted server would have to first disable the backend in the UI, as
self-hosted realms are created with all backend flags enabled. A backend
doesn't show up in the org settings UI until it is first enabled in
AUTHENTICATION_BACKENDS in settings.py - that's why this is a rare
state. A sequence of steps like this has to be followed to reproduce:
1. Add the backend to AUTHENTICATION_BACKENDS in settings.py.
2. Disable the backend in the org settings UI.
3. Now try to re-enable it, which fails due to the bug.
This commit removes create_web_public_stream_policy setting
since web-public channel creation permissions are now
handled by group-based setting.
We still pass "realm_create_web_public_stream_policy" in
"/register" response though for older clients with its
value being set depending on the value of group based
setting. If we cannot set its value to an appropriate enum
corresponding to the group setting, then we set it to
"Admins and moderators" considering that server will not
allow the users without permissions to create web-public
channels but the client can make sure that UI is
available to the users who have permission.
Messages are rendered outside of a transaction, for performance
reasons, and then sent inside of one. This opens thumbnailing up to a
race where the thumbnails have not yet been written when the message
is rendered, but the message has not been sent when thumbnailing
completes, causing `rewrite_thumbnailed_images` to be a no-op and the
message being left with a spinner which never resolves.
Explicitly lock and use he ImageAttachment data inside the
message-sending transaction, to rewrite the message content with the
latest information about the existing thumbnails.
Despite the thumbnailing worker taking a lock on Message rows to
update them, this does not lead to deadlocks -- the INSERT of the
Message rows happens in a transaction, ensuring that either the
message rending blocks the thumbnailing until the Message row is
created, or that the `rewrite_thumbnailed_images` and Message INSERT
waits until thumbnailing is complete (and updated no Message rows).
Migrate the following endpoints from @has_request_variables
to @typed_endpoint:
- get_user_group()
- delete_user_group()
- update_user_group_backend()
- update_subgroups_of_user_group()
- get_is_user_group_member()
- get_user_group_members()
- get_subgroups_of_user_group()
With tweaks from tabbott to avoid calling thunks unnecessarily.
Earlier, we were using 'send_event' in 'do_update_user_status' which
can lead to a situation, if any db operation is added after the
'send_event' in future, where we enqueue events but the action
function fails at a later stage.
Events should not be sent until we know we're not rolling back.
Fixes part of #30489.
Providing a signed Camo URL for arbitrary URLs opened the server up to
being an open redirector. Return 403 if the URL is not a user upload,
and the backend image if it is. Since we do not have ImageAttachment
rows for uploads at a time we wrote `/thumbnail?` URLs, return the
full-size content.
Modern browsers respect the EXIF orientation information of images,
applying rotation and/or mirroring as specified in those tags. The
the `width="..."` and `height="..."` tags are to size the image
_after_ applying those orientation transformations.
The `.width` and `.height` properties of libvips' images are _before_
any transformations are applied. Since we intend to use these to hint
to rendering clients the size that the image should be _rendered at_,
change to storing (and providing to clients) the dimensions of the
rendered image, not the stored bytes.
If the email subject is something like `Fwd:`, it gets stripped to an
empty string, activating the "(no topic)" override. This however leads
to failure if the organization enables the setting forcing every message
to have a topic. Such emails should still go through, so we should just
change the topic value used.
This allows clients to potentially lay out the thumbnails more
intelligently, or to provide a better "progressive-load" experience
when enlarging the thumbnail.
Previously animated images were automatically played in the
message feed of the web app.
Now that we have still thumbnails available for them, we can add a new
personal setting, "web_animate_image_previews", which controls how the
animated images would be played in the web app message feed -- always
played, on hover, or only in the image viewer.
Fixes#31016.
In 'fetch_initial_state_data' we were doing one database query
per announcement stream.
This commit updates the logic to prefetch those streams using
select_related hence avoiding the extra db queries.
Fixes#28909.
Fixes warnings like “ResourceWarning: unclosed file <_io.FileIO
name='/srv/zulip/var/044e5d44-87aa-4c43-abbb-28a144fa6654/test-backend/run_1238680/worker_0/test_uploads/files/thumbnail/2/1e/jmUuDhQC8WlaSRCuc0zQyx7D/img.tif/100x75.webp'
mode='rb' closefd=True>” with warnings enabled.
deque(…, 0) is an efficient way to consume an iterator documented at
https://docs.python.org/3/library/itertools.html#itertools-recipes
under consume.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Previously, if someone changed the font-size or line height
settings to some value other than the legacy values, we set
dense_mode to False if it was True. This commit changes the
code to require clients to pass dense_mode as False in such
cases and raise an error otherwise.
Instead of the PUSH_NOTIFICATIONS_BOUNCER_URL and
SUBMIT_USAGE_STATISTICS settings, we want servers to configure
individual ZULIP_SERVICE_* settings, while maintaining backward
compatibility with the old settings. Thus, if all the new
ZULIP_SERVICE_* are at their default False value, but the legacy
settings are activated, they need to be translated in computed_settings
to the modern way.
In an interleaved view when composing a message we fade messages
which the user is not replying to, to reduce the chance they send
a message to a recipient they didn't intend to.
Also, it reduces the visual/cognitive processing required
to figure out where their message is going to go.
But, it's not necessarily clear to users that what the
fading means, so this commit adds a one-time compose banner
to explain what's going on the first time this comes up.
Fixes part of #29076.
In a non interleaved view when composing a message to another
conversation we fade messages which the user is not replying to,
to reduce the chance they send a message to a recipient they didn't
intend to. Also, it reduces the visual/cognitive processing required
to figure out where their message is going to go.
But, it's not necessarily clear to users that what the
fading means, so this commit adds a one-time compose banner
to explain what's going on the first time this comes up.
Fixes part of #29076.
A new table is created to track which path_id attachments are images,
and for those their metadata, and which thumbnails have been created.
Using path_id as the effective primary key lets us ignore if the
attachment is archived or not, saving some foreign key messes.
A new worker is added to observe events when rows are added to this
table, and to generate and store thumbnails for those images in
differing sizes and formats.
SHA1PasswordHasher will be removed in Django 5.1. MD5PasswordHasher
will remain for the purpose of speeding up tests.
Followup to commit ac5161f439 (#29620).
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Alex Vandiver
Tim Abbott
Lauryn Menard
roanster007
Prakhar Pratyush
Mateusz Mandera
Kenneth Rodrigues
Ujjawal Modi
Sahil Batra
bedo
Shubham Padia
Prakhar Pratyush
Vector73
Anders Kaseorg