On the frontend, the selection is still a dropdown of system groups but
on the API level, we have started accepting anonymous groups similar to
other settings.
We've kept require system groups true for now until we switch to group
picker on the frontend.
With the introduction of `assignee_updated` parameter in the library,
- Github, Gitea, Gogs can display the assignee in assignment events.
- Github can display the user unassigned in unassignment events.
Fixes https://chat.zulip.org/#narrow/channel/127-integrations/near/1965136
We didn't have thumbnailing for images coming from data import and this
commit adds the functionality.
There are a few fundamental issues that the implementation needs to
solve.
1. The images come from an untrusted source and therefore we don't want
to just pass them through to thumbnailing without checking. For that
reason, we cannot just import ImageAttachment rows from the export
data, even for zulip=>zulip imports.
The right way to process images is to pass them to maybe_thumbail(),
which runs libvips_check_image() on them to verify we're okay with
thumbnailing, creates ImageAttachment rows for them and sends them
to the thumbnailing queue worker. This approach lets us handle both
zulip=>zulip and 3rd party=>zulip imports in the same way,
2. There is a somewhat circular dependency between the Message,
Attachment and ImageAttachment import process:
- ImageAttachments would ideally be created after importing
Attachments, but they need to already exist at the time of Message
import. Otherwise, the markdown processor doesn't know it has to add
HTML for image previews to messages that reference images. This would
mean that messages imported from 3rd party tools don't get image
previews.
- Attachments only get created after Message import however, due to the
many-to-many relationship between Message and Attachment.
This is solved by fixing up some data of Attachments pre-emptively, such
as the path_ids. This gives us the necessary information for creating
ImageAttachments before importing Messages.
While we generate ImageAttachment rows synchronously, the actual
thumbnailing job is sent to the queue worker. Theoretically, the worker
could be very backlogged and not process the thumbnails anytime soon.
This is fine - if the app is loaded and tries to display a message with
such a not-yet-generated thumbnail, the code in `serve_file` will
generate the thumbnails synchronously on the fly and the user will see
the image preview displayed normally. See:
1b47134d0d/zerver/views/upload.py (L333-L342)
This commit reformats hashed medium static avatar files("-medium.png"
files). Unlike user-uploaded avatar files, avatars served alongside
static files are hashed by Django. To implement the URL patterns for
finding medium-size avatar URLs that are hardcoded into current
versions of the mobile apps, the medium avatar files need to be
adjusted to include the "-medium" string just before the file type.
For example, the URL format will now be:
Before: welcome-bot-medium.123123321.png
After: welcome-bot.123123123-medium.png
This commit introduces an assertion to verify that the avatar file for
system bots exists and findable.
In development, it'll assert that the avatar file exists in the static
directory. This isn't done in production environment to avoid
unnecessary overhead. It helps verify that the protocol to fetch system
bot avatars still works when making changes during development.
In production it'll check if the avatar file exists in the STATIC_ROOT
and return a default avatar png if it doesn't.
Previously, requesting system bots URLs did not return any -medium.png
variants and SVG file was also used for notification bots' avatar, which
was problematic.
In this commit, the -medium.png variants is added for the avatars of
system bots and zulip-icon-square.svg is also converted into
notification-bot.png for the notification bot. The get_avatar_url method
has been updated to return the "medium" file variants for the system
bots.
Additionally, the system bots' avatar files is moved to a dedicated
directory to simplify the hashing logic for these files. Now, all files
in the "images/static_avatars/" directory will be hashed.
The old endpoint for updating a user worked only via user id. Now we add
a different entry to this functionality, fetching the user by
.delivery_email.
update_user_backend becomes the main function handling all the logic,
invoked by the two endpoints.
This adds a new special UserProfile flag can_change_user_emails(disabled
by default) and the ability for changing the email address of users in
the realm via update_user_backend. This is useful for allowing
organizations to update user emails without needing to set up a SCIM
integration, but since it gives the ability to hijack user accounts, it
needs to be behind this additional permission and can't be just given to
organization owners by default. Analogical to how the
create_user_backend endpoint works.
This commit makes sure system bots avatar files are hashed when served
as static files. This way, requests for these avatar files will be
served with long-lived caching headers by our nginx (see #22275).
We don't need to worry about stale caches for these files because they
will only be used by system bots.
Fixes#31458.
This makes a Zulip server more isolated than relying on gravatar, and
avoids complex logistics if in the future we move system bots to live
inside individual realms.
Co-authored-by: PieterCK <pieterceka123@gmail.com>
This commit adds support to add subgroups to a group while
creating it.
User can add the subgroups to group irrespective of permissions
like user can add members during creating it.
This commit updates code to allow users with permission
to add members to add subgroups as well. And only users
with permission to manage the group can remove subgroups.
Also updated tests to check permissions in separate tests
and removed them from the existing test.
The comment about non-admins and non-moderators who are not
member of the group cannot update subgroups of that group
is not correct since there is no such restriction now after
c9d527603. The test passes because the member user is not
part of can_manage_group or can_manage_all_groups.
This is not the best factored version of this, but it saves effort
changing the tests, and importantly should make failures involving
metadata only take a couple seconds rather than first doing a giant
BSON read before learning about them.
Removed `move_messages_between_streams_policy` property, as the permission
to move messages between channels is now controlled by
`can_move_messages_between_channels_group` setting.
Users with permission to manage the group have all the permissions
including joining/leaving the group, adding others group which also
have a separate setting to control them.
So, it makes sense to just check managing permissions first in
access_user_group_for_update and then check the specific permission.
There is no behavioral change in this commit, it only changes the
order of checking permissions.
We want to allow the user, who can add others to group, to
join the group as well irrespective of can_join_group setting.
Previously, the permission to add others (or say anyone) was
controlled by can_manage_group setting, but now it is controlled
by can_add_members_group setting. This commit fixes the code to
use can_add_members_group setting to check permission for joining
the group.
This commit also improves the tests for checking permission to
join the group such that different settings are tested in isolation.
This commit makes the third-party data converters check for invalid user
emails. If it finds any, it’ll raise an Exception and show an error
message with all the bad emails listed out.
Fixes: #31783
The main change is redefining ALLOW_GROUP_VALUED_SETTINGS to not
control code, but instead to instead control the configuration for
whether settings that have not been converted to use our modern UI
patterns should require system groups.
Fundamentally, it's the same for the realm/stream group-valued
settings, which don't have the new UI patterns yet.
We remove the visual hiding of the "can manage group" setting, which
was hidden only due to transitions being incomplete.
Fixes ##31935.
do_update_user_custom_profile_data_if_change can't be durable as it's
invoked within `sync_ldap_user_data`, which is already in
transaction.atomic.
This change requires a few additional tweaks to untangle other related
transactions. The top level view functions up the codepath now use
durable=True. check_remove_custom_profile_field_value is called inside
do_update_user, so it no longer can be durable and should be switched to
savepoint=False. In turn, its remaining caller - the view
remove_user_custom_profile_data - gets switched to durable=True.
Fixes a spurious error that’s logged and ignored during the Puppeteer
tests, introduced by commit eef65d7e30
(#31438).
Signed-off-by: Anders Kaseorg <anders@zulip.com>
'realm_upload_quota_mib` is updated when `plan_type` changes.
Earlier, we were including 'upload_quota' to update
`realm_upload_quota_mib` in extra_data field of 'realm op: update'
event format when property='plan_type'.
This commit migrate those two parameters to `realm op: update_dict`
event format.
* None of the clients processes these fields, so no compatibility
code required.
* Renamed `upload_quota` to `upload_quota_mib` as it better aligns
with our goal to encode units in the client-facing API names.
Also, it helps to avoid extra code to update 'realm_upload_quota_mib`
in web client, web client simply aligns with
'realm["realm_" + key] = value'.
Few of the properties in the data field of 'realm/update_dict'
event format were no longer in sorted order (by alphabet).
This commit rearranges them to maintain the order.
In 'do_change_realm_plan_type' function we use 'do_set_realm_property'
to set 'enable_spectator_access' to False.
There is no need to again update that field.
This change was included in d2f2fbf6b9.
Earlier we use to restrict admins, moderators or members of a group to
manage that group if they were part of the realm wide
`can_manage_all_groups`. We will not do that anymore and even
non-members of a group regardless of role can manage a group if they are
part of `can_manage_all_groups`.
See
https://chat.zulip.org/#narrow/stream/101-design/topic/Group.20add.20members.20dropdown/near/1952902
to check more about the migration plan for which this is the last step.
Fixes#25942.
Users with permission to manage the group (either on the group level or
realm level) should be able to add members to the group without being
present in can_add_members_group.
Removing members will be controlled by `can_manage_group` until we add
`can_remove_members_group` in the future.
Users with permission to manage a group can add members to that group by
default without being present in `can_add_members_group`.
Earlier, only public data export was possible via `POST /export/realm`
endpoint. This commit adds support to create full data export with
member consent via that endpoint.
Also, this adds a 'export_type' parameter to the dictionaries
in `realm_export` event type and `GET /export/realm` response.
Fixes part of #31201.
The user groups, fetched to send events when deactivating or
reactivating a user, are ordered by ID so that we can avoid
flaky behavior in tests when verifying event details in
test_do_deactivate_user and test_do_reactivate_user tests in
test_events.py.
Document setting the keys that will send a message or create a new
line when composting a message via the compose box and via the
personal settings overlay.
Updates links and redirect from former help center article about
this feature.
Fixes#31620.
This commit updates backend code to not allow adding deactivated
users to groups including when creating groups and also to not
allow removing deactivated users from groups.
There is no behavioral changes to deactivated users as we do
not create UserMessage rows or call the notification code path
for deactivated users in a user group mention. But it is better
to not include the deactivated users in fields like
"mention_user_ids", so this commit updates the code to not
include deactivated users in the computed mention data.
This commit updates code to not include deactivated users in the
anonymous group settings data sent to clients, where the setting
value is sent as a dict containing members and subgroups of the
anonymous group.
This commit updates code to not include deactivated users in
members list in the user groups object sent in "/register"
and "GET /user_groups" response and also in the response
returned by endpoint like "GET /user_groups/{group_id}/members".
The events code is also update to handle this -
- We expect clients to update the members list on receiving
"realm_user/update" event on deactivation. But for guests
who cannot access the user, "user_group/remove_members"
event is sent to update the group members list on deactivation.
- "user_group/add_members" event is sent to all the users on
reactivating the user.
We previously did not update the subscribers list for unsubscribed
and never subscribed streams when a user is deactivated or a
guest user loses access to some user.
For the six realm settings mentioned in the main description of
the /api/update-message endpoint, link back to that page in the
/api/register-queue and /api/get-events endpoints. This way we
can maintain a centralized point of documentation for how these
settings work for message content edits and moving messages.
The descriptions in the events and register pages focuses on the
specifics for each realm setting, e.g., when a value is added or
changed for a particular realm setting.
We show a confirmation dialog explaining the "resolve topics"
feature when the user marks a topic resolved for the first time.
If the user confirms the action, we mark the
topic resolved, else we don't.
We don't show anything the first time a topic is marked
unresolved.
Fixes#31242
Limiting lookups by delivery_email to users with "everyone" email
visibility is overly simplistic. We can successfully do these lookups
whenever the requester has the permission to view the real email address
of the user they're looking up.
This is helpful for taking an "acting user" and getting the list of
email_address_visibility values such that the UserProfiles with those
values of the setting permit the acting user to view their
deliver_email.
This can be used for a query "all users whose delivery_email is viewable
by <requester>" in an upcoming commit.
The added code is ugly, but at least it lets us simplify some similarly
ugly logic in can_access_delivery_email.
Ths hardcoded documentation of which values are possible was destined
to end up inaccurate and out-of-date; and meanwhile, we do have a part
of the API that already has these data in machine-readable format.
This commit updates the 'notify_reaction_update' function to use
the generic 'event_recipient_ids_for_action_on_messages' function.
It helps to add hardening such that if the invariant "no usermessage
row corresponding to a message exists if the user loses access to the
message" is violated due to some bug, it has minimal user impact.
Earlier, submessage was not live-updated for users who joined
the stream after the message was sent.
This commit fixes that bug.
Also, now we use 'event_recipient_ids_for_action_on_messages'.
It helps to add hardening such that if the invariant "no usermessage
row corresponding to a message exists if the user loses access to the
message" is violated due to some bug, it has minimal user impact.
Earlier, we were sending 'delete_message' event to all active
subscribers of the stream.
We shouldn't send event to those users who don't have access
to the deleted message in a private stream with protected history.
This commit fixes that bug.
Also, now we use 'event_recipient_ids_for_action_on_messages'.
It helps to add hardening such that if the invariant "no usermessage
row corresponding to a message exists if the user loses access to the
message" is violated due to some bug, it has minimal user impact.
Added `result_` prefix to differentiate it from upcoming `message_ids`
parameter to the API request. Also, this is final `message_ids` that
we will fetch the messages for. So, a `result` prefix makes sense here.
Earlier, we used to store the key data related to realm exports
in RealmAuditLog. This commit adds a separate table to store
those data.
It includes the code to migrate the concerned existing data in
RealmAuditLog to RealmExport.
Fixes part of #31201.
This commit updates code to store the realm export stats in
json format instead of plain text.
This will help in storing the stats as JsonField in RealmExport table.
This prevents a deadlock between the thumbnailing worker and message
sending, as follows:
1. A user uploads an image, making Attachment and ImageAttachment
rows, as well as enqueuing a job in the thumbnailing queue.
2. Message sending starts a transaction, creates the Message row,
and calls `do_claim_attachments`, which edits the Attachment row
of the upload (implicitly locking it).
3. The thumbnailing worker starts a transaction, locks the
ImageAttachment row for its image, thumbnails it, and then
attempts to `select_for_update()` the message objects (joined to
the Attachments table) to find the ones which link to the
attachment in question. This query blocks, since "a locking
clause without a table list affects all tables used in the
statement"[^1] and the message-send request already has a write
lock on the Attachments row in question.
4. The message-send request attempts to re-fetch the ImageAttachment
row inside the transaction, which tries to pull a lock on it.
5. Deadlock, because the message-send request has the Attachment
lock, and waits for the ImageAttachment lock; the thumbnailing
worker has the ImageAttachment lock, and waits for the Attachment
lock.
We break this deadlock by limiting the
`update_message_rendered_content` `select_for_update` to only take
the lock on the Message table, and not also the Attachments table --
no changes will be made to the Attachments, so no lock is necessary
there. This allows the thumbnailing worker to successfully pull the
empty list of messages (since the message-send request has not
commits its transaction, and thus the Message row is not visible
yet), and release its ImageAttachment lock so that the message-send
request can proceed.
[^1]: https://www.postgresql.org/docs/current/sql-select.html#SQL-FOR-UPDATE-SHARE
This better simulates the Slack API, which is important, since some
integrations check this response and decide whether the Slack endpoint
is working based on what they receive.
These files are necessary for the protocol to verify that the file
upload was completed successfully. Rather than delete them, we update
their StorageClass if it is non-STANDARD.
Because the main indexes on end_time either don't include realm_id or
do include subgroup, passing an explicit subgroup=None for
single-realm queries to read CountStats that don't use the subgroups
feature greatly improves the query plans.
We create an unnamed user group with just the group creator as it's
member when trying to set the default. The pattern I've followed across
most of the acting_user additions is to just put the user declared
somewhere before the check_add_user_group and see if the test passes.
If it does not, then I'll look at what kind of user it needs to be set
to `acting_user`.
Shubham Padia
Tim Abbott
Mateusz Mandera
Niloth P
PieterCK
Sahil Batra
Anders Kaseorg
Prakhar Pratyush
Vector73
Harsh
Alya Abbott
Lauryn Menard
Kislay Udbhav Verma
Aman Agrawal
Alex Vandiver
Pierre Carru