We previously did not update the subscribers list for unsubscribed
and never subscribed streams when a user is deactivated or a
guest user loses access to some user.
For the six realm settings mentioned in the main description of
the /api/update-message endpoint, link back to that page in the
/api/register-queue and /api/get-events endpoints. This way we
can maintain a centralized point of documentation for how these
settings work for message content edits and moving messages.
The descriptions in the events and register pages focuses on the
specifics for each realm setting, e.g., when a value is added or
changed for a particular realm setting.
We show a confirmation dialog explaining the "resolve topics"
feature when the user marks a topic resolved for the first time.
If the user confirms the action, we mark the
topic resolved, else we don't.
We don't show anything the first time a topic is marked
unresolved.
Fixes#31242
Limiting lookups by delivery_email to users with "everyone" email
visibility is overly simplistic. We can successfully do these lookups
whenever the requester has the permission to view the real email address
of the user they're looking up.
This is helpful for taking an "acting user" and getting the list of
email_address_visibility values such that the UserProfiles with those
values of the setting permit the acting user to view their
deliver_email.
This can be used for a query "all users whose delivery_email is viewable
by <requester>" in an upcoming commit.
The added code is ugly, but at least it lets us simplify some similarly
ugly logic in can_access_delivery_email.
Ths hardcoded documentation of which values are possible was destined
to end up inaccurate and out-of-date; and meanwhile, we do have a part
of the API that already has these data in machine-readable format.
This commit updates the 'notify_reaction_update' function to use
the generic 'event_recipient_ids_for_action_on_messages' function.
It helps to add hardening such that if the invariant "no usermessage
row corresponding to a message exists if the user loses access to the
message" is violated due to some bug, it has minimal user impact.
Earlier, submessage was not live-updated for users who joined
the stream after the message was sent.
This commit fixes that bug.
Also, now we use 'event_recipient_ids_for_action_on_messages'.
It helps to add hardening such that if the invariant "no usermessage
row corresponding to a message exists if the user loses access to the
message" is violated due to some bug, it has minimal user impact.
Earlier, we were sending 'delete_message' event to all active
subscribers of the stream.
We shouldn't send event to those users who don't have access
to the deleted message in a private stream with protected history.
This commit fixes that bug.
Also, now we use 'event_recipient_ids_for_action_on_messages'.
It helps to add hardening such that if the invariant "no usermessage
row corresponding to a message exists if the user loses access to the
message" is violated due to some bug, it has minimal user impact.
Added `result_` prefix to differentiate it from upcoming `message_ids`
parameter to the API request. Also, this is final `message_ids` that
we will fetch the messages for. So, a `result` prefix makes sense here.
Earlier, we used to store the key data related to realm exports
in RealmAuditLog. This commit adds a separate table to store
those data.
It includes the code to migrate the concerned existing data in
RealmAuditLog to RealmExport.
Fixes part of #31201.
This commit updates code to store the realm export stats in
json format instead of plain text.
This will help in storing the stats as JsonField in RealmExport table.
This prevents a deadlock between the thumbnailing worker and message
sending, as follows:
1. A user uploads an image, making Attachment and ImageAttachment
rows, as well as enqueuing a job in the thumbnailing queue.
2. Message sending starts a transaction, creates the Message row,
and calls `do_claim_attachments`, which edits the Attachment row
of the upload (implicitly locking it).
3. The thumbnailing worker starts a transaction, locks the
ImageAttachment row for its image, thumbnails it, and then
attempts to `select_for_update()` the message objects (joined to
the Attachments table) to find the ones which link to the
attachment in question. This query blocks, since "a locking
clause without a table list affects all tables used in the
statement"[^1] and the message-send request already has a write
lock on the Attachments row in question.
4. The message-send request attempts to re-fetch the ImageAttachment
row inside the transaction, which tries to pull a lock on it.
5. Deadlock, because the message-send request has the Attachment
lock, and waits for the ImageAttachment lock; the thumbnailing
worker has the ImageAttachment lock, and waits for the Attachment
lock.
We break this deadlock by limiting the
`update_message_rendered_content` `select_for_update` to only take
the lock on the Message table, and not also the Attachments table --
no changes will be made to the Attachments, so no lock is necessary
there. This allows the thumbnailing worker to successfully pull the
empty list of messages (since the message-send request has not
commits its transaction, and thus the Message row is not visible
yet), and release its ImageAttachment lock so that the message-send
request can proceed.
[^1]: https://www.postgresql.org/docs/current/sql-select.html#SQL-FOR-UPDATE-SHARE
This better simulates the Slack API, which is important, since some
integrations check this response and decide whether the Slack endpoint
is working based on what they receive.
These files are necessary for the protocol to verify that the file
upload was completed successfully. Rather than delete them, we update
their StorageClass if it is non-STANDARD.
Because the main indexes on end_time either don't include realm_id or
do include subgroup, passing an explicit subgroup=None for
single-realm queries to read CountStats that don't use the subgroups
feature greatly improves the query plans.
We create an unnamed user group with just the group creator as it's
member when trying to set the default. The pattern I've followed across
most of the acting_user additions is to just put the user declared
somewhere before the check_add_user_group and see if the test passes.
If it does not, then I'll look at what kind of user it needs to be set
to `acting_user`.
This commit does not add the logic of using this setting to actually
check the permission on the backend. That will be done in a later
commit.
Only owners can modify this setting, but we will add that logic in a
later commit in order to keep changes in this commit minimal.
Adding the setting breaks the frontend, since the frontend tries to find
a dropdown widget for the setting automatically. To avoid this, we've
added a small temporary if statement to `settings_org.js`.
Although, most lists where we insert this setting follow an unofficial
alphabetical order, `can_manage_all_groups` has been bunched together
with `can_create_groups` since keeping those similar settings together
would be nicer when checking any code related to creating/managing a
user group.
We will not remove `user_group_edit_policy` yet. That will be removed
once we have introduced a user group setting to manage edit permissions
to groups.
We might introduce a generic testing function similar to
do_test_changing_settings_by_owners_only later, but not right now, since
there is only 1 setting at the moment needing that test.
This commit does not add the logic of using this setting to actually
check the permission on the backend. That will be done in a later
commit.
Adding the setting breaks the frontend, since the frontend tries to find
a dropdown widget for the setting automatically. To avoid this, we've
added a small temporary if statement to `settings_org.js`.
In docker-zulip installs, /etc/zulip/zulip.conf,
/etc/zulip/zulip-secrets.conf, and /home/zulip/uploads are all
symlinks into the `/data` directory which is mounted as a Docker
Volume. By default, `tar` does not dereference symlinks, leading to
backups that are missing these critical pieces.
Add `-h` to the `tar` invocation, to follow symlinks, so backups in
Docker have all of their pieces. Since none of the contents of the
backup intentionally use symlinks, this is safe.
Co-authored-by: Alex Vandiver <alexmv@zulip.com>
The error response when a user group cannot be deactivated due
to it being used as a subgroup or for a setting includes details
about the supergroups, streams, user groups as well the settings
for which it is used.
This commit adds access_user_group_to_read_membership function
so that we can avoid calling get_user_group_by_id_in_realm with
"for_read=True" from views functions, which is better for security
since that function does not do any access checks.
Previously, if the user_group_edit_policy was set to allow
members or full members to manage the group, the user had
to be the direct member of the group being managed.
This commit updates the code to allow members of the subgroups
as well to manage the group as technically members of the
subgroups are member of the group.
This also improves the code to not fetch all the group members
to check this, and instead directly call is_user_in_group
which uses "exists" to check it.
This commit renames has_user_group_access function to
has_user_group_access_for_subgroup, since the function
is only used to check access for using a group as subgroup.
This commit refactors the code to check permission for
accessing user group in such a way that we can avoid
duplicate code in future when we will have different
settings controlling the permissions for editing group
details and settings, joining the group, adding others
to group, etc.
The use of the robust Address(...)-based address generation was added in
b945aa3443 but then this couple of
instances of the naive approach were added later.
Creates a URLRedirect for this help center article to go to the
new "Moving to Zulip" guide.
Updates the astro.config.mjs file for the changes to the help
center sidebar that have been made as part of the replacement
of this help center guide.
Fixes#31499.
Replaces links to "Getting your organization started with Zulip"
in onboarding emails and Welcome bot direct message for owners of
new organizations.
Revises text in those emails and messages to reflect the new
"Moving to Zulip" help center guide that is now used.
In order to only generate relative links for Zulip Cloud billing
specific gear menu options in relevant help center articles, we
pass down settings.CORPORATE_ENABLED to be set as a global variable
for zerver/lib/markdown/help_relative_links.py so that self-hosted
servers' help center documentation will not have these links.
It's nicer to have these indexes properly registered, rather than hidden
in RunSQL operations. Now that Django has had support for unique
functional indexes for a while, let's clean this up.
This lets slack conversions be done on development hosts, which have a
trailing :9991 on their EXTERNAL_HOST; otherwise, we generate fake
emails like `imported-slack-bot@host.name:9991` which fail to
validate.
The Content-Type, Content-Disposition, StorageClass, and general
metadata are not set according to our patterns by tusd; copy the file
to itself to update those properties.
Setting `ResponseContentDisposition=attachment` means that we override
the stored `ContentDisposition`, which includes a filename. This
means that using the "Download" link on servers with S3 storage
produced a file named the sanitized version we stored.
Explicitly build a `ContentDisposition` to tell S3 to return, which
includes both `attachment` as well as the filename (if we have it
locally).
Apparently, Outlook ignores height/width CSS rules, but does support
the attribute on the image element itself, so specify that instead.
I don't think there are likely to be image tag implementations that
don't support the attribute, given that's the only thing that works in
Outlook.
This test was written back when Django accepted view function names as
strings that might be wrong; that’s not possible in Django ≥ 1.10.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
For exporting full with consent:
* Earlier, a message advertising users to react with thumbs up
was sent and later used to determine the users who consented.
* Now, we no longer need to send such a message. This commit
updates the logic to use `allow_private_data_export` user-setting
to determine users who consented.
Fixes part of #31201.
This commit adds test coverage for the
'if user_profile.realm != message.get_realm():' block in
'set_visibility_policy_possible' function.
Currently, 'test_command_with_consented_message_id' adds the
test coverage for this code block. But we plan to remove that
test in the next commit as a part of removing 'react on consent
message' approach.
So, this commit explicitly adds code for the test coverage of
the concerned code block.
Removes the ignore_pull_requests URL parameter from the Travis CI
integration because its functionality is now fully covered by the
standard incoming webhook event filtering framework.
Fixes#30934.
This new property allows organization administrators to specify whether
users can modify the custom profile field value on their own account.
This property is configurable for individual fields.
By default, existing and newly created fields have this property set to
true, that is, they allow users to edit the value of the fields.
Fixes part of #22883.
Co-Authored-By: Ujjawal Modi <umodi2003@gmail.com>
This allows finer-grained access control and auditing. The links
generated also expire after one week, and the suggested configuration
is that the underlying data does as well.
Co-authored-by: Prakhar Pratyush <prakhar@zulip.com>
The `get_signed_upload_url` code is called for every S3 file serve
request, and is thus in the hot path. The boto3 client caching
optimization is thus potentially useful as a performance optimization.
This commit renames the 'send_event' function to
'send_event_rollback_unsafe' to reflect the fact that it doesn't
wait for the db transaction (within which it gets called, if any)
to commit and sends event irrespective of commit or rollback.
In most of the cases we don't want to send event in the case of
rollbacks, so the caller should be aware that calling the function
directly is rollback unsafe.
Adds a check for changing an existing guest user's role before
calling do_update_user in the case that a realm has a current
paid plan with manual license management.
In addition to checking for available licenses in the current
billing period when adding or inviting new non-guest users, for
manual billing, we also verify that the number of licenses set
for the next billing period will be enough when adding/inviting
new users.
Realms that are exempt from license number checks do not have
this restriction applied.
Admins are notified via group direct message when a user fails
to register due to this restriction.
If do_delete_messages (and friends) are called for a massive number of
messages, the giant list of message ids is passed to Postgres even
though chunk_size makes all but the first chunk_size of message ids
useless.
Sahil Batra
Lauryn Menard
Kislay Udbhav Verma
Mateusz Mandera
Tim Abbott
Prakhar Pratyush
Aman Agrawal
Alex Vandiver
Shubham Padia
Anders Kaseorg
Pierre Carru
bao-qian
Vector73
Aditya
tnmkr