Earlier, we were using 'send_event' in 'do_mark_onboarding_step_as_read'
which can lead to a situation, if any db operation is added after the
'send_event' in future, where we enqueue events but the action
function fails at a later stage.
Events should not be sent until we know we're not rolling back.
Fixes part of #30489.
This deletes them directly, rather than move them into archival,
because that would be slow, and bloat the archival table in a way
which might interfere with other deletions.
Previously, comment related notifications only displayed the issue
title as a plain string. This commit reformats the issue title to
include a link back to the Jira issue.
Adjusted the Jira documentation for recent changes in their UI
when setting up webhooks, reformatted the note about compatible
Jira version, and added a link to Jira's official webhook guide.
Note that the link in zulip_update_announcements.py is not updated
so that the content in the source code reflects what users actually
received in the update announcement message.
This bug was introduced in da9e4e6e54.
validate validate_plan_for_authentication_methods is already called
inside validate_authentication_methods_dict_from_api, conditionally on
settings.BILLING_ENABLED. This additional, redundant call runs
regardless of BILLING_ENABLED, and thus prevents a self-hosted server
from enabling certain backends in the organization settings UI.
The impact of this is limited - in order to encounter this bug, a
self-hosted server would have to first disable the backend in the UI, as
self-hosted realms are created with all backend flags enabled. A backend
doesn't show up in the org settings UI until it is first enabled in
AUTHENTICATION_BACKENDS in settings.py - that's why this is a rare
state. A sequence of steps like this has to be followed to reproduce:
1. Add the backend to AUTHENTICATION_BACKENDS in settings.py.
2. Disable the backend in the org settings UI.
3. Now try to re-enable it, which fails due to the bug.
This commit removes create_web_public_stream_policy setting
since web-public channel creation permissions are now
handled by group-based setting.
We still pass "realm_create_web_public_stream_policy" in
"/register" response though for older clients with its
value being set depending on the value of group based
setting. If we cannot set its value to an appropriate enum
corresponding to the group setting, then we set it to
"Admins and moderators" considering that server will not
allow the users without permissions to create web-public
channels but the client can make sure that UI is
available to the users who have permission.
Messages are rendered outside of a transaction, for performance
reasons, and then sent inside of one. This opens thumbnailing up to a
race where the thumbnails have not yet been written when the message
is rendered, but the message has not been sent when thumbnailing
completes, causing `rewrite_thumbnailed_images` to be a no-op and the
message being left with a spinner which never resolves.
Explicitly lock and use he ImageAttachment data inside the
message-sending transaction, to rewrite the message content with the
latest information about the existing thumbnails.
Despite the thumbnailing worker taking a lock on Message rows to
update them, this does not lead to deadlocks -- the INSERT of the
Message rows happens in a transaction, ensuring that either the
message rending blocks the thumbnailing until the Message row is
created, or that the `rewrite_thumbnailed_images` and Message INSERT
waits until thumbnailing is complete (and updated no Message rows).
This migration references the "confirmation" app for the first time,
which means we must have migrated at least part of it by this point.
Set the migration to depend on the latest "confirmation" migration at
the time of this migration.
Migrate the following endpoints from @has_request_variables
to @typed_endpoint:
- get_user_group()
- delete_user_group()
- update_user_group_backend()
- update_subgroups_of_user_group()
- get_is_user_group_member()
- get_user_group_members()
- get_subgroups_of_user_group()
With tweaks from tabbott to avoid calling thunks unnecessarily.
Earlier, we were using 'send_event' in 'do_schedule_messages' which
can lead to a situation, if any db operation is added after the
'send_event' in future, where we enqueue events but the action
function fails at a later stage.
Events should not be sent until we know we're not rolling back.
Fixes part of #30489.
Earlier, we were using 'send_event' in 'do_update_user_status' which
can lead to a situation, if any db operation is added after the
'send_event' in future, where we enqueue events but the action
function fails at a later stage.
Events should not be sent until we know we're not rolling back.
Fixes part of #30489.
Earlier, we were immediately enqueueing event in
'do_remove_alert_words' which can lead to a situation, if any
db operation is added after enqueueing event in future, where the
action function fails at a later stage.
Events should not be sent until we know we're not rolling back.
Fixes part of #30489.
Earlier, we were using 'send_event' in 'do_add_alert_words' which
can lead to a situation, if any db operation is added after the
'send_event' in future, where we enqueue events but the action
function fails at a later stage.
Events should not be sent until we know we're not rolling back.
Fixes part of #30489.
GitHub sends two almost identical payloads when a pull
request is reviewed, which results in two duplicative
notification messages.
The payloads have different "action" value, with one
having the "submitted" action, whereas the other is
"edited" and has an empty "changes" dict.
We now ignore the payload with the "edited" action type
and an empty "changes" dict.
Fixes#26145.
Co-authored-by: Pieter CK <pieterceka123@gmail.com>
Providing a signed Camo URL for arbitrary URLs opened the server up to
being an open redirector. Return 403 if the URL is not a user upload,
and the backend image if it is. Since we do not have ImageAttachment
rows for uploads at a time we wrote `/thumbnail?` URLs, return the
full-size content.
47683144ff switched the web client to prefer the 840x560 size, as the
mobile apps prefer; remove the now-unused 300x200 size. No client was
using the generated `.jpg` formats, as all clients support `.webp`, so
remove the unused `.jpg` thumbnail as well.
Modern browsers respect the EXIF orientation information of images,
applying rotation and/or mirroring as specified in those tags. The
the `width="..."` and `height="..."` tags are to size the image
_after_ applying those orientation transformations.
The `.width` and `.height` properties of libvips' images are _before_
any transformations are applied. Since we intend to use these to hint
to rendering clients the size that the image should be _rendered at_,
change to storing (and providing to clients) the dimensions of the
rendered image, not the stored bytes.
If the email subject is something like `Fwd:`, it gets stripped to an
empty string, activating the "(no topic)" override. This however leads
to failure if the organization enables the setting forcing every message
to have a topic. Such emails should still go through, so we should just
change the topic value used.
This allows clients to potentially lay out the thumbnails more
intelligently, or to provide a better "progressive-load" experience
when enlarging the thumbnail.
Previously animated images were automatically played in the
message feed of the web app.
Now that we have still thumbnails available for them, we can add a new
personal setting, "web_animate_image_previews", which controls how the
animated images would be played in the web app message feed -- always
played, on hover, or only in the image viewer.
Fixes#31016.
In 'fetch_initial_state_data' we were doing one database query
per announcement stream.
This commit updates the logic to prefetch those streams using
select_related hence avoiding the extra db queries.
Fixes#28909.
The libvips cache is 100MB, 100 operations, or 100 files, whichever is
less. A single Django process or worker is extremely unlikely to ever
see the same image twice, much less within those timeframes.
Disable the cache, since it is mostly useless memory usage for our use
case.
The emoji dir is present in the data from our export tool. This was
added in 468afe4840.
This comment hasn't been updated since
c4b886d8ae, so probably we just forgot to
refresh it when custom emoji export was added.
Fixes warnings like “ResourceWarning: unclosed file <_io.FileIO
name='/srv/zulip/var/044e5d44-87aa-4c43-abbb-28a144fa6654/test-backend/run_1238680/worker_0/test_uploads/files/thumbnail/2/1e/jmUuDhQC8WlaSRCuc0zQyx7D/img.tif/100x75.webp'
mode='rb' closefd=True>” with warnings enabled.
deque(…, 0) is an efficient way to consume an iterator documented at
https://docs.python.org/3/library/itertools.html#itertools-recipes
under consume.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Previously, if someone changed the font-size or line height
settings to some value other than the legacy values, we set
dense_mode to False if it was True. This commit changes the
code to require clients to pass dense_mode as False in such
cases and raise an error otherwise.
We should flush the cache after commiting changes to DB to
make sure that the cache does not contain stale data when
updating multiple settings in a single request.
Instead of the PUSH_NOTIFICATIONS_BOUNCER_URL and
SUBMIT_USAGE_STATISTICS settings, we want servers to configure
individual ZULIP_SERVICE_* settings, while maintaining backward
compatibility with the old settings. Thus, if all the new
ZULIP_SERVICE_* are at their default False value, but the legacy
settings are activated, they need to be translated in computed_settings
to the modern way.
This is a small optimization to avoid DB queries if the bot owner
is not changed.
This also helps in avoiding showing the error message in UI if
anything other than the owner is updated for a deactivated bot.
Ideally if the bot owner is not changed, the bot_owner_id field
should not be passed in the request, but we would handle that
later given this is anyways a small nice optimization and we
follow this pattern at other places as well in the API.
Adds a header to the section of changes notes about new and
deprecated narrow filters, so that it's easier to directly
link to the relevant content in the construct a narrow article
from the main API changelog and endpoint documentation.
The HookPress plugin has been unavailable since July 2019, so it
doesn't make sense to continue supporting these events in the
WordPress webhook integration.
Tornado ≥ 5 uses AsyncIOLoop and AsyncIOMainLoop automatically when
appropriate. Fixes “DeprecationWarning: make_current is deprecated;
start the event loop first” with warnings enabled.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
In an interleaved view when composing a message we fade messages
which the user is not replying to, to reduce the chance they send
a message to a recipient they didn't intend to.
Also, it reduces the visual/cognitive processing required
to figure out where their message is going to go.
But, it's not necessarily clear to users that what the
fading means, so this commit adds a one-time compose banner
to explain what's going on the first time this comes up.
Fixes part of #29076.
In a non interleaved view when composing a message to another
conversation we fade messages which the user is not replying to,
to reduce the chance they send a message to a recipient they didn't
intend to. Also, it reduces the visual/cognitive processing required
to figure out where their message is going to go.
But, it's not necessarily clear to users that what the
fading means, so this commit adds a one-time compose banner
to explain what's going on the first time this comes up.
Fixes part of #29076.
A new table is created to track which path_id attachments are images,
and for those their metadata, and which thumbnails have been created.
Using path_id as the effective primary key lets us ignore if the
attachment is archived or not, saving some foreign key messes.
A new worker is added to observe events when rows are added to this
table, and to generate and store thumbnails for those images in
differing sizes and formats.
SHA1PasswordHasher will be removed in Django 5.1. MD5PasswordHasher
will remain for the purpose of speeding up tests.
Followup to commit ac5161f439 (#29620).
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Imports (including conversions from Slack, Hipchat, etc) do not write
the `.original` of the file (nor the still versions, for animations),
only the thumbnailed (and possibly animated) version. Looking for
`.original` would thus fail, and overwrite the (working) emoji with
the "?" and disable it.
In the event that we do not find the `.original`, fall back to the
bytes from the non-`.original`, to handle these import cases.
This commit improves the assert statements to verify
that the table name is not "usermessage' instead of
verifying that table name doesn't include a substring
"usermessage".
This prep commit will help to avoid assertion error when
importing "onboardingusermessage" table.
Earlier, the export tool was logging a warning:
"??? NO DATA EXPORTED FOR TABLE zerver_onboardingusermessage!!!"
This bug was due to not configuring a Config object for
'OnboardingUserMessage' in 'get_realm_config()'.
This commit fixes the bug to export the table properly.
This is needed for the command to work in docker-zulip, where the zulip
user may not have write permissions to the parent directory of the
secrets file. By default, crudini creates a temporary file, which makes
the command fail under those conditions.
With --inplace, the secret file gets written to directly without
creation of temporary files.
Adds server and web app support for processing the new `with`
search operator.
Fixes part of #21505.
Co-authored-by: roanster007 <rohan.gudimetla07@gmail.com>
Co-authored-by: Tim Abbott <tabbott@zulip.com>
bulk fetch query of UserPfrofile against which
user_ids are validated, instead of looping
over user_ids and fetchingeach UserPfrofile resulting
in O(n) queries.
We use a truncated SHA256 of the id and a server-side secret to make
emoji have non-guessable filenames, while also making collisions
unlikely.
We also adjust the Slack import to use the same SHA-based name,
instead of taking the same name as it had in Slack.
We thumbnail and serve emoji with the same format as they were
uploaded. However, we preserved the original extension, which might
mismatch with the provided content-type.
Limit the content-type to a subset which is both (a) an image format
we can thumbnail, and (b) a media format which is widely-enough
supported that we are willing to provide it to all browsers. This
prevents uploading a `.tiff` emoji, for instance.
Based on this limited content-type, we then reverse to find the
reasonable extension to use when storing it. This is particularly
important because the local file storage uses the file extension to
choose what content-type to re-serve the emoji as.
This does nothing for existing emoji, which may have odd or missing
file extensions.
Aside from updating the doc structure, this commit
also rephrases step 4 to make it read better. The
list of supported events in step 5 is also removed.
Part of #29592.
We use the already existing server level setting to only allow
settings to be set to system groups, not a named user defined
group as well, in production. But we allow to settings to be set
to any named or anonymous user group in tests and development server.
"can_mention_group" setting can be set to user defined groups
because some of the realms already do that in production.
The existing server level setting is also renamed to make it clear
that both user defined groups and anonymous groups are not allowed
if that setting is set to False.
This commit also changes the error message to be consistent for the
case when a setting cannot be set to user defined groups as per
server level and setting and when a particular setting cannot be set
to user defined groups due to the configuration of that particular
setting. For this we add a new class SystemGroupRequiredError in
exceptions.py so that we need not re-write the error message in
multiple places.
Besides "RE:" and "FWD:", "AW:" (from German "Antwort" for answer) is
another common prefix in email subjects. Let email_mirror automatically
remove it as well.
When using the sub-expression purely for filtering, and not for
accessing the value in the resultset, .alias() is potentially faster
since it does not pull the value in as well.
This commit adds an 'transaction.atomic' decorator to the
'add_subscriptions_backend' view thus making the db operations
within the view atomic and helps to avoid race between events sent.
In tests where we make POST requests to this view, we have
wrapped the API calls with a transaction.atomic() context
manager. It helps us with NOT rolling back the entire test
transaction due to error responses.
In 'test_subs' we were making POST request to add
subscrption using 'self.api_post'.
This commit updates the code to use the test helper
function 'common_subscribe_to_streams' instead.
This prep commit will also help us to avoid adding
'transaction.atomic' context manager to these API calls
individually in the case of error response in the next commit.
Prakhar Pratyush
Alex Vandiver
PieterCK
adnan-td
Lauryn Menard
Vector73
Mateusz Mandera
Sahil Batra
Anders Kaseorg
bedo
roanster007
Prakhar Pratyush
Kenneth Rodrigues
lumpleme
Tim Abbott
Alya Abbott
Niloth P
Aman Agrawal
Niklas Fiekas