Commit Graph

18789 Commits

Author SHA1 Message Date
Mateusz Mandera aaca394813 presence: Remove the queue worker. 2024-07-31 16:46:42 -07:00
Anders Kaseorg 8f1a97f168 users: Reduce date_joined precision to minutes.
Fixes #31188, removing a lot of useless high-entropy data from the
/register response.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-31 12:53:09 -07:00
lumpleme b5c63cfb85 integrations: Prevent duplicate GitHub pull request review messages.
GitHub sends two almost identical payloads when a pull
request is reviewed, which results in two duplicative
notification messages.

The payloads have different "action" value, with one
having the "submitted" action, whereas the other is
"edited" and has an empty "changes" dict.

We now ignore the payload with the "edited" action type
and an empty "changes" dict.

Fixes #26145.

Co-authored-by: Pieter CK <pieterceka123@gmail.com>
2024-07-31 10:42:41 -07:00
adnan-td 76df435dab integration-docs: Update Semaphore to the new doc format.
Part of #29592.

Co-authored-by: Lauryn Menard <lauryn@zulip.com>
2024-07-31 10:16:54 -07:00
adnan-td bdbe3adf1b integration-docs: Update Sentry to the new doc format.
Part of #29592.
2024-07-31 10:16:54 -07:00
Tim Abbott e7cc9090fd api docs: Clarify event structure when moving channels.
The previous language would most naturally be read as suggesting that
moving topics between channels would have these fields.
2024-07-31 09:22:44 -07:00
Mateusz Mandera a0971934d9 thumbnail: Fix typo in comment. 2024-07-30 00:17:59 +02:00
Anders Kaseorg 3a17946c85 zulip_update_announcements: Fix December 2024 typo.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-26 10:20:27 -07:00
Alya Abbott 7a1f5765b7 updates: Add update message (level=8). 2024-07-25 13:19:25 -07:00
Alex Vandiver c726d2ec01 thumbnail: Do not Camo old thumbor URLs; serve images directly.
Providing a signed Camo URL for arbitrary URLs opened the server up to
being an open redirector.  Return 403 if the URL is not a user upload,
and the backend image if it is.  Since we do not have ImageAttachment
rows for uploads at a time we wrote `/thumbnail?` URLs, return the
full-size content.
2024-07-24 16:04:34 -07:00
Alex Vandiver e3a238fc89 thumbnail: Remove unused thumbnail sizes.
47683144ff switched the web client to prefer the 840x560 size, as the
mobile apps prefer; remove the now-unused 300x200 size.  No client was
using the generated `.jpg` formats, as all clients support `.webp`, so
remove the unused `.jpg` thumbnail as well.
2024-07-24 09:57:20 -07:00
Alex Vandiver e4a8304f57 thumbnail: Store the post-orientation-transformation dimensions.
Modern browsers respect the EXIF orientation information of images,
applying rotation and/or mirroring as specified in those tags.  The
the `width="..."` and `height="..."` tags are to size the image
_after_ applying those orientation transformations.

The `.width` and `.height` properties of libvips' images are _before_
any transformations are applied.  Since we intend to use these to hint
to rendering clients the size that the image should be _rendered at_,
change to storing (and providing to clients) the dimensions of the
rendered image, not the stored bytes.
2024-07-24 09:56:42 -07:00
Alex Vandiver e7ac62aad7 tests: Add a test that the thumb is actually the expected size. 2024-07-24 09:56:42 -07:00
Alex Vandiver d824988bc4 tests: Remove THUMBNAIL_IMAGES overrides, since the setting was dropped.
These were omitted from b42863be4b.
2024-07-24 09:53:14 -07:00
Mateusz Mandera 8f461b3c03 email_mirror: Change default topic name if subject ends up empty.
If the email subject is something like `Fwd:`, it gets stripped to an
empty string, activating the "(no topic)" override. This however leads
to failure if the organization enables the setting forcing every message
to have a topic. Such emails should still go through, so we should just
change the topic value used.
2024-07-22 22:31:45 -07:00
Alex Vandiver 2ea0cc0005 thumbnail: Add a data-original-dimensions attribute.
This allows clients to potentially lay out the thumbnails more
intelligently, or to provide a better "progressive-load" experience
when enlarging the thumbnail.
2024-07-22 22:41:10 -04:00
Alex Vandiver 65828b20e9 thumbnail: Factor out a dataclass for markdown image metadata. 2024-07-22 22:41:10 -04:00
roanster007 66a96bee71 settings: Add setting to control how animated images are played.
Previously animated images were automatically played in the
message feed of the web app.

Now that we have still thumbnails available for them, we can add a new
personal setting, "web_animate_image_previews", which controls how the
animated images would be played in the web app message feed -- always
played, on hover, or only in the image viewer.

Fixes #31016.
2024-07-22 14:53:31 -07:00
Prakhar Pratyush 6838a7302d test_import_export: Add test coverage for OnboardingUserMessage. 2024-07-22 10:26:33 -07:00
Prakhar Pratyush f48e87cd3c fetch_initial_state: Avoid doing one db query per announcement stream.
In 'fetch_initial_state_data' we were doing one database query
per announcement stream.

This commit updates the logic to prefetch those streams using
select_related hence avoiding the extra db queries.

Fixes #28909.
2024-07-22 10:21:22 -07:00
Alex Vandiver 3ac14632d8 thumbnail: Disable libvips cache.
The libvips cache is 100MB, 100 operations, or 100 files, whichever is
less.  A single Django process or worker is extremely unlikely to ever
see the same image twice, much less within those timeframes.

Disable the cache, since it is mostly useless memory usage for our use
case.
2024-07-22 10:19:33 -07:00
Alex Vandiver b42863be4b markdown: Show thumbnails for uploaded images.
Fixes: #16210.
2024-07-21 18:41:59 -07:00
Alex Vandiver 71406ac767 thumbnail: Factor frames into account for IMAGE_BOMB_TOTAL_PIXELS. 2024-07-21 18:41:59 -07:00
Alex Vandiver aacf28f7e3 test_classes: Extract a thumbnailing output format helper. 2024-07-21 18:41:59 -07:00
Alex Vandiver 94ff443c00 test_classes: Merge verbose assertEquals into ZulipTestCase. 2024-07-21 18:41:59 -07:00
Niloth P e1d35b1e72 integration-docs: Update Opsgenie for new doc format.
Part of zulip#29592.
2024-07-21 18:09:13 -07:00
Niloth P d972d76725 integration-docs: Update Pivotal for new doc format.
Part of zulip#29592.
2024-07-21 18:09:13 -07:00
Niloth P d571410ba3 integration-docs: Update Rundeck for new doc format.
Part of zulip#29592.
2024-07-21 18:09:13 -07:00
Niloth P 650a1384ca integration-docs: Update RhodeCode for new doc format.
Part of zulip#29592.
2024-07-21 18:09:13 -07:00
Niloth P 8fe7f4e409 integration-docs: Update Review Board for new doc format.
Part of zulip#29592.
2024-07-21 18:09:13 -07:00
Niloth P ec018780b5 integration-docs: Update Raygun for new doc format.
Part of zulip#29592.
2024-07-21 18:09:13 -07:00
Niloth P 335f9b8119 integration-docs: Update Radarr for new doc format.
Part of zulip#29592.
2024-07-21 18:09:13 -07:00
Mateusz Mandera f50104d475 import_realm: Update incorrect comment about emoji dir in exported data.
The emoji dir is present in the data from our export tool. This was
added in 468afe4840.

This comment hasn't been updated since
c4b886d8ae, so probably we just forgot to
refresh it when custom emoji export was added.
2024-07-21 13:16:49 -07:00
Kenneth Rodrigues 8b489f4b96 streams: Convert to typed_enpoint. 2024-07-21 12:57:50 -07:00
Kenneth Rodrigues 6815cded83 zerver: Migrate some files to typed_endpoint.
Migrates `invite.py`, `registration.py` and
`email_mirror.py` to use `typed_endpoint`.
2024-07-20 15:46:48 -07:00
Niloth P 16abd82fa5 integrations: Remove Solano integration. 2024-07-20 15:44:17 -07:00
Sahil Batra c6e9173297 exceptions: Add new error class to be used for invalid parameter values.
This would help us in avoiding adding translation everytime we use
this error for a new pair of parameters.
2024-07-19 09:24:54 -07:00
Anders Kaseorg d574200423 tests: Consume streaming responses.
Fixes warnings like “ResourceWarning: unclosed file <_io.FileIO
name='/srv/zulip/var/044e5d44-87aa-4c43-abbb-28a144fa6654/test-backend/run_1238680/worker_0/test_uploads/files/thumbnail/2/1e/jmUuDhQC8WlaSRCuc0zQyx7D/img.tif/100x75.webp'
mode='rb' closefd=True>” with warnings enabled.

deque(…, 0) is an efficient way to consume an iterator documented at
https://docs.python.org/3/library/itertools.html#itertools-recipes
under consume.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-19 09:20:56 -07:00
Anders Kaseorg c7281b406f test_delete_unclaimed_attachments: Close file in make_attachment.
Fixes “ResourceWarning: unclosed file <_io.BufferedReader
name='/srv/zulip/zerver/tests/images/text.txt'>” with warnings enabled.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-19 09:20:56 -07:00
Tim Abbott de35f44624 zulip_updates: Mention default for send navigation.
Apparently I forgot to merge this before -beta1.
2024-07-18 14:06:04 -07:00
Alex Vandiver 4351cc5914 thumbnail: Move get_image_thumbnail_path and split_thumbnail_path. 2024-07-18 13:50:28 -07:00
Alex Vandiver d474600056 tests: Remove a nonsensical no-op vimeo test.
Embeds are disabled in this test, and it tests that it does not embed
anything.
2024-07-18 13:50:28 -07:00
Alex Vandiver 424f3dd503 tests: Break up and reorganize test_markdown.py. 2024-07-18 13:50:28 -07:00
Alex Vandiver ecdf5713c2 tests: Factor out ZulipVerboseEqualTest class, and use more. 2024-07-18 13:50:28 -07:00
Alex Vandiver 08191d3f69 tests: Default ENABLE_FILE_LINKS to False.
Test configuration should mirror default production as much as possible.
2024-07-18 13:50:28 -07:00
Alex Vandiver e02e9c9181 tests: INLINE_URL_EMBED_PREVIEW=False is the default in tests. 2024-07-18 13:50:28 -07:00
Alex Vandiver 060754980c tests: INLINE_IMAGE_PREVIEW=True is already the default. 2024-07-18 13:50:28 -07:00
Alya Abbott d775b6ffff updates: Add update message (level=7). 2024-07-18 12:38:26 -07:00
Tim Abbott f37082c4ca models: Switch font size default to non-compact.
This is the intended longer-term default, and it's polished enough for
testing.
2024-07-18 12:37:06 -07:00
Niloth P cbe2677a2a integration-docs: Include suggestion for HTTPS hint in Stripe.
With some wording tweaks by tabbott to make it more obviously only
shown in certain environments.
2024-07-18 12:28:08 -07:00
Niloth P ed885ec806 integration-docs: Format keyboard shortcuts in Trello doc. 2024-07-18 12:24:45 -07:00
Niloth P 525f1e28d4 integration-docs: Fix typo in Splunk doc hint. 2024-07-18 12:24:36 -07:00
Sahil Batra c1c4c95ce7 settings: Require clients to pass dense_mode value when needed.
Previously, if someone changed the font-size or line height
settings to some value other than the legacy values, we set
dense_mode to False if it was True. This commit changes the
code to require clients to pass dense_mode as False in such
cases and raise an error otherwise.
2024-07-18 11:36:53 -07:00
Sahil Batra 0ea5daece8 user_settings: Flush cache after commiting changes to DB.
We should flush the cache after commiting changes to DB to
make sure that the cache does not contain stale data when
updating multiple settings in a single request.
2024-07-18 11:36:53 -07:00
Anders Kaseorg 97753fe3b0 emoji_names: Rebuild with CLDR 45, emoji-data 15.1.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-17 22:49:52 -07:00
Mateusz Mandera 4a93149435 settings: Rework how push notifications service is configured.
Instead of the PUSH_NOTIFICATIONS_BOUNCER_URL and
SUBMIT_USAGE_STATISTICS settings, we want servers to configure
individual ZULIP_SERVICE_* settings, while maintaining backward
compatibility with the old settings. Thus, if all the new
ZULIP_SERVICE_* are at their default False value, but the legacy
settings are activated, they need to be translated in computed_settings
to the modern way.
2024-07-17 17:14:06 -07:00
Anders Kaseorg 722842a0aa rocketchat: Remove unnecessary SHA-1 hashing of direct message groups.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-17 15:56:00 -07:00
Anders Kaseorg 541699a1c6 mattermost: Remove unnecessary MD5 hashing of direct message groups.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-17 15:56:00 -07:00
Anders Kaseorg 90cc88c88e mattermost: Split get_recipient_id_from_receiver_name to 3 functions.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-17 15:56:00 -07:00
Anders Kaseorg 27b0618704 data_import: Fix IdMapper typing.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-17 15:56:00 -07:00
Anders Kaseorg 1fd3f983a5 data_import: Remove int detection from IdMapper.
This seems to have been used only for HipChat.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-17 15:56:00 -07:00
Alex Vandiver ab6b6639e6 migrations: Derive emoji content-type from the bytes. 2024-07-17 11:39:04 -07:00
Sahil Batra cf14d8f611 users: Change bot owner only if needed.
This is a small optimization to avoid DB queries if the bot owner
is not changed.

This also helps in avoiding showing the error message in UI if
anything other than the owner is updated for a deactivated bot.
Ideally if the bot owner is not changed, the bot_owner_id field
should not be passed in the request, but we would handle that
later given this is anyways a small nice optimization and we
follow this pattern at other places as well in the API.
2024-07-17 10:07:51 -07:00
Lauryn Menard b9c33ce5f1 api-docs: Add changes section in construct a narrow article.
Adds a header to the section of changes notes about new and
deprecated narrow filters, so that it's easier to directly
link to the relevant content in the construct a narrow article
from the main API changelog and endpoint documentation.
2024-07-16 23:40:45 -07:00
Vector73 7a80fcf042 events: Send `delete_message` event to user who deleted the message.
Fixes #29826.

Co-authored-by: Mukul Goyal <goyal.mukul7689@gmail.com>
Co-authored-by: Aman Agrawal <amanagr@zulip.com>
2024-07-16 23:29:02 -07:00
Niloth P 56aa72af90 integration-docs: Update Trello for new doc format.
Part of zulip#29592.
2024-07-16 17:02:55 -07:00
Niloth P c8781b4de7 integration-docs: Update TeamCity for new doc format.
Part of zulip#29592.
2024-07-16 17:02:55 -07:00
Alex Vandiver 9983dc761f migrations: Further fix emoji re-thumbnailing.
90701978d9 was an incomplete fix.
2024-07-16 15:38:22 -07:00
Niloth P f663544353 integration-docs: Update Slack incoming-webhook for new doc format.
Part of zulip#29592.
2024-07-16 14:55:56 -07:00
Niloth P 32177a9a1c integration-docs: Update Sonarqube for new doc format.
Part of zulip#29592.
2024-07-16 14:55:56 -07:00
Niloth P a546fa719a integration-docs: Update Sonarr for new doc format.
Part of zulip#29592.
2024-07-16 14:55:56 -07:00
Niloth P 1ddf92a98c integration-docs: Update Splunk for new doc format.
Part of zulip#29592.
2024-07-16 14:55:56 -07:00
Niloth P 8549a6dc8a integration-docs: Update Statuspage for new doc format.
Part of zulip#29592.
2024-07-16 14:55:56 -07:00
Niloth P c0ab89ec52 integration-docs: Update Stripe for new doc format.
Part of zulip#29592.
2024-07-16 14:55:56 -07:00
Alex Vandiver e8a049ea57 migrations: Handle missing emoji/images/still/ directory. 2024-07-16 14:55:45 -07:00
Niloth P a17a54b077 integration-docs: Update Zabbix for new doc format.
Part of zulip#29592.
2024-07-16 14:55:27 -07:00
Niloth P c271cdb113 integration-docs: Update WordPress for new doc format.
Part of zulip#29592.

Co-authored-by: Lauryn Menard <lauryn@zulip.com>
2024-07-16 14:55:02 -07:00
Lauryn Menard 3e100d872a integrations: Remove HookPress events from WordPress webhook.
The HookPress plugin has been unavailable since July 2019, so it
doesn't make sense to continue supporting these events in the
WordPress webhook integration.
2024-07-16 14:55:02 -07:00
Niloth P 1e327cc054 integration-docs: Update Wekan for new doc format.
Part of zulip#29592.
2024-07-16 14:55:02 -07:00
Anders Kaseorg dd0d482d76 narrow: Migrate legacy SQLAlchemy select syntax.
Fixes “sqlalchemy.exc.RemovedIn20Warning: Deprecated API features
detected! These feature(s) are not compatible with SQLAlchemy 2.0.”
with warnings enabled.

https://docs.sqlalchemy.org/en/14/changelog/migration_14.html#change-5284

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-16 14:50:30 -07:00
Sahil Batra a0efc0ee76 settings: Refetch realm and user profile objects.
This is a temporary solution to make sure stale objects from
cache are not used when previous request updated multiple settings.
2024-07-16 14:04:19 -07:00
Sahil Batra 83c4c27b18 settings: Do not allow invalid combinations for information density settings. 2024-07-16 14:04:19 -07:00
Sahil Batra 17541ddacb settings: Toggle dense mode if font and line height values are changed. 2024-07-16 14:04:19 -07:00
Anders Kaseorg ad9837911e runtornado: Remove explicit AsyncIOMainLoop installation.
Tornado ≥ 5 uses AsyncIOLoop and AsyncIOMainLoop automatically when
appropriate.  Fixes “DeprecationWarning: make_current is deprecated;
start the event loop first” with warnings enabled.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-16 14:01:32 -07:00
Prakhar Pratyush df7ed437c2 compose: Show banner to explain interleaved view messages fading.
In an interleaved view when composing a message we fade messages
which the user is not replying to, to reduce the chance they send
a message to a recipient they didn't intend to.
Also, it reduces the visual/cognitive processing required
to figure out where their message is going to go.

But, it's not necessarily clear to users that what the
fading means, so this commit adds a one-time compose banner
to explain what's going on the first time this comes up.

Fixes part of #29076.
2024-07-16 13:52:30 -07:00
Prakhar Pratyush 35380b095f compose: Show banner to explain non interleaved view messages fading.
In a non interleaved view when composing a message to another
conversation we fade messages which the user is not replying to,
to reduce the chance they send a message to a recipient they didn't
intend to. Also, it reduces the visual/cognitive processing required
to figure out where their message is going to go.

But, it's not necessarily clear to users that what the
fading means, so this commit adds a one-time compose banner
to explain what's going on the first time this comes up.

Fixes part of #29076.
2024-07-16 13:52:29 -07:00
Alex Vandiver 556b92810b thumbnail: Advertize the thumbnail formats at client registration. 2024-07-16 13:22:15 -07:00
Alex Vandiver 6c624805ce upload: Return the closest-rendered thumbnail. 2024-07-16 13:22:15 -07:00
Alex Vandiver d121a80b78 upload: Serve thumbnailed images. 2024-07-16 13:22:15 -07:00
Alex Vandiver 2e38f426f4 upload: Generate thumbnails when images are uploaded.
A new table is created to track which path_id attachments are images,
and for those their metadata, and which thumbnails have been created.
Using path_id as the effective primary key lets us ignore if the
attachment is archived or not, saving some foreign key messes.

A new worker is added to observe events when rows are added to this
table, and to generate and store thumbnails for those images in
differing sizes and formats.
2024-07-16 13:22:15 -07:00
Alex Vandiver 7aa5bb233d tests: Clarify tests in test_thumbnail.py. 2024-07-16 13:22:15 -07:00
Alex Vandiver 63dbf67120 tests: Ensure we start with a local storage backend. 2024-07-16 13:22:15 -07:00
Alex Vandiver 229dcd0218 upload: Clean up empty directories in local storage. 2024-07-16 13:22:15 -07:00
Alex Vandiver 5f25fae0fa upload: Batch deleting old attachments.
delete_message_attachments() is faster than calling
delete_message_attachment() one-by-one.
2024-07-16 13:22:15 -07:00
Anders Kaseorg 8843f9f62a tests: Remove deprecated SHA1PasswordHasher.
SHA1PasswordHasher will be removed in Django 5.1.  MD5PasswordHasher
will remain for the purpose of speeding up tests.

Followup to commit ac5161f439 (#29620).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-16 13:06:31 -07:00
Alex Vandiver 90701978d9 migrations: Fix emoji re-thumbnailing for imports.
Imports (including conversions from Slack, Hipchat, etc) do not write
the `.original` of the file (nor the still versions, for animations),
only the thumbnailed (and possibly animated) version.  Looking for
`.original` would thus fail, and overwrite the (working) emoji with
the "?" and disable it.

In the event that we do not find the `.original`, fall back to the
bytes from the non-`.original`, to handle these import cases.
2024-07-16 12:57:49 -07:00
Alex Vandiver 1745d87f2e migrations: Add enough information to logs to reverse bad emoji "fixes." 2024-07-16 12:57:49 -07:00
Niloth P 6afed1432c integration-docs: Update Travis CI for new doc format.
Part of zulip#29592.
2024-07-16 09:40:45 -07:00
Niloth P 5db83f87e2 integration-docs: Update Transifex for new doc format.
Part of zulip#29592.
2024-07-16 09:40:45 -07:00
Niloth P 1f4310f893 integration-docs: Update Thinkst for new doc format.
Part of zulip#29592.
2024-07-16 09:40:45 -07:00