Commit Graph

8793 Commits

Author SHA1 Message Date
bedo 00c9f36434 create stream: Bulk conversion of principals to UserProfiles.
Previously, this logic did the database queries to look up UserProfile
objects in a loop.

Fixes #21820.

Significantly improves Stream creation time and also unsusbcribing users.
Tested stream creation with 10k stream subscribers:

- before: 127 seconds ~2 mins
- after: 17 seconds ~0.3 min

Add a test case for user unsubscribing themself.
2024-08-16 17:46:19 -07:00
Shubham Padia 2b3a41be58 user_card_popover: Show popover on all .messagebox user mention.
Show user card popover for scheduled messages overlay, compose box
preview, message edit preview, message edit history.

`.messagebox` was chosen as the selector since that was the nearest
parent class that was common for all of the above.

`@all` does not have a popover and that's why it will have the same
pointer as its parent element. We also introduce a new class called
`.user-mention-all` for managing css rules specific to that mention.
2024-08-15 16:28:08 -07:00
Prakhar Pratyush 52a9846cdf user_profile: Remove 'tutorial_status' field.
The 'tutorial_status' field on 'UserProfile' model is
no longer used to show onboarding tutorial.

This commit removes the 'tutorial_status' field,
'POST users/me/tutorial_status' endpoint, and
'needs_tutorial' parameter in 'page_params'.

Fixes part of zulip#30043.
2024-08-15 13:31:40 -07:00
Prakhar Pratyush ee806c49b9 ui_init: Use OnboardingStep for narrowing on first web app load.
We plan to remove the 'tutorial_status' field from UserProfile
table as it is no longer used to show tutorial.

The field is also used to narrow a new user in DM with
welcome bot on the first load.

This prep commit updates the logic to use a new OnboardingStep
for the narrowing behaviour on the first load. This will help
in removing the 'tutorial_status' field.
2024-08-15 13:31:40 -07:00
PieterCK d974c10efc integration-docs: Update Open Collective for new doc format.
Other than reformatting documentation for Open Collective, this
commit also moves it to the "Financial" category from "Communications".
This is because Open Collective is mainly a fundrising + legal status +
money management platform, as stated in https://opencollective.com/.

Part of #29592.
2024-08-15 13:16:14 -07:00
PieterCK 9494620022 integration: Delete Opbeat integration.
Opbeat was acquired by Elastic in 2017[1] and maintained until Elastic
APM went live[2]. Now that Elastic APM is online, it looks like Opbeat
repositories have been marked as archived/unmaintained[3].

[1]: https://www.elastic.co/blog/welcome-opbeat-to-the-elastic-family
[2]:
https://www.elastic.co/observability/application-performance-monitoring
[3]: https://github.com/opbeat
2024-08-15 13:16:14 -07:00
Vector73 80969a62e6 settings: Add `can_delete_any_message_group` setting.
Fixes #30717.
2024-08-15 12:03:49 -07:00
Alex Vandiver 2bfdccd065 narrow: Do not error when given an empty DM recipient list. 2024-08-14 10:39:40 -07:00
Shubham Padia 8b96aa277a css: Use classname instead of `div` selector for .blocks.
Having the `div` selector there affects the performance. See
https://chat.zulip.org/#narrow/stream/6-frontend/topic/CSS.20selector.20performance/near/1845719
for more details.
2024-08-13 22:36:35 -07:00
Prakhar Pratyush 0fddda232d drafts: Update do_delete_draft to send event on commit.
Earlier, we were using 'send_event' in 'do_delete_draft' which
can lead to a situation, if any db operation is added after the
'send_event' in future, where we enqueue events but the action
function fails at a later stage.

Events should not be sent until we know we're not rolling back.

Fixes part of #30489.
2024-08-12 12:16:14 -07:00
Prakhar Pratyush 096647fbd5 drafts: Update do_edit_draft to send event on commit.
Earlier, we were using 'send_event' in 'do_edit_draft' which
can lead to a situation, if any db operation is added after the
'send_event' in future, where we enqueue events but the action
function fails at a later stage.

Events should not be sent until we know we're not rolling back.

Fixes part of #30489.
2024-08-12 12:16:14 -07:00
Prakhar Pratyush ed7c399644 drafts: Update do_create_drafts to send event on commit.
Earlier, we were using 'send_event' in 'do_create_drafts' which
can lead to a situation, if any db operation is added after the
'send_event' in future, where we enqueue events but the action
function fails at a later stage.

Events should not be sent until we know we're not rolling back.

Fixes part of #30489.
2024-08-12 12:16:14 -07:00
Alex Vandiver 1f045bc794 migrations: Allow zulipinternal bots to delete their own messages. 2024-08-12 12:02:04 -07:00
adnan-td 83094860b1 integration-docs: Update Pingdom to the new doc format.
Replaced the image to show **Add Integration** with text
instructions.

Part of #29592.
2024-08-12 11:43:16 -07:00
Lauryn Menard 481fb8e518 help: Rename and redirect allow-image-link-previews article.
Note that the link in zulip_update_announcements.py is not updated
so that the content in the source code reflects what users actually
received in the update announcement message.
2024-08-12 11:22:37 -07:00
Sahil Batra 6919448917 settings: Remove create_web_public_stream_policy setting.
This commit removes create_web_public_stream_policy setting
since web-public channel creation permissions are now
handled by group-based setting.

We still pass "realm_create_web_public_stream_policy" in
"/register" response though for older clients with its
value being set depending on the value of group based
setting. If we cannot set its value to an appropriate enum
corresponding to the group setting, then we set it to
"Admins and moderators" considering that server will not
allow the users without permissions to create web-public
channels but the client can make sure that UI is
available to the users who have permission.
2024-08-01 22:49:33 -07:00
Sahil Batra 09102c5780 streams: Check permission to create web-public streams based on group setting. 2024-08-01 22:49:33 -07:00
Sahil Batra a15bc5e264 realm: Add new group setting for web-public stream creation. 2024-08-01 22:49:33 -07:00
Alex Vandiver 6f20c15ae9 thumbnail: Resolve a race condition when rendering messages.
Messages are rendered outside of a transaction, for performance
reasons, and then sent inside of one.  This opens thumbnailing up to a
race where the thumbnails have not yet been written when the message
is rendered, but the message has not been sent when thumbnailing
completes, causing `rewrite_thumbnailed_images` to be a no-op and the
message being left with a spinner which never resolves.

Explicitly lock and use he ImageAttachment data inside the
message-sending transaction, to rewrite the message content with the
latest information about the existing thumbnails.

Despite the thumbnailing worker taking a lock on Message rows to
update them, this does not lead to deadlocks -- the INSERT of the
Message rows happens in a transaction, ensuring that either the
message rending blocks the thumbnailing until the Message row is
created, or that the `rewrite_thumbnailed_images` and Message INSERT
waits until thumbnailing is complete (and updated no Message rows).
2024-08-01 16:48:16 -07:00
roanster007 7b3e163d55 refactor: Rename `huddle` to `direct_message_group` in non api files.
This commit completes rename of "huddle" to "direct_message_group"
in all the non API files.

Part of #28640
2024-07-31 23:25:56 -07:00
Prakhar Pratyush dc27711399 alert_words: Update remove_alert_word codepath to send event on commit.
Earlier, we were immediately enqueueing event in
'do_remove_alert_words' which can lead to a situation, if any
db operation is added after enqueueing event in future, where the
action function fails at a later stage.

Events should not be sent until we know we're not rolling back.

Fixes part of #30489.
2024-07-31 22:33:52 -07:00
Prakhar Pratyush ba80266b7e alert_words: Update add_alert_words codepath to send event on commit.
Earlier, we were using 'send_event' in 'do_add_alert_words' which
can lead to a situation, if any db operation is added after the
'send_event' in future, where we enqueue events but the action
function fails at a later stage.

Events should not be sent until we know we're not rolling back.

Fixes part of #30489.
2024-07-31 22:33:52 -07:00
Kenneth Rodrigues 97f15d8811 events: Migrate to typed_endpoint.
Migrate `event_register.py` and `tornado` to typed_endpoint.
Modified the tests to work with the migrated endpoints.
2024-07-31 17:10:06 -07:00
Kenneth Rodrigues 0f692436ca user_settings: Migrate to typed_endpoint.
Migrate `user_settings.py` to `typed_endpoint`.
Fix the error messages for the tests.
Migrate required validators.
2024-07-31 17:10:06 -07:00
Anders Kaseorg 8f1a97f168 users: Reduce date_joined precision to minutes.
Fixes #31188, removing a lot of useless high-entropy data from the
/register response.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-31 12:53:09 -07:00
Mateusz Mandera a0971934d9 thumbnail: Fix typo in comment. 2024-07-30 00:17:59 +02:00
Anders Kaseorg 3a17946c85 zulip_update_announcements: Fix December 2024 typo.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-26 10:20:27 -07:00
Alya Abbott 7a1f5765b7 updates: Add update message (level=8). 2024-07-25 13:19:25 -07:00
Alex Vandiver c726d2ec01 thumbnail: Do not Camo old thumbor URLs; serve images directly.
Providing a signed Camo URL for arbitrary URLs opened the server up to
being an open redirector.  Return 403 if the URL is not a user upload,
and the backend image if it is.  Since we do not have ImageAttachment
rows for uploads at a time we wrote `/thumbnail?` URLs, return the
full-size content.
2024-07-24 16:04:34 -07:00
Alex Vandiver e3a238fc89 thumbnail: Remove unused thumbnail sizes.
47683144ff switched the web client to prefer the 840x560 size, as the
mobile apps prefer; remove the now-unused 300x200 size.  No client was
using the generated `.jpg` formats, as all clients support `.webp`, so
remove the unused `.jpg` thumbnail as well.
2024-07-24 09:57:20 -07:00
Alex Vandiver e4a8304f57 thumbnail: Store the post-orientation-transformation dimensions.
Modern browsers respect the EXIF orientation information of images,
applying rotation and/or mirroring as specified in those tags.  The
the `width="..."` and `height="..."` tags are to size the image
_after_ applying those orientation transformations.

The `.width` and `.height` properties of libvips' images are _before_
any transformations are applied.  Since we intend to use these to hint
to rendering clients the size that the image should be _rendered at_,
change to storing (and providing to clients) the dimensions of the
rendered image, not the stored bytes.
2024-07-24 09:56:42 -07:00
Mateusz Mandera 8f461b3c03 email_mirror: Change default topic name if subject ends up empty.
If the email subject is something like `Fwd:`, it gets stripped to an
empty string, activating the "(no topic)" override. This however leads
to failure if the organization enables the setting forcing every message
to have a topic. Such emails should still go through, so we should just
change the topic value used.
2024-07-22 22:31:45 -07:00
Alex Vandiver 2ea0cc0005 thumbnail: Add a data-original-dimensions attribute.
This allows clients to potentially lay out the thumbnails more
intelligently, or to provide a better "progressive-load" experience
when enlarging the thumbnail.
2024-07-22 22:41:10 -04:00
Alex Vandiver 65828b20e9 thumbnail: Factor out a dataclass for markdown image metadata. 2024-07-22 22:41:10 -04:00
Prakhar Pratyush f48e87cd3c fetch_initial_state: Avoid doing one db query per announcement stream.
In 'fetch_initial_state_data' we were doing one database query
per announcement stream.

This commit updates the logic to prefetch those streams using
select_related hence avoiding the extra db queries.

Fixes #28909.
2024-07-22 10:21:22 -07:00
Alex Vandiver 3ac14632d8 thumbnail: Disable libvips cache.
The libvips cache is 100MB, 100 operations, or 100 files, whichever is
less.  A single Django process or worker is extremely unlikely to ever
see the same image twice, much less within those timeframes.

Disable the cache, since it is mostly useless memory usage for our use
case.
2024-07-22 10:19:33 -07:00
Alex Vandiver b42863be4b markdown: Show thumbnails for uploaded images.
Fixes: #16210.
2024-07-21 18:41:59 -07:00
Alex Vandiver 71406ac767 thumbnail: Factor frames into account for IMAGE_BOMB_TOTAL_PIXELS. 2024-07-21 18:41:59 -07:00
Alex Vandiver aacf28f7e3 test_classes: Extract a thumbnailing output format helper. 2024-07-21 18:41:59 -07:00
Alex Vandiver 94ff443c00 test_classes: Merge verbose assertEquals into ZulipTestCase. 2024-07-21 18:41:59 -07:00
Mateusz Mandera f50104d475 import_realm: Update incorrect comment about emoji dir in exported data.
The emoji dir is present in the data from our export tool. This was
added in 468afe4840.

This comment hasn't been updated since
c4b886d8ae, so probably we just forgot to
refresh it when custom emoji export was added.
2024-07-21 13:16:49 -07:00
Kenneth Rodrigues 8b489f4b96 streams: Convert to typed_enpoint. 2024-07-21 12:57:50 -07:00
Kenneth Rodrigues 6815cded83 zerver: Migrate some files to typed_endpoint.
Migrates `invite.py`, `registration.py` and
`email_mirror.py` to use `typed_endpoint`.
2024-07-20 15:46:48 -07:00
Niloth P 16abd82fa5 integrations: Remove Solano integration. 2024-07-20 15:44:17 -07:00
Sahil Batra c6e9173297 exceptions: Add new error class to be used for invalid parameter values.
This would help us in avoiding adding translation everytime we use
this error for a new pair of parameters.
2024-07-19 09:24:54 -07:00
Anders Kaseorg d574200423 tests: Consume streaming responses.
Fixes warnings like “ResourceWarning: unclosed file <_io.FileIO
name='/srv/zulip/var/044e5d44-87aa-4c43-abbb-28a144fa6654/test-backend/run_1238680/worker_0/test_uploads/files/thumbnail/2/1e/jmUuDhQC8WlaSRCuc0zQyx7D/img.tif/100x75.webp'
mode='rb' closefd=True>” with warnings enabled.

deque(…, 0) is an efficient way to consume an iterator documented at
https://docs.python.org/3/library/itertools.html#itertools-recipes
under consume.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-19 09:20:56 -07:00
Tim Abbott de35f44624 zulip_updates: Mention default for send navigation.
Apparently I forgot to merge this before -beta1.
2024-07-18 14:06:04 -07:00
Alex Vandiver 4351cc5914 thumbnail: Move get_image_thumbnail_path and split_thumbnail_path. 2024-07-18 13:50:28 -07:00
Alex Vandiver ecdf5713c2 tests: Factor out ZulipVerboseEqualTest class, and use more. 2024-07-18 13:50:28 -07:00
Alex Vandiver 08191d3f69 tests: Default ENABLE_FILE_LINKS to False.
Test configuration should mirror default production as much as possible.
2024-07-18 13:50:28 -07:00
Alya Abbott d775b6ffff updates: Add update message (level=7). 2024-07-18 12:38:26 -07:00
Mateusz Mandera 4a93149435 settings: Rework how push notifications service is configured.
Instead of the PUSH_NOTIFICATIONS_BOUNCER_URL and
SUBMIT_USAGE_STATISTICS settings, we want servers to configure
individual ZULIP_SERVICE_* settings, while maintaining backward
compatibility with the old settings. Thus, if all the new
ZULIP_SERVICE_* are at their default False value, but the legacy
settings are activated, they need to be translated in computed_settings
to the modern way.
2024-07-17 17:14:06 -07:00
Anders Kaseorg dd0d482d76 narrow: Migrate legacy SQLAlchemy select syntax.
Fixes “sqlalchemy.exc.RemovedIn20Warning: Deprecated API features
detected! These feature(s) are not compatible with SQLAlchemy 2.0.”
with warnings enabled.

https://docs.sqlalchemy.org/en/14/changelog/migration_14.html#change-5284

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-16 14:50:30 -07:00
Prakhar Pratyush df7ed437c2 compose: Show banner to explain interleaved view messages fading.
In an interleaved view when composing a message we fade messages
which the user is not replying to, to reduce the chance they send
a message to a recipient they didn't intend to.
Also, it reduces the visual/cognitive processing required
to figure out where their message is going to go.

But, it's not necessarily clear to users that what the
fading means, so this commit adds a one-time compose banner
to explain what's going on the first time this comes up.

Fixes part of #29076.
2024-07-16 13:52:30 -07:00
Prakhar Pratyush 35380b095f compose: Show banner to explain non interleaved view messages fading.
In a non interleaved view when composing a message to another
conversation we fade messages which the user is not replying to,
to reduce the chance they send a message to a recipient they didn't
intend to. Also, it reduces the visual/cognitive processing required
to figure out where their message is going to go.

But, it's not necessarily clear to users that what the
fading means, so this commit adds a one-time compose banner
to explain what's going on the first time this comes up.

Fixes part of #29076.
2024-07-16 13:52:29 -07:00
Alex Vandiver 556b92810b thumbnail: Advertize the thumbnail formats at client registration. 2024-07-16 13:22:15 -07:00
Alex Vandiver 2e38f426f4 upload: Generate thumbnails when images are uploaded.
A new table is created to track which path_id attachments are images,
and for those their metadata, and which thumbnails have been created.
Using path_id as the effective primary key lets us ignore if the
attachment is archived or not, saving some foreign key messes.

A new worker is added to observe events when rows are added to this
table, and to generate and store thumbnails for those images in
differing sizes and formats.
2024-07-16 13:22:15 -07:00
Alex Vandiver 63dbf67120 tests: Ensure we start with a local storage backend. 2024-07-16 13:22:15 -07:00
Alex Vandiver 229dcd0218 upload: Clean up empty directories in local storage. 2024-07-16 13:22:15 -07:00
Prakhar Pratyush 0de9f7f5cf import_realm: Add support to import 'OnboardingUserMessage' table. 2024-07-16 09:36:02 -07:00
Prakhar Pratyush 762b1c0f5b import_realm: Improve assert statements verifying table name.
This commit improves the assert statements to verify
that the table name is not "usermessage' instead of
verifying that table name doesn't include a substring
"usermessage".

This prep commit will help to avoid assertion error when
importing "onboardingusermessage" table.
2024-07-16 09:36:02 -07:00
Prakhar Pratyush 7d379e00b0 export: Fix 'OnboardingUserMessage' table not being exported.
Earlier, the export tool was logging a warning:
"??? NO DATA EXPORTED FOR TABLE zerver_onboardingusermessage!!!"

This bug was due to not configuring a Config object for
'OnboardingUserMessage' in 'get_realm_config()'.

This commit fixes the bug to export the table properly.
2024-07-16 09:36:02 -07:00
Kenneth Rodrigues a7da24a36f validators: Use cleaner syntax for AfterValidator.
Created a function that returns an `AfterValidator` for `check_int_in`
and `check_string_in` instead of having to use a
`lambda` wraper everytime.
2024-07-15 16:49:32 -07:00
Vector73 a07ebba860 zerver: Replace uri with url in local variables and comments. 2024-07-14 22:30:28 -07:00
Anders Kaseorg 80d3e60df8 ruff: Fix FURB180 Use of `metaclass=abc.ABCMeta`.
This is a preview rule, not yet enabled by default.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-14 13:53:40 -07:00
Anders Kaseorg e3a191b99b ruff: Fix FURB154 Use of repeated consecutive `global`, `nonlocal`.
This is a preview rule, not yet enabled by default.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-14 13:53:18 -07:00
Anders Kaseorg 6412c2d630 ruff: Fix FURB142 Use of set.add() in a for loop.
This is a preview rule, not yet enabled by default.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-14 13:52:59 -07:00
Anders Kaseorg 1e9b6445a9 ruff: Fix PLR6104 Use `+=` to perform an augmented assignment directly.
This is a preview rule, not yet enabled by default.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-14 13:49:51 -07:00
Anders Kaseorg b96feb34f6 ruff: Fix SIM117 Use a single `with` statement with multiple contexts.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-14 13:48:32 -07:00
Tim Abbott b0f144327d narrow: Update new function for modern Python typing syntax.
I apparently needed to be more careful rebasing the typing syntax
modernization work.
2024-07-13 23:06:47 -07:00
Anders Kaseorg 48202389b8 ruff: Bump target-version from py38 to py310.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-13 22:28:22 -07:00
Anders Kaseorg 3f29bc42b1 ruff: Fix B905 `zip()` without an explicit `strict=` parameter.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-13 22:28:22 -07:00
Anders Kaseorg 1464009fae ruff: Fix UP038 Use `X | Y` in `isinstance` call instead of `(X, Y)`.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-13 22:28:22 -07:00
Anders Kaseorg 0fa5e7f629 ruff: Fix UP035 Import from `collections.abc`, `typing` instead.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-13 22:28:22 -07:00
Anders Kaseorg 531b34cb4c ruff: Fix UP007 Use `X | Y` for type annotations.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-13 22:28:22 -07:00
Anders Kaseorg e08a24e47f ruff: Fix UP006 Use `list` instead of `List` for type annotation.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-13 22:28:22 -07:00
Anders Kaseorg a2fad74a28 typed_endpoint: Make is_optional understand T | None.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-13 22:28:22 -07:00
Aman Agrawal 70be9e8c51 narrow: Implement rendering of `with` narrow operators.
Adds server and web app support for processing the new `with`
search operator.

Fixes part of #21505.

Co-authored-by: roanster007 <rohan.gudimetla07@gmail.com>
Co-authored-by: Tim Abbott <tabbott@zulip.com>
2024-07-13 07:18:24 -07:00
Alya Abbott bd04a30bbc updates: Add update message (level=6). 2024-07-12 16:40:41 -07:00
Alex Vandiver 4df9b03451 migrations: Re-thumbnail and rename existing emoji. 2024-07-12 13:26:47 -07:00
Alex Vandiver 0385e5bab9 emoji: Store in S3 with a long public cache-control. 2024-07-12 13:26:47 -07:00
Alex Vandiver 0442e95276 emoji: Use a non-predictable filename.
We use a truncated SHA256 of the id and a server-side secret to make
emoji have non-guessable filenames, while also making collisions
unlikely.

We also adjust the Slack import to use the same SHA-based name,
instead of taking the same name as it had in Slack.
2024-07-12 13:26:47 -07:00
Alex Vandiver 262689da76 thumbnail: Fix MAX_EMOJI_GIF_FILE_SIZE_BYTES check to be post-resize.
This check was intended to check the post-resized image size, not the
pre-resized image.
2024-07-12 13:26:47 -07:00
Alex Vandiver 544d3df057 thumbnail: Stop applying MAX_EMOJI_GIF_FILE_SIZE_BYTES before resizing.
b14a33c659 attempted to make the 128k limit apply _after_ resizing,
but left this check, which examines the pre-resized image size.
2024-07-12 13:26:47 -07:00
Alex Vandiver 54f2fabac0 thumbnail: Still emoji are always pngs. 2024-07-12 13:26:47 -07:00
Alex Vandiver f6b99171ce emoji: Derive the file extension from a limited set of content-types.
We thumbnail and serve emoji with the same format as they were
uploaded.  However, we preserved the original extension, which might
mismatch with the provided content-type.

Limit the content-type to a subset which is both (a) an image format
we can thumbnail, and (b) a media format which is widely-enough
supported that we are willing to provide it to all browsers.  This
prevents uploading a `.tiff` emoji, for instance.

Based on this limited content-type, we then reverse to find the
reasonable extension to use when storing it.  This is particularly
important because the local file storage uses the file extension to
choose what content-type to re-serve the emoji as.

This does nothing for existing emoji, which may have odd or missing
file extensions.
2024-07-12 13:26:47 -07:00
Alex Vandiver 62a0611ddb emoji: Pass down content-type, rather than guessing from extension. 2024-07-12 13:26:47 -07:00
Alex Vandiver f3473defe1 transfer: Log when local emoji files are missing. 2024-07-12 13:26:47 -07:00
Sahil Batra ffb7744974 settings: Allow settings to be set to system groups only in production.
We use the already existing server level setting to only allow
settings to be set to system groups, not a named user defined
group as well, in production. But we allow to settings to be set
to any named or anonymous user group in tests and development server.

"can_mention_group" setting can be set to user defined groups
because some of the realms already do that in production.

The existing server level setting is also renamed to make it clear
that both user defined groups and anonymous groups are not allowed
if that setting is set to False.

This commit also changes the error message to be consistent for the
case when a setting cannot be set to user defined groups as per
server level and setting and when a particular setting cannot be set
to user defined groups due to the configuration of that particular
setting. For this we add a new class SystemGroupRequiredError in
exceptions.py so that we need not re-write the error message in
multiple places.
2024-07-12 09:51:49 -07:00
Niklas Fiekas af3a2500df
email_mirror: Also strip "AW:" from subject.
Besides "RE:" and "FWD:", "AW:" (from German "Antwort" for answer) is
another common prefix in email subjects. Let email_mirror automatically
remove it as well.
2024-07-11 11:16:24 -07:00
Alex Vandiver d5a4941691 django: Switch to .alias() instead .annotate() where possible.
When using the sub-expression purely for filtering, and not for
accessing the value in the resultset, .alias() is potentially faster
since it does not pull the value in as well.
2024-07-11 09:26:23 -07:00
Alex Vandiver 382cb5bb13 thumbnail: Lock down which formats we parse. 2024-07-11 07:31:39 -07:00
Alex Vandiver 4bc563128e thumbnail: Use a consistent set of supported image types. 2024-07-11 07:31:39 -07:00
Alex Vandiver a091b9ef81 thumbnail: Provide a more explicit hint than "Bad". 2024-07-11 07:31:39 -07:00
Alex Vandiver ff90e5355f upload: Pass down content-type of realm icon/logo to backend.
This saves having to try to re-derive it from the file extension,
which may be ".original" in some cases.
2024-07-11 07:31:39 -07:00
Vector73 a43d8159a9 exceptions: Update DM permission error string. 2024-07-10 18:43:50 -07:00
Prakhar Pratyush d92eb77c98 add_subscriptions_backend: Add 'transaction.atomic' decorator.
This commit adds an 'transaction.atomic' decorator to the
'add_subscriptions_backend' view thus making the db operations
within the view atomic and helps to avoid race between events sent.

In tests where we make POST requests to this view, we have
wrapped the API calls with a transaction.atomic() context
manager. It helps us with NOT rolling back the entire test
transaction due to error responses.
2024-07-10 10:50:37 -07:00
Prakhar Pratyush 9c614531fb test_subs: Use 'common_subscribe_to_streams' helper function.
In 'test_subs' we were making POST request to add
subscrption using 'self.api_post'.

This commit updates the code to use the test helper
function 'common_subscribe_to_streams' instead.

This prep commit will also help us to avoid adding
'transaction.atomic' context manager to these API calls
individually in the case of error response in the next commit.
2024-07-10 10:50:37 -07:00
Tim Abbott c9af4d571b onboarding: Fix whitespace for code block examples. 2024-07-09 16:42:23 -07:00
Alya Abbott b6e2a1b50d onboarding: Update Welcome Bot's responses to commands. 2024-07-09 13:08:02 -07:00
codewithnick b926ae0904 onboarding: Modify message that welcome bot does not understand.
Previously the bot sent bot commands whenever an undefined message
was sent by the user. This commit intends to fix the problem so that
the bot will only respond to the first message it does not understand
and not reply to any future undefined messages.

Fixes part of #30049.
2024-07-09 13:08:02 -07:00
codewithnick 4b79a77427 onboarding: Change default reply message of welcome bot.
The reply given by welcome bot when it does not understand a message,
sent by the user is changed, so that the bot is more clear to the user.

Fixes part of #30049.
2024-07-09 13:08:02 -07:00
codewithnick e41fda78d4 onboarding: Remove welcome bot message.
The previous message that welcome bot sent with
the bot commands has been removed in order to be more
clear and concise to the user. Previously weclome bot
also sent bot commands in intial message, this commit
intends to remove those commands.

Fixes part of #30049.
2024-07-09 13:08:02 -07:00
Vector73 6098c2cebe settings: Add two realm settings to restrict direct messages.
Fixes #24467.
2024-07-08 19:34:17 -07:00
Vector73 318d3e3cca user_groups: Add `is_any_user_in_group` function in `user_groups.py`. 2024-07-08 19:34:17 -07:00
bedo 88a0a3061e corporate-upgrade: Migrate to @typed_endpoint.
migrate the following endpoints from @has_request_variables
to @typed_endpoint :

- upgrade()
- remote_realm_upgrade()
- upgrade_page()
- remote_server_upgrade()
2024-07-08 16:14:57 -07:00
Lauryn Menard fb3ec350c7 help-settings-links: Fix for new invitations section in users tab.
Updates the instructions for `{settings_tab|invitations}` to work
like the deactivated users link, and updates the one article that
currently uses it: `help/invite-new-users.md`.
2024-07-08 13:56:18 -07:00
Prakhar Pratyush 5bb66e6c99 onboarding: Update initial direct message content.
This commit updates the Welcome Bot's initial
direct message content.

We inform about the tracked onboarding messages
via direct message only if it exists.

Fixes #30051.
2024-07-08 10:32:19 -07:00
roanster007 02d0566dc5 refactor: Rename `Huddle` Django model class to `DirectMessageGroup`.
This commit renames the "Huddle" Django model class to
"DirectMessageGroup", while maintaining the same table --
"zerver_huddle".

Fixes part of #28640.
2024-07-07 21:31:30 -07:00
Alex Vandiver 79f858b4b8 upload: Pass bytes to create_attachment.
This will be used to analyze the bytes for image metadata.
2024-07-07 14:40:07 -07:00
Alex Vandiver f97a30f240 upload: Reorder arguments to parallel upload_message_attachment. 2024-07-07 14:40:07 -07:00
Alex Vandiver f52a93bc14 upload: Stop requiring callers pass in the file size.
This can be calculated because we have the contents.
2024-07-07 14:40:07 -07:00
Alex Vandiver 58a9fe9af1 upload: Drop unused parameters to upload_message_attachment. 2024-07-07 14:40:07 -07:00
Alex Vandiver 0a296b2a6e upload: Start storing content-type for new uploads. 2024-07-07 14:40:07 -07:00
Alex Vandiver e29a455b2d avatars: Encode version into the filename.
Hash the salt, user-id, and now avatar version into the filename.
This allows the URL contents to be immutable, and thus to be marked as
immutable and cacheable.  Since avatars are served unauthenticated,
hashing with a server-side salt makes the current and past avatars not
enumerable.

This requires plumbing the current (or future) avatar version through
various parts of the upload process.

Since this already requires a full migration of current avatars, also
take the opportunity to fix the missing `.png` on S3 uploads (#12852).

We switch from SHA-1 to SHA-256, but truncate it such that avatar URL
data does not substantially increase in size.

Fixes: #12852.
2024-07-07 14:40:07 -07:00
Alex Vandiver feca9939bb s3: Support setting a cache-control on uploads. 2024-07-07 14:40:07 -07:00
Alex Vandiver 6258817bfd s3: Stop setting empty Content-Disposition header. 2024-07-07 14:40:07 -07:00
Prakhar Pratyush ed605328fb onboarding: Mark a few onboarding messages as starred.
To improve onboarding experience following onboarding
messages are marked as starred:
* First message in each onboarding topic.
* Initial DM sent by Welcome bot

Note: The onboarding topic messages needs to be tracked
in 'OnboardingUserMessage' model to get starred.

Fixes #29298.
2024-07-05 15:39:32 -07:00
Prakhar Pratyush 1fd6b09598 onboarding: Create OnboardingUserMessage rows during realm creation.
This commit updates the 'send_initial_realm_messages' function
to create OnboardingUserMessage rows corresponding to the
initial realm messages we send.

It helps to distinguish new onboarding messages from the
onboarding messages in the old realms.
2024-07-05 15:39:32 -07:00
Prakhar Pratyush fb836a4f0a onboarding: Add 'OnboardingUserMessage' model.
This prep commit adds a new OnboardingUserMessage model
that will be used to mark the new onboarding messages
for new users as unread and the first message of each
onboarding topic as starred.

This table won't include the old onboarding messages.
2024-07-05 15:39:32 -07:00
roanster007 52692a6448 refactor: Rename `huddle` to `direct_message_group` in non API.
This commit performs a sweep on the first batch of non API
files to rename "huddle" to "direct_message_group`.

It also renames variables and methods of type -
"huddle_message" to "group_direct_message".

This is a part of #28640
2024-07-04 07:56:31 -07:00
Alya Abbott cb1531b061 help: Don't link to the direct message feed.
We should teach users how to get there.
2024-07-03 16:30:52 -07:00
Tim Abbott da6c2e0c3c settings: Rename Deactivated users settings subtab.
Includes help center updates to better link to the tab.

Fixes https://chat.zulip.org/#narrow/stream/9-issues/topic/deactivated.20users.20tab/near/1843114
2024-07-02 13:53:31 -07:00
Sahil Batra 5ef14c3a8e users: Fix uploading user avatars.
Due to recent refactoring in 9fb03cb2c7, a user could not
upload avatar if the server uses local upload backend and there
was already an avatar file for that user.

This commit fixes it to just check if there exists a file only
when importing and not when the user is actually trying to
change the avatar.

Fixes #30676.
2024-07-02 13:26:21 -07:00
Lauryn Menard 1140e8402b integrations: Remove Google Code-in integration.
The Google Code-in program ended in 2019/2020 so there is no need
to maintain this Webhook integration.
2024-07-02 10:04:34 -07:00
Sayam Samal 073b116c55 zcommand: Rename night->dark and day->light in the color scheme backend.
As a follow up for f49a11c810, this
commit standardizes the naming of the day and night themes to light
and dark, respectively in the backend. This makes the backend
consistent with the naming used in the frontend and UI.

This also solves a regression introduced in
f49a11c810, where the frontend was sending
"/light" and "/dark" commands to the backend, but the backend was
expecting "/day" and "/night" commands.
2024-06-30 22:45:26 -07:00
Anders Kaseorg 5af8dfda3e ruff: Fix FURB167 Use of regular expression alias.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-06-30 22:37:15 -07:00
Anders Kaseorg b115d44b6a requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-06-27 15:31:43 -07:00
Anders Kaseorg 9fdc0eeb8e scim: Fix PLR1704 Redefining argument with the local name `path`.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-06-27 15:31:43 -07:00
Alex Vandiver 2eaf098c5d upload: Content-type is always defined. 2024-06-26 16:43:11 -07:00
Alex Vandiver 17fb23746f upload: Move methods into zerver.lib.upload from .base. 2024-06-26 16:43:11 -07:00
Alex Vandiver c826d80061 upload: Factor out common code into zerver.lib.upload. 2024-06-26 16:43:11 -07:00
Alex Vandiver c0d9a95aa8 email_mirror: Fix a misleading variable name. 2024-06-26 16:43:11 -07:00
Alex Vandiver 5cd10ce51d s3: Allow setting a CloudFront URL prefix for avatar and emoji images. 2024-06-26 16:43:11 -07:00
Alex Vandiver 08b24484d1 upload: Remove redundant acting_user_profile argument.
This argument, effectively added in 9eb47f108c, was never actually
used.
2024-06-26 16:43:11 -07:00
Alex Vandiver fb929ca218 thumbnailing: Remove unnecessary third return value from resize_emoji. 2024-06-26 16:43:09 -07:00
Alex Vandiver b14a33c659 thumbnailing: Switch to libvips, from PIL/pillow.
This is done in as much of a drop-in fashion as possible.  Note that
libvips does not support animated PNGs[^1], and as such this
conversion removes support for them as emoji; however, libvips
includes support for webp images, which future commits will take
advantage of.

This removes the MAX_EMOJI_GIF_SIZE limit, since that existed to work
around bugs in Pillow.  MAX_EMOJI_GIF_FILE_SIZE_BYTES is fixed to
actually be 128KiB (not 128MiB, as it actually was), and is counted
_after_ resizing, since the point is to limit the amount of data
transfer to clients.

[^1]: https://github.com/libvips/libvips/discussions/2000
2024-06-26 16:42:57 -07:00
Alex Vandiver 9fb03cb2c7 upload: Factor out common avatar logic. 2024-06-26 16:38:01 -07:00
Alex Vandiver d92993c972 upload: Factor out common emoji logic. 2024-06-26 16:38:01 -07:00
Vector73 b285e631e9 api_docs: Fix missing content from API documentation. 2024-06-25 15:37:31 -07:00
Prakhar Pratyush 1b9143b511 onboarding: Update message contents to use triple quotes.
This commit updates the message contents to use triple quotes.

It helps to maintain a consistent pattern for multi-line strings
in 'zerver/lib/onboarding.py' file and it'll be a lot more
convenient to edit in that form in the future.
2024-06-24 10:44:30 -07:00
Sahil Batra b44d30add7 events: Optimize computing can_create_streams and related fields.
Currently, for computing fields like can_create_public_streams
and can_create_private_steams fields, is_user_in_group is called
to check whether the user is part of the group which has the
permission. This means that there will be one DB query for each
field.

To optimize this, we now first fetch all the groups that the
user is member of, including the anonymous groups which are
used for settings, such that we can then just check whether
the user is part of the group which has the permission meaning
we would need only one query to compute all the fields.

This would be helpful when settings for other similar fields
will also be migrated to groups framework.
2024-06-24 10:00:08 -07:00
Mateusz Mandera f8616fa013 analytics: Send ZULIP_MERGE_BASE to the bouncer. 2024-06-23 07:44:11 -07:00
Anders Kaseorg f7eecb0e03 sentry: Untangle from page_params.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-06-22 18:53:56 -07:00
Alex Vandiver be9369541a create_user: Copy the avatar before announcing it. 2024-06-20 23:06:08 -04:00
Alex Vandiver 35f8c3c455 avatar: Clean up one callsite of deprecated avatar_url_from_dict. 2024-06-20 23:06:08 -04:00
Alex Vandiver 0153d6dbcd thumbnailing: Move resizing functions into zerver.lib.thumbnail. 2024-06-20 23:06:08 -04:00
Alex Vandiver 2c5dff7f59 thumbnailing: Remove unnecessary os.path adjustment.
This is a library file, not a binary; os.path is already set up.
2024-06-20 23:06:08 -04:00
Alex Vandiver 846cffe2fa name_restrictions: Reserve "avatar". 2024-06-20 23:06:08 -04:00
Sahil Batra 6164a41864 realm: Remove create_private_stream_policy setting.
This commit removes create_private_stream_policy setting as
we now use new group based setting.

The "/register" response includes realm_create_private_stream_policy
field to return a value representing superset of users who have the
permission to create private channels, as older clients still expect
this field.
2024-06-20 15:48:14 -07:00
Sahil Batra 0bae59fa4b streams: Check permission to create channels based on group setting.
This commit updates code to use new group based setting when
checking permission to create private channels.
2024-06-20 15:48:14 -07:00
Sahil Batra 27171d103e realm: Add new group setting for who can create private channels.
This commit adds new group based setting to control who can create
private channels.
2024-06-20 15:48:14 -07:00
Sahil Batra 428d4284f9 users: Update has_permission to accept realm as argument.
This commit helps in using the realm object which has the
prefetched group settings so that we can avoid extra queries
when calculating fields like can_create_public_streams.
2024-06-20 15:48:14 -07:00
Sahil Batra 19373d61e8 events: Do not call "can_create_..." functions twice.
There is no need to call settings_user.can_create_public_streams
and similar functions for private and web-public streams twice,
once to compute the field for a single stream type and one to
compute can_create_streams.

The value for each stream type can be used to compute value of
can_create_streams field.
2024-06-20 15:48:14 -07:00
Alya Abbott 2b9a0cbe52 updates: Add update about Reactions view, hiding sidebars, is:followed. 2024-06-20 15:30:47 -07:00
Anders Kaseorg 5f053c4aa7 upload: Serve more cross-browser audio and image formats as inline.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-06-20 15:29:20 -07:00
Anders Kaseorg fb4ad1422e mime_types: Add audio and image types missing from Python library.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-06-20 15:29:20 -07:00
Kenneth Rodrigues 15accfc983 narrow: Use NarrowParameter instead of dictionary.
Earlier we were using the type `OptionalNarrowListT` for all functions
that required "narrow" as a parameter.
This commit changes all the functions accepting a "narrow"
to use a list of the new `NarrowParameter`
instead of `OptionalNarrowListT` which is a list of dicts.
2024-06-20 12:22:37 -07:00
Kenneth Rodrigues 4e05381897 message_fetch: Migrate to typed_endpoint.
Removed the old `narrow_parameter` as we have shifted to
the new `NarrowParameter` Pydantic Object.
This new object provides better error messages for data validation,
hence changed the error messages in `test_message_fetch`.
2024-06-20 12:22:37 -07:00
Kenneth Rodrigues 387fba7156 custom_profile: Convert to typed_endpoint.
Convert `custom_profile_fields.py` to use `typed_endpoint`.

Use `TypedDict` from `typing_extensions` instead of `typing`,
to support Pydantic's type checking.
2024-06-19 18:29:28 -07:00
Prakhar Pratyush c10ef38b32 docs: Add subsystem doc for 'Onboarding Steps'.
Fixes part of #30043.
2024-06-18 09:39:50 -07:00
Prakhar Pratyush 0d59ff8c96 docs: Remove 'Hotspots' subsystem doc.
We no longer have 'Hotspots' to introduce users
to important UI elements.

We now use one time banners for it.
2024-06-18 09:39:50 -07:00
Mateusz Mandera 4917e01ffb push_notifications: Migrate to FCM HTTP v1 API.
The legacy API we use via python-gcm is deprecated and about to be
disabled.

Fixes #29768.
2024-06-17 18:26:26 -07:00
Mateusz Mandera 297141f8c4 push_notifs: Rename parse_gcm_options to parse_fcm_options. 2024-06-17 18:22:59 -07:00
Mateusz Mandera 497be370f5 push_notifs: Rename GCM->FCM in a bunch of comments.
This is called FCM now, even though we're using the legacy API that used
to be called GCM.
2024-06-17 18:22:59 -07:00
Mateusz Mandera 9c10b0d700 push_notifs: Rename has_gcm_credentials to has_fcm_credentials. 2024-06-17 18:22:59 -07:00
Mateusz Mandera 00b8cce50e push_notifs: Rename PushDeviceToken.GCM to FCM. 2024-06-17 18:22:59 -07:00
Mateusz Mandera 4591202032 push_notifs: Remove "canonical" handling in FCM API.
We believe this to already be obsolete and dead code and is about to be
removed with the migration to the FCM HTTP v1 API, where the concept
doesn't exist anymore.
2024-06-17 18:22:59 -07:00
nimish 81ea09be19 search: Add is:followed filter.
Create the is:followed search operator.
Fetch all messages that are from followed topics
using exists.
Update API documentation and changelog.

Co-authored-by: Kenneth Rodrigues <kenneth.nrk123@gmail.com>

Fixes #27309.
2024-06-17 18:22:50 -07:00
Shubham Padia 8ab6e71593 settings: Unite user settings into a single panel.
Previously, there were three different sections for managing active
users, deactivated users and invitations.
This commit combines users section has into a single tabbed panel.

Fixes: #26949.

Co-authored-by: shashank-23002 <21bec103@iiitdmj.ac.in>
2024-06-17 17:16:21 -07:00
Shubham Padia e4c89771fd settings: Rename `user-list-admin` to `users`.
Renaming the url for better readability on the user's side.
We've also added a redirect for this, which will change the `section`
from `user-list-admin` to `users` whenever required.
See https://github.com/zulip/zulip/pull/30268#discussion_r1631539656
2024-06-17 17:16:21 -07:00
Prakhar Pratyush 5e42c8cd2b user_topics: Handle IntegrityError during bulk insertion.
When there was a race during bulk insertion of UserTopic
rows, it resulted in Integrity error.

We update the 'last_updated' and 'visibility_policy'
columns for conflicting rows.

We also removed the separate update query to update
visibility_policy because now the new SQL query can
handle the updates too. This leads to have fewer round
trips to the database.
2024-06-17 11:21:30 -07:00
Prakhar Pratyush 7754483c3f user_topics: Clean up the 'duplicate_request' variable.
In the 'bulk_set_user_topic_visibility_policy_in_database' function,
the 'duplicate_request' variable wasn't improving any readability.

This commit cleans up that variable.
2024-06-17 11:21:30 -07:00
Prakhar Pratyush 55a8bdfbd9 push_notifications: Use is_same_server_message_link lib function.
This commit replaces the local 'is_same_server_message_link'
function used in 'get_mobile_push_content' with the
'is_same_server_message_link' lib function.

The lib function is the same logically but uses urllib instead
of regex for parsing and is backed by tests, hence more robust.
2024-06-17 10:42:32 -07:00
Prakhar Pratyush 484befe9ce url_decoding: Add 'is_same_server_message_link' function.
This prep commit adds a lib function 'is_same_server_message_link'.

This will be currently used while compressing quote and reply
in push notifications and later can be used at other places.
2024-06-17 10:42:32 -07:00
Sahil Batra e23af20079 realm: Prefetch group settings only for "/register" response.
This commit updates code to prefetch realm group settings like
"can_create_public_channel_group" only when computing settings
for "/register" response by refetching the realm object with
select_related instead of fetching those settings in UserProfile
query.

This change is done because we do not need to prefetch these
settings for every UserProfile object and for most of the cases
where these settings are actually accessed, we can afford extra
query like when checking permission to create streams. But we
cannot afford one query extra for each setting when computing
these settings for "/register" response, so we re-fetch the
realm object with select_related leading to only one extra
query.

The query count changes in tests are -
- Query count increases by 1 when calling fetch_initial_state_data
for computing can_create_public_streams because Realm object from
UserProfile does not have prefetched setting fields.

- Query count increases by one in test_subs where streams are
created which is as expected due to the setting not being prefetched.

- Query count increases by 2 in tests in test_home.py where one
query is to refetch the realm object and one for computing
can_create_public_streams as mentioned above.
2024-06-16 11:55:28 -07:00
Sahil Batra dea62a3fd9 events: Pass realm to fetch_initial_state_data always.
This commit makes passing realm mandatory to fetch_initial_state_data.

This is a prep commit for refetching the realm object with
select_related for group setting fields so that extra queries
can be avoided when computing "/registe" response.
2024-06-16 11:55:28 -07:00
Prakhar Pratyush de0c22592f push_notifications: Compress blockquotes when replying to a message.
To make better use of the limited characters in mobile push
notifications for messages quoting another message, we compress
the blockquotes and "user said" paragraphs to make space for the
actual message.

Fixes #28951.
2024-06-14 17:26:22 -07:00
Lauryn Menard 0bd2f0e545 api-docs: Extract logic for rendering events table and use templates.
Extracts code for generating the events table strings to a separate
function and uses templates so that the HTML is clearer.

Updates events table classes for CSS to start with "api-" for
clarity.
2024-06-14 10:29:53 -07:00
Lauryn Menard 963c871858 api-docs: Extract logic for generating individual event strings.
Creates a dataclass for getting the data needed for an individual
event's documentation and uses that dataclass in a separate
function for generating the strings for each event's documentation.

Uses templates for parts of event documentation with HTML and CSS,
and updates CSS classes to have clearer names.
2024-06-14 10:29:53 -07:00
Lauryn Menard 9caf8b8b2b help: Update billing articles for current billing management features.
Updates the self-hosted and Zulip Cloud billing articles for features
that have been added to billing management such as paying by invoice,
adding billing contact information, viewing past invoices, etc.

Revises text to be cleaner and clearer in some places.

More consistently uses bold (when not linked) for plan names.

Fixes a few errors or omissions in existing instructions.

Co-authored-by: Alya Abbott <alya@zulip.com>
2024-06-13 07:00:39 -07:00
Vector73 e8a5cb99bf api_docs: Improve events table and fix font style of events title. 2024-06-13 06:53:51 -07:00
Ryan Crisanti fc2aac6baa integration: Update topic & content of grafana webhook.
Sending to a topic based on the number of firing alerts makes no
sense, and leads to conversations and alerts scattered randomly across
topics based on how on fire the alerting is.

Send a separate message for each alert in the Grafana webhook payload,
with the alert's name as its topic; if no alert name can be found,
fall back to the alert's fingerprint.  Also include all alert values
in the body of the message, along with links to the alert generator,
silence, and image, if available.

Co-authored-by: Alex Vandiver <alexmv@zulip.com>
2024-06-12 06:44:14 -07:00
Alex Vandiver 18067794ae grafana: Rename fixtures to contain Grafana versions.
These are guessed, based on the shape of their contents and their
dates.
2024-06-12 06:44:14 -07:00
Vector73 f4c111b68a api_docs: Convert event names to lowercase in api docs. 2024-06-11 11:17:38 -07:00
Vector73 a4108e05ad api_docs: Add table of links for easy navigation in events api docs. 2024-06-11 11:17:38 -07:00
Sahil Batra 222995b1be settings: Remove create_public_stream_policy.
This commit removes create_public_stream_policy setting
since public channel creation permissions are now handled
by group-based setting.

We still pass "realm_create_public_stream_policy" in
"/register" response though for older clients with its
value being set depending on the value of group based
setting. If we cannot set its value to an appropriate
enum corresponding to the group setting, then we set
it to "Members only" considering that server will not
allow the users without permissions to create public
channels but the client can make sure that UI is
available to the users who have permission.
2024-06-10 12:24:45 -07:00
Sahil Batra 9365604c03 streams: Check permission to create streams based on group setting. 2024-06-10 12:24:45 -07:00
Sahil Batra 3bda17949e realm: Add new group setting for who can create public streams. 2024-06-10 12:24:44 -07:00
Mateusz Mandera f0ec4f5732 presence: Return last_update_id=-1 on zephyr mirror realm.
This was a bug, where in the realm.presence_disabled (synonymous to
being a zephyr mirror realm) case we would return None. We have decided
on the convention of using only integers here, and -1 representing lack
of data.
2024-06-07 13:57:40 -07:00
Mateusz Mandera 512f4d1476 presence: Backend implementation of the last_update_id API.
This builds on top of 016880f54d which
maintains correct .last_update_id for UserPresence objects; now we add
the related API changes to utilize it.
2024-06-06 17:25:54 -07:00
Kenneth Rodrigues 428ca713d0 users: Migrate to typed_endpoint. 2024-06-06 16:50:56 -07:00
Alex Vandiver 03864db5a2 streams: Move enforcement of stripped-stream-names lower.
This catches things like trailing spaces in internationalized
default-stream names.
2024-06-05 10:44:53 -07:00
Prakhar Pratyush 22f3aebb33 compose: Show a one-time banner for jump to sent message conversation.
We immediately navigate the user to the conversation they just
sent a message to if they are not already in the appropriate
conversation view.

This commit adds a first-time banner to explain the same.

Fixes #29575.
2024-06-05 09:31:17 -07:00
Alex Vandiver 46a57e37aa push_notifications: Support test notifications from old servers.
4430ab9cbe changed this, assuming that all servers would send
`realm_url` -- however, only servers running that commit do.  Update
to accept either `realm_url` or `realm_uri` payload properties.
2024-06-04 23:07:39 -07:00
Sahil Batra aac6d6cddb user_groups: Move get_group_setting_value_for_audit_log_data.
This commit moves get_group_setting_value_for_audit_log_data
to zerver.lib.user_groups.
2024-06-04 09:55:27 -07:00
Vector73 4430ab9cbe zerver: Replace `realm_uri` with `realm_url` in backend files.
Co-authored-by: Junyao Chen <junyao.chen@socitydao.org>
2024-06-03 10:07:10 -07:00
Sahil Batra d3ea6520dc user_groups: Add server level setting disallow anonymous groups for settings.
This commit adds a server level setting which controls whether the setting
can be set to anonymous user groups. We only allow it in the tests for
now because the UI can only handle named user groups.
2024-06-03 09:45:26 -07:00
Sahil Batra 6711f0af09 message_delete: Fix event sent for updating first_message_id.
This commit fixes the event sent for updating first_message_id
when a message is deleted, to include the name field as it is
required for all "stream/update" events.

This commit also adds a test in test_events for the case when
first_message_id of a stream is updated on deleting a message.
2024-06-03 08:52:11 -07:00
Mateusz Mandera b1d50b511c presence: Handle PresenceSequence in the export/import system. 2024-06-02 22:08:28 -07:00
Mateusz Mandera 0dca8f2a38 models: New PresenceSequence model and UserPresence.last_update_id col.
Migration plan:
1. Add NULLable .last_update_id column to UserPresence with default 0
   for new objects.
2. Backfill the value to 0 for old UserPresences, can be done in the
   background while server is running.
3. Make the column non-NULL.
4. Add new model PresenceSequence and create its rows for old realms.
2024-06-02 22:08:28 -07:00
Mateusz Mandera 355f05ffbc import_realm: Import message.edit_history correctly.
Fixes #26369.

There are two important fixes to make to the dicts in edit_history:
1. Update the user_id so that it points to the imported sender.
2. Apply fix_message_rendered_content to the prev_rendered_content data
to fix up mentions and other such syntax.
2024-06-02 21:10:50 -07:00
Mateusz Mandera 878d46ea49 import_realm: Fix docstring of fix_message_rendered_content.
The docstring was misleading talking about the import from non-Zulip
platforms, when this function is also very much applicable for
Zulip-to-Zulip imports.
2024-06-02 21:10:50 -07:00
roanster007 384a43c79c narrow: Fix server error of operand of "id" operator at large values.
Previously, when the operand of id operator was more than
2147483647, it was raising server error. This is because the
maximum permissible PostgreSQL integers value is 2147483647.

This is fixed by raising a BadNarrowOperatorError in case the
id operand is larger than 2147483647.
2024-06-01 22:38:18 -07:00
Alex Vandiver 5814583694 rabbitmq: Support non-/ vhosts.
Fixes: #30255.
2024-05-31 15:41:19 -07:00
Anders Kaseorg e53e3af0f6 codespell: Fix spelling mistakes caught by codespell.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-05-31 14:32:33 -07:00
Prakhar Pratyush f9fca5b469
remove_single_newlines: Fix regex to parse list syntax correctly.
Earlier, the 'remove_single_newlines' function wasn't working
correctly when '\n' was followed by asterix, hyphen, or number.
Specifically, they were not added as a syntax for ordered or
unordered list in markdown.

For example, see the workaround fix in 330439a83b.

This commit updates the function to replace '\n' with ' ', when
"\n" is not preceded by "\n" and not followed by:
* Another newline (\n)
* A hyphen followed by a space
* An asterisk followed by a space
* A number followed by a dot and a space

We won't have to do fixes like 330439a83b in the future.
2024-05-30 11:32:52 -07:00
Prakhar Pratyush 4deecfa58d zulip_updates: Fix assertion error while sending update messages.
Earlier, for new realm with zulip_update_announcements_stream
set to None, an assertion error was raised in
'is_group_direct_message_sent_to_admins_within_days' because
no 'None' to new level change took place for such realm.

A new realm is on the latest level and a default stream is set.

This commit updates the logic to simply skip sending
update messages as the stream was manually set to None
in such cases.
2024-05-30 10:52:32 -07:00
Prakhar Pratyush 1ce7c11705 user_profile: Remove 'onboarding_steps' field.
Removes the 'onboarding_steps' field on 'UserProfile'
model which is no longer used.

It was introduced back in 2013 (b5e22bf), and is no
longer used.

Fixes part of #30043.
2024-05-29 16:12:35 -07:00
Sahil Batra aa84080ad6 tests: Add a helper function to create anonymous groups.
This commit adds a new helper function to create or update
a UserGroup object for a setting. We could have used existing
update_or_create_user_group_for_setting but that also validates
user IDs and subgroup IDs which we can skip in tests.
2024-05-28 07:24:07 -07:00
Sahil Batra 81765a1e83 user_groups: Fix audit log data when changing group settings.
This commit fixes the code store correct old value in audit
log data when changing can_mention_group setting from a
anonymous group to another anonymous group. The bug was
because the old value was being computed after updating
the UserGroup object with new members and subgroups and
is fixed by computing the old value for all the cases
and passing it to do_change_user_group_permission_setting.
2024-05-28 07:24:07 -07:00
Mateusz Mandera c0bf67d6da import: Check off old, redundant TODO about huddle import.
This was added long ago in
38dd9e49de

Double-checked that this is right and added a brief mention of what
exactly the dependency here is.
2024-05-25 23:01:41 -07:00
Alex Vandiver b7bf9e41c7 markdown: Catch urllib3 exceptions which are raised during streaming.
requests transforms the base urllib3 exceptions into
requests.RequestExceptions -- but only within code that it is
running.  When parsing the streaming body in fetch_open_graph_image,
the read itself (inside lxml) may trigger urllib3 to raise its own
timeout error -- which escapes the current catch of
requests.RequestExceptions.

Catch both requests and urllib3 exceptions.
2024-05-24 14:54:53 -07:00
Vector73 93262e03ad emails: Replace occurrences of uri with url in email templates.
This commit replaces occurrences of realm_uri with realm_url in email templates
and other related backend files.

Co-authored-by: Junyao Chen <junyao.chen@socitydao.org>
2024-05-24 11:18:35 -07:00
Alex Vandiver cb2a105de2 message: Respect first_visible_message_id when computing unreads.
This prevents unread counts for messages which the web UI does not
display, and does not let you mark as unread via "mark all as read."
2024-05-24 10:50:19 -07:00
Alex Vandiver 2218de0399 management: Disable Sentry for management commands run interactively.
This adds `--automated` and `--no-automated` flags to all Zulip
management commands, whose default is based on if STDIN is a TTY.
This enables cron jobs and supervisor commands to continue to report
to Sentry, and manually-run commands (when reporting to Sentry does
not provide value, since the user can see them) to not.

Note that this only applies to Zulip commands -- core Django
commands (e.g. `./manage.py`) do not grow support for `--automated`
and will always report exceptions to Sentry.

`manage.py` subcommands in the `upgrade` and `restart-server` paths
are marked as `--automated`, since those may be run semi-unattended,
and they are useful to log to Sentry.
2024-05-24 10:30:16 -07:00
Alex Vandiver 174ae2b409 zulip_update: Catch and carry on if a realm fails.
It is better to log the exception and _try_ all realms, than to fail
early due to a bad realm and fail to notify the rest.
2024-05-24 10:21:45 -07:00
Alex Vandiver 5c2fd1de5a docs: Update Django links to our current version. 2024-05-24 10:18:37 -07:00
Vector73 62dfd93a83 api: Add "users/<int:user_id>/status" endpoint.
The documentation Creates a shared UserStatus schema that's used for
the return value of this new endpoint and for the existing user_status
objects returned by the register queue endpoint.

Co-authored-by: Suyash Vardhan Mathur <suyash.mathur@research.iiit.ac.in>

Fixes #19079.
2024-05-22 18:07:22 -07:00
Vector73 d6672c57ff user_status: Rename `get_user_status_dict` function.
Renames `get_user_status_dict` function to `get_all_users_status_dict`
to distinguish it from new `get_user_status` function.
2024-05-22 18:07:22 -07:00
Sahil Batra d1bcac0822 realm: Use enums for accessing setting values.
This commit updates code, majorly in tests, to use
setting values from enums instead of directly using
the constants defined in Realm.

We still have those constants defined Realm as they
are used in a couple of places where the same code
is used for different settings. These will be
handled later.
2024-05-22 17:20:37 -07:00
Sahil Batra 52b0199189 realm: Migrate "PATCH /realm" endpoint to typed_endpoint. 2024-05-22 17:20:37 -07:00
Prakhar Pratyush c91d018b38 user_topics: Make bulk_set_user_topic_..._in_database atomic.
In e726012244, we removed the
@transaction.atomic decorator from the function
'bulk_set_user_topic_visibility_policy_in_database'.

We should keep it to avoid adding it to the caller
of this function, when reused in the future.
2024-05-21 09:24:43 -07:00
Prakhar Pratyush 508c5611d1 claim_attachment: Remove the stale 'user_profile' parameter.
This commit removes the unused 'user_profile' parameter
of the 'claim_attachement' function.
2024-05-21 09:24:43 -07:00
Tim Abbott 330439a83b zulip_updates: Wrap differently to workaround bug.
The bold syntax appearing on the start of a line made this render badly.
2024-05-20 18:01:04 -07:00
Alya Abbott 0decaef284 updates: Add update about stream->channel rename and faster loading. 2024-05-20 16:31:55 -07:00
Sahil Batra 2007a582a1 message: Optimize checking if mentioning user group is allowed.
This commit updates the code to not call is_user_in_group
function if can_mention_group setting is set to "role:everyone"
group.
2024-05-20 14:43:07 -07:00
Sahil Batra eec80a4a5a message: Fix code to check group mention permission.
This commits fixes the code which checks group mention permission
to handle anonymous user groups correctly. Basically we were
not checking whether the UserGroup is linked to a NamedUserGroup
and directly accessing named_user_group which results in an
error.

We also update the error messages to include the group name
which has permission to mention the groups since now there
might be a comnbination of groups and users who has permission
to mention the group.

This commit also adds tests to check sending and editing messages
when can_mention_group is set to a anonymous user group.
2024-05-20 14:43:07 -07:00
Kenneth Rodrigues ba92cd8b23
search: Include history for is:resolved queries.
The previous logic assumed that all `is:` operators were those like `is:muted`
or `is:starred`, which depend on personal data.
2024-05-20 14:37:08 -07:00
Anders Kaseorg b545abe1e2 typos: Fix typos caught by mwic.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-05-20 13:55:00 -07:00
Anders Kaseorg 38cae070ed typos: Fix typos caught by typos.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-05-20 13:55:00 -07:00
bedo acec5108c5 import/export: Map each message id to its attachments.
Fixes #29350
2024-05-20 09:18:49 -07:00
Prakhar Pratyush c798d192dc message_send: Update do_send_messages codepath to send event on commit.
Earlier, we were using 'send_event' & 'queue_json_publish' in
'do_send_messages' which can lead to a situation where we enqueue
events but the transaction fails at a later stage.

Events should not be sent until we know we're not rolling back.
2024-05-19 23:18:43 -07:00
evykassirer 96c9950115 composebox_typeahead: Convert module to typescript. 2024-05-16 14:53:26 -07:00
Sahil Batra aa83812b1b user_groups: Rename are_both_setting_values_equal function.
This commit renames are_both_setting_values_equal function
to are_both_group_setting_values_equal to make it clear
that this function is used to compare value of group
settings.
2024-05-16 12:58:20 -07:00
Sahil Batra 7db7b3f94d groups: Move functions for updating settings to "lib.user_groups".
This commit moves validate_group_setting_value_change,
are_both_setting_values_equal and parse_group_setting_value
functions, which are used for updating the group settings, to
"zerver.lib.user_groups" as these functions will also be used for
group based realm and stream settings and "zerver.lib.user_groups"
file seems a better place to place such functions which are used
at multiple places.

For same reasons, we also move GroupSettingChangeRequest dataclass
to "zerver.lib.user_groups" file.
2024-05-16 12:58:20 -07:00
Tomas Fonseca 753b8e1f6f markdown: Added error handling for invalid time zones.
If an invalid timezone (such as +32h) was provided, the
timestamp.astimezone call would throw an exception, causing the
message send to fail. Replace that with a user-facing error.

Fixes #19658.
2024-05-16 09:38:25 -07:00
Aman Agrawal 7203661d99 support: Set discounted price instead percentage for customers.
This allows us to set the price of a plan exactly as discussed with
the customer.
2024-05-16 02:18:43 -07:00
tnmkr 51635c81fc streams: Set creator for some initial streams in populate_db.
This is a follow-up to #29425. It assigns creator to three streams
that are used in tests and another two that aren't. For tests,
we assign a guest, an admin and an owner as creator.
2024-05-15 11:21:46 -07:00
Prakhar Pratyush e264b12066 zulip_updates: Send an introductory message before the first update.
To help users focus on the onboarding experience, we no longer
send the introductory "Zulip updates" message as a part of
onboarding.

Now, we send the introductory message just before the first
update message.

Fixes #30053.
2024-05-15 11:19:03 -07:00
Sahil Batra 8bca565218 groups: Pass old setting value for can_mention_group.
This commit adds support to pass object containing both old and new
values of the can_mention_group setting, as well as detailed API
documentation for this part of the API system.

Co-authored-by: Tim Abbott <tabbott@zulip.com>
Co-authored-by: Greg PRice <greg@zulip.com>
2024-05-14 12:17:15 -07:00
Mateusz Mandera 160076fdfa import: Fix import of Stream.creator field.
When importing a Stream, UserProfiles don't yet exist. so trying to
import stream.creator fails with something like:

```
psycopg2.errors.ForeignKeyViolation: insert or update on table
"zerver_stream" violates foreign key constraint
"zerver_stream_creator_id_65aeba7e_fk_zerver_userprofile_id"
DETAIL:  Key (creator_id)=(5) is not present in table "zerver_userprofile".
```
2024-05-13 20:11:28 -07:00
Prakhar Pratyush 0cddc8a6a6 onboarding: Fix topic names not being translated correctly.
In 'send_initial_realm_messages', the topics names were not
being translated properly as they were computed outside the
with `override_language` block.

Now, we use 'send_initial_realm_messages' inside a with
'override_language' block in the caller. This fixes the bug.

Note: We are using 'override_language' block in the caller
and not within the function as it helps to avoid indenting
everything inside the function.
2024-05-13 16:06:10 -07:00
Prakhar Pratyush aad66674e7 onboarding: Fix the query for 'greetings_message'.
The query to fetch one of the greetings message based on
content wasn't taking translation into account that would
result in a bug in realms using non-english language.

This commit updates the query to fix the bug.
2024-05-13 16:06:10 -07:00
Alya Abbott 8a51c42c07 onboarding: Modify content on onboarding messages.
- Tighten up the content.
- Use pretty quotes.
- Reorder "experiments" topic to appear after "start a conversation"
  in Inbox.
- Drop reference to messages being starred (not implemented yet).
2024-05-12 17:24:17 -07:00
Prakhar Pratyush 15a7870d2e onboarding: Use 'remove_single_newlines' once, while sending message.
Instead of applying 'remove_single_newlines()' to individual
contents when declared, we use it once while prepping up
message in 'internal_prep_stream_message_by_name'.

Just for the code to be less finnicky.
2024-05-12 17:24:17 -07:00
Prakhar Pratyush 643998fcfa onboarding_steps: Rename 'hotspots' files to 'onboarding_steps'. 2024-05-10 12:30:22 -07:00
Prakhar Pratyush bf2360bcf2 onboarding_steps: Remove hotspot as an onboarding_step.
Earlier, hotspots and one-time notices were the valid
type of onboarding step.

Now, one-time notice is the only valid type.

Fixes #29296.
2024-05-10 12:30:22 -07:00
Prakhar Pratyush e8769f80a6 onboarding_steps: Rename 'copy_hotspots' function.
The function is used for any type of onboarding step
i.e. not limited to Hotspot.

This commit updates the function and variables name
to reflect the above mentioned behaviour.
2024-05-10 12:30:22 -07:00
Prakhar Pratyush ac2152fdfd onboarding: Replace channel messages in new organizations.
As a part of improving onboarding experience, this
commit updates the channels and messages initially
available in new orgs.

Fixes #29297.
2024-05-09 14:05:35 -07:00