Commit Graph

18206 Commits

Author SHA1 Message Date
Anders Kaseorg 3b4527843d requirements: Upgrade Django to 5.0.5.dev.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-04-18 12:39:47 -07:00
Anders Kaseorg 72018cc26b timeout: Rename to unsafe_timeout.
This timeout strategy using asynchronous exceptions has a number of
safety caveats (read the docstring!!) and should only be used in very
specific circumstances.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-04-18 11:50:38 -07:00
Anders Kaseorg 631c2f7d4c test_helpers: Remove unused timeout_mock.
We don’t want to encourage more uses of zerver.lib.timeout.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-04-18 11:50:38 -07:00
Anders Kaseorg da979bc65c streams: Remove inappropriate use of zerver.lib.timeout.
zerver.lib.timeout abuses asynchronous exceptions, so it’s only safe
to use on CPU computations with no side effects.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-04-18 11:50:38 -07:00
Anders Kaseorg 95a1481f99 message_flags: Remove inappropriate use of zerver.lib.timeout.
zerver.lib.timeout abuses asynchronous exceptions, so it’s only safe
to use on CPU computations with no side effects.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-04-18 11:50:38 -07:00
Anders Kaseorg 72aeaf8d52 db: Split reset_queries into a new module zerver.lib.db_connections.
Fixes an import cycle that breaks mypy inference with django-stubs:

zproject.settings → zproject.computed_settings → zerver.lib.db →
django.db → django.db.backends.base.base →
django.db.backends.base.features → django.db.models.base →
django.db.models.options → django.contrib.contenttypes.fields →
django.contrib.contenttypes.models → confirmation.models → django.conf
→ zproject.settings

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-04-17 16:49:03 -07:00
Kislay Verma 245357c868 stream_settings_ui: Go to newly created stream.
- On creating a stream, the user is redirected to the
 "stream events" topic of the newly created stream.
- If this is the first time the user has created a stream,
an explanatory modal is shown.

test_hotspots.py is modified to account for the new addition
of the first_stream_created_banner.

stream_create.test.ts is also modified accordingly.

Fixes #29375.
2024-04-17 15:35:19 -07:00
Mahhheshh d82f33a3c8 zilencer: Migrate to `@typed_endpoint`.
migrated views:
- `zilencer.views.register_remote_server`
- `zilencer.views.register_remote_push_device`
- `zilencer.views.unregister_remote_push_device`
- `zilencer.views.unregister_all_remote_push_devices`
- `zilencer.views.remote_server_notify_push`

to make sure the previous checks for `remote_server_notify_push` matches
to old one, The `RemoteServerNotificationPayload` is defined.
2024-04-17 13:45:54 -07:00
Tim Abbott 2c62152a8b left-sidebar: Rename "All direct messages" => "Direct message feed".
This aligns with the new naming scheme for other left sidebar
elements.
2024-04-17 12:42:21 -07:00
nimishmedatwal 1594011b67 left_sidebar: Rename All messages to Global Feed.
Fixes part of #27802.
2024-04-17 12:42:21 -07:00
Anders Kaseorg f31579a220 python: Avoid relying on Collection supertype of QuerySet.
QuerySet doesn’t implement __contains__, so it can’t be a subtype of
Container or Collection (https://code.djangoproject.com/ticket/35154).
This incorrect subtyping annotation was removed in
https://github.com/typeddjango/django-stubs/pull/1925, so we need to
stop relying on it before upgrading to django-stubs 5.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-04-16 23:02:16 -07:00
Alex Vandiver 5654d051f7 worker: Split into separate files.
This makes each worker faster to start up.
2024-04-16 23:00:02 -07:00
Alex Vandiver db86027cbd migrations: Add a migration to fix emoji metadata in S3.
bcd179be0e fixed this for recent imports, but did not address
imports from before it was merged.
2024-04-16 17:13:06 -07:00
Anders Kaseorg b64bf2d425 test_console_output: Mark regexes as raw strings.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-04-16 14:51:39 -07:00
roanster007 68b4298d8e settings: Add option to disable seeing typing notifications.
This commit adds an option to the advanced section of
Preferences settings, that would allow users to choose
whether to receive typing notifications from other
users.

Fixes #29642
2024-04-16 11:38:12 -07:00
Alex Vandiver 693b959656 markdown: Switch to directly URL-escaping CSS URLs.
soupsieve is a heavy-weight dependency, and Tornado pulls it in by way
of markdown rendering; since we are only using it for a very simple
process, perform that manually.

Per CSS spec[^1]:

> In quoted <string> url()s, only newlines and the character used to
> quote the string need to be escaped.

[^1]: https://drafts.csswg.org/css-values/#urls
2024-04-16 10:48:51 -07:00
Alex Vandiver 1424a2e748 documentation: Move OpenGraph description updating out of middleware.
This middleware was highly-specific to a set of URLs, and pulled in a
beautifulsoup dependency for Tornado.  Move it closer to where it is
used, minimizing action at a distance, as well as trimming out a
dependency.
2024-04-16 10:41:45 -07:00
Alex Vandiver 16a08a8b6b narrow: Do not create huddles when fetching messages.
9a682fb20a started performing message fetching in a read-only
transaction.  However, our use of `get_or_create_huddle` can violate
the read-only promise, and result in a user-facing 500.

In the cases where we're attempting to narrow to a huddle that does
not exist, this is equivalent to a false condition; catch those,
without making the huddle row, and insert a false.
2024-04-16 09:57:55 -07:00
Alex Vandiver 823f732132 narrow: Move python-only implemenation into its own file.
Tornado uses this, and none of the heavy-weight SQLAlchemy code; so it
just adds to the startup time of the Tornado server.
2024-04-16 09:48:11 -07:00
Alex Vandiver 57ff573535 topic: Move sqlalchemy methods into their own file.
Loading sqlalchemy can take a significant amount of time, so splitting
these into these own file can be a significant startup-time savings.
2024-04-16 09:48:11 -07:00
Shubham Padia 1abd356a91 custom_profile_fields: Make required field optional during update. 2024-04-15 16:52:11 -07:00
Shubham Padia 8c30b61201 custom_profile_fields: Display_in_profile_summary optional for update. 2024-04-15 16:35:42 -07:00
Shubham Padia 137776b092 custom_profile_fields: Make field_data optional during update. 2024-04-15 16:35:42 -07:00
Shubham Padia c6c392bcc7 custom_profile_fields: Make name & hint optional during update.
Fixes #23022.
Added a new validation function that validates updates to custom
profile fields.
2024-04-15 16:35:39 -07:00
Mateusz Mandera 4a2a9176c2 realms: Add caching to the get_seat_count calculation for upload limit.
For simiplicty's sake, we can avoid trying to do cache invalidation in
the variety of events that can cause the seat count to change - since
having an up to 1 day delay between users being added and the upload
limit going up is quite reasonable.
2024-04-15 15:08:56 -07:00
Mateusz Mandera 066de96a86 uploads: Implement 5GB/user quota for paid orgs on Zulip Cloud.
Fixes #28621

Till now, this was actually a flat 50GB despite what the /plans/ page
says and was adjusted flexibly when somebody asked for a higher limit.

This actually implements the advertised formula, but changing it to
5GB/user since that's a more reasonable limit.

Keeps the 50GB limit for sponsored Standard Free organizations and also
places it as the floor for the quota for paid orgs, to not lower this
for tiny orgs with less than 5 users.
2024-04-15 15:08:56 -07:00
Mateusz Mandera 8038e2322c realm: Change implementation approach for upload_quota_gb.
Most importantly, fixes a bug where a realm with a custom
.upload_quota_gb value (set by changing it in the database via e.g.
manage.py shell) would end up having it lowered while upgrading their
plan via the do_change_realm_plan_type function, which used to just set
it to the value implied by the new plan without caring about whether
that isn't lower than the original limit.

The new approach is cleaner since we don't do db queries by
upload_quota_gb so it's nicer to just generate these dynamically, making
changes to our limit-per-plan rules much easier - skipping the need for
migrations.
2024-04-15 15:08:56 -07:00
Mateusz Mandera 540d419ef7 events: Fix realm_upload_quota_mib value to be in MiB.
This was bytes until now.
2024-04-15 15:08:56 -07:00
Alex Vandiver d341e6ced1 runtornado: Honor the --skip-checks flag.
This is particularly important because they can be quite heavy-weight.
2024-04-15 14:29:27 -07:00
Alex Vandiver 043d3127eb upload: Only load S3 backend (and thus boto3) if necessary.
Because loading boto3 is so slow, this saves a significant amount of
time (0.3s or so) in process startup on servers which are not using
the S3 file storage backend.
2024-04-15 13:12:51 -07:00
N-Shar-ma d8a8364d1a todo_widget: Allow tasks to be added through `/todo` command.
Uptil now, users could add tasks to a todo widget only after creating
it through the `/todo` command in the compose box.

Users can now add an initial list of tasks using the `/todo` command,
with each task on a new line in the compose box, where the 1st `:`
would separate a task from its (optional) description. Example:
`/todo\nTask1:description1\nTask2 without description`.

Fixes part of #20213.
2024-04-13 21:56:33 -07:00
N-Shar-ma 6df3ad251a todo_widget: Allow task list title to be set and edited by author.
Users can now name task lists by providing the task list title in the
`/todo` command on the same line. Example: `/todo School Work`. If no
title is provided by the user, "Task list" (which is also the
placeholder) is used as default.

The author of a task list can later edit / update the task list title
in the todo widget, just like the question in the poll widget.

Fixes part of #20213.
2024-04-13 21:56:33 -07:00
N-Shar-ma b30eb4c4fc widget: Fix bug where 2nd line became poll question.
Uptil now, if the user did not specify a poll question, but included
options when using the `/poll` command, the 1st option wrongly
became the poll question.

Now on leaving the poll question space blank, the next line will not be
confused for it, and the poll widget will have an input for adding a
question.
2024-04-13 21:56:33 -07:00
Alex Vandiver 5990835bc1 realm_export: Use .count(), instead of len(). 2024-04-13 21:50:42 -07:00
Alex Vandiver a49111c89a invites: Use query.count() not len(query). 2024-04-12 17:37:11 -07:00
Lauryn Menard 32d5b4fe3e api-docs: Document the "channel" and "channels" search/narrow operators.
Documents the addition of "channel" and "channels" as operators for
search/narrow filters, noting that they are aliases of the "stream"
and "streams" operators.

This change does not yet deprecate the "stream" and "streams"
operators.

Part of stream to channel rename project.
2024-04-12 13:44:25 -07:00
Lauryn Menard 608b30577a narrow: Add backend support for "channels" operator.
Adds backend support for "channels" operator.

This will deprecate/replace the "streams" operator eventually, but
we will keep support of the operator for backwards compatibility
for a while.

Part of renaming stream to channel project.
2024-04-12 13:44:25 -07:00
Lauryn Menard 0e972e26da narrow: Add backend support for "channel" operator.
Adds backend support for "channel" operator.

This will deprecate/replace the "stream" operator eventually, but
we will keep support of the operator for backwards compatibility
for a while.

Part of renaming stream to channel project.
2024-04-12 13:44:25 -07:00
Lauryn Menard 692d5e9706 narrow: Update check to prevent conflicting dm and channel narrow.
Prep commit for deprecating the "stream" search operator and
replacing it with the "channel" operator, and for depreacting the
"streams" operator and replacing it with the "channels" operator.
2024-04-12 13:44:25 -07:00
Lauryn Menard 94ac3f2b58 test-message-fetch: Replace stream with channel for variables/names.
Prep commit for deprecating the "stream" search operator and
replacing it with the "channel" operator, and for depreacting the
"streams" operator and replacing it with the "channels" operator.

Replaces test names, variables and strings to use channel instead
of stream, so that subsequent commits that change the code
are clearer.
2024-04-12 13:44:25 -07:00
Kenneth Rodrigues c3408b56f0 search: Add server support for has:reaction search operator.
Web app support will be a follow-up commit.
2024-04-12 11:57:57 -07:00
Alex Vandiver 436dab0e01 messages: Remove use of @overload in access_message.
f92d43c690 added uses of `@overload` to probide multiple type
signatures for `access_message`, based on the `get_user_message`
parameter.  Unfortunately, mypy does not check the function body
against overload signatures, so it allows type errors to go
undetected.

Replace the overloads with two functions, for one of which also
returns the usermessage.  The third form, of only returning if the
usermessage exists, is not in a high-enough performance endpoint that
a third form is worth maintaining; it uses the usermessage form.
2024-04-12 11:41:27 -07:00
Alex Vandiver 936c2b54cb push_notifications: Use ignore_conflicts, over catching IntegrityError.
The IntegrityError shows up in the database logs, which looks
unnecessarily concerning.  Use `ON CONFLICT IGNORE` to mark this as
expected, especially since the return value is never used.
2024-04-12 11:36:40 -07:00
Alex Vandiver 9a682fb20a message_fetch: Take a REPEATABLE READ READ ONLY when fetching messages. 2024-04-12 09:35:10 -07:00
Pietro Albini 0f521fba41 video_calls: Add default password to zoom meetings.
The password will be included in the meeting link, so users won't have
to type it in manually. Example of the newly generated URLs:

https://DOMAIN.zoom.us/j/MEETING_NUMBER?pwd=ENCODED_PASSWORD

This improves the experience in Zoom organizations requiring
authentication for all meetings, as otherwise the waiting room is
forcibly enabled.
2024-04-12 08:49:08 -07:00
Tim Abbott 1cd11a4c61 zulip_updates: Adjust formatting for initial updates. 2024-04-11 16:13:42 -07:00
Prakhar Pratyush bc0325922e commands: Add 'skip-delay' option to send_zulip_update_announcements.
This commit adds a 'skip-delay' option to the
'send_zulip_update_announcements' management command.

It will be useful for self-hosted servers after 9.0 upgrade to
avoid the 24 hour delay to receive update messages after group
DM is sent to the admins.

One can run the management command with the --skip-delay flag
to immediately send the update messages.
2024-04-11 16:13:42 -07:00
Prakhar Pratyush 507d9c3281 test_zulip_update_announcement: Remove global announcements list used.
Earlier, we were using 'test_zulip_update_announcements' global
list in 'ZulipUpdateAnnouncementsTest' tests and extending
that list within tests.

The previous behaviour can lead to flaky tests depending on the
order in which individual tests are run.

This commit replaces the global list with an instance variable
'zulip_update_announcements' declared in setUp(). It resets the
variable to an initial value before executing each tests making
it independent of the order of tests execution.
2024-04-11 16:13:42 -07:00
Mateusz Mandera c9ca4e68e5 scim: Add config option to disable initial streams for guests.
When an organization (without open ability for anyone to join) invites a
guest user, the invitation prompts allows them to choose whether the
guest should be added to default streams or not. This is useful, because
since we don't have per-role default streams configs, they may want
default streams to be for full Members.

SCIM provisioning doesn't have this control, since a newly provisioned
user gets created via a direct do_create_user call, thus adding them to
the organization's default streams, with no workaround possible aside of
just getting rid of default streams in the organization.

To make provisioning guests in such an organization usable, we add a
simple config option to create them with no streams. It's configured by
adding
```
"create_guests_without_streams": True
```

to the config dict in settings.SCIM_CONFIG.
2024-04-11 12:28:26 -07:00
rht c77ed52fa9
slack import: Refer the archive file as a file or dir. 2024-04-11 12:05:56 -07:00
sujal shah dd91157993 integration: Split 'opened_or_update_pull_request' event type.
Modified the 'get_zulip_event_name' function to
differentiate between 'opened' and 'updated' actions
for pull requests within the 'pull_request' event type.
Created separate event types for 'opened_pull_request' and
'updated_pull_request'. Updated the 'EVENT_FUNCTION_MAPPER'
dictionary to include handlers for these new event types.
Adjusted the'get_opened_or_update_pull_request_body' function to handle
both 'opened' and 'updated' actions appropriately.

Fixes #29594.
2024-04-11 11:23:08 -07:00
Alex Vandiver d834f1fc01 unarchive_stream: Prevent unarchiving a stream in the wrong realm. 2024-04-11 09:41:41 -07:00
Lauryn Menard 01b59c5aa2 message-type: Add support for "channel" as value for type parameter.
For endpoints with a type parameter to indicate whether a message is
a direct or stream message, adds support for passing "channel" as a
value for stream messages.

Part of stream to channel rename project.
2024-04-11 09:40:25 -07:00
Sahil Singh 6c2535fe46 bots: Avoid multiple active bots with the same name.
Creating a bot with a name that is already in use
will raise an error. However, by deactivating
the existing bot, creating a new bot with the
same name, and then reactivating the original bot,
it is possible to have multiple bots with the same name.

To fix this, we check if the bot name is already
in use in the active bots list. If it is,
an error will be raised, prompting either the
name of the existing bot to be changed or
the bot to be deactivated.

Co-authored-by: Sujal Shah <sujalshah28092004@gmail.com>
2024-04-10 13:14:25 -07:00
Vector73 2734a9483b api_documentation: Document "/invites/multiuse" endpoint.
Adds "/invites/multiuse" endpoint to the API documentation.

Creates a shared schema for the invite_as and invite_expires_in_minutes
parameters that are the same for the "POST /invites" endpoint.

Also, updates the response documented for the "GET /invites" endpoint
to match the information in the "POST /invites" and "/invites/multiuse"
documentation.
2024-04-10 11:29:14 -07:00
Anders Kaseorg d8ebb2db95 auth: Avoid deprecated django.contrib.auth.views.logout_then_login.
It’s removed in Django 5.0.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-04-04 16:27:58 -07:00
JohnLu2004 11c9e5bb3d api-docs: Document optimization of is:unread filter.
Documents that the is:unread narrow/filter can be an important
optimization for clients to consider using in certain cases.

Fixes #28328.
2024-04-04 12:54:25 -07:00
Lauryn Menard e700e818e5 mentions: Add "channel" as a wildcard mention.
Adds "channel" to the `stream_wildcards` frozenset for stream
wildcard notifications on the backend/server.

Updates frontend/web-app to handle "channel" as the other stream
wildcards are handled in the typeahead and composebox modules.

Updates the API version and documentation for the addition of
"channel" as a wildcard mention. But does not change any of the
functionailty of (or deprecate) the "stream" wildcard at this
point.

Part of project to rename "stream" to "channel".
2024-04-04 11:16:07 -07:00
Lauryn Menard 92f236ffea mentions: Update backend tests to not use only "stream" wildcard.
Use "all" or the set of stream wildcard mentions for backend tests.

Part of project to rename "stream" to "channel".
2024-04-04 11:16:07 -07:00
Prakhar Pratyush e3f8c62e34 registration: Send a group DM to admins when no spare licenses left.
Earlier, when adding a new user failed due to no spare licenses
available, a message was sent to the "New user announcements"
stream.

We plan to disable the stream by default as a part of improving
onboarding experience.

Now, we send a group DM to admins when adding a new user fails
due to no spare licenses available. It makes it independent of
the "New user announcements" setting. These warning messages
are important and shouldn't be missed.
2024-04-03 12:28:05 -07:00
Prakhar Pratyush 2abc25ac4c create_user: Send a group DM to admins for low licenses warning.
Earlier, low licenses warning message was sent to the
"New user announcements" stream.

We plan to disable the stream by default as a part of improving
onboarding experience.

Now, we send a group DM to admins for low licenses warning
to make it independent of the setting. These warning messages
are important and shouldn't be missed.
2024-04-03 12:28:05 -07:00
Prakhar Pratyush db214c8377 message: Add recipient_users parameter to internal_send_huddle_message.
This is a prep commit to add a 'recipient_users' parameter to
the 'internal_send_huddle_message' function.

'emails' is no longer a required parameter. We can use either
of the 'emails' or 'recipient_users' parameter. 'emails' is
eventually used to fetch 'recipient_users', so if the
'recipient_users' is already available we should use that to
skip database query.
2024-04-03 12:28:05 -07:00
Vector73 6af9a6635c api_docs: Fix "/invites" enpoint documentation. 2024-04-03 10:45:05 -07:00
Lauryn Menard db8e130110 emails: Update text version of find team email to match HTML version.
The HTML version of the email was updated in commit 5410df2a7b.
2024-04-03 10:44:31 -07:00
Alex Vandiver ebe9437ccc migrations: Match dependencies from 8.x branch.
The `zerver/0501_delete_dangling_usermessages` was backported to the
`8.x` branch (and the 8.3 release) in 3db1733310.  However, because
`main` contained migrations which `8.x` did not, it was backported
with a different `dependencies`:

```
    dependencies = [
        ("zerver", "0496_alter_scheduledmessage_read_by_sender"),
    ]
```

...as opposed to in `main`:

```
    dependencies = [
        ("zerver", "0500_realm_zulip_update_announcements_stream"),
    ]
```

This causes upgrades from 8.3 to `main` to fail:
```
django.db.migrations.exceptions.InconsistentMigrationHistory:
Migration zerver.0501_delete_dangling_usermessages is applied before
its dependency zerver.0500_realm_zulip_update_announcements_stream on
database 'default'.
```

Adjust the dependencies in `main` to match those in `8.x` where many
deploys will first have encountered the migration.
2024-04-03 09:16:33 -07:00
Prakhar Pratyush bcebcb334c zulip_news: Wait for one day after sending group DM if stream is set.
For organizations with "Zulip update announcements" stream set
to a default value, we wait for one day after sending group
DM to admins to allow them to change the stream from it's
default value if they wish to.
2024-04-02 15:12:50 -07:00
Prakhar Pratyush d4fe4ca934 zulip_news: Refactor 'is_group_direct_...' to allow flexible timeframe.
This prep commit refactors the function
'is_group_direct_message_sent_to_admins_atleast_one_week_ago' to
'is_group_direct_message_sent_to_admins_within_days' allowing us
to use a flexible timeframe instead of hardcoded 1 week.

We will reuse this function as a part of determinig whether
group DM to admins was sent within 1 day.
2024-04-02 15:12:46 -07:00
Alya Abbott b8e5954c0b zulip_updates: Announce three features.
- Pasting formatted content.
- Quote-and-reply selection.
- Hide user list.
2024-04-02 15:07:22 -07:00
Tim Abbott 563485a37e zulip_updates: Fix line removal logic for bulleted lists. 2024-04-02 15:07:22 -07:00
roanster007 c7a08f3b77 settings: Add permission to enforce unique names in realm.
Previously, users were allowed to signup or change their names to
those which already existed in the realm.

This commit adds an Organization Permission, that shall enforce
users to use unique names while signing up or changing their
names. If a same or normalized full name is found in realm,
then a validation error is thrown.

Fixes #7830.
2024-04-02 14:55:59 -07:00
swayam0322 5410df2a7b find-account: Send email when no Zulip accounts found.
Previously, email addresses that weren't connected to a Zulip account
were ignored but now they receive an email stating their email isn't
connected to a Zulip account.

Also, removes the "Thanks for using Zulip!" line at the end of the
find accounts email that's sent when a Zulip account is found.
Updates the i18n test that used this string with another in the
German translation from this a successful account found email.

Fixes part of #3128

Co-authored-by: Lauryn Menard <lauryn@zulip.com>
2024-04-02 10:39:47 -07:00
Lauryn Menard 968059c7ef emails: Update help link in find team email to use external host url.
Updates the help link in the find team emails to use the external
host information.

Removes the link for the external host since the realm links are
what the user should click on to login.

Also, passes corporate_enabled to the find team email to adjust
the text for Zulip Cloud emails.
2024-04-02 10:39:47 -07:00
Lauryn Menard 8bb8240c27 integrations: Restructure documentation pages.
Restructures the integration documentation pages to use a style
that's more similar to the help center documentation, with an
instruction block for setting up the integration, and sections
for additional configuration information and related documentation
links.

Updates the doc pages for the airbrake, azuredevops and gitlab
integrations as examples of the updated style.

Also updates the URL specification section of the incoming webhook
overview in the API documentation so that the documented URL
parameters can be linked to directly in the integration doc pages.

Co-authored-by: Alya Abbott <alya@zulip.com>
2024-04-02 09:30:48 -07:00
Anders Kaseorg 7e2ef11f61 ruff: Fix UP041 Replace aliased errors with `TimeoutError`.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-04-01 18:32:52 -07:00
Anders Kaseorg 6e871e7731 ruff: Fix UP036 Version block is outdated for minimum Python version.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-04-01 18:32:52 -07:00
Anders Kaseorg a82a3eb4d7 ruff: Fix UP033 Use `@functools.cache`.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-04-01 18:32:52 -07:00
Anders Kaseorg ae47de36c9 python: Add missing TypeAlias annotations.
See commit c2c96eb0cf (#26405).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-04-01 16:19:56 -07:00
Vector73 7a55bb218b api_documentation: Add "/invites" endpoint to the api documentation. 2024-04-01 13:16:31 -07:00
Aman Agrawal d21f5c9b75 registration: Ask user how they found Zulip. 2024-04-01 12:44:12 -07:00
Anders Kaseorg 8a7f61fcad test_decorators: Replace some deleted tests.
Commit 807a4428f6 (#29485) deleted the
corresponding tests in TestAuthenticatedJsonPostViewDecorator.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-03-29 09:39:34 -07:00
Anders Kaseorg 807a4428f6 compose_validate: Remove autosubscribe feature.
This was only used in the undocumented narrow_stream mode, and relied
on a deprecated synchronous XHR request.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-03-28 09:06:48 -07:00
Prakhar Pratyush a7dc7c0734 message: Add function to remove single newline in triple quoted string.
For multiline strings in triple quotes, a '\n' is included
at the end of each line.

Earlier, to skip '\n' we used to add an escape character '\'
at the end of each line.

This commit adds a function to avoid manually adding '\'.
2024-03-28 09:03:59 -07:00
Karl Stolley 286d44bf33 settings: Add new web information density settings.
Note that these settings are not operative at present, and are only
visible in the settings UI in the development environment.
2024-03-27 12:58:32 -07:00
Prakhar Pratyush b9d9676b43 zulip_news: Update the group DM sent initially to admins.
As a part of the zulip news feature, we send an initial
group DM to admins suggesting them to update or set
the 'zulip_update_announcements_stream'.

This commit updates those messages to make it better.
2024-03-27 11:42:00 -07:00
Lauryn Menard c53a96a7b0 i18n-test: Update translated strings in test_email_translation.
Updates the check email translation test for updated email text in
confirm_new_email.html and onboarding_zulip_topics.html for current
translated strings in German.
2024-03-27 11:16:59 -07:00
Alex Vandiver 7de527f3bc soft_deactivation: Use much faster bulk_insert_all_ums.
As noted in the docstring for `bulk_insert_ums`, this is at least one
order of magnitude faster than using `bulk_create`.  This also
includes a `ON CONFLICT DO NOTHING` which allows multiple
soft-reactivations to run at once without failing.  We also adjust the
update of `last_active_message_id` to be safe against races.
2024-03-26 10:52:37 -07:00
Alex Vandiver 7988aad159 user_message: Use INSERT ... ON CONFLICT for historical UM creation.
Rather than use a bulk insert via Django, use the faster
`bulk_insert_all_ums` that we already have.  This also adds a `ON
CONFLICT` clause, to make the insert resilient to race conditions.

There are currently two callsites, with different desired `ON
CONFLICT` behaviours:
 - For `notify_reaction_update`, if the `UserMessage` had already been
   created, we would have done nothing to change it.
 - For `do_update_message_flags`, we would have ensured a specific bit
   was (un)set.

Extend `create_historical_user_messages` and `bulk_insert_all_ums` to
support `ON CONFLICT (...) UPDATE SET flags = ...`.
2024-03-26 10:52:37 -07:00
Alex Vandiver 52e3c8e1b2 user_message: Move create_historical_user_messages to zerver.lib.user_message.
It is not even used in zerver.actions.create_user.
2024-03-26 10:52:37 -07:00
Alex Vandiver a7ec7e924e import: Skip "active" check, and analytics, for crossrealm bots.
The bots do not exist in the user table to look up their active
status, and attempting to import them into the analytics table will
result in duplicate rows.
2024-03-25 10:21:48 -07:00
Aditya Bajaj a20b454ebb
notifications: Shorten string for redacted content.
Replace the long string for organisations that have notification
body/content disabled (settings.PUSH_NOTIFICATION_REDACT_CONTENT
set to true) with "New message".

This allows more of the limited space on the mobile device screen to
be used for additional messages rather than this verbose content.

Fixes #29152
2024-03-22 09:37:51 -07:00
Mateusz Mandera 89a22fb076 register_server: Check write access to secrets file when rotate_key.
Using --rotate-key without write access to the secrets file is currently
quite painful, since you end up rotating your registration's secret with
no local record of it; so effectively you lose your registration and
need help from support. We should just prevent this failure mode.
2024-03-22 09:33:00 -07:00
Alex Vandiver 7d0c12aa3a semgrep: Enforce no bare select_related() calls.
Previously, #26419 addressed the majority of these calls, but did not
prevent more from creeping in.  Remove the one remaining
callsite (after the cleanup from the previous commits), and ban any
future use of the pattern.
2024-03-22 09:30:17 -07:00
Alex Vandiver fd5a091b30 messages: Only check the UserMessage row if necessary.
For the common case of not needing to reference the UserMessage row
later, and for being a stream without private history, the UserMessage
row is irrelevant.  Convert `has_user_message` to a thunk, and defer
loading it unless necessary.
2024-03-22 09:30:17 -07:00
Alex Vandiver f92d43c690 messages: Use overloads to only return a user_message if needed. 2024-03-22 09:30:17 -07:00
Alex Vandiver 6ace34c374 messages: Stop joining through every table.
Calling `.select_related()` with no arguments joins through every
possible table, recursively.  In this case, this currently produces a
query which joins through forty-three tables.

This is rather inefficient, particularly for what is a very common
call which should be very fast.

No callsite depends on having prefetched any joined table on the
object; drop all of the joins.
2024-03-22 09:30:17 -07:00
John Lu a5cf0ec526
refactor: Replace HUDDLE with DIRECT_MESSAGE_GROUP.
Replaced HUDDLE attribute with DIRECT_MESSAGE_GROUP using VS Code search,
part of a general renaming of the object class.

Fixes part of #28640.

Co-authored-by: JohnLu2004 <JohnLu10212004@gmail.com>
2024-03-21 16:39:33 -07:00
Anders Kaseorg 224b484195 test_classes: Skip OpenAPI validation on exceptions.
This gives more helpful error tracebacks from failing tests.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-03-21 16:36:48 -07:00
Alex Vandiver 9d8d2d138b missedmessage_emails: Add Sentry spans to worker thread. 2024-03-21 12:46:13 -07:00
Alex Vandiver 9451d08bb9 worker: Split out worker sampling rate, and add Sentry transactions. 2024-03-21 12:46:13 -07:00
Alex Vandiver 3cbce0c5c7 missedmessage_emails: Clear caches and db query tracking per-loop.
Otherwise, these accumulate and leak memory.
2024-03-21 12:46:13 -07:00
Alex Vandiver 6ad777c86f dev_urls: Make requests to /docs redirect to the index.
This makes it easier to browse to the docs in development mode, and
more closely matches ReadTheDocs production.
2024-03-21 11:27:14 -07:00
Vector73 f758ca596b custom_profile_fields: Add "required" parameter to the profile fields.
Fixes #28512.
2024-03-21 10:48:54 -07:00
Anders Kaseorg a1a341f0ae ruff: Fix UP032 Use f-string instead of `format` call.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-03-21 08:59:17 -07:00
Lauryn Menard 373671080d remote-support: Include billing entity name in internal billing notice.
Adds a line to the top of the internal_billing_notice email with
the billing entity's display name.

Makes sure all internal_billng_notice email subjects also include
the billing entity's display name.

Makes small updates to the notice text for some cases.
2024-03-20 11:58:15 -07:00
Prakhar Pratyush 118a7e8d9d zulip_updates: Send zulip updates based on zulip_update_*_level.
This commit adds a management command that will run regularly
as a cron job to send zulip updates to realms based on their
current and latest zulip_update_announcements_level.

For realms with:
* level = None: Send a group DM to admins notifying them about
this new feature & suggestion to set the stream accordingly.

* level = 0:
  * If stream is still not configured, wait for a week
    before setting their level to latest level. They will
    miss updates until their configure the stream.
  * If stream is configured, send updates.

* level > 0: Send one message/update per level & increase
  the level by 1 till the latest level.

Fixes #28604.
2024-03-20 11:48:06 -07:00
Prakhar Pratyush 336d46001c message_send: Add 'internal_prep_huddle_message' function.
This is a prep commit to extract out the logic to
create message from 'internal_send_huddle_message'
into a separate function 'internal_prep_huddle_message'.

We will use this new function to get the huddle message
without sending it immediately.
2024-03-20 11:48:06 -07:00
Prakhar Pratyush 23fc04577b message_send: Don't create savepoint in do_send_messages.
In general, we never want to use savepoints.

This prep commit adds savepoint=False in do_send_messages
as we don't want to just rollback to this savepoint and
proceed if we encounter any error while sending zulip updates
via cron.
2024-03-20 11:48:06 -07:00
Prakhar Pratyush 6484532830 help: Migrate configure-notification-bot help page to a new URL.
This commit migrates the 'help/configure-notification-bot' help
page to a new URL i.e. 'help/configure-automated-notices'.
2024-03-20 11:48:06 -07:00
Alex Vandiver 6bf9b7a7f7 migrations: Merge duplicated migration heads. 2024-03-19 19:04:32 -04:00
Alex Vandiver 0f0631813f message_edit: Do set differences in QuerySets. 2024-03-19 18:22:24 -04:00
Alex Vandiver c71b7afe9f message_edit: Handle user, not subscription, collections.
Nothing about the subscription is relevant -- we instead use
collections of User objects for readability.
2024-03-19 18:22:24 -04:00
Alex Vandiver f15e006873 message_edit: Perform cross join in SQL, not Python.
The database can do this faster than we can, and it reduces the amount
of information transmitted over the wire.
2024-03-19 18:22:24 -04:00
Alex Vandiver a8795c5d0b message_edit: Pull user_ids_gaining_usermessages higher, rename. 2024-03-19 18:22:24 -04:00
Alex Vandiver 7ce6095003 message_edit: Handle previous subscribers and private-history streams. 2024-03-19 18:22:24 -04:00
Alex Vandiver cf8b9adad4 CVE-2024-27286: Delete dangling UserMessage rows.
This cleans up dangling UserMessage rows for moved messages which were
affected by bugs in one of the previous two commits.
2024-03-19 18:22:24 -04:00
Alex Vandiver 7b1feac06a CVE-2024-27286: Remove UserMessage rows for non-active Subscriptions.
A user who was no longer subscribed to a private stream kept their
UserMessage row for a message sent while they were in it; this is
expected.  However, they _also_ kept that row even if the message was
moved to a different private stream that they were also not subscribed
to.  This violates the invariant that users without subscriptions
never have UserMessage rows.
2024-03-19 18:22:24 -04:00
Alex Vandiver e964536139 CVE-2024-27286: Run usermessage modifications even for change_one.
This `if new_stream is not None` block was improperly indented,
causing it to only run if the propagation mode was not `change_one`.
Since the block controlled creation and deletion of UserMessage rows,
this led to messages being improperly still visible to members of the
old stream if they were being moved from public to private streams.
Clients also failed to receive `delete_message` events, so the
messages remained visible in their feeds until they reloaded the
application.
2024-03-19 18:22:24 -04:00
Alex Vandiver 64df8c7075 send_custom_email: is_realm_admin is a property, not a real column. 2024-03-19 13:19:45 -07:00
Prakhar Pratyush de53e372dd onboarding: Don't show introduce zulip view modals to existing users.
We don't want to show one-time modals introducing 'Inbox' and
'Recent conversations' views to existing users.

When a user views a modal, we mark it as read by storing a row
in the 'OnboardingStep' model. Once marked as read, the user will
no longer see the modal.

This commit adds a migration to create
two rows per user (mark them as read) in OnboardingStep model.
2024-03-19 09:32:59 -07:00
Prakhar Pratyush ad3603c0aa onboarding: Show one-time modal to introduce recent conversations view.
To improve onboarding experience, this commit adds a
one-time modal which introduces the recent conversations view.

Users see this one-time modal on visiting the recent
conversations view.

Fixes #29073.
2024-03-19 09:32:59 -07:00
Prakhar Pratyush 9a7634d527 onboarding: Show one-time modal to introduce inbox view.
To improve onboarding experience, this commit adds
a one-time modal which introduces the inbox view.

Users see this one-time modal on visiting the inbox view.

Fixes part of #29073.
2024-03-19 09:32:59 -07:00
Alex Vandiver c13e3dee24 katex: Replace subprocess call with minimal external service.
Replace a separate call to subprocess, starting `node` from scratch,
with an optional standalone node Express service which performs the
rendering.  In benchmarking, this reduces the overhead of a KaTeX call
from 120ms to 2.8ms.  This is notable because enough calls to KaTeX in
a single message would previously time out the whole message
rendering.

The service is optional because he majority of deployments do not use
enough LaTeX to merit the additional memory usage (60Mb).

Fixes: #17425.
2024-03-15 15:34:12 -07:00
Alex Vandiver c7e92ee2f3 outgoing_http: Support fractional-second timeouts. 2024-03-15 15:34:12 -07:00
Lauryn Menard 2b9220ff58 api-docs: Update feature level 224 changelog and **Changes** notes.
Links to the available message flag table in the feature level 224
changelog entry, as there are relevant **Changes** notes for this
feature level in that part of the API documentation.

Updates the order and formatting of these new and deprecated flags
in the available flags table. Also, adds a link to the topic
wildcard mentions section of the help center documentation.

Makes small clean ups to the changes notes for this feature level,
as well as the changelog entry itself.

The original commit for these feature level 224 API changes was
c597de6a1d.
2024-03-13 16:50:48 -07:00
Mahhheshh 6c0818ff51 email_page: Migrate has_request_variables to typed_endpoint.
Refactor `email_page` view to use `typed_endpoint` decorator instead of
`has_request_variables`.
2024-03-13 16:47:34 -07:00
Mahhheshh 22bd8048b1 parse_client: Migrate has_request_variables to typed_endpoint.
Refactor `parse_client` view to use `typed_endpoint decorator`
instead of `has_request_variables`. This change improves code consistency
and enhances codebase comprehension.
2024-03-12 13:27:21 -07:00
Aman Agrawal 5548a28980 message_fetch: Allow spectators to access all messages view. 2024-03-11 14:13:26 -07:00
Aman Agrawal 2b12c8b4b9 narrow: Fix comment. 2024-03-11 14:13:26 -07:00
Alex Vandiver 6e91e326e9 deferred_work: Reduce batch size due to bad statistics.
PostgreSQL's estimate of the number of usermessage rows for a single
message can be wildly off, due to poor statistics generation.  This
causes this query, with 100-message batch sizes, to incorrectly
estimate millions of matched rows, causing it to perform a full-table
index scan, rather than piecemeal using the `message_id` index.

Reduce the batch size to 50, which is enough to tip in favor of a
rational query plan.
2024-03-11 09:24:59 -07:00
Mahhheshh 073b70a516 csp_violations: Migrate has_request_variables to typed_endpoint.
Refactor `report_csp_violations` view to use `typed_endpoint` decorator
instead of `has_request_variables`. This change improves code
consistency and enhances codebase comprehension.
2024-03-08 11:26:00 -08:00
Alex Vandiver 597704fa5f tests: Clarify test_inaccessible_msg_after_stream_change. 2024-03-06 17:31:16 -08:00
Alex Vandiver 40eb5bf42d tests: Split test_message_edit.py. 2024-03-06 17:31:16 -08:00
Alex Vandiver 90abb2281d tests: Split DeleteMessageTest from test_message_edit.py. 2024-03-06 17:31:16 -08:00
Mateusz Mandera b572fa93cd context_managers: Open file in write mode in lockfile_nonblocking.
Otherwise this fails if the file doesn't yet exist.
2024-03-06 10:35:02 -08:00
Mateusz Mandera c1857b2c86 config_error: Support passing arguments specifying the "go back" link.
Depending on the kind of config error being shown, different "go back"
links may be more appropriate.
We probably hard-coded /login/ for it, because these config errors are
most commonly used for authentication backend config error, where it
makes sense to have /login/ as "go back", because the user most likely
indeed got there from the login page.

However, for remote_billing_bouncer_not_configured, it doesn't make
sense, because the user almost surely is already logged in and got there
by clicking "Plan management" inside the gear menu in the logged in app.
2024-03-05 11:53:51 -08:00
Mateusz Mandera e952c3b627 remote_billing: Tweak /self-hosted-billing/ endpoints access model.
It's best for these to just be consistent. Therefore:
1. The .../not-configured/ error page endpoint should be restricted to
   .has_billing_access users only.
2. For consistency, self_hosting_auth_view_common is tweaked to also do
   the .has_billing_access check as the first thing, to avoid revealing
   configuration information via its redirect/error-handling behavior.

The revealed configuration information seems super harmless, but it's
simpler to not have to worry about it and just be consistent.
2024-03-05 11:53:51 -08:00
Mateusz Mandera 6593ccd7fc test_helpers: Remove /self-hosted-billing/ from exempt_patterns.
This was added early on and hasn't been removed since, even though we
have tests for the endpoint for a while now.
2024-03-05 11:53:51 -08:00
Mateusz Mandera e39f400f94 remote_billing: Make "plan management" always available.
Just shows a config error page if the bouncer is not enabled. Uses a new
endpoint for this so that it can work nicely for both browser and
desktop app clients.
It's necessary, because the desktop app expects to get a json response
with either an error or billing_access_url to redirect to. Showing a
nice config error page can't be done via the json error mechanism, so
instead we just serve a redirect to the new error page, which the app
will open in the browser in a new window or tab.
2024-03-05 11:53:51 -08:00
Mateusz Mandera d5fb3f3176 authentication_methods: Bump feature level and add changelog entries. 2024-03-05 11:48:58 -08:00
Mateusz Mandera 1ede8da46c events: Change format of authentication_methods realm_update_dict event.
Makes the event in line with state_data.realm_authentication_methods.
2024-03-05 11:48:58 -08:00
Mateusz Mandera da9e4e6e54 backends: Implementation of restricting certain backends by plan.
Only affects zulipchat, by being based on the BILLING_ENABLED setting.

The restricted backends in this commit are
- AzureAD - restricted to Standard plan
- SAML - restricted to Plus plan, although it was already practically
  restricted due to requiring server-side configuration to be done by us

This restriction is placed upon **enabling** a backend - so
organizations that already have a backend enabled, will continue to be
able to use it. This allows us to make exceptions and enable a backend
for an org manually via the shell, and to grandfather organizations into
keeping the backend they have been relying on.
2024-03-05 11:48:58 -08:00
David Rosa 3d63ea20b6 help: Add "View stream subscribers" page.
Documents how to view stream subscribers via settings and
the right sidebar.

Fixes #28887.
2024-03-05 10:22:18 -08:00
Mateusz Mandera 634015411a update_analytics_count: Use a correct lock mechanism.
Adds a re-usable lockfile_nonblocking helper to context_managers.

Relying on naive `os.mkdir` is not enough especially now that the
successful operation of this command is necessary for push notifications
to work for many servers.

We can't use `lockfile` context manager from
`zerver.lib.context_managers`, because we want the custom behavior of
failing if the lock can't be acquired, instead of waiting.
That's because if an instance of this gets stuck, we don't want to start
queueing up more processes waiting forever whenever the cronjob runs
again and fail->exit is preferrable instead.
2024-03-05 10:21:14 -08:00
Alex Vandiver 352dbf9387 caches: Only take the realm_id, not the Realm, as a cache key function.
This saves a hit to the database to fetch the Realm of a UserProfile
that we are trying to flush.
2024-03-04 16:35:57 -08:00
Anders Kaseorg 59b0548433 timezone: Only look up canonical time zones from the system.
Legacy time zone aliases were removed from the Debian tzdata package
in tzdata 2023c-8.

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1040997

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-03-01 17:38:08 -08:00
Anders Kaseorg 7b1bb984b3 ruff: Fix RUF022 `__all__` is not sorted.
This is a preview rule, not yet enabled by default.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-03-01 09:30:04 -08:00
Anders Kaseorg dbb20d636d ruff: Fix E721 Use `is` and `is not` for type comparisons.
This is a preview rule, not yet enabled by default.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-03-01 09:30:04 -08:00
Anders Kaseorg 0e71f52e86 ruff: Fix E203 Whitespace before ','
This is a preview rule, not yet enabled by default.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-03-01 09:30:04 -08:00
Anders Kaseorg 869d9d9a79 ruff: Fix RUF025 Unnecessary dict comprehension for iterable.
This is a preview rule, not yet enabled by default.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-03-01 09:30:04 -08:00
Anders Kaseorg 3b114c516c ruff: Fix PLR2044 Line with empty comment.
This is a preview rule, not yet enabled by default.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-03-01 09:30:04 -08:00
Anders Kaseorg 87992b8b29 ruff: Fix PERF403 Use a dictionary comprehension instead of a for-loop.
This is a preview rule, not yet enabled by default.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-03-01 09:30:04 -08:00
Anders Kaseorg 82a9fd927b ruff: Fix E226 Missing whitespace around arithmetic operator.
This is a preview rule, not yet enabled by default.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-03-01 09:30:04 -08:00
Anders Kaseorg d748ec8d52 ruff: Fix PLW0108 Lambda may be unnecessary.
This is a preview rule, not yet enabled by default.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-03-01 09:30:04 -08:00
Anders Kaseorg 553f268b04 ruff: Fix RUF027 Possible f-string without an `f` prefix.
This is a preview rule, not yet enabled by default.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-03-01 09:30:04 -08:00
Anders Kaseorg 865febb307 ruff: Fix RUF021 Parenthesize when chaining `and` and `or` together.
This is a preview rule, not yet enabled by default.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-03-01 09:30:04 -08:00
Anders Kaseorg 2655fe7182 ruff: Fix TCH005 Found empty type-checking block.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-03-01 09:27:48 -08:00
Anders Kaseorg 570f3dd447 python: Reformat with Ruff formatter.
https://docs.astral.sh/ruff/formatter/

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-02-29 17:07:16 -08:00
Prakhar Pratyush d66b7ad853 zilencer: Notify when paid plan attached to now-deleted remote realm.
When a server doesn't submit a remote realm info which was
previously submitted, we mark it as locally deleted.

If such a realm has paid plan attached to it, we should investigate.

This commit adds logic to send an email to sales@zulip.com for
investigation.
2024-02-29 12:50:23 -08:00
Sahil Batra 0669e4f631 realm: Allow everyone to delete their own messages by default.
This commit updates default for delete_own_message_policy
setting to "Everyone" as it is helpful to allow everyone
to delete their own messages in a new organization where
users might be using Zulip for the first time.
2024-02-28 13:13:46 -08:00
Sahil Batra 4fbc137c2b realm: Allow members to move msgs between streams by default.
This commit updates default for move_messages_between_streams_policy
setting to "Members and above" as it is helpful to allow members
to move messages between streams in new organizations where users
might be using Zulip for first time.
2024-02-28 13:13:46 -08:00
Alex Vandiver a808c730bc deferred_work: Use an id high-water-mark instead of offsets.
This solves the problem listed in the now-removed comment.
2024-02-27 17:02:34 -08:00
Alya Abbott ad8bdfbe20 portico: Consistently describe Zulip as organized team chat. 2024-02-27 17:01:05 -08:00
Alex Vandiver 58f0669997 deferred_work: Re-queue remaining "mark all as read" work after 30s. 2024-02-27 10:21:04 -08:00
Alex Vandiver 75e9903be5 deferred_work: Move all queries into the transaction.
The presence of `len(messages)` outside the transaction caused the
full resultset to be fetched outside of the transaction.  This should
ideally be inside the transaction, and also only need be the count.

However, also note that the process of counting matching rows, and
then executing a second query which embeds the same query, is
susceptible to phantom reads, where a query with the same conditions
returns different resultsets, under PostgreSQL's default transaction
isolation of "read committed."  While this is possible to resolve by
pulling the returned IDs into a Python list, it would not address the
issue that concurrent updates which change the resultset would make
the overall algorithm still incorrect.

Add a comment clarifying the conditions under which the algorithm is
correct.  A more correct algorithm would walk the UserMessage rows
which are unread and in the stream, but this requires a
whole-UserMessage index which would be quite large for such an
infrequent use case.
2024-02-27 10:21:04 -08:00
Alex Vandiver 4712031528 email_validation: Merge two implementations of a disposable-email check.
Only one of them properly checked for improperly-formed email
addresses, thanks to #22723.
2024-02-27 10:19:23 -08:00
Alex Vandiver 4b512b3409 email_validation: Rename to validate_is_not_disposable.
This clarifies what we are asserting -- namely, that the email is
_not_ disposable.
2024-02-27 10:19:23 -08:00
Alex Vandiver c1aadbe52e events: Do nothing in apply_event for restart events.
These signal that the Tornado process restarted, which in itself is
not notable for apply_events.
2024-02-27 10:18:11 -08:00
Alex Vandiver be59ef9c02 runtornado: Swap deferred reload events to the default.
This makes no immediate reloads the default for runtornado, matching
the production configuration, and changes the development incantation
to be the one to specify the departure from the norm, with
--immediate-reloads.
2024-02-26 22:29:19 -08:00
Lauryn Menard cf82d3316b push-bouncer: Exclude LoggingCountStats with partial data.
LoggingCountStats with a daily duration and that are directly stored
on the RealmCount table (not via aggregation in process_count_stat),
can be in a state, after the hourly cron job to update analytics
counts, where the logged value will be live-updated later, because
the end time for the stat is still in the future.

As these logging counts are designed to be used on the self-hosted
installation for either debugging or rate limiting, sending these
partial/incomplete counts to the bouncer has low value.
2024-02-26 17:53:12 -08:00
Lauryn Menard 3a03e14938 integrations: Update Slack webhook to use check_send_webhook_message.
Due to the channel_map_to_topics URL parameter in the Slack webhook,
it was not migrated to use the check_send_webhook_message.

By using check_send_webhook_message, any topic parameter in the
webhook URL will be prioritized over mapping Slack channels to
topics, e.g. when channel_map_to_topics is true. This is because
the default behaviour for incoming webhooks is to send a default
topic as a parameter to check_send_webhook_message in case there
is no topic specified in the URL.

In contrast, we can override the stream passed in the URL when
channel_map_to_topics is false by passing the Slack channel name
to check_send_webhook_message. The default behaviour for incoming
webhooks is to send a direct message if there is no specified
stream in the URL, so a default stream is not generally passed
to check_send_webhook_message.

Fixes #27601.
2024-02-25 16:47:34 -08:00
Alex Vandiver 8e7ab57bf2 export_search: Support exporting DMs or huddles. 2024-02-25 09:34:17 -08:00
Prakhar Pratyush fe1a20ebb3 settings: Add realm-level setting 'zulip_update_announcements_stream'.
This commit adds a realm-level setting named
'zulip_update_announcements_stream' that configures the
stream to which zulip updates should be posted.

Fixes part of #28604.
2024-02-25 09:33:00 -08:00
Mateusz Mandera d7cf4336e4 zilencer: Flip realm_locally_deleted when realm re-appears during sync. 2024-02-24 08:52:11 -08:00
David Rosa 5215e3963c help: Clean up beta app installation instructions.
Converts the beta app installation instructions into our standard
tabbed instructions format.

Fixes #28885.
2024-02-24 08:12:34 -08:00
David Rosa 948b427de7 api-docs: Add instructions to download `zuliprc` files.
- Adds instructions for downloading a zuliprc file for a bot or for
  yourself.
- Updates the button label to "Download zuliprc", since that's the
  filename it downloads.

Fixes #28881.
2024-02-23 09:43:57 -08:00
Tim Abbott d0c276d863 corporate: Fix billing_session variable reuse confusion.
The previous logic incorrectly used the server-level number of users
even when a (presumably smaller) realm-level count was available.

Fixes a bug introduced in 2e1ed4431a.
2024-02-21 17:51:30 -08:00
Tim Abbott 6ee123be21 migrations: Renumber misnumbered migrations. 2024-02-21 09:19:34 -08:00
Prakhar Pratyush ee612dafac settings: Rename signup_notifications_stream realm setting.
This commit renames the realm-level setting
'signup_notifications_stream' to 'signup_announcements_stream'.

The new name reflects better what the setting does.
2024-02-21 09:04:23 -08:00
Prakhar Pratyush ab453fbe20 settings: Rename notifications_stream to new_stream_announce..._stream.
This commit renames the realm-level setting 'notifications_stream'
to 'new_stream_announcements_stream'.

The new name reflects better what the setting does.
2024-02-21 09:04:23 -08:00
Prakhar Pratyush 0c83bca81e events: Use 'get_notifications_stream' realm method.
This commit updates the code in 'fetch_initial_state_data'
function to use the 'get_notifications_stream' method
instead of re-writing the logic.
2024-02-21 09:04:23 -08:00
Alex Vandiver 7e9d8cd192 tornado: Fix GC of old queues. 2024-02-20 22:07:25 -08:00
Alex Vandiver b747ea285f topic: Fix history order for topic moves.
5c96f94206 mistakenly appended, rather than prepended, the edit to
the history.  This caused AssertionErrors when attempting to view the
history of moved messages, which check that the `last_edit_time`
matches the timestamp of the first edit in the list.

Fix the ordering, and update the `edit_history` for messages that were
affected.  We limit to only messages edited since the commit was
merged, since that helps bound the affected messages somewhat.
2024-02-20 21:30:32 -08:00
Mateusz Mandera be03dabf76 zilencer: Implement do_reactivate_remote_server utility function.
The inverse of do_deactivate_remote_server. It's just flipping the
.deactivated flag, but we also should have an AuditLog for these events.
2024-02-19 20:26:47 -08:00
Tim Abbott 2e1ed4431a corporate: Fix plan precedence issues with expired plans.
RemoteRealm customer takes precedence over RemoteServer
in general. But if an inactive plan is associated with
RemoteRealm and an active plan with RemoteServer, the
ACTIVE plan takes precendence.

Co-authored-by: Prakhar Pratyush <prakhar@zulip.com>
2024-02-19 17:58:49 -08:00
Anders Kaseorg e568064f4e page_params: Set and expect state_data = null for spectators.
Fix a ZodError when spectators load the app, introduced by commit
a4938d3760 (#28971).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-02-19 03:03:58 -05:00
Anders Kaseorg a4938d3760 page_params: Parse page_params and state_data with Zod.
This establishes a runtime check that their types continue to reflect
reality going forward.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-02-17 00:02:38 -08:00
Alya Abbott 4f1659fe8f billing: Make push notifications error message more clear. 2024-02-16 16:01:53 -08:00
sayyedarib 5c993f0479 direct_message: Allow DMs to bots and self regardless of policy.
Previously, in DM disabled org messaging to bot was not working when
starting new conversation and adding bot as recipient because of not
updating on recipient change. And secondly, self messaging was not
allowed.
This commit ensures that the DM to bot and self are allowed irrespective
of dm restrictions.

tests: Verify DMs adhere to DM restriction policy.

Fixes #28412

Signed-off-by: sayyedarib <sayyedaribhussain4321@gmail.com>
2024-02-15 16:13:36 -08:00
Alex Vandiver 0079688c49 tornado: Drop WebReloadClientError logic.
The widening of the time between when a process is marked for
reload (at Tornado startup) and when it sends reload events makes it
unlikely-to-impossible that a single `/` request will span both of
them, and thus hit the WebReloadClientError corner case.

Remove it, as it is not worth the complication.  The bad behaviour it
is attempting to prevent (of a reload right after opening `/`) was
always still possible  -- if the `/` request completed right before
Tornado restarted -- so it is not clear that it was ever worth the
complication.
2024-02-15 15:42:50 -08:00
Alex Vandiver da6b0b1cc6 tornado: Add a web_reload_clients endpoint to notify web clients. 2024-02-15 15:42:50 -08:00
Alex Vandiver b25d966352 tornado: Add a --no-immediate-reloads flag to leave web clients alone. 2024-02-15 15:42:50 -08:00
Alex Vandiver 1d3813ec4f tornado: Track which queues were inherited from old Tornado instances. 2024-02-15 15:42:50 -08:00
Alex Vandiver fc41d6085b tornado: Split server restart events from web client reload events. 2024-02-15 15:42:50 -08:00
Alex Vandiver a6287faea4 tornado: Stop collapsing "restart" events via virtual events.
Collapsing was done incorrectly, as 65c400e06d added `zulip_version`
and `zulip_feature_level`, but did not update the virtual event logic
to copy those new values into the virtual event.

However, it is unlikely that a server will be upgraded multiple times
in quick enough succession for this to ever be relevant.  Remove the
logic, which is additional complication for little or no gain.
2024-02-15 15:42:50 -08:00
Alex Vandiver 52a80e1d78 tornado: Switch to time.perf_counter for timing data. 2024-02-15 15:42:50 -08:00
Alex Vandiver ab683fac29 decorator: Rename internal_notify_view.
As noted in the previous commit, this decorator is not just used for
"notify" endpoints anymore.
2024-02-15 15:42:50 -08:00
Alex Vandiver be75736a95 decorator: Rename authenticate_notify.
This was originally used for more Django -> Tornado communication than
it now does; it is primarily not used to "notify" Tornado anymore.
2024-02-15 15:42:50 -08:00
Alex Vandiver e6fa1f5436 email_mirror_message: Move endpoint under /api/internal/.
This also has the side effect that it saves the endpint from being
improperly redirected by the HostDomainMiddleware's canonicalization
code.
2024-02-15 15:42:50 -08:00
Alex Vandiver 84fa9be73a tornado: Move the /notify-tornado endpoint, and document it better. 2024-02-15 15:42:50 -08:00
Alya Abbott 8bebabab8f api docs: Clarify Zulip Cloud requirements for user creation endpoint. 2024-02-15 15:08:53 -08:00