Commit Graph

16872 Commits

Author SHA1 Message Date
Lauryn Menard b0b3230917 scheduled-messages: Update scheduled_message_id parameter for DELETE.
Updates the scheduled_message_id parameter for deleting scheduled
messages to use the to_non_negative_int converter function for
validation, which is used in other endpoints/views with an ID in
the request path.
2023-05-31 08:15:03 -07:00
rht 1c84f02f57 slack import: Convert threads to nicely named Zulip topics.
Fixes #9006.
2023-05-30 16:35:19 -07:00
sbansal1999 2d6e6369f3 integrations: Add Linear webhook integration.
Fixes part of #23118.
2023-05-30 10:48:19 -07:00
Prakhar Pratyush dcb163bbe3 user_topics: Update API documentation.
This commit adds the missing 'UNMUTED' visibility policy
to the documentation for 'api/get-events' and 'api/register-queue'.

It replaces INHERIT with NONE for a clearer name
in the 'api/update-user-topic' documentation.

Other smaller changes in wording to improve readability.
2023-05-30 10:46:05 -07:00
Sahil Batra a143d6b62c message_edit: Add period to error message raised due to time limit. 2023-05-29 14:49:32 -07:00
Satyam Bansal caf6506811 integrations: Simplify regex checks in GitHub Integration.
The regex check is replaced with a simpler "startswith" function.
2023-05-29 10:55:31 -07:00
Tim Abbott dce4a3c98e markdown: Remove most of Twitter integration.
Twitter removed their v1 API. We take care to keep the existing cached
results around for now, and to not poison that cache, since we might
be able replace this with something that can still use the existing
cache.
2023-05-29 10:43:35 -07:00
Alya Abbott 3ca20e3ea8 help: Document quick way to check Zulip version.
Version number was added to top section of gear menu in
4df8c6610f.

Version number was added to "About Zulip" in
668b5137b0.
2023-05-29 08:39:01 -07:00
Mateusz Mandera db7a7d589c actions: Handle cross-realm messages in do_scrub_realm.
This is necessary to properly scrub all data from the realm.
2023-05-28 15:24:31 -07:00
Anders Kaseorg 66501a8eae requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-05-26 22:09:18 -07:00
Anders Kaseorg 9797de52a0 ruff: Fix RUF010 Use conversion in f-string.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-05-26 22:09:18 -07:00
Lauryn Menard 0c09a4fffe scheduled-messages: Add API documentation for new endpoint.
Part of splitting creating and editing scheduled messages.
Final commit. Should be merged with previous commits in series.

Updates the API documentation for the new endpoint for editing
scheduled messages.
2023-05-26 18:05:55 -07:00
Lauryn Menard 957382253a scheduled-messages: Split out edit scheduled message endpoint.
Part of splitting creating and editing scheduled messages.
Should be merged with final commit in series. Breaks tests.

Splits out editing an existing scheduled message into a new
view function and updated `edit_scheduled_message` function.
2023-05-26 18:05:55 -07:00
Lauryn Menard 154af5bb6b scheduled-messages: Remove ID from create scheduled message.
Part of splitting creating and editing scheduled messages.
Should be merged with final commit in series. Breaks tests.

Removes `scheduled_message_id` parameter from the create scheduled
message path.
2023-05-26 18:05:55 -07:00
Lauryn Menard 7af5ceb1c5 scheduled-messages: Add direct scheduled message to populate_db.
Prep commit for splitting create/edit endpoint for scheduled
messages.

Because of `test-api` runs the tests in alphabetical order based on
the `operationId`, we need two scheduled messages in the test database.
The first for the curl example delete (delete-scheduled-message) and
the second for the curl example update (update-scheduled-message).
2023-05-26 18:05:55 -07:00
Lauryn Menard 2f196bff19 api-docs: Add when moving messages fields/parameters were added.
Adds API changelog feature level 1 and associated Changes notes
for when the `stream_id` parameter in the `PATCH /messages/message_id`
was added, and for when the `prev_stream` field was added to edit
history information for messages.

We're adding these to the Zulip 3.0 feature level 1 because
commit 843345dfee that introduced this field and this parameter
to the server / backend code was merged before the commit that added
the API feature level tracking, commit e3b90a5ec8, at level 1.
2023-05-26 11:57:08 -07:00
Lauryn Menard bb735460e5 api-docs: Update avatar_url descriptions for feature level 163 change.
Updates the descriptions of the `avatar_url` field in message and
user objects to be clear that the current user must have access
to the other user's real email address in order for the value to
ever be `null`.

Also adds a bullet point to the API changelog feature level 163
entry about this change.
2023-05-26 11:51:05 -07:00
Lauryn Menard 389e45625a api-docs: Clarify if "email" is the "Zulip API email".
Clarifies additional areas of the API documentation where a user's
email is mentioned / used where it could be useful to clarify
that the email in question is the "Zulip API email".
2023-05-26 11:51:05 -07:00
Lauryn Menard 8c7e18fd75 api-docs: Use reference link in `/register-queue` main description. 2023-05-26 11:51:05 -07:00
Lauryn Menard ee58910f1a api-docs: Update `/get-own-user` to consistently use "requesting user". 2023-05-26 11:51:05 -07:00
Lauryn Menard 7a4565e833 api-changelog: Update feature level 163 entry and changes notes.
Original commits for this feature level's changes were part
of #22994.
2023-05-26 11:51:05 -07:00
Lauryn Menard 526fd7f24b api-docs: Add "delivery_email" field to example responses. 2023-05-26 11:51:05 -07:00
Lauryn Menard 58a64b33c9 api-docs: Revise main description for /get-user-by-email endpoint. 2023-05-26 11:51:05 -07:00
Lauryn Menard 87e9482672 api-docs: Update "Zulip display email" to "Zulip API email".
In some instances, the email was not clearly defined as
Zulip display email, but the Zulip API email is what is being
returned.
2023-05-26 11:51:05 -07:00
Sahil Batra 48e99657ad events: Remove realm_community_topic_editing_limit_seconds.
This commit removes realm_community_topic_editing_limit_seconds
field from register response since topic edit limit is now
controlled by move_messages_within_streams_limit_seconds
setting.
We also remove DEFAULT_COMMUNITY_TOPIC_EDITING_LIMIT_SECONDS
constant since it is no longer used.
2023-05-25 17:26:21 -07:00
Lauryn Menard e41c235657 api-docs: Link client_capabilities parameter in property descriptions.
In the register response properties deprecated at feature level 89,
update the descriptions to link to the client_capabilities parameter
when referenced.

Also, moves the enter_send property to be in the same section of the
register response as other properties deprecated at this feature level.

These descriptions were originally added to these properties in
commit e6f828a8e2.
2023-05-25 12:04:03 -07:00
Satyam Bansal cae02dbca4 integrations: Ignore merge queue push events for GitHub Integration.
Previosuly, the server would send a 500 whenever a merge queue push
event was encountered.

Fixes #25629.
2023-05-24 15:33:50 -07:00
Lauryn Menard 1beebb9fa1 api-changelog: Update feature level 168 entry and changes notes.
Original API changelog entry from commit ae72777c77.
2023-05-24 11:38:00 -07:00
Mateusz Mandera dcbcb05655 saml: Make SP-initiated SLO work after signup. 2023-05-23 13:01:15 -07:00
Mateusz Mandera 04f5358a76 tests: Flush session in verify_desktop_flow_end_page in social auth.
As the relevant comment elaborates - what happens next in the test in
simulating the step that happens in the desktop app. Thus a new session
needs to be used. Otherwise, the old session created normally in the
browser pollutes the state and can give falsely passing tests.

This should be happening for all social auth tests using this, not just
in that one SAML test, thus moving it inside the helper method.
2023-05-23 13:01:15 -07:00
Mateusz Mandera 8fb0fe96c6 saml: Save SessionIndex in session and use when making a LogoutRequest.
This is a useful improvement in general for making correct
LogoutRequests to Idps and a necessary one to make SP-initiated logout
fully work properly in the desktop application. During desktop auth
flow, the user goes through the browser, where they log in through their
IdP. This gives them a logged in  browser session at the IdP. However,
SAML SP-initiated logout is fully conducted within the desktop
application. This means that proper information needs to be given to the
the IdP in the LogoutRequest to let it associate the LogoutRequest with
that logged in session that was established in the browser. SessionIndex
is exactly the tool for that in the SAML spec.
2023-05-23 13:01:15 -07:00
Mateusz Mandera 5dd4dcdebb saml: Make SP-initiated SLO work in the desktop application. 2023-05-23 13:01:15 -07:00
Mateusz Mandera 3f55c10685 saml: Rework SP-initiated logout config to support IdP-level config.
This gives more flexibility on a server with multiple organizations and
SAML IdPs. Such a server can have some organizations handled by IdPs
with SLO set up, and some without it set up. In such a scenario, having
a generic True/False server-wide setting is insufficient and instead
being able to specify the IdPs/orgs for SLO is needed.
2023-05-23 13:01:15 -07:00
Mateusz Mandera 0bb0220ebb saml: Implement SP-initiated Logout.
Closes #20084

This is the flow that this implements:
1. A logged-in user clicks "Logout".
2. If they didn't auth via SAML, just do normal logout. Otherwise:
3. Form a LogoutRequest and redirect the user to
https://idp.example.com/slo-endpoint?SAMLRequest=<LogoutRequest here>
4. The IdP validates the LogoutRequest, terminates its own user session
and redirects the user to
https://thezuliporg.example.com/complete/saml/?SAMLRequest=<LogoutResponse>
with the appropriate LogoutResponse. In case of failure, the
LogoutResponse is expected to express that.
5. Zulip validates the LogoutResponse and if the response is a success
response, it executes the regular Zulip logout and the full flow is
finished.
2023-05-23 13:01:15 -07:00
Mateusz Mandera dda4603f94 auth: Extract EXPIRABLE_SESSION_VAR_DEFAULT_EXPIRY_SECS. 2023-05-23 13:01:15 -07:00
Lauryn Menard eba83b9d87 api-docs: Expand `/update-message` main description for realm settings.
Expands the main description for the `/update-message` documentation
to include a list of the realm settings in the API that are relevant
to when users can update a message's content, topic or stream.
2023-05-23 11:11:20 -07:00
Lauryn Menard 8ec9abe00d api-changelog: Clarify feature level 159 descriptive text.
Original API changelog entries: commit 02eee3a04f and
commit 815bf609fa.
2023-05-23 11:11:20 -07:00
Lauryn Menard e4ae8b79c3 api-changelog: Clarify feature level 162 descriptive text.
Original API changelog entries: commit 73f0eae394 and
commit 2c4e076fef and commit 891f83601d.
2023-05-23 11:11:20 -07:00
Lauryn Menard 0e9c34a975 api-changelog: Clarify feature level 172 descriptive text.
Original API feature level entries: commit 440f9e397a and
commit bd7f728796.
2023-05-23 11:11:20 -07:00
Lauryn Menard 6d6a335e32 api-docs: Make realm_linkifiers current API clear in description.
Adjusts the descriptions of realm_linkifiers (and deprecated
realm_filters) events and register response fields so that the
description of the current API is complete without the feature
level 176 **Changes** notes.
2023-05-22 15:20:07 -07:00
Lauryn Menard fd02648b0e api-changelog: Update feature level 175 entry and related changes notes. 2023-05-22 13:21:03 -07:00
Lauryn Menard 3cb6c9aea9 api-docs: Add examples to `realm_filters` event prose description.
Adds examples of the regex pattern and old URL string format to
the deprecated `realm_filters` event and register response field.
The examples are in the prose description since the events are
no longer sent and therefore no longer tested.
2023-05-19 13:23:07 -07:00
Lauryn Menard 03a2c2da6a api-changelog: Update and clarify docs for feature level 176 entry.
Revises API changelog entry for missing endpoint method and to
clarify the overall text.

Updates Changes notes for feature level 176 to not have repetitive
text, so that the updates were clearer and more concise.

The original commit with the changes related to this API changelog
entry is commit 268f858f39.
2023-05-19 13:23:07 -07:00
Sahil Batra 4c4caa7be4 CVE-2023-32677: Check permission to subscribe other users in invites.
This commit updates the API to check the permission to subscribe other
users while inviting.  The API will error if the user passes the
"stream_ids" parameter (even when it contains only default streams)
and the calling user does not having permission to subscribe others to
streams.

For users who do not have permission to subscribe others, the
invitee will be subscribed to default streams at the time of
accepting the invite.

There is no change for multiuse invites, since only admins are allowed
to send them, and admins always have the permission to subscribe
others to streams.
2023-05-19 16:13:32 -04:00
Mateusz Mandera a23b077b79 CVE-2023-28623: Prevent unauthorized signup with ldap + external auth.
Since 74dd21c8fa in Zulip Server 2.1.0, if:
- ZulipLDAPAuthBackend and an external authentication backend (any aside
  of ZulipLDAPAuthBackend and EmailAuthBackend) are the only ones
  enabled in AUTHENTICATION_BACKENDS in /etc/zulip/settings.py
- The organization permissions don't require invitations to join

...then an attacker can create a new account in the organization with
an arbitrary email address in their control that's not in the
organization's LDAP directory.

The impact is limited to installations which have the specific
combination of authentication backends described above, in addition to
having the "Invitations are required for joining this organization
organization" permission disabled.
2023-05-19 16:13:00 -04:00
Mateusz Mandera 3ed2a30e01 maybe_send_to_registration: Remove password_required arg.
This argument was added with the default incorrectly set to `True` in
bb0eb76bf3 - despite
`maybe_send_to_registration` only ever being called in production code
in a single place, with `password_required=False` explicitly. And then
it just got carried forward through refactors.

`maybe_send_to_registration` was/is also called twice in tests, falling
back to the default, but the `password_required` value is irrelevant to
the tests - and if anything letting it use the `True` has been wrong,
due to not matching how this function is actually used.
2023-05-19 16:13:00 -04:00
Mateusz Mandera b55adbef3d export: Handle RealmAuditLog with .acting_user in different realm. 2023-05-19 11:12:19 -07:00
Alex Vandiver c978bfaa32 models: Add a unique index on UserProfile.api_key.
This prevents `get_user_profile_by_api_key` from doing a sequential
scan.

Doing this requires moving the generation of initial api_key values
into the column definition, so that even bare calls to
`UserProfile.objects.create` (e.g. from tests) call appropriately
generate a random initial value.
2023-05-19 11:11:04 -07:00
Lauryn Menard 145d6915c1 api-changelog: Add 2.1.0 entry for realm_default_external_accounts.
Adds an API changelog note to 2.1 for the addition of
realm_default_external_accounts to the `/register-queue` response.

Also adds a Changes note to the field in the endpoint's response
API documentation.

The original commit that added it to that endpoint's response was
commit d7ee2aced1.
2023-05-19 10:50:17 -07:00
Alex Vandiver b312a86ada sentry: Add the observed user's IP address before forwarding.
The default for Javascript reporting is that Sentry sets the IP
address of the user to the IP address that the report was observed to
come from[^1].  Since all reports come through the Zulip server, this
results in all reports being "from" one IP address, thus undercounting
the number of affected unauthenticated users, and making it difficult
to correlate Sentry reports with server logs.

Consume the Sentry Envelope format[^2] to inject the submitting
client's observed IP address, when possible.  This ensures that Sentry
reports contain the same IP address that Zulip's server logs do.

[^1]: https://docs.sentry.io/platforms/python/guides/logging/enriching-events/identify-user/
[^2]: https://develop.sentry.dev/sdk/envelopes/
2023-05-18 16:25:54 -07:00
Lauryn Menard 5802f7775f api-changelog: Update and clarify docs for feature level 178 entry.
Updates the descriptions and examples for there only being two key
values: "website" and "aggregated".

Also, clarifies that email keys are the Zulip display email.

And removes any descriptive text that says presence objects have
information about the clients the user is logged into.
2023-05-18 11:57:02 -07:00
Alex Vandiver 1184bdc934 push_notifications: Lock message while we mark it pending for push.
Deleting a message can race with sending a push notification for it.
b47535d8bb handled the case where the Message row has gone away --
but in such cases, it is also possible for `access_message` to
succeed, but for the save of `user_message.flags` to fail, because the
UserMessage row has been deleted by then.

Take a lock on the Message row over the accesses of, and updates to,
the relevant UserMessage row.  This guarantees that the
message's (non-)existence is consistent across that transaction.

Partial fix for #16502.
2023-05-18 11:53:21 -07:00
Anders Kaseorg 4b19863065 test_timeout: Skip test_timeout_warn on Python 3.11 for coverage issue.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-05-18 11:52:22 -07:00
Lauryn Menard 2f96b1756e api-changelog: Add notes for feature level 3 `zulip_version` change.
Original API changelog note was made in commit 2c63130195.
2023-05-18 08:51:46 -07:00
Lauryn Menard 3023091caa scheduled-messages: Add help center link to fetch / delete descriptions. 2023-05-18 08:45:51 -07:00
Lauryn Menard 67e1889817 scheduled-messages: Add explicit test with emails for direct message. 2023-05-18 08:45:51 -07:00
Lauryn Menard a2a50e1408 scheduled-messages: Simplify message_to type in check_schedule_message.
As of commit 38f6807af1, we accept only stream and user IDs for
the recipient information for scheduled messages, which means we
can simplify the type for `message_to` in `check_schedule_message`.
2023-05-18 08:45:51 -07:00
Lauryn Menard 1ad0ad8ece tests-scheduled-messages: Use "direct" instead of "private". 2023-05-18 08:45:51 -07:00
Lauryn Menard 8f4de3afc0 api-docs: Update create/edit scheduled message errors for IDs.
In commit 38f6807af1, we updated the `POST /scheduled_messages`
endpoint to only accept user IDs for direct messages. The endpoint
alread only accepted a stream ID for stream messages.

But the API documentation was not updated for the errors returned
when either a stream or user with the specified ID does not exist.

Updates the API documentation for the correct error responses.
2023-05-16 15:14:54 -07:00
Alex Vandiver 24c3e25f86 middleware: Redirect non-canonical realm domain names.
If a host is in REALM_HOSTS, it has its own domain name.  Redirect
access from other domain names to that name.
2023-05-16 15:13:51 -07:00
Alex Vandiver 724de9cd49 rocketchat: Treat users with "bot" roles as bots when importing.
We previously relied on `type`, but we have observed bots typed with a
`bot` role as well.
2023-05-16 15:10:58 -07:00
Alex Vandiver 34394cec9a rocketchat: Handle users with no email address set.
Fixes: #25596.
2023-05-16 15:10:58 -07:00
Alex Vandiver 3160c3cce0 realm_export: Return export id from POST which create it. 2023-05-16 14:05:01 -07:00
Alex Vandiver 7811e99548 realm_export: Handle hard head-of-queue failures.
Realm exports may OOM on deployments with low memory; to ensure
forward progress, log the start time in the RealmAuditLog entry, and
key off of the existence of that to prevent re-attempting an export
which was already tried once.
2023-05-16 14:05:01 -07:00
Alex Vandiver 4a43856ba7 realm_export: Do not assume null extra_data is special.
Fixes: #20197.
2023-05-16 14:05:01 -07:00
Alex Vandiver 5eeb616666 realm_export: Test failures during export. 2023-05-16 14:05:01 -07:00
Alex Vandiver 362177b788 workers: Run realm export with one thread if in low-memory environment.
We previously hard-coded 6 threads for the realm export; in low-memory
environments, spawning 6 threads for an export can lean to an OOM,
which kills the process and leaves a partial export on disk -- which
is then tried again, since the export was never completed.  This leads
to excessive disk consumption and brief repeated outages of all other
workers, until the failing export job is manually de-queued somehow.

Lower the export to only use on thread if it is already running in a
multi-threaded environment.  Note that this does not guarantee forward
progress, it merely makes it more likely that exports will succeed in
low-memory deployments.
2023-05-16 14:05:01 -07:00
Alex Vandiver 9f231322c9 workers: Pass down if they are running multi-threaded.
This allows them to decide for themselves if they should enable
timeouts.
2023-05-16 14:05:01 -07:00
Alex Vandiver 5329fed387 test_queue_worker: Do a full recursion on subclasses.
This makes it less likely we will accidentally fail to include a class
if the subclassing of QueueProcessingWorker changes, and lets mypy
more accurately understand the typing.
2023-05-16 14:05:01 -07:00
Sahil Batra 007a51f277 accounts: Allow user to change email visibility during first login.
We now allow users to change email address visibility setting
on the "Terms of service" page during first login. This page is
not shown for users creating account using normal registration
process, but is useful for imported users and users created
through API, LDAP, SCIM and management commands.
2023-05-16 13:52:56 -07:00
Sahil Batra 7f01b3fb63 users: Set tos_version to -1 for users who have not logged-in yet.
We now set tos_version to "-1" for imported users and the ones
created using API or using other methods like LDAP, SCIM and
management commands. This value will help us to allow users to
change email address visibility setting during first login.
2023-05-16 13:52:56 -07:00
Lauryn Menard 8d06fa7e38 api-docs: Update descriptions for display_emoji_reaction_users. 2023-05-16 12:31:03 -07:00
Lauryn Menard 8962565154 api-docs: Fix capitalization in descriptions with int values lists. 2023-05-16 12:31:03 -07:00
Lauryn Menard 239458a173 api-docs: Remove instances of "=>" in API documentation descriptions. 2023-05-16 12:31:03 -07:00
Lauryn Menard 63d51e8114 api-docs: Clarify uses of "=" in API documentation descriptions.
For cases of `name=value` in descriptions in the API documentation,
update to either use JSON style of `"name": value` when correct or
revise the descriptive text.
2023-05-16 12:31:03 -07:00
Lauryn Menard 09ab1be34f api-docs: Use backticks for descriptions referencing "null" values. 2023-05-16 12:31:03 -07:00
Lauryn Menard c43ea96c06 api-docs: Clean up instances of "None" that should be "null". 2023-05-16 12:31:03 -07:00
Lauryn Menard 4fcb243769 api-docs: Standardize on lowercase true, false and null.
Creates a custom linter rule for `zerver/openapi/zulip.yaml` to
only allow lowercase versions of "true", "false" and "null".

Updates existing documentation for new rules.
2023-05-16 12:31:03 -07:00
Alex Vandiver ffaccb8af2 outgoing_webhook: Respect settings.OUTGOING_WEBHOOK_TIMEOUT_SECONDS.
The use of the setting was accidentally removed in b88d7a741e, and
replaced with a static 10 seconds.
2023-05-16 07:00:37 -07:00
David Rosa be14ec2cab help: Rename "Starting a new private thread" to "... new direct message".
With the private messages -> direct messages migration, we should
rename the "Starting a new private thread" help center article.

- Renames article to "Starting a new direct message"
- Updates relevant section in /help/getting-started-with-zulip
- Fixes typo in /help/send-group-dm
- Updates file names and adds URL redirect.

Fixes #25506.
2023-05-15 16:13:55 -07:00
Alex Vandiver ca2ca030d2 migrations: Backfill missing RealmAuditLog entries for subscriptions.
Backfill subscription realm audit log SUBSCRIPTION_CREATED events for
users which are currently subscribed but don't have any subscription
events, presumably due to some historical bug.  This is important
because those rows are necessary when reactivating a user who is
currently soft-deactivated.

For each stream, we find the subscribed users who have no
subscription-related realm audit log entries, and create a
`backfill=True` subscription audit log entry which is the latest it
could have been, based on UserMessage rows.  We then optionally insert
a `DEACTIVATION` if the current subscription is not active.
2023-05-15 16:09:44 -07:00
Aman Agrawal c2145a8993 scheduled_message: Send `update` event on failed delivery. 2023-05-14 16:46:20 -07:00
Aman Agrawal 4dd32a4d85 scheduled_messages: Extract notify_update_scheduled_message. 2023-05-14 16:46:20 -07:00
Ujjawal Modi 2a6146110c subscriptions: Change in API used for adding new subscriptions.
Earlier when a user who is not allowed to add subscribers to a
stream because of realm level setting "Who can add users to streams"
is subscribing other users while creating a new stream than new stream
was created but no one is subscribed to stream.

To fix this issue this commit makes changes in the API used
for adding subscriptions. Now stream will be created only when user
has permissions to add other users.

With a rewrite of the test by Tim Abbott.
2023-05-14 11:19:05 -07:00
Ujjawal Modi a47569bf47 backend_tests: Add a test for subsribing others to public streams.
Earlier there was no backend test for subscribing others to
public streams in zephyr realm.

This commit adds a backend test for it.
2023-05-14 11:19:04 -07:00
Tim Abbott 2d3d3f6072 message_send: Don't mark scheduled messages to self as read.
The only reasonable intent for such a scheduled message is to remind
oneself of something at that time, which requires it being unread.

Fixes #25523.
2023-05-12 17:55:46 -07:00
Tim Abbott daf6fb17bf do_send_messages: Use mandatory kwargs pattern. 2023-05-12 17:55:46 -07:00
Mateusz Mandera 254ea4b0c8 social_auth: Save authentication method information in the session.
The immediate application of this will be for SAML SP-initiated logout,
where information about which IdP was used for authenticating the
session needs to be accessed. Aside of that, this seems like generally
valuable session information to keep that other features may benefit
from in the future.
2023-05-12 16:21:26 -07:00
Mateusz Mandera af9d1a7dfb register_remote_user: Use explicit kwargs list.
This is nicer that .pop()ing specified keys - e.g. we no longer will
have to update this chunk of code whenever adding a new key to
ExternalAuthDataDict.
2023-05-12 16:21:25 -07:00
Lauryn Menard 90cc2716f0 scheduled-messages: Update API dicts for `failed` boolean field.
Adds the `failed` boolean from the ScheduledMessage to the API dict
returned by scheduled message events and register response, and by
fetching the user's scheduled messages.

`failed` will only be true when the server has tried to send the
scheduled message and failed due to an error.
2023-05-12 15:48:59 -07:00
Lauryn Menard c7c67c01ce scheduled-messages: Update failure to send message fields for edits.
In the case of a user editing a scheduled message that the server
had failed to send at the scheduled time due to an error, we want
to update the `failed` and `failure_message` fields as the intent
is for the server to retry to send the scheduled message based on
the updated information provided by the user.
2023-05-12 15:48:59 -07:00
Lauryn Menard cc648a2c19 scheduled-messages: Send notification if send scheduled message fails.
In the case that there is an error when sending a scheduled message,
we now send a message from the notification bot to the user who
scheduled the message about the failure/error.

The notification message is not sent if the error when sending the
scheduled message was due to the realm or sender being deactivated.
2023-05-12 15:48:59 -07:00
Alex Vandiver a2ed0302ce streams: Prevent already-deactivated streams from being deactivated. 2023-05-12 13:26:43 -07:00
Alex Vandiver 5be7bc58fe upload: Use content_disposition_header from Django 4.2.
The code for this was merged in Django 4.2:
https://code.djangoproject.com/ticket/34194
2023-05-11 14:51:28 -07:00
Sahil Batra ddb0bb58ed tests: Add tests to update visibility policy when target topic is empty.
This commit adds a new test to check how the visibility policy updates
when moving messages to a topic that didn't exist previously.

This test also helps us adding coverage for the code which just
skips setting visibility_policy if there is no need to update the
value because both previous and new value of visibility policy
is INHERIT. The "actions/message_edit.py" file has 100% coverage
now and thus is removed from "not_yet_fully_covered" list.
2023-05-11 12:13:50 -07:00
Sahil Batra d645d5c0ec message_edit: Fix code to set visibility policy on moving messages.
The code for updating visibility policy values on moving messages
had two bugs.

- There was a typo in elif condition where "user_profile" was being
used instead of "user_profile_with_policy".

This commit fixes the typo.

- It was assumed that there would be no UserTopic rows for target
topic if the target topic didn't exist. But there can be such case
where some messages were sent to that topic and the user muted
the topic. But then the messages in that topic was deleted. In
such case there can be UserTopic rows for a stream-topic pair
that does not exist.

This commit fixes the code to handle such case as well and set
the visibility policy of new topic to what was set for the original
topic. This change simplifies the condition to just check whether
new_visibility_policy is equal to target_topic_visibility_policy
and skip if so, and update the visibility policy otherwise.

Due to this change, we now do not try to mute the already muted
topic if the topic is moved to a topic which didn't exist
previously and thus we modify the existing test to not expect
any INFO logs.
2023-05-11 12:13:50 -07:00
Sahil Batra a2600a2b97 tests: Add coverage to actions/message_edit.py.
This commit adds tests to cover the case of message editing
not allowed due to allow_message_editing set to False and
the case when there is no limit set when moving all messages
in a topic.

The "actions/message_edit.py" file does not have 100% coverage
still and it will be addressed in the next commit.
2023-05-11 12:13:50 -07:00
Sahil Batra 9fa67f0fa9 tests: Add coverage to actions/create_realm.py.
This commit adds test coverage to actions/create_realm.py.
The file is also removed from not_yet_fully_covered list
since it has 100% coverage now.
2023-05-11 12:13:50 -07:00
Sahil Batra 5e7d49d129 create_realm: Remove unused argument from do_create_realm.
We do not pass "email_address_visibility" to do_create_realm
anymore. It was passed before to set the setting for realms in
development database, but it has been changed since we changed
email_address_visibility to be a user-level setting instead
of realm-level setting since now it is set on RealmUserDefault
table.
2023-05-11 12:13:50 -07:00
Alex Vandiver 6b58f9f9fa integrations: Add support for MovieAdded Radarr hook. 2023-05-11 12:08:25 -07:00
Alex Vandiver 1d5bccf4c3 integrations: Add support for MovieFileDelete Radarr hook. 2023-05-11 12:08:25 -07:00
Alex Vandiver 79c1123700 validator: Generalize type of check_string_in argument. 2023-05-11 12:08:25 -07:00
Alex Vandiver 01241f0e72 integrations: Add support for MovieDelete Radarr hook. 2023-05-11 12:08:25 -07:00
Alex Vandiver 84b440f0e4 integrations: Add support for ApplicationUpdate Radarr hook. 2023-05-11 12:08:25 -07:00
Alex Vandiver 5931787f11 integrations: Support scans of untagged images in Harbor. 2023-05-11 11:42:54 -07:00
Alex Vandiver 6e842f54c1 webhooks: Add support for gmail and rules sources to Front integration. 2023-05-11 11:36:58 -07:00
Lauryn Menard 1d209220dd tests: Add coverage for error when editing a sent scheduled message.
Adds test coverage for the error sent for editing a scheduled
message that was successfully sent.

`zerver/actions/scheduled_messages.py` now has 100% test coverage
again.
2023-05-11 10:52:01 -07:00
Anders Kaseorg e88b2caeef Revert "db: Force use of TimeTrackingCursor to work around Django 4.2 bug."
This reverts commit f1925487e8.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-05-10 19:44:47 -07:00
Anders Kaseorg d0481be3e5 requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-05-10 19:44:47 -07:00
Tim Abbott 2aae32f642 events: Fix apply_events when raw_unread_msgs not present.
We were missing a few checks for raw_unread_msgs being present before
trying to parse and update it.

The test only covers 2/3 of the cases, but I wasn't convinced it was
worth adding another test just for the corner case of removing a
message flag; this seems fairly unlikely to regress.
2023-05-10 13:44:35 -07:00
Alya Abbott a9d0656605 help: Document stream/DM switcher on "Mastering the compose box" page. 2023-05-10 11:37:36 -07:00
Sahil Batra 4ca887bade invites: Allow users to invite without specifying any stream to join.
We now allow users to invite without specifying any stream to join.
In such cases, the user would join the default streams, if any, during
the process of account creation after accepting the invite.

It is also fine if there are no default streams and user isn't
subscribed to any stream initially.
2023-05-09 17:05:17 -07:00
Sahil Batra 875bcb183e invites: Do not add user to default streams if streams list is empty.
We do not add user to the default streams if the streams list passed
while sending the invite (both email and multi-use) was empty since
invite explicitly selected to not subscribe the user to default
streams.
2023-05-09 16:59:06 -07:00
Sahil Batra ded6b7ede4 test_invite: Add test to verify stream subscriptions.
This commit adds test to verify whether the user was
subscribed to the streams selected when inviting them.
2023-05-09 16:59:06 -07:00
Sahil Batra aebf7bf0e1 test_invite: Fix typo in comment. 2023-05-09 16:59:06 -07:00
Aman Agrawal f025415f2b scheduled_messages: Add tests for delivery of scheduled messages. 2023-05-09 13:48:28 -07:00
Tim Abbott 602e4c2aa3 scheduled_messages: Add focused scheduled message indexes. 2023-05-09 13:48:28 -07:00
Tim Abbott 835f62617e scheduled_messages: Store the final delivered message ID.
This could be useful for debugging problems with the system
operationally.
2023-05-09 13:48:28 -07:00
Tim Abbott fe9656237e scheduled_message: Move remaining logic out of management command.
While this doesn't include tests yet, it will make it much easier to
write tests for the core server logic here.
2023-05-09 13:48:28 -07:00
Tim Abbott 20be0759b5 scheduled_message: Avoid delivering scheduled messages late. 2023-05-09 13:48:28 -07:00
Tim Abbott a065576db7 scheduled messages: Call check_message at send time. 2023-05-09 13:48:28 -07:00
Tim Abbott 7051d3416b scheduled_messages: Add reasonable failure handling.
Previously, it seemed possible for the scheduled messages API to try
to send infinite copies of a message if we had the very poor luck of a
persistent failure happening after a message was sent.

The failure_message field supports being able to display what happened
in the scheduled messages modal, though that's not exposed to the API
yet.
2023-05-09 13:48:28 -07:00
Tim Abbott 147e296e0a scheduled_messages: Add defensive assert for double sends. 2023-05-09 13:48:28 -07:00
Tim Abbott 24bb03c89f scheduled_messages: Add logging on successful send. 2023-05-09 13:48:28 -07:00
Tim Abbott 1fc6ed224a scheduled_messages: Remove batching logic.
The previous logic would attempt to send a large number of unrelated
messages in a single transaction, which is just asking for trouble in
the event that one of the attempts fails.
2023-05-09 13:48:28 -07:00
Tim Abbott d1dfab5b70 scheduled_messages: Live update #scheduled view after sending.
We apparently had neglected to send the actual event to clients that
the message was successfully sent.
2023-05-09 13:48:28 -07:00
Tim Abbott c8f3443fb6 schedules_messages: Move logic out of management command. 2023-05-09 13:48:28 -07:00
Tim Abbott 611cee177d deliver_scheduled_messages: Rename misleading message variable. 2023-05-09 13:48:28 -07:00
Alex Vandiver 1285c39098 zerver: Remove now-unused report/ endpoints. 2023-05-09 13:16:28 -07:00
Lauryn Menard e44520c8fe scheduled-messages: Limit `to` parameter to user and stream IDs.
For scheduled stream messages, we already limited the `to`
parameter to be the stream ID, but here we return a JsonableError
in the case of a ValueError when the passed value is not an integer.

For scheduled direct messages, we limit the list for the `to`
parameter to be user IDs. Previously, we accepted emails like
we do when sending messages.
2023-05-09 12:45:11 -07:00
sbansal1999 63fcfd543a integrations: Add documentation for Notion-Zapier Integration.
Fixes #25314.
2023-05-09 12:25:18 -07:00
sbansal1999 6e2423fe64 integrations: Update text in Zapier Documentation.
This commit updates the text to match the current version
displayed on the website.
2023-05-09 12:25:18 -07:00
Lauryn Menard df17a1bf28 test-events: Test do_delete_message with no messages specified.
Test coverage for `zerver/actions/message_delete.py`.

Both callers of this function would already return if there were
no Messages specified to delete, which is why existing tests did
not cover this.
2023-05-09 09:58:33 -07:00
Alex Vandiver 89d4737ab2 message_edit: Do not reuse user_profile variable in "for" loop.
Doing so causes the "username resolved this topic" or "this topic was
moved by username" notifications to be attributed to a random user who
had a visibility policy on the topic.
2023-05-09 09:38:38 -07:00
Lauryn Menard 02fafb0376 models: Update the references for API dicts for scheduled messages. 2023-05-09 07:36:05 -07:00
Mateusz Mandera 682a4d6f9e import_realm: Improve comment about sequencing of imported models.
The comment was outdated, currently we import UserProfiles before
realm_tables - because some models in realm_tables have a dependency on
UserProfile.

Also makes sense to elaborate a bit more in the comment that it's just
an outline of the ordering, not an exhaustive list.
2023-05-08 15:55:06 -07:00
Mateusz Mandera 0abf60fd01 scheduled_message: Make export/import work.
Closes #25130 by addressing the import/export part of it.
2023-05-08 15:55:06 -07:00
Mateusz Mandera 780ef71891 export: Fix typo in variable name. 2023-05-08 15:55:06 -07:00
Mateusz Mandera 414658fc8e scheduled_message: Handle attachments properly.
Fixes #25414.

We add Attachment.scheduled_messages relation to track ScheduledMessages
which reference the attachment.

The import bits can be done after merging this, by updating #25345.
2023-05-08 09:56:02 -07:00
Mateusz Mandera 4598607a46 test_uploads: Fix two typos. 2023-05-08 09:56:02 -07:00
sbansal1999 2a3d4041e0 integrations: Improve GitHub force push notifications.
Previously, we didn't explicitly display something special for force-pushes.

Fixes #21969.
2023-05-04 14:31:25 -07:00
sbansal1999 dba6f84b97 integrations: Add force push fixtures for GitHub Integration.
One fixture is just removing commits; the other removes some commits
but adds others.
2023-05-04 14:31:25 -07:00
Mateusz Mandera d0e9a77a57 migrations: Fix performance of migration 0436.
This was doing bulk_create in a loop for each realm, which is too slow
on very large servers. Just do a single bulk_create with a reasonable
batch_size at the end.
2023-05-03 11:09:23 -07:00
Tim Abbott 9bdb044ff4 i18n: Update translation data from Transifex. 2023-05-02 13:16:25 -07:00
Lauryn Menard b5c4064b7b api-docs: Clarify topic parameter for create-or-edit-scheduled-message. 2023-05-02 12:46:41 -07:00
Lauryn Menard a5b527f321 onboarding: Specialize Welcome Bot message for education organizations.
Because education organizations and users have slightly specialized
use cases, we update the Welcome Bot message content sent to new
users and new organization owners for these types of organizations
to link to help center articles/guides geared toward these users
and organizations.

Also, updates the demo organization warning to only go to the new
demo organization owner because the 30 day deletion text is only
definitely accurate when the organization is created.

Fixes #21694.
2023-05-01 16:48:48 -07:00
Lauryn Menard 06dd7a3a68 emails: Add corporate_enabled to context for emails.
In commit fc58c35c0, we added a check in various emails for the
settings.CORPORATE_ENABLED value, but that context is only always
included for views/templates with a request.

Here we add that to common_context, which is often used when there
is not a request (like with emails). And we manually add it to the
email context in various cases when there is not a user account to
call with common_context: new user invitations, registration emails,
and realm reactivation emails.
2023-05-01 10:32:43 -07:00
Lauryn Menard d41f12495d api-docs: Document create/edit scheduled message endpoint and events.
Adds API changelog update for the new endpoint added for creating
or editing scheduled messages, for the events sent for scheduled
messages (add, update, delete), and for the `scheduled_messages`
field in the register response.

Adds a specific API markdown documentation file for the create
or update scheduled messages endpoint, so that the parameters in
the cURL examples are accurate for the type of message (direct or
stream) and action (create or edit).

Bumps the API feature level, adds Changes notes to API documentation
and adds the scheduled message endpoints to the API documentation
sidebar.
2023-04-28 17:25:00 -07:00
Lauryn Menard 15c6d67e9c populate-db: Add scheduled message to test database.
Prep commit for adding the scheduled-message endpoints to the API
documentation.

Adds a scheduled message for Iago in the test database so that it
can be deleted in the delete cURL example in the api-test suite.
2023-04-28 17:25:00 -07:00
Aman Agrawal bd2545b0d7 scheduled_message: Send CRUD events to clients. 2023-04-28 17:25:00 -07:00
Aman Agrawal f40855bad2 reminder: Remove feature from zulip.
This is being removed to make the code simpler. We have plans
to add it as a feature in the future , but it will most likely
not use the same code.
2023-04-28 17:25:00 -07:00
Aman Agrawal d60d6e9115 urls: Add new endpoint to create scheduled messages.
This will help us remove scheduled message and reminder logic
from `/messages` code path.

Removes `deliver_at`/`defer_until` and `tz_guess` parameters. And
adds the `scheduled_delivery_timestamp` instead. Also updates the
scheduled message dicts to return `scheduled_delivery_timestamp`.

Also, revises some text in `/delete-scheduled-message` endpoint
and in the `ScheduledMessage` schema in the API documentation.
2023-04-28 17:25:00 -07:00
Lauryn Menard 7739703111 scheduled-messages: Update scheduled message objects in the API for type.
Updates the objects in the API for scheduled messages so that those
for stream messages return the `to` property as an integer since it
is always the unique stream ID and so that those for direct messages
do not have a `topic` property since direct messages never have a
topic.

Also makes small update so that web app scheduled messages overlay
has the correct stream ID.
2023-04-28 17:25:00 -07:00
Aman Agrawal 4718eaa213 scheduled_messages: Extract method to get undelivered scheduled messages.
This also changes key for `ID` of scheduled message from `message_id`
to `scheduled_message_id`.
2023-04-28 17:25:00 -07:00
Aman Agrawal 4cb238fb6e models: Add method to convert ScheduleMessage objects into dicts. 2023-04-28 17:25:00 -07:00
Aman Agrawal 963fe566d7 scheduled_messages: Use scheduled_message_id instead of message_id.
Using `message_id` can be confusing for API users since it can be
mistaken for the ID of the message that will be sent.
2023-04-28 17:25:00 -07:00
Aman Agrawal 7bf0793c94 scheduled_messages: Move database related function to /actions.
This would match the pattern we use for other modules and also
shrink the giant message_send.py.
2023-04-28 17:25:00 -07:00
Aman Agrawal cddf25656f test_message_send: Move out scheduled message tests. 2023-04-28 17:25:00 -07:00
Alex Vandiver 4f2417cfc4 soft_reactivation: Add a partial index to speed up event lookups.
The full auditlog table is moderately large, and the previously-chosen
index (on `modified_user_id`) is not terribly specific.
2023-04-28 12:43:34 -07:00
Alex Vandiver a56da4be76 soft_deactivation: Only fetch necessary columns.
Existing tests verify that this does not add more queries.
2023-04-28 12:43:34 -07:00
Alex Vandiver ae7485a96e soft_deactivation: Do not bother to fetch stream data as well.
This prefetch is unnecessary and makes this query load more data than
needed.

Existing tests verify that this does not add more queries.
2023-04-28 12:43:34 -07:00
Tim Abbott a03dca93ca message_edit: Clarify losing_access_user_ids calculation.
This was previously called delete_event_notify_user_ids, which seemed
to narrow its purpose in a way that was confusing given that it's also
used for other calculations.

Further, calculate it as soon as we know it, not when we're first
going to use it.
2023-04-27 18:35:32 -07:00
Tim Abbott 71e57d2022 message_edit: Clarify name for gaining_usermessage_user_ids.
This list contains user IDs, not subscription objects.
2023-04-27 18:35:32 -07:00
Tim Abbott 3ff361a1f0 message_edit: Use sets for old/new stream IDs.
This fixes a quadratic performance issue iterating through these
lists. Give these variables slightly better names while we're at it.
2023-04-27 18:35:32 -07:00
Prakhar Pratyush c8a9c0ee04 realm_redirect: Redirect always to the login page with the next parameter.
Previously, entering an organization via 'accounts/go' with the
web-public stream enabled took the user to the web-public view
even if the user was not logged in.

Now, a user is always redirected to the 'login_page' with
the next parameter, if present.

The 'login_page' view is updated to redirect an authenticated
user based on the 'next' parameter instead of always redirecting
to 'realm.uri'.

Fixes #23344.
2023-04-27 16:50:10 -07:00
Sahil Batra afc5066e36 registration: Fix "Resend" link not working for realm creation.
The "Resend" link for realm creation was not working correctly
because it is implemented by basically submiting the registration
form again which results in resending the email but all the
required parameters were not passed to the form after recent
changes in the realm creation flow.

This commit fixes it by passing all the required parameters -
email, realm name, realm type and realm subdomain, when submitting
form again by clicking on the "resend" link.

Fixes #25249.
2023-04-27 12:28:37 -07:00
Sahil Batra f8f4fa4c5e tests: Extract realm name and string_id values in variables.
This is a prep commit so that we can use these variables to
verify the urls in next commit.
2023-04-27 12:28:37 -07:00
Sahil Batra 850e0046eb models: Add ORG_TYPE_IDS constant field to Realm.
This commit adds ORG_TYPE_IDS constant field to Realm class
such that it can be used when we want to validate the org_type
passed in request. This was previously defined in realm.py, but
we move it inside Realm class such that we can use it at other
places as well.
2023-04-27 12:28:37 -07:00
Daniil Fadeev db37880d08 emails: Fix some css not being applied to emails.
This commit places the email CSS into the `style` tag located in the
`head` section. This resolves the issue of being unable to apply
certain CSS styles that cannot be inlined, such as media queries and
pseudo-classes.
2023-04-27 08:55:24 -07:00
AcKindle3 521487f444 tornado_url: Replace occurrences of `uri` with `url`.
In #23380 we want to replace all occurrences of `uri` with `url`.
This commit replaces the occurrences appeared in a variable name
`tornado_uri` and a function name `get_tornado_uri`.
2023-04-26 16:37:16 -07:00
AcKindle3 0a1ccb3d89 api_url_context: Replace `uri` with `url`.
In #23380 we want to change all occurrences of `uri` with `url`.
This commit changes the occurrences in a context key `api_uri_context`
and a function name `add_api_uri_context`.
2023-04-26 16:37:16 -07:00
AcKindle3 98c9a0366a settings: Replace occurrences of `uri` with `url`.
In #23380 we want to change all occurrences of `uri` with `url`.
This commit changes the names of two variables `external_uri_scheme`
and `main_site_uri`, who are constructed using `settings` constants.
2023-04-26 16:37:16 -07:00
AcKindle3 4544eb4576 email: Replace `uri` with `url` in templates and backend.
In #23380 we want to change all ocurrences of `uri` to `url`. This
commit changes the ocurrences of `uri` appeared in files related to
email, including templates (`.html`, `.txt`) and backend (`.py`)
codes.

In `email.md`, `base_images_uri` is changed to `images_base_url` -
the words `base` and `images` are swapped and plural form is added
for `image`.  This is becasue the former is not found anywhere in
the codebase while the later appears a lot. To reduce confusion,
this doccumentation changed accordingly.
2023-04-26 16:37:16 -07:00
Lauryn Menard a22168d8b3 templates: Add CSS formatting to invalid email redirect page.
Adds CSS formatting for `invalid_email.html`.

Uses the `white-box` style because this page is a redirect when
there is an error with the email the user provided during
registration.

Also, updates the text of this page for some grammar errors and
to clarify the language between an invalid email and an email that
is not allowed by the Zulip organization in question.

Finally, makes any references to the `realm_name` also link to
the Zulip organization with the `realm_uri`.
2023-04-26 15:41:20 -07:00
Lauryn Menard 601d8101f5 templates: Add CSS formatting to no available licenses page.
Adds CSS formatting for `no_spare_licenses.html`.

Uses the `white-box` style because this page is a redirect when
a user tries to register for a Zulip Cloud organization that does
not have any available licenses for new users.

Updates reference to `realm_name` to be a link to the `realm_uri`.
2023-04-26 15:41:20 -07:00
Alya Abbott 977bec25ba portico: Add Atolio case study. 2023-04-26 14:33:30 -07:00
Tim Abbott 10b893e1e3 api_docs: Update API docs for presence changes. 2023-04-26 14:26:47 -07:00
Mateusz Mandera a9f40a64fd presence: Support null values in UserPresence. 2023-04-26 14:26:47 -07:00
Mateusz Mandera 0d79f6dd27 presence: Deduplicate code formatting legacy presence info.
This also removes the error in one of these functions that was using a
different constant instead of
PRESENCE_LEGACY_EVENT_OFFSET_FOR_ACTIVITY_SECONDS.
2023-04-26 14:26:47 -07:00
Mateusz Mandera ed91e72c5c presence: Rewrite the backend data model - final migration. 2023-04-26 14:26:47 -07:00
Tim Abbott 027b67be80 presence: Rewrite the backend data model.
This implements the core of the rewrite described in:

For the backend data model for UserPresence to one that supports much
more efficient queries and is more correct around handling of multiple
clients.  The main loss of functionality is that we no longer track
which Client sent presence data (so we will no longer be able to say
using UserPresence "the user was last online on their desktop 15
minutes ago, but was online with their phone 3 minutes ago").  If we
consider that information important for the occasional investigation
query, we have can construct that answer data via UserActivity
already.  It's not worth making Presence much more expensive/complex
to support it.

For slim_presence clients, this sends the same data format we sent
before, albeit with less complexity involved in constructing it.  Note
that we at present will always send both last_active_time and
last_connected_time; we may revisit that in the future.

This commit doesn't include the finalizing migration, which drops the
UserPresenceOld table.
The way to deploy is to start the backfill migration with the server
down and then start the server *without* the user_presence queue worker,
to let the migration finish without having new data interfering with it.
Once the migration is done, the queue worker can be started, leading to
the presence data catching up to the current state as the queue worker
goes over the queued up events and updating the UserPresence table.

Co-authored-by: Mateusz Mandera <mateusz.mandera@zulip.com>
2023-04-26 14:26:47 -07:00
Tanmay Munjal 80c01e9fc3
auth: Refactor JWT code to push REQ to top level functions.
Fixes #25015.
2023-04-26 10:57:22 -07:00
Anders Kaseorg 03b3c8522d requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-04-25 21:20:33 -07:00
Anders Kaseorg f74cfe0ed5 test_console_output: Fix WrappedIO.writelines with iterator.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-04-25 21:20:33 -07:00
Anders Kaseorg 5ecff88c1b ruff: Fix C418 Unnecessary `dict` literal passed to `dict()`.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-04-25 21:20:33 -07:00
Anders Kaseorg 9db3451333 Remove statsd support.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-04-25 19:58:16 -07:00
Anders Kaseorg ae7a21834e utils: Fix make_safe_digest type safety.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-04-25 19:01:01 -07:00
Anders Kaseorg 408ef63404 utils: Fix process_list_in_batches type safety.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-04-25 19:01:01 -07:00
Daniil Fadeev d79f4d4f46 emails: Fix style regression for dynamically generated content.
The migration to css-inline "fixed" the fact that styles from this
file previously were never applied to the internals of missed-message
emails.

Rewrite much of the CSS to more correctly scope to apply to the
appropriate elements, and document with comments the purpose of most
blocks.

Co-authored-by: Tim Abbott <tabbott@zulip.com>
2023-04-25 17:53:14 -07:00
Alya Abbott dd4b391229 help: Cover unmuting topics in /help/mute-a-topic. 2023-04-25 15:04:24 -07:00
Lauryn Menard 9880fcb936 narrow: Document new filters `is:dm`, `dm` and `dm-including`.
Documents narrows now have support for new filters for direct messages:
`is:dm`, `dm`, and `dm-including`. Also documents that `is:private`,
`pm-with` and `group-pm-with` are now legacy aliases for these three
new filters respectively.

Note that API documentation references the help center documentation
for search/narrow filters.

Fixes #24806.
2023-04-25 09:07:08 -07:00
Lauryn Menard 243216cb60 url-encoding: Update URLs for direct messages.
In a previous commit, the frontend of the web app was updated for
URLs with `#narrow/dm/...` for direct messages (group and 1-on-1).

Updates the URLs generated for email notifications and outgoing
webhook notification messages to use the new `/dm/...` format.
2023-04-25 09:07:08 -07:00
Lauryn Menard 33886575b2 narrow: Add backend support for `dm-including` operator.
Adds backend support for `dm-including` operator. This will
deprecate the `group-pm-with` operator, but we keep support
for backwards-compatibility.

For testing updates, because the messages returned by these
two operators are different, most of the tests for `group-pm-with`
remain unchanged, but added comments about deprecated state.

Also, cleans up remaining instance of "PM" in `narrow.py` to
be "DM".

The general API changelog and documentation updates will be done
in a final commit in the series of commits that adds support for
the various new direct message narrows.
2023-04-25 09:07:08 -07:00
Lauryn Menard 29832de5f6 narrow: Extract function for calculating huddle recipients.
Extracts the code from `by_group_pm_with` that calculates the
huddle recipients. Prep commit for implementing new `by_*`
method for "dm-including".
2023-04-25 09:07:08 -07:00
Lauryn Menard 665e435b58 narrow: Add backend support for `dm` operator.
Adds backend support for `dm` operator. This will deprecate the
`pm-with` operator, but we keep support for backwards-compatibility.

For testing updates, updates the existing tests for `pm-with` to
use `dm`, and adds one basic test for `pm-with` in the `add_term`
tests as the two operators refer to the same `by_*` method.

The general API changelog and documentation updates will be done
in a final commit in the series of commits that adds support for
the various new direct message narrows.
2023-04-25 09:07:08 -07:00
Lauryn Menard ece752014c narrow: Add backend support for `is:dm` narrow.
Adds backend support for `is` operator with the `dm` operand. This
will deprecate the `is` operator with the `private` operand, but we
keep support for backwards-compatibility.

Note that there is some clean up of references to private messages
in the updated backend test. In commit 43ec7ed, the documentation
for `build_narrow_filter` wasn't updated for the rename of
`BuildNarrowFilterTest` to `NarrowLibraryTest`, so that's also
corrected in these changes.

The general API changelog and documentation updates will be done
in a final commit in the series of commits that adds support for
the various new direct message narrows.
2023-04-25 09:07:08 -07:00
sbansal1999 77ae3176ca webhooks: Rephrase message for Issue Assigned and Unassigned Events.
This commit rephrases the message for Issue assigned and unassigned
events for GitHub Integration.
2023-04-24 23:16:48 -07:00
sbansal1999 31a9d0e69e webhooks: Change assignee info for Issue events in Github Integration.
This commit changes the assignee info to the correct one.

Earlier, the current assignee info of the issue was used
to create the message but when an issue was unassigned
this would give incorrect information if there are still
assignees left on the issue. Now, it uses the details of
the user who has been unassigned from the issue.
2023-04-24 23:16:48 -07:00
sbansal1999 e15df084ad webhooks: Remove description from Issue Unassigned event message.
This commit removes the description from the Issue Unassigned
event message that is sent by the Github Integration.
2023-04-24 23:16:48 -07:00
sbansal1999 85689a5343 webhooks: Remove description from Issue Assigned event message.
This commit removes the description from the Issue Assigned
event message that is sent by the Github Integration.

Previously when an issue was assigned, the message also had the
description of the issue, which is redundant as the description
has been already sent through the issue created event. After
this commit, the description no longer appears in the message.

Fixes #24554
2023-04-24 23:16:48 -07:00