Commit Graph

17142 Commits

Author SHA1 Message Date
Lauryn Menard 89794ce47c api-docs: Update mute and unmute user endpoint documentation.
Updates the main descriptions for the mute a user and unmute a
user endpoint documentation. Also, revises the `muted_user_id`
parameter description and changes note for feature level 188.

The original feature level changes were made in #26005.
2023-06-27 11:24:30 -07:00
Alex Vandiver 21aeb4a040 slack: Handle the special case of permissions denied on team.info call.
This is a follow-up to 4c8915c8e4, for
the case when the `team:read` permission is missing, which causes the
`team.info` call itself to fail.  The error message supplies
information about the provided and missing permissions -- but it also
still sends the `X-OAuth-Scopes` header which we normall read, so we can
use that as normal.
2023-06-27 11:04:41 -07:00
Lauryn Menard e1c809a71e api-docs: Clarify what topics are returned by `get-stream-topics`.
Updates the main description for the `get-stream-topics` endpoint
so that it is clear that the topics for private streams with protected
history are limited to the topics / messages the user has access to.

And updates that documentation and the help center documentation for
bot permissions / abilities, to clarify that bots have the same
restriction and can only access messages / topics that are sent after
the bot (not the bot's owner) subscribed to the stream.
2023-06-26 20:26:30 -07:00
Satyam Bansal 92a1e3464d
integrations: Allow null pull request review body in GitHub Integration.
There are instances in which the pull request review body
can be null.

Fixes #26055.
2023-06-26 11:35:14 -07:00
Satyam Bansal 0945f2dee6 integrations: Separate issue labeled events in GitHub Integration.
This commit creates separate events for issue labeled and
unlabeled notifications. This allows the end-users to choose
whether they want these notifications or not.

Fixes #25789.
2023-06-26 11:04:32 -07:00
Satyam Bansal 842e9d1aca integrations: Improve GitHub issue labeled and unlabeled notifications.
Earlier, the notifications had no information about the labels
being added or removed.
2023-06-26 11:04:32 -07:00
Satyam Bansal 8fc28be8ca integrations: Add issue unlabeled fixture to GitHub Integration. 2023-06-26 11:04:32 -07:00
Satyam Bansal b4674892a4 integrations: Add issue labeled fixture to GitHub Integration. 2023-06-26 11:04:32 -07:00
David Rosa 04ead84c4d lib-markdown: Rename confusing variable name in `tabbed_sections.py`.
- Renames `data_language` -> `data_tab_key`.
- Renames `tab_name` -> `tab_key`.
2023-06-24 07:47:25 -07:00
David Rosa 0e0512df92 widgets: Rename confusing variable name in `tabbed_instructions.ts`.
The `tabbed_instructions` widget used for both language toggles in our
API documentation and app toggles in our Help Center documentation
misleadingly calls the identifier for the tab `language` in local
variables and its interface.

- Renames local variables `language` -> `tab_key`.
- Renames HTML data attributes `data-language` -> `data-tab-key`.

Fixes #24669.
2023-06-24 07:47:25 -07:00
Lauryn Menard 73fd729c4b message-flags: Rename classes for direct message dicts.
Renames `UnreadPrivateMessageInfo` and `RawUnreadPrivateMessageDict`
to be `UnreadDirectMessageInfo` and `RawUnreadDirectMessageDict`
instead.
2023-06-23 11:24:13 -07:00
Lauryn Menard 8c0d89a225 delete-user: Update management command help doc-string.
Updates the help doc-string for the delete_user management command
to use "direct message" instead of "private message".
2023-06-23 11:24:13 -07:00
Lauryn Menard d3f7cfccbc zerver: Update comments with "private message" or "PM".
Updates comments/doc-strings that use "private message" or "PM" in
files in the `/zerver` directory to instead use "direct message".
2023-06-23 11:24:13 -07:00
Lauryn Menard b75c99b893 webhooks: Update references to "private message" and "PM".
Updates references to "private message" and "PM" to instead be
"direct message".
2023-06-23 11:24:13 -07:00
Lauryn Menard 2eeeda7694 mattermost: Update references to "private message" and "PM".
Updates references to "private message" and "PM" in the data import
and related tests for Mattermost to be "direct message" or "DM"
instead.
2023-06-23 11:24:13 -07:00
Lauryn Menard d53b854a7c backend-tests: Update "private message" or "PM" to "direct message".
Updates comments and test strings/names with "private message" or
"PM" to use "direct message" instead.
2023-06-23 11:24:13 -07:00
Alex Vandiver 38d1b3314a convert_slack_data: Document "--token" more correctly. 2023-06-23 11:09:45 -07:00
Alex Vandiver 4c8915c8e4 slack: Provide more information when a Slack token fails to validate. 2023-06-23 11:09:45 -07:00
Alex Vandiver 1b2ba4e09d test_slack_importer: Switch to xoxb tokens, which is what we accept. 2023-06-23 11:09:45 -07:00
Lauryn Menard f461f28feb api-docs: Update endpoints in streams section for permissions info.
Updates the `api/subscribe` and `api/update-stream` endpoint docs
to note that streams' permissions impact whether a user/admin can
subscribe users and/or update a stream's permissions settings.

Updates the `api/archive-stream` and `api/delete-topic` endpoint
docs to note that they are only available to org admins.
2023-06-23 11:03:40 -07:00
Alex Vandiver 6c3969f893 name_restrictions: Reject anything with zulip or kandra in it.
This is primarily to prevent impersonation, such as `zulipteam`.  We
only enable these protections for CORPORATE_ENABLED, since `zulip` is
a reasonable test name for self-hosters.
2023-06-23 10:45:40 -07:00
Anders Kaseorg c09e7d6407 codespell: Correct “requestor” to “requester”.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-06-20 16:17:55 -07:00
Alex Vandiver d30ab34e3d circleci: Support the "ping" event type. 2023-06-20 14:26:31 -04:00
Lauryn Menard 27878cce87 api-docs: Update `zulip.yaml` for "direct message".
Updates the API documentation in `zerver/openapi/zulip.yaml` to use
"direct message" instead of "private message".
2023-06-19 16:56:15 -07:00
Anders Kaseorg 98310f269b middleware: Do not consume StreamingHttpResponse.streaming_content.
streaming_content is an iterator. Consuming it within middleware
prevents it from being sent to the browser.

https://docs.djangoproject.com/en/4.2/ref/request-response/#streaminghttpresponse-objects

“The StreamingHttpResponse … has no content attribute. Instead, it has
a streaming_content attribute. This can be used in middleware to wrap
the response iterable, but should not be consumed.”

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-06-19 15:04:18 -07:00
Anders Kaseorg 7657cb4a0f django_api: Extract send_event_on_commit helper.
django-stubs 4.2.1 gives transaction.on_commit a more accurate type
annotation, but this exposed that mypy can’t handle the lambda default
parameters that we use to recapture loop variables such as

    for stream_id in public_stream_ids:
        peer_user_ids = …
        event = …

        transaction.on_commit(
            lambda event=event, peer_user_ids=peer_user_ids: send_event(
                realm, event, peer_user_ids
            )
        )

https://github.com/python/mypy/issues/15459

A workaround that mypy accepts is

        transaction.on_commit(
            (
                lambda event, peer_user_ids: lambda: send_event(
                    realm, event, peer_user_ids
                )
            )(event, peer_user_ids)
        )

But that’s kind of ugly and potentially error-prone, so let’s make a
helper function for this very common pattern.

        send_event_on_commit(realm, event, peer_user_ids)

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-06-19 13:42:40 -07:00
Alex Vandiver 77c146b8b0 send_email: Delete ScheduledEmail objects with no recipients.
9d97af6ebb addressed the one major source of inconsistent data which
would be solved by simply re-attempting the ScheduledEmail row.  Every
other instance that we have seen since then has been a corrupt or
modified database in some way, which does not self-resolve.  This
results in an endless stream of emails to the administrator, and no
forward progress.

Drop this to a warning, and make it remove the offending row.  This
ensures we make forward progress.
2023-06-19 13:40:50 -07:00
Anders Kaseorg 92c83c1df4 tests: Remove assert_streaming_content helper in favor of getvalue.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-06-15 16:49:27 -07:00
Zixuan James Li 5901ffb0ab models: Fix typo in comment.
"This consistent ordering is important to prevent to prevent"
->
"This consistent ordering is important to prevent"
2023-06-15 17:50:34 -04:00
Prakhar Pratyush 134058b06d settings: Configure 'enable_followed_topic_audible_notifications'.
This commit makes it possible for users to control the
audible desktop notifications for messages sent to followed topics
via a global notification setting.

There is no support for configuring this setting through the UI yet.
2023-06-13 18:01:41 -07:00
Prakhar Pratyush a848c744c3 settings: Configure 'enable_followed_topic_desktop_notifications'.
This commit makes it possible for users to control the
visual desktop notifications for messages sent to followed topics
via a global notification setting.

There is no support for configuring this setting through the UI yet.
2023-06-13 18:01:41 -07:00
Prakhar Pratyush e71d3ada87 settings: Add wildcard mention notifications for the followed topics.
This commit makes it possible for users to control the wildcard
mention notifications for messages sent to followed topics
via a global notification setting.

There is no support for configuring this setting
through the UI yet.
2023-06-13 18:01:41 -07:00
Prakhar Pratyush d73c715dc2 settings: Add push notifications for the followed topics.
This commit makes it possible for users to control
the push notifications for messages sent to followed topics
via a global notification setting.

There is no support for configuring this setting
through the UI yet.
2023-06-13 18:01:41 -07:00
Prakhar Pratyush 5e5538886f settings: Add email notifications for the followed topics.
This commit makes it possible for users to control
the email notifications for messages sent to followed topics
via a global notification setting.
Although there is no support for configuring this setting
through the UI yet.

Add five new fields to the UserBaseSettings class for
the "followed topic notifications" feature, similar to
stream notifications. But this commit consists only of
the implementation of email notifications.
2023-06-13 18:01:41 -07:00
Zixuan James Li 693b3679e3 muted users: Add support to muting bots.
We intentionally disallow muting bots previously upon
a pending design decision in #16915.
This lifts that constraint.

Fixes #22693.
2023-06-13 16:44:12 -07:00
Satyam Bansal 06c49969a0 integrations: Update links for various events in GitLab Integration.
After the release of GitLab 16.0, legacy URLs were modified to
include a "/-/" delimiter in the URLs.

Fixes #25643.
2023-06-12 17:14:18 -07:00
Satyam Bansal 45757ca7a1 integrations: Modify URL for note events in GitLab Integration.
Previosly, the URL was created manually by joining different
fields of the payload.
2023-06-12 17:14:18 -07:00
Satyam Bansal 508514c870 integrations: Update note fixtures for GitLab Integration. 2023-06-12 17:14:18 -07:00
Satyam Bansal 16563a3217 integrations: Add support for sample events in Sentry Integration.
Fixes #25778.
2023-06-12 17:05:12 -07:00
Satyam Bansal 580d8c4dfe integrations: Add exception fixture for Vue in Sentry Integration. 2023-06-12 17:05:12 -07:00
Satyam Bansal 3bdb806fba integrations: Check for Raven SDK only on python in Sentry Integration.
Fixes part of #25778.
2023-06-12 17:05:12 -07:00
Satyam Bansal 142e455d81 integrations: Add support for Rails backend in Sentry Integration. 2023-06-12 17:05:12 -07:00
Satyam Bansal 9e793c37e6 integrations: Update documentation for Sentry Integration. 2023-06-12 17:05:12 -07:00
Alex Vandiver 76d7a5a53a dev_settings: Remove `THUMBNAIL_IMAGES` from test_extra_settings.
THUMBNAIL_IMAGES was previously set to true as there were tests on a new
thumbnail functionality. The feature was never stable enough to remain in
the codebase and the setting was left enabled. This setting also doesn't
reflect how the production deployments are and it has been decided that we
should drop setting from test_extra_settings altogether.

Co-authored-by: Joseph Ho <josephho678@gmail.com>
2023-06-12 16:26:55 -07:00
Alex Vandiver fbb831ff3b uploads: Allow access to the /download/ variant anonymously.
This was mistakenly left off of b799ec32b0.
2023-06-12 12:55:27 -07:00
Alex Vandiver 0dbe111ab3 test_helpers: Switch add/remove_ratelimit to a contextmanager.
Failing to remove all of the rules which were added causes action at a
distance with other tests.  The two methods were also only used by
test code, making their existence in zerver.lib.rate_limiter clearly
misplaced.

This fixes one instance of a mis-balanced add/remove, which caused
tests to start failing if run non-parallel and one more anonymous
request was added within a rate-limit-enabled block.
2023-06-12 12:55:27 -07:00
Sahil Batra ea1357be66 user_groups: Prevent cycles when adding subgroups for a user group.
The user group depedency graph should always be a DAG.
This commit adds code to make sure we keep the graph DAG
while adding subgroups to a user group.

Fixes #25913.
2023-06-12 11:06:49 -07:00
Anders Kaseorg 85681546ce openapi: Convert deprecated Spec.create to Spec.from_dict.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-06-11 13:51:17 -07:00
Tim Abbott f63973c0cf integrations: Remove ancient legacy trello plugin docs.
This has been "legacy" since 2016, so surely nobody is using it.
2023-06-09 15:08:24 -07:00
Zixuan James Li b67c354826 user_groups: Make system groups creation atomic.
We want to make sure that the system groups, once created, will always
have the GroupGroupMemberships fully set up.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2023-06-09 15:07:37 -07:00
Prakhar Pratyush 79e5d32ef6 mention: Refactor 'possible_mentions' to return a dataclass.
This prep commit refactors 'possible_mentions' to
return a dataclass instead of a tuple for better readability.
2023-06-07 16:55:31 -07:00
Prakhar Pratyush d78bdc092f mention: Refactor 'extract_mention_text' to return dataclass.
This prep commit refactors 'extract_mention_text' to
return a dataclass instead of a tuple for better readability.
2023-06-07 16:55:31 -07:00
Prakhar Pratyush 0ea999b348 refactor: Convert the 'wildcards' list to a frozenset.
This commit converts the 'wildcards' list to a frozenset,
as we never intend to mutate 'wildcards'.
2023-06-07 16:55:30 -07:00
Zixuan Li e39e04c3ce
migration: Add `extra_data_json` for audit log models.
Note that we use the DjangoJSONEncoder so that we have builtin support
for parsing Decimal and datetime.

During this intermediate state, the migration that creates
extra_data_json field has been run. We prepare for running the backfilling
migration that populates extra_data_json from extra_data.

This change implements double-write, which is important to keep the
state of extra data consistent. For most extra_data usage, this is
handled by the overriden `save` method on `AbstractRealmAuditLog`, where
we either generates extra_data_json using orjson.loads or
ast.literal_eval.

While backfilling ensures that old realm audit log entries have
extra_data_json populated, double-write ensures that any new entries
generated will also have extra_data_json set. So that we can then safely
rename extra_data_json to extra_data while ensuring the non-nullable
invariant.

For completeness, we additionally set RealmAuditLog.NEW_VALUE for
the USER_FULL_NAME_CHANGED event. This cannot be handled with the
overridden `save`.

This addresses: https://github.com/zulip/zulip/pull/23116#discussion_r1040277795

Note that extra_data_json at this point is not used yet. So the test
cases do not need to switch to testing extra_data_json. This is later
done after we rename extra_data_json to extra_data.

Double-write for the remote server audit logs is special, because we only
get the dumped bytes from an external source. Luckily, none of the
payload carries extra_data that is not generated using orjson.dumps for
audit logs of event types in SYNC_BILLING_EVENTS. This can be verified
by looking at:

`git grep -A 6 -E "event_type=.*(USER_CREATED|USER_ACTIVATED|USER_DEACTIVATED|USER_REACTIVATED|USER_ROLE_CHANGED|REALM_DEACTIVATED|REALM_REACTIVATED)"`

Therefore, we just need to populate extra_data_json doing an
orjson.loads call after a None-check.

Co-authored-by: Zixuan James Li <p359101898@gmail.com>
2023-06-07 12:14:43 -07:00
evykassirer aa270bcef0 search: Remove support for experimental search pills.
This in-progress feature was started in 2018 and hasn't
been worked on much since. It's already in a broken state,
which makes it hard to iterate on the existing search bar
since it's hard to know how those changes will affect search
pills.

We do still want to add search pills eventually, and when
we work on that, we can refer to this diff to readd the
changes back.
2023-06-06 18:36:02 -07:00
Anders Kaseorg b7909db987 ruff: Fix PLC0208 Use a sequence type when iterating over values.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-06-06 14:58:11 -07:00
Anders Kaseorg b907ad0dcb ruff: Fix more of RUF010 Use conversion in f-string.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-06-06 14:58:11 -07:00
Anders Kaseorg 92db6eba78 test_helpers: Convert TypedDict from queries_captured to dataclass.
An implicit coercion from an untyped dict to the TypedDict was hiding
a type error: CapturedQuery.sql was really str, not bytes.  We should
always prefer dataclass over TypedDict to prevent such errors.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-06-06 14:58:11 -07:00
Zixuan James Li 28ec7baaef zilencer: Make analytics bouncer forward-compatible with JSONField.
This adds support to accepting extra_data being dict from remote
servers' RealmAuditLog entries. So that it is forward-compatible with
servers that have migrated to use JSONField for RealmAuditLog just in
case. This prepares us for migrating zilencer's audit log models to use
JSONField for extra_data.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2023-06-05 17:38:10 -07:00
Zixuan James Li 71ab77db9a zilencer: Use more realistic audit log extra_data.
This prepares for the audit log migration which requires us to populate
a JSONField from the extra_data field. "data" is not representative of
the actual extra_data field for RealmAuditLog entries of event types
in SYNC_BILLING_EVENTS.

We intentionally leave the test cases unchanged without bothering to
verify if the extra_data arrives as-is to keep this change minimal.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2023-06-05 17:38:10 -07:00
Mateusz Mandera 06143943d8 migrations: Handle duplicate fk constraint in 0443.
It turns out that for some some deployments, there exists a second,
duplicate, foreign key constraint for user_profile_id. The logic below
would try to rename both to the same name, which would fail on the
second:

```
psycopg2.errors.DuplicateObject: constraint "zerver_userpresenceo_user_profile_id_d75366d6_fk_zerver_us" for relation "zerver_userpresence" already exists
```

Eliminate the duplicate constraint, rather than attempting to rename
it.  Also add a block, in case of future reuse of this pattern, which
caveats that this approach will not work in the presence of
explicitly-named indexes.  UserPresence happens to not have any, so
this technique is safe in this instance.

Co-authored-by: Alex Vandiver <alexmv@zulip.com>
2023-06-05 17:34:17 -07:00
Zixuan James Li a5cc3c5d45 users: Use a less generic response for unauthorized user creation.
This reduces confusion when an admin user tries to create users.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2023-06-05 17:33:26 -07:00
Lalit 0ced086821 docs: Fix data type for `submessages` field on `message` type.
Before this commit our docs mentioned `string[]` data type for
`submessages` field on the `message` object. This commit changes the
type to `object[]` and correctly mentions all fields of the `submessage`
object.
2023-06-05 16:21:01 -07:00
dhruv302003 34e16c1553 integrations: Refactor Github Integration to use get_sender_name. 2023-06-05 12:01:27 -07:00
dhruv302003 67125acaa6 integrations: Add user url for star event in Github Integration.
Fixes #25672.
2023-06-05 12:01:27 -07:00
Mateusz Mandera a55901aa67 migrations: Fix bug in migration 0439.
This code clearly meant to return host and returning realm.host is a
mistake. realm.host is not accessible in a migration due to being a
@property-decorated method. The code constructs the host var value just
above this line.
2023-06-05 12:16:41 -04:00
Anders Kaseorg b5e5728112 coverage: Clean up coverage configuration.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-05-31 13:53:04 -07:00
Anders Kaseorg 2db9c0bc21 openapi: Remove unused prune_schema_by_type function.
It’s unused since commit a881918a05
(#24979).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-05-31 13:53:04 -07:00
Anders Kaseorg 0eff862459 openapi: Remove unused get_event_type function.
It’s unused since commit 8108acbdfd
(#16000).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-05-31 13:53:04 -07:00
Lauryn Menard bca7683f58 api-docs: Revisions for feature level 161 changes.
Revises the API changelog entry for feature level 161 to document
the changes to `DELETE /users/me/subscriptions` and to explain
more clearly what the new `can_remove_subscribers_group_id`
parameter does.

Updates the feature level 161 changes notes and related descriptions
to include links and also more clearly explain the updates.

Also, updates the `GET /user_groups` example to better reflect what
is returned for system groups since this is now referenced in the
`can_remove_subscribers_group_id` parameter description.

The original API feature level 161 API documentation changes were
made in commit c3759814be and commit 73f11853ec.
2023-05-31 08:16:22 -07:00
Lauryn Menard b0b3230917 scheduled-messages: Update scheduled_message_id parameter for DELETE.
Updates the scheduled_message_id parameter for deleting scheduled
messages to use the to_non_negative_int converter function for
validation, which is used in other endpoints/views with an ID in
the request path.
2023-05-31 08:15:03 -07:00
rht 1c84f02f57 slack import: Convert threads to nicely named Zulip topics.
Fixes #9006.
2023-05-30 16:35:19 -07:00
sbansal1999 2d6e6369f3 integrations: Add Linear webhook integration.
Fixes part of #23118.
2023-05-30 10:48:19 -07:00
Prakhar Pratyush dcb163bbe3 user_topics: Update API documentation.
This commit adds the missing 'UNMUTED' visibility policy
to the documentation for 'api/get-events' and 'api/register-queue'.

It replaces INHERIT with NONE for a clearer name
in the 'api/update-user-topic' documentation.

Other smaller changes in wording to improve readability.
2023-05-30 10:46:05 -07:00
Sahil Batra a143d6b62c message_edit: Add period to error message raised due to time limit. 2023-05-29 14:49:32 -07:00
Satyam Bansal caf6506811 integrations: Simplify regex checks in GitHub Integration.
The regex check is replaced with a simpler "startswith" function.
2023-05-29 10:55:31 -07:00
Tim Abbott dce4a3c98e markdown: Remove most of Twitter integration.
Twitter removed their v1 API. We take care to keep the existing cached
results around for now, and to not poison that cache, since we might
be able replace this with something that can still use the existing
cache.
2023-05-29 10:43:35 -07:00
Alya Abbott 3ca20e3ea8 help: Document quick way to check Zulip version.
Version number was added to top section of gear menu in
4df8c6610f.

Version number was added to "About Zulip" in
668b5137b0.
2023-05-29 08:39:01 -07:00
Mateusz Mandera db7a7d589c actions: Handle cross-realm messages in do_scrub_realm.
This is necessary to properly scrub all data from the realm.
2023-05-28 15:24:31 -07:00
Anders Kaseorg 66501a8eae requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-05-26 22:09:18 -07:00
Anders Kaseorg 9797de52a0 ruff: Fix RUF010 Use conversion in f-string.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-05-26 22:09:18 -07:00
Lauryn Menard 0c09a4fffe scheduled-messages: Add API documentation for new endpoint.
Part of splitting creating and editing scheduled messages.
Final commit. Should be merged with previous commits in series.

Updates the API documentation for the new endpoint for editing
scheduled messages.
2023-05-26 18:05:55 -07:00
Lauryn Menard 957382253a scheduled-messages: Split out edit scheduled message endpoint.
Part of splitting creating and editing scheduled messages.
Should be merged with final commit in series. Breaks tests.

Splits out editing an existing scheduled message into a new
view function and updated `edit_scheduled_message` function.
2023-05-26 18:05:55 -07:00
Lauryn Menard 154af5bb6b scheduled-messages: Remove ID from create scheduled message.
Part of splitting creating and editing scheduled messages.
Should be merged with final commit in series. Breaks tests.

Removes `scheduled_message_id` parameter from the create scheduled
message path.
2023-05-26 18:05:55 -07:00
Lauryn Menard 7af5ceb1c5 scheduled-messages: Add direct scheduled message to populate_db.
Prep commit for splitting create/edit endpoint for scheduled
messages.

Because of `test-api` runs the tests in alphabetical order based on
the `operationId`, we need two scheduled messages in the test database.
The first for the curl example delete (delete-scheduled-message) and
the second for the curl example update (update-scheduled-message).
2023-05-26 18:05:55 -07:00
Lauryn Menard 2f196bff19 api-docs: Add when moving messages fields/parameters were added.
Adds API changelog feature level 1 and associated Changes notes
for when the `stream_id` parameter in the `PATCH /messages/message_id`
was added, and for when the `prev_stream` field was added to edit
history information for messages.

We're adding these to the Zulip 3.0 feature level 1 because
commit 843345dfee that introduced this field and this parameter
to the server / backend code was merged before the commit that added
the API feature level tracking, commit e3b90a5ec8, at level 1.
2023-05-26 11:57:08 -07:00
Lauryn Menard bb735460e5 api-docs: Update avatar_url descriptions for feature level 163 change.
Updates the descriptions of the `avatar_url` field in message and
user objects to be clear that the current user must have access
to the other user's real email address in order for the value to
ever be `null`.

Also adds a bullet point to the API changelog feature level 163
entry about this change.
2023-05-26 11:51:05 -07:00
Lauryn Menard 389e45625a api-docs: Clarify if "email" is the "Zulip API email".
Clarifies additional areas of the API documentation where a user's
email is mentioned / used where it could be useful to clarify
that the email in question is the "Zulip API email".
2023-05-26 11:51:05 -07:00
Lauryn Menard 8c7e18fd75 api-docs: Use reference link in `/register-queue` main description. 2023-05-26 11:51:05 -07:00
Lauryn Menard ee58910f1a api-docs: Update `/get-own-user` to consistently use "requesting user". 2023-05-26 11:51:05 -07:00
Lauryn Menard 7a4565e833 api-changelog: Update feature level 163 entry and changes notes.
Original commits for this feature level's changes were part
of #22994.
2023-05-26 11:51:05 -07:00
Lauryn Menard 526fd7f24b api-docs: Add "delivery_email" field to example responses. 2023-05-26 11:51:05 -07:00
Lauryn Menard 58a64b33c9 api-docs: Revise main description for /get-user-by-email endpoint. 2023-05-26 11:51:05 -07:00
Lauryn Menard 87e9482672 api-docs: Update "Zulip display email" to "Zulip API email".
In some instances, the email was not clearly defined as
Zulip display email, but the Zulip API email is what is being
returned.
2023-05-26 11:51:05 -07:00
Sahil Batra 48e99657ad events: Remove realm_community_topic_editing_limit_seconds.
This commit removes realm_community_topic_editing_limit_seconds
field from register response since topic edit limit is now
controlled by move_messages_within_streams_limit_seconds
setting.
We also remove DEFAULT_COMMUNITY_TOPIC_EDITING_LIMIT_SECONDS
constant since it is no longer used.
2023-05-25 17:26:21 -07:00
Lauryn Menard e41c235657 api-docs: Link client_capabilities parameter in property descriptions.
In the register response properties deprecated at feature level 89,
update the descriptions to link to the client_capabilities parameter
when referenced.

Also, moves the enter_send property to be in the same section of the
register response as other properties deprecated at this feature level.

These descriptions were originally added to these properties in
commit e6f828a8e2.
2023-05-25 12:04:03 -07:00
Satyam Bansal cae02dbca4 integrations: Ignore merge queue push events for GitHub Integration.
Previosuly, the server would send a 500 whenever a merge queue push
event was encountered.

Fixes #25629.
2023-05-24 15:33:50 -07:00
Lauryn Menard 1beebb9fa1 api-changelog: Update feature level 168 entry and changes notes.
Original API changelog entry from commit ae72777c77.
2023-05-24 11:38:00 -07:00
Mateusz Mandera dcbcb05655 saml: Make SP-initiated SLO work after signup. 2023-05-23 13:01:15 -07:00
Mateusz Mandera 04f5358a76 tests: Flush session in verify_desktop_flow_end_page in social auth.
As the relevant comment elaborates - what happens next in the test in
simulating the step that happens in the desktop app. Thus a new session
needs to be used. Otherwise, the old session created normally in the
browser pollutes the state and can give falsely passing tests.

This should be happening for all social auth tests using this, not just
in that one SAML test, thus moving it inside the helper method.
2023-05-23 13:01:15 -07:00
Mateusz Mandera 8fb0fe96c6 saml: Save SessionIndex in session and use when making a LogoutRequest.
This is a useful improvement in general for making correct
LogoutRequests to Idps and a necessary one to make SP-initiated logout
fully work properly in the desktop application. During desktop auth
flow, the user goes through the browser, where they log in through their
IdP. This gives them a logged in  browser session at the IdP. However,
SAML SP-initiated logout is fully conducted within the desktop
application. This means that proper information needs to be given to the
the IdP in the LogoutRequest to let it associate the LogoutRequest with
that logged in session that was established in the browser. SessionIndex
is exactly the tool for that in the SAML spec.
2023-05-23 13:01:15 -07:00
Mateusz Mandera 5dd4dcdebb saml: Make SP-initiated SLO work in the desktop application. 2023-05-23 13:01:15 -07:00
Mateusz Mandera 3f55c10685 saml: Rework SP-initiated logout config to support IdP-level config.
This gives more flexibility on a server with multiple organizations and
SAML IdPs. Such a server can have some organizations handled by IdPs
with SLO set up, and some without it set up. In such a scenario, having
a generic True/False server-wide setting is insufficient and instead
being able to specify the IdPs/orgs for SLO is needed.
2023-05-23 13:01:15 -07:00
Mateusz Mandera 0bb0220ebb saml: Implement SP-initiated Logout.
Closes #20084

This is the flow that this implements:
1. A logged-in user clicks "Logout".
2. If they didn't auth via SAML, just do normal logout. Otherwise:
3. Form a LogoutRequest and redirect the user to
https://idp.example.com/slo-endpoint?SAMLRequest=<LogoutRequest here>
4. The IdP validates the LogoutRequest, terminates its own user session
and redirects the user to
https://thezuliporg.example.com/complete/saml/?SAMLRequest=<LogoutResponse>
with the appropriate LogoutResponse. In case of failure, the
LogoutResponse is expected to express that.
5. Zulip validates the LogoutResponse and if the response is a success
response, it executes the regular Zulip logout and the full flow is
finished.
2023-05-23 13:01:15 -07:00
Mateusz Mandera dda4603f94 auth: Extract EXPIRABLE_SESSION_VAR_DEFAULT_EXPIRY_SECS. 2023-05-23 13:01:15 -07:00
Lauryn Menard eba83b9d87 api-docs: Expand `/update-message` main description for realm settings.
Expands the main description for the `/update-message` documentation
to include a list of the realm settings in the API that are relevant
to when users can update a message's content, topic or stream.
2023-05-23 11:11:20 -07:00
Lauryn Menard 8ec9abe00d api-changelog: Clarify feature level 159 descriptive text.
Original API changelog entries: commit 02eee3a04f and
commit 815bf609fa.
2023-05-23 11:11:20 -07:00
Lauryn Menard e4ae8b79c3 api-changelog: Clarify feature level 162 descriptive text.
Original API changelog entries: commit 73f0eae394 and
commit 2c4e076fef and commit 891f83601d.
2023-05-23 11:11:20 -07:00
Lauryn Menard 0e9c34a975 api-changelog: Clarify feature level 172 descriptive text.
Original API feature level entries: commit 440f9e397a and
commit bd7f728796.
2023-05-23 11:11:20 -07:00
Lauryn Menard 6d6a335e32 api-docs: Make realm_linkifiers current API clear in description.
Adjusts the descriptions of realm_linkifiers (and deprecated
realm_filters) events and register response fields so that the
description of the current API is complete without the feature
level 176 **Changes** notes.
2023-05-22 15:20:07 -07:00
Lauryn Menard fd02648b0e api-changelog: Update feature level 175 entry and related changes notes. 2023-05-22 13:21:03 -07:00
Lauryn Menard 3cb6c9aea9 api-docs: Add examples to `realm_filters` event prose description.
Adds examples of the regex pattern and old URL string format to
the deprecated `realm_filters` event and register response field.
The examples are in the prose description since the events are
no longer sent and therefore no longer tested.
2023-05-19 13:23:07 -07:00
Lauryn Menard 03a2c2da6a api-changelog: Update and clarify docs for feature level 176 entry.
Revises API changelog entry for missing endpoint method and to
clarify the overall text.

Updates Changes notes for feature level 176 to not have repetitive
text, so that the updates were clearer and more concise.

The original commit with the changes related to this API changelog
entry is commit 268f858f39.
2023-05-19 13:23:07 -07:00
Sahil Batra 4c4caa7be4 CVE-2023-32677: Check permission to subscribe other users in invites.
This commit updates the API to check the permission to subscribe other
users while inviting.  The API will error if the user passes the
"stream_ids" parameter (even when it contains only default streams)
and the calling user does not having permission to subscribe others to
streams.

For users who do not have permission to subscribe others, the
invitee will be subscribed to default streams at the time of
accepting the invite.

There is no change for multiuse invites, since only admins are allowed
to send them, and admins always have the permission to subscribe
others to streams.
2023-05-19 16:13:32 -04:00
Mateusz Mandera a23b077b79 CVE-2023-28623: Prevent unauthorized signup with ldap + external auth.
Since 74dd21c8fa in Zulip Server 2.1.0, if:
- ZulipLDAPAuthBackend and an external authentication backend (any aside
  of ZulipLDAPAuthBackend and EmailAuthBackend) are the only ones
  enabled in AUTHENTICATION_BACKENDS in /etc/zulip/settings.py
- The organization permissions don't require invitations to join

...then an attacker can create a new account in the organization with
an arbitrary email address in their control that's not in the
organization's LDAP directory.

The impact is limited to installations which have the specific
combination of authentication backends described above, in addition to
having the "Invitations are required for joining this organization
organization" permission disabled.
2023-05-19 16:13:00 -04:00
Mateusz Mandera 3ed2a30e01 maybe_send_to_registration: Remove password_required arg.
This argument was added with the default incorrectly set to `True` in
bb0eb76bf3 - despite
`maybe_send_to_registration` only ever being called in production code
in a single place, with `password_required=False` explicitly. And then
it just got carried forward through refactors.

`maybe_send_to_registration` was/is also called twice in tests, falling
back to the default, but the `password_required` value is irrelevant to
the tests - and if anything letting it use the `True` has been wrong,
due to not matching how this function is actually used.
2023-05-19 16:13:00 -04:00
Mateusz Mandera b55adbef3d export: Handle RealmAuditLog with .acting_user in different realm. 2023-05-19 11:12:19 -07:00
Alex Vandiver c978bfaa32 models: Add a unique index on UserProfile.api_key.
This prevents `get_user_profile_by_api_key` from doing a sequential
scan.

Doing this requires moving the generation of initial api_key values
into the column definition, so that even bare calls to
`UserProfile.objects.create` (e.g. from tests) call appropriately
generate a random initial value.
2023-05-19 11:11:04 -07:00
Lauryn Menard 145d6915c1 api-changelog: Add 2.1.0 entry for realm_default_external_accounts.
Adds an API changelog note to 2.1 for the addition of
realm_default_external_accounts to the `/register-queue` response.

Also adds a Changes note to the field in the endpoint's response
API documentation.

The original commit that added it to that endpoint's response was
commit d7ee2aced1.
2023-05-19 10:50:17 -07:00
Alex Vandiver b312a86ada sentry: Add the observed user's IP address before forwarding.
The default for Javascript reporting is that Sentry sets the IP
address of the user to the IP address that the report was observed to
come from[^1].  Since all reports come through the Zulip server, this
results in all reports being "from" one IP address, thus undercounting
the number of affected unauthenticated users, and making it difficult
to correlate Sentry reports with server logs.

Consume the Sentry Envelope format[^2] to inject the submitting
client's observed IP address, when possible.  This ensures that Sentry
reports contain the same IP address that Zulip's server logs do.

[^1]: https://docs.sentry.io/platforms/python/guides/logging/enriching-events/identify-user/
[^2]: https://develop.sentry.dev/sdk/envelopes/
2023-05-18 16:25:54 -07:00
Lauryn Menard 5802f7775f api-changelog: Update and clarify docs for feature level 178 entry.
Updates the descriptions and examples for there only being two key
values: "website" and "aggregated".

Also, clarifies that email keys are the Zulip display email.

And removes any descriptive text that says presence objects have
information about the clients the user is logged into.
2023-05-18 11:57:02 -07:00
Alex Vandiver 1184bdc934 push_notifications: Lock message while we mark it pending for push.
Deleting a message can race with sending a push notification for it.
b47535d8bb handled the case where the Message row has gone away --
but in such cases, it is also possible for `access_message` to
succeed, but for the save of `user_message.flags` to fail, because the
UserMessage row has been deleted by then.

Take a lock on the Message row over the accesses of, and updates to,
the relevant UserMessage row.  This guarantees that the
message's (non-)existence is consistent across that transaction.

Partial fix for #16502.
2023-05-18 11:53:21 -07:00
Anders Kaseorg 4b19863065 test_timeout: Skip test_timeout_warn on Python 3.11 for coverage issue.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-05-18 11:52:22 -07:00
Lauryn Menard 2f96b1756e api-changelog: Add notes for feature level 3 `zulip_version` change.
Original API changelog note was made in commit 2c63130195.
2023-05-18 08:51:46 -07:00
Lauryn Menard 3023091caa scheduled-messages: Add help center link to fetch / delete descriptions. 2023-05-18 08:45:51 -07:00
Lauryn Menard 67e1889817 scheduled-messages: Add explicit test with emails for direct message. 2023-05-18 08:45:51 -07:00
Lauryn Menard a2a50e1408 scheduled-messages: Simplify message_to type in check_schedule_message.
As of commit 38f6807af1, we accept only stream and user IDs for
the recipient information for scheduled messages, which means we
can simplify the type for `message_to` in `check_schedule_message`.
2023-05-18 08:45:51 -07:00
Lauryn Menard 1ad0ad8ece tests-scheduled-messages: Use "direct" instead of "private". 2023-05-18 08:45:51 -07:00
Lauryn Menard 8f4de3afc0 api-docs: Update create/edit scheduled message errors for IDs.
In commit 38f6807af1, we updated the `POST /scheduled_messages`
endpoint to only accept user IDs for direct messages. The endpoint
alread only accepted a stream ID for stream messages.

But the API documentation was not updated for the errors returned
when either a stream or user with the specified ID does not exist.

Updates the API documentation for the correct error responses.
2023-05-16 15:14:54 -07:00
Alex Vandiver 24c3e25f86 middleware: Redirect non-canonical realm domain names.
If a host is in REALM_HOSTS, it has its own domain name.  Redirect
access from other domain names to that name.
2023-05-16 15:13:51 -07:00
Alex Vandiver 724de9cd49 rocketchat: Treat users with "bot" roles as bots when importing.
We previously relied on `type`, but we have observed bots typed with a
`bot` role as well.
2023-05-16 15:10:58 -07:00
Alex Vandiver 34394cec9a rocketchat: Handle users with no email address set.
Fixes: #25596.
2023-05-16 15:10:58 -07:00
Alex Vandiver 3160c3cce0 realm_export: Return export id from POST which create it. 2023-05-16 14:05:01 -07:00
Alex Vandiver 7811e99548 realm_export: Handle hard head-of-queue failures.
Realm exports may OOM on deployments with low memory; to ensure
forward progress, log the start time in the RealmAuditLog entry, and
key off of the existence of that to prevent re-attempting an export
which was already tried once.
2023-05-16 14:05:01 -07:00
Alex Vandiver 4a43856ba7 realm_export: Do not assume null extra_data is special.
Fixes: #20197.
2023-05-16 14:05:01 -07:00
Alex Vandiver 5eeb616666 realm_export: Test failures during export. 2023-05-16 14:05:01 -07:00
Alex Vandiver 362177b788 workers: Run realm export with one thread if in low-memory environment.
We previously hard-coded 6 threads for the realm export; in low-memory
environments, spawning 6 threads for an export can lean to an OOM,
which kills the process and leaves a partial export on disk -- which
is then tried again, since the export was never completed.  This leads
to excessive disk consumption and brief repeated outages of all other
workers, until the failing export job is manually de-queued somehow.

Lower the export to only use on thread if it is already running in a
multi-threaded environment.  Note that this does not guarantee forward
progress, it merely makes it more likely that exports will succeed in
low-memory deployments.
2023-05-16 14:05:01 -07:00
Alex Vandiver 9f231322c9 workers: Pass down if they are running multi-threaded.
This allows them to decide for themselves if they should enable
timeouts.
2023-05-16 14:05:01 -07:00
Alex Vandiver 5329fed387 test_queue_worker: Do a full recursion on subclasses.
This makes it less likely we will accidentally fail to include a class
if the subclassing of QueueProcessingWorker changes, and lets mypy
more accurately understand the typing.
2023-05-16 14:05:01 -07:00
Sahil Batra 007a51f277 accounts: Allow user to change email visibility during first login.
We now allow users to change email address visibility setting
on the "Terms of service" page during first login. This page is
not shown for users creating account using normal registration
process, but is useful for imported users and users created
through API, LDAP, SCIM and management commands.
2023-05-16 13:52:56 -07:00
Sahil Batra 7f01b3fb63 users: Set tos_version to -1 for users who have not logged-in yet.
We now set tos_version to "-1" for imported users and the ones
created using API or using other methods like LDAP, SCIM and
management commands. This value will help us to allow users to
change email address visibility setting during first login.
2023-05-16 13:52:56 -07:00
Lauryn Menard 8d06fa7e38 api-docs: Update descriptions for display_emoji_reaction_users. 2023-05-16 12:31:03 -07:00
Lauryn Menard 8962565154 api-docs: Fix capitalization in descriptions with int values lists. 2023-05-16 12:31:03 -07:00
Lauryn Menard 239458a173 api-docs: Remove instances of "=>" in API documentation descriptions. 2023-05-16 12:31:03 -07:00
Lauryn Menard 63d51e8114 api-docs: Clarify uses of "=" in API documentation descriptions.
For cases of `name=value` in descriptions in the API documentation,
update to either use JSON style of `"name": value` when correct or
revise the descriptive text.
2023-05-16 12:31:03 -07:00
Lauryn Menard 09ab1be34f api-docs: Use backticks for descriptions referencing "null" values. 2023-05-16 12:31:03 -07:00
Lauryn Menard c43ea96c06 api-docs: Clean up instances of "None" that should be "null". 2023-05-16 12:31:03 -07:00
Lauryn Menard 4fcb243769 api-docs: Standardize on lowercase true, false and null.
Creates a custom linter rule for `zerver/openapi/zulip.yaml` to
only allow lowercase versions of "true", "false" and "null".

Updates existing documentation for new rules.
2023-05-16 12:31:03 -07:00
Alex Vandiver ffaccb8af2 outgoing_webhook: Respect settings.OUTGOING_WEBHOOK_TIMEOUT_SECONDS.
The use of the setting was accidentally removed in b88d7a741e, and
replaced with a static 10 seconds.
2023-05-16 07:00:37 -07:00
David Rosa be14ec2cab help: Rename "Starting a new private thread" to "... new direct message".
With the private messages -> direct messages migration, we should
rename the "Starting a new private thread" help center article.

- Renames article to "Starting a new direct message"
- Updates relevant section in /help/getting-started-with-zulip
- Fixes typo in /help/send-group-dm
- Updates file names and adds URL redirect.

Fixes #25506.
2023-05-15 16:13:55 -07:00
Alex Vandiver ca2ca030d2 migrations: Backfill missing RealmAuditLog entries for subscriptions.
Backfill subscription realm audit log SUBSCRIPTION_CREATED events for
users which are currently subscribed but don't have any subscription
events, presumably due to some historical bug.  This is important
because those rows are necessary when reactivating a user who is
currently soft-deactivated.

For each stream, we find the subscribed users who have no
subscription-related realm audit log entries, and create a
`backfill=True` subscription audit log entry which is the latest it
could have been, based on UserMessage rows.  We then optionally insert
a `DEACTIVATION` if the current subscription is not active.
2023-05-15 16:09:44 -07:00
Aman Agrawal c2145a8993 scheduled_message: Send `update` event on failed delivery. 2023-05-14 16:46:20 -07:00
Aman Agrawal 4dd32a4d85 scheduled_messages: Extract notify_update_scheduled_message. 2023-05-14 16:46:20 -07:00
Ujjawal Modi 2a6146110c subscriptions: Change in API used for adding new subscriptions.
Earlier when a user who is not allowed to add subscribers to a
stream because of realm level setting "Who can add users to streams"
is subscribing other users while creating a new stream than new stream
was created but no one is subscribed to stream.

To fix this issue this commit makes changes in the API used
for adding subscriptions. Now stream will be created only when user
has permissions to add other users.

With a rewrite of the test by Tim Abbott.
2023-05-14 11:19:05 -07:00
Ujjawal Modi a47569bf47 backend_tests: Add a test for subsribing others to public streams.
Earlier there was no backend test for subscribing others to
public streams in zephyr realm.

This commit adds a backend test for it.
2023-05-14 11:19:04 -07:00
Tim Abbott 2d3d3f6072 message_send: Don't mark scheduled messages to self as read.
The only reasonable intent for such a scheduled message is to remind
oneself of something at that time, which requires it being unread.

Fixes #25523.
2023-05-12 17:55:46 -07:00
Tim Abbott daf6fb17bf do_send_messages: Use mandatory kwargs pattern. 2023-05-12 17:55:46 -07:00
Mateusz Mandera 254ea4b0c8 social_auth: Save authentication method information in the session.
The immediate application of this will be for SAML SP-initiated logout,
where information about which IdP was used for authenticating the
session needs to be accessed. Aside of that, this seems like generally
valuable session information to keep that other features may benefit
from in the future.
2023-05-12 16:21:26 -07:00
Mateusz Mandera af9d1a7dfb register_remote_user: Use explicit kwargs list.
This is nicer that .pop()ing specified keys - e.g. we no longer will
have to update this chunk of code whenever adding a new key to
ExternalAuthDataDict.
2023-05-12 16:21:25 -07:00
Lauryn Menard 90cc2716f0 scheduled-messages: Update API dicts for `failed` boolean field.
Adds the `failed` boolean from the ScheduledMessage to the API dict
returned by scheduled message events and register response, and by
fetching the user's scheduled messages.

`failed` will only be true when the server has tried to send the
scheduled message and failed due to an error.
2023-05-12 15:48:59 -07:00
Lauryn Menard c7c67c01ce scheduled-messages: Update failure to send message fields for edits.
In the case of a user editing a scheduled message that the server
had failed to send at the scheduled time due to an error, we want
to update the `failed` and `failure_message` fields as the intent
is for the server to retry to send the scheduled message based on
the updated information provided by the user.
2023-05-12 15:48:59 -07:00
Lauryn Menard cc648a2c19 scheduled-messages: Send notification if send scheduled message fails.
In the case that there is an error when sending a scheduled message,
we now send a message from the notification bot to the user who
scheduled the message about the failure/error.

The notification message is not sent if the error when sending the
scheduled message was due to the realm or sender being deactivated.
2023-05-12 15:48:59 -07:00
Alex Vandiver a2ed0302ce streams: Prevent already-deactivated streams from being deactivated. 2023-05-12 13:26:43 -07:00
Alex Vandiver 5be7bc58fe upload: Use content_disposition_header from Django 4.2.
The code for this was merged in Django 4.2:
https://code.djangoproject.com/ticket/34194
2023-05-11 14:51:28 -07:00
Sahil Batra ddb0bb58ed tests: Add tests to update visibility policy when target topic is empty.
This commit adds a new test to check how the visibility policy updates
when moving messages to a topic that didn't exist previously.

This test also helps us adding coverage for the code which just
skips setting visibility_policy if there is no need to update the
value because both previous and new value of visibility policy
is INHERIT. The "actions/message_edit.py" file has 100% coverage
now and thus is removed from "not_yet_fully_covered" list.
2023-05-11 12:13:50 -07:00
Sahil Batra d645d5c0ec message_edit: Fix code to set visibility policy on moving messages.
The code for updating visibility policy values on moving messages
had two bugs.

- There was a typo in elif condition where "user_profile" was being
used instead of "user_profile_with_policy".

This commit fixes the typo.

- It was assumed that there would be no UserTopic rows for target
topic if the target topic didn't exist. But there can be such case
where some messages were sent to that topic and the user muted
the topic. But then the messages in that topic was deleted. In
such case there can be UserTopic rows for a stream-topic pair
that does not exist.

This commit fixes the code to handle such case as well and set
the visibility policy of new topic to what was set for the original
topic. This change simplifies the condition to just check whether
new_visibility_policy is equal to target_topic_visibility_policy
and skip if so, and update the visibility policy otherwise.

Due to this change, we now do not try to mute the already muted
topic if the topic is moved to a topic which didn't exist
previously and thus we modify the existing test to not expect
any INFO logs.
2023-05-11 12:13:50 -07:00
Sahil Batra a2600a2b97 tests: Add coverage to actions/message_edit.py.
This commit adds tests to cover the case of message editing
not allowed due to allow_message_editing set to False and
the case when there is no limit set when moving all messages
in a topic.

The "actions/message_edit.py" file does not have 100% coverage
still and it will be addressed in the next commit.
2023-05-11 12:13:50 -07:00
Sahil Batra 9fa67f0fa9 tests: Add coverage to actions/create_realm.py.
This commit adds test coverage to actions/create_realm.py.
The file is also removed from not_yet_fully_covered list
since it has 100% coverage now.
2023-05-11 12:13:50 -07:00
Sahil Batra 5e7d49d129 create_realm: Remove unused argument from do_create_realm.
We do not pass "email_address_visibility" to do_create_realm
anymore. It was passed before to set the setting for realms in
development database, but it has been changed since we changed
email_address_visibility to be a user-level setting instead
of realm-level setting since now it is set on RealmUserDefault
table.
2023-05-11 12:13:50 -07:00
Alex Vandiver 6b58f9f9fa integrations: Add support for MovieAdded Radarr hook. 2023-05-11 12:08:25 -07:00
Alex Vandiver 1d5bccf4c3 integrations: Add support for MovieFileDelete Radarr hook. 2023-05-11 12:08:25 -07:00
Alex Vandiver 79c1123700 validator: Generalize type of check_string_in argument. 2023-05-11 12:08:25 -07:00
Alex Vandiver 01241f0e72 integrations: Add support for MovieDelete Radarr hook. 2023-05-11 12:08:25 -07:00
Alex Vandiver 84b440f0e4 integrations: Add support for ApplicationUpdate Radarr hook. 2023-05-11 12:08:25 -07:00
Alex Vandiver 5931787f11 integrations: Support scans of untagged images in Harbor. 2023-05-11 11:42:54 -07:00
Alex Vandiver 6e842f54c1 webhooks: Add support for gmail and rules sources to Front integration. 2023-05-11 11:36:58 -07:00
Lauryn Menard 1d209220dd tests: Add coverage for error when editing a sent scheduled message.
Adds test coverage for the error sent for editing a scheduled
message that was successfully sent.

`zerver/actions/scheduled_messages.py` now has 100% test coverage
again.
2023-05-11 10:52:01 -07:00
Anders Kaseorg e88b2caeef Revert "db: Force use of TimeTrackingCursor to work around Django 4.2 bug."
This reverts commit f1925487e8.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-05-10 19:44:47 -07:00
Anders Kaseorg d0481be3e5 requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-05-10 19:44:47 -07:00
Tim Abbott 2aae32f642 events: Fix apply_events when raw_unread_msgs not present.
We were missing a few checks for raw_unread_msgs being present before
trying to parse and update it.

The test only covers 2/3 of the cases, but I wasn't convinced it was
worth adding another test just for the corner case of removing a
message flag; this seems fairly unlikely to regress.
2023-05-10 13:44:35 -07:00
Alya Abbott a9d0656605 help: Document stream/DM switcher on "Mastering the compose box" page. 2023-05-10 11:37:36 -07:00
Sahil Batra 4ca887bade invites: Allow users to invite without specifying any stream to join.
We now allow users to invite without specifying any stream to join.
In such cases, the user would join the default streams, if any, during
the process of account creation after accepting the invite.

It is also fine if there are no default streams and user isn't
subscribed to any stream initially.
2023-05-09 17:05:17 -07:00
Sahil Batra 875bcb183e invites: Do not add user to default streams if streams list is empty.
We do not add user to the default streams if the streams list passed
while sending the invite (both email and multi-use) was empty since
invite explicitly selected to not subscribe the user to default
streams.
2023-05-09 16:59:06 -07:00
Sahil Batra ded6b7ede4 test_invite: Add test to verify stream subscriptions.
This commit adds test to verify whether the user was
subscribed to the streams selected when inviting them.
2023-05-09 16:59:06 -07:00
Sahil Batra aebf7bf0e1 test_invite: Fix typo in comment. 2023-05-09 16:59:06 -07:00
Aman Agrawal f025415f2b scheduled_messages: Add tests for delivery of scheduled messages. 2023-05-09 13:48:28 -07:00
Tim Abbott 602e4c2aa3 scheduled_messages: Add focused scheduled message indexes. 2023-05-09 13:48:28 -07:00
Tim Abbott 835f62617e scheduled_messages: Store the final delivered message ID.
This could be useful for debugging problems with the system
operationally.
2023-05-09 13:48:28 -07:00
Tim Abbott fe9656237e scheduled_message: Move remaining logic out of management command.
While this doesn't include tests yet, it will make it much easier to
write tests for the core server logic here.
2023-05-09 13:48:28 -07:00
Tim Abbott 20be0759b5 scheduled_message: Avoid delivering scheduled messages late. 2023-05-09 13:48:28 -07:00
Tim Abbott a065576db7 scheduled messages: Call check_message at send time. 2023-05-09 13:48:28 -07:00
Tim Abbott 7051d3416b scheduled_messages: Add reasonable failure handling.
Previously, it seemed possible for the scheduled messages API to try
to send infinite copies of a message if we had the very poor luck of a
persistent failure happening after a message was sent.

The failure_message field supports being able to display what happened
in the scheduled messages modal, though that's not exposed to the API
yet.
2023-05-09 13:48:28 -07:00
Tim Abbott 147e296e0a scheduled_messages: Add defensive assert for double sends. 2023-05-09 13:48:28 -07:00
Tim Abbott 24bb03c89f scheduled_messages: Add logging on successful send. 2023-05-09 13:48:28 -07:00
Tim Abbott 1fc6ed224a scheduled_messages: Remove batching logic.
The previous logic would attempt to send a large number of unrelated
messages in a single transaction, which is just asking for trouble in
the event that one of the attempts fails.
2023-05-09 13:48:28 -07:00
Tim Abbott d1dfab5b70 scheduled_messages: Live update #scheduled view after sending.
We apparently had neglected to send the actual event to clients that
the message was successfully sent.
2023-05-09 13:48:28 -07:00
Tim Abbott c8f3443fb6 schedules_messages: Move logic out of management command. 2023-05-09 13:48:28 -07:00
Tim Abbott 611cee177d deliver_scheduled_messages: Rename misleading message variable. 2023-05-09 13:48:28 -07:00
Alex Vandiver 1285c39098 zerver: Remove now-unused report/ endpoints. 2023-05-09 13:16:28 -07:00
Lauryn Menard e44520c8fe scheduled-messages: Limit `to` parameter to user and stream IDs.
For scheduled stream messages, we already limited the `to`
parameter to be the stream ID, but here we return a JsonableError
in the case of a ValueError when the passed value is not an integer.

For scheduled direct messages, we limit the list for the `to`
parameter to be user IDs. Previously, we accepted emails like
we do when sending messages.
2023-05-09 12:45:11 -07:00
sbansal1999 63fcfd543a integrations: Add documentation for Notion-Zapier Integration.
Fixes #25314.
2023-05-09 12:25:18 -07:00
sbansal1999 6e2423fe64 integrations: Update text in Zapier Documentation.
This commit updates the text to match the current version
displayed on the website.
2023-05-09 12:25:18 -07:00
Lauryn Menard df17a1bf28 test-events: Test do_delete_message with no messages specified.
Test coverage for `zerver/actions/message_delete.py`.

Both callers of this function would already return if there were
no Messages specified to delete, which is why existing tests did
not cover this.
2023-05-09 09:58:33 -07:00
Alex Vandiver 89d4737ab2 message_edit: Do not reuse user_profile variable in "for" loop.
Doing so causes the "username resolved this topic" or "this topic was
moved by username" notifications to be attributed to a random user who
had a visibility policy on the topic.
2023-05-09 09:38:38 -07:00
Lauryn Menard 02fafb0376 models: Update the references for API dicts for scheduled messages. 2023-05-09 07:36:05 -07:00
Mateusz Mandera 682a4d6f9e import_realm: Improve comment about sequencing of imported models.
The comment was outdated, currently we import UserProfiles before
realm_tables - because some models in realm_tables have a dependency on
UserProfile.

Also makes sense to elaborate a bit more in the comment that it's just
an outline of the ordering, not an exhaustive list.
2023-05-08 15:55:06 -07:00
Mateusz Mandera 0abf60fd01 scheduled_message: Make export/import work.
Closes #25130 by addressing the import/export part of it.
2023-05-08 15:55:06 -07:00
Mateusz Mandera 780ef71891 export: Fix typo in variable name. 2023-05-08 15:55:06 -07:00
Mateusz Mandera 414658fc8e scheduled_message: Handle attachments properly.
Fixes #25414.

We add Attachment.scheduled_messages relation to track ScheduledMessages
which reference the attachment.

The import bits can be done after merging this, by updating #25345.
2023-05-08 09:56:02 -07:00
Mateusz Mandera 4598607a46 test_uploads: Fix two typos. 2023-05-08 09:56:02 -07:00
sbansal1999 2a3d4041e0 integrations: Improve GitHub force push notifications.
Previously, we didn't explicitly display something special for force-pushes.

Fixes #21969.
2023-05-04 14:31:25 -07:00
sbansal1999 dba6f84b97 integrations: Add force push fixtures for GitHub Integration.
One fixture is just removing commits; the other removes some commits
but adds others.
2023-05-04 14:31:25 -07:00
Mateusz Mandera d0e9a77a57 migrations: Fix performance of migration 0436.
This was doing bulk_create in a loop for each realm, which is too slow
on very large servers. Just do a single bulk_create with a reasonable
batch_size at the end.
2023-05-03 11:09:23 -07:00
Tim Abbott 9bdb044ff4 i18n: Update translation data from Transifex. 2023-05-02 13:16:25 -07:00
Lauryn Menard b5c4064b7b api-docs: Clarify topic parameter for create-or-edit-scheduled-message. 2023-05-02 12:46:41 -07:00
Lauryn Menard a5b527f321 onboarding: Specialize Welcome Bot message for education organizations.
Because education organizations and users have slightly specialized
use cases, we update the Welcome Bot message content sent to new
users and new organization owners for these types of organizations
to link to help center articles/guides geared toward these users
and organizations.

Also, updates the demo organization warning to only go to the new
demo organization owner because the 30 day deletion text is only
definitely accurate when the organization is created.

Fixes #21694.
2023-05-01 16:48:48 -07:00
Lauryn Menard 06dd7a3a68 emails: Add corporate_enabled to context for emails.
In commit fc58c35c0, we added a check in various emails for the
settings.CORPORATE_ENABLED value, but that context is only always
included for views/templates with a request.

Here we add that to common_context, which is often used when there
is not a request (like with emails). And we manually add it to the
email context in various cases when there is not a user account to
call with common_context: new user invitations, registration emails,
and realm reactivation emails.
2023-05-01 10:32:43 -07:00
Lauryn Menard d41f12495d api-docs: Document create/edit scheduled message endpoint and events.
Adds API changelog update for the new endpoint added for creating
or editing scheduled messages, for the events sent for scheduled
messages (add, update, delete), and for the `scheduled_messages`
field in the register response.

Adds a specific API markdown documentation file for the create
or update scheduled messages endpoint, so that the parameters in
the cURL examples are accurate for the type of message (direct or
stream) and action (create or edit).

Bumps the API feature level, adds Changes notes to API documentation
and adds the scheduled message endpoints to the API documentation
sidebar.
2023-04-28 17:25:00 -07:00
Lauryn Menard 15c6d67e9c populate-db: Add scheduled message to test database.
Prep commit for adding the scheduled-message endpoints to the API
documentation.

Adds a scheduled message for Iago in the test database so that it
can be deleted in the delete cURL example in the api-test suite.
2023-04-28 17:25:00 -07:00
Aman Agrawal bd2545b0d7 scheduled_message: Send CRUD events to clients. 2023-04-28 17:25:00 -07:00
Aman Agrawal f40855bad2 reminder: Remove feature from zulip.
This is being removed to make the code simpler. We have plans
to add it as a feature in the future , but it will most likely
not use the same code.
2023-04-28 17:25:00 -07:00
Aman Agrawal d60d6e9115 urls: Add new endpoint to create scheduled messages.
This will help us remove scheduled message and reminder logic
from `/messages` code path.

Removes `deliver_at`/`defer_until` and `tz_guess` parameters. And
adds the `scheduled_delivery_timestamp` instead. Also updates the
scheduled message dicts to return `scheduled_delivery_timestamp`.

Also, revises some text in `/delete-scheduled-message` endpoint
and in the `ScheduledMessage` schema in the API documentation.
2023-04-28 17:25:00 -07:00
Lauryn Menard 7739703111 scheduled-messages: Update scheduled message objects in the API for type.
Updates the objects in the API for scheduled messages so that those
for stream messages return the `to` property as an integer since it
is always the unique stream ID and so that those for direct messages
do not have a `topic` property since direct messages never have a
topic.

Also makes small update so that web app scheduled messages overlay
has the correct stream ID.
2023-04-28 17:25:00 -07:00
Aman Agrawal 4718eaa213 scheduled_messages: Extract method to get undelivered scheduled messages.
This also changes key for `ID` of scheduled message from `message_id`
to `scheduled_message_id`.
2023-04-28 17:25:00 -07:00
Aman Agrawal 4cb238fb6e models: Add method to convert ScheduleMessage objects into dicts. 2023-04-28 17:25:00 -07:00
Aman Agrawal 963fe566d7 scheduled_messages: Use scheduled_message_id instead of message_id.
Using `message_id` can be confusing for API users since it can be
mistaken for the ID of the message that will be sent.
2023-04-28 17:25:00 -07:00
Aman Agrawal 7bf0793c94 scheduled_messages: Move database related function to /actions.
This would match the pattern we use for other modules and also
shrink the giant message_send.py.
2023-04-28 17:25:00 -07:00
Aman Agrawal cddf25656f test_message_send: Move out scheduled message tests. 2023-04-28 17:25:00 -07:00
Alex Vandiver 4f2417cfc4 soft_reactivation: Add a partial index to speed up event lookups.
The full auditlog table is moderately large, and the previously-chosen
index (on `modified_user_id`) is not terribly specific.
2023-04-28 12:43:34 -07:00
Alex Vandiver a56da4be76 soft_deactivation: Only fetch necessary columns.
Existing tests verify that this does not add more queries.
2023-04-28 12:43:34 -07:00
Alex Vandiver ae7485a96e soft_deactivation: Do not bother to fetch stream data as well.
This prefetch is unnecessary and makes this query load more data than
needed.

Existing tests verify that this does not add more queries.
2023-04-28 12:43:34 -07:00
Tim Abbott a03dca93ca message_edit: Clarify losing_access_user_ids calculation.
This was previously called delete_event_notify_user_ids, which seemed
to narrow its purpose in a way that was confusing given that it's also
used for other calculations.

Further, calculate it as soon as we know it, not when we're first
going to use it.
2023-04-27 18:35:32 -07:00
Tim Abbott 71e57d2022 message_edit: Clarify name for gaining_usermessage_user_ids.
This list contains user IDs, not subscription objects.
2023-04-27 18:35:32 -07:00
Tim Abbott 3ff361a1f0 message_edit: Use sets for old/new stream IDs.
This fixes a quadratic performance issue iterating through these
lists. Give these variables slightly better names while we're at it.
2023-04-27 18:35:32 -07:00
Prakhar Pratyush c8a9c0ee04 realm_redirect: Redirect always to the login page with the next parameter.
Previously, entering an organization via 'accounts/go' with the
web-public stream enabled took the user to the web-public view
even if the user was not logged in.

Now, a user is always redirected to the 'login_page' with
the next parameter, if present.

The 'login_page' view is updated to redirect an authenticated
user based on the 'next' parameter instead of always redirecting
to 'realm.uri'.

Fixes #23344.
2023-04-27 16:50:10 -07:00
Sahil Batra afc5066e36 registration: Fix "Resend" link not working for realm creation.
The "Resend" link for realm creation was not working correctly
because it is implemented by basically submiting the registration
form again which results in resending the email but all the
required parameters were not passed to the form after recent
changes in the realm creation flow.

This commit fixes it by passing all the required parameters -
email, realm name, realm type and realm subdomain, when submitting
form again by clicking on the "resend" link.

Fixes #25249.
2023-04-27 12:28:37 -07:00
Sahil Batra f8f4fa4c5e tests: Extract realm name and string_id values in variables.
This is a prep commit so that we can use these variables to
verify the urls in next commit.
2023-04-27 12:28:37 -07:00
Sahil Batra 850e0046eb models: Add ORG_TYPE_IDS constant field to Realm.
This commit adds ORG_TYPE_IDS constant field to Realm class
such that it can be used when we want to validate the org_type
passed in request. This was previously defined in realm.py, but
we move it inside Realm class such that we can use it at other
places as well.
2023-04-27 12:28:37 -07:00
Daniil Fadeev db37880d08 emails: Fix some css not being applied to emails.
This commit places the email CSS into the `style` tag located in the
`head` section. This resolves the issue of being unable to apply
certain CSS styles that cannot be inlined, such as media queries and
pseudo-classes.
2023-04-27 08:55:24 -07:00
AcKindle3 521487f444 tornado_url: Replace occurrences of `uri` with `url`.
In #23380 we want to replace all occurrences of `uri` with `url`.
This commit replaces the occurrences appeared in a variable name
`tornado_uri` and a function name `get_tornado_uri`.
2023-04-26 16:37:16 -07:00
AcKindle3 0a1ccb3d89 api_url_context: Replace `uri` with `url`.
In #23380 we want to change all occurrences of `uri` with `url`.
This commit changes the occurrences in a context key `api_uri_context`
and a function name `add_api_uri_context`.
2023-04-26 16:37:16 -07:00
AcKindle3 98c9a0366a settings: Replace occurrences of `uri` with `url`.
In #23380 we want to change all occurrences of `uri` with `url`.
This commit changes the names of two variables `external_uri_scheme`
and `main_site_uri`, who are constructed using `settings` constants.
2023-04-26 16:37:16 -07:00
AcKindle3 4544eb4576 email: Replace `uri` with `url` in templates and backend.
In #23380 we want to change all ocurrences of `uri` to `url`. This
commit changes the ocurrences of `uri` appeared in files related to
email, including templates (`.html`, `.txt`) and backend (`.py`)
codes.

In `email.md`, `base_images_uri` is changed to `images_base_url` -
the words `base` and `images` are swapped and plural form is added
for `image`.  This is becasue the former is not found anywhere in
the codebase while the later appears a lot. To reduce confusion,
this doccumentation changed accordingly.
2023-04-26 16:37:16 -07:00
Lauryn Menard a22168d8b3 templates: Add CSS formatting to invalid email redirect page.
Adds CSS formatting for `invalid_email.html`.

Uses the `white-box` style because this page is a redirect when
there is an error with the email the user provided during
registration.

Also, updates the text of this page for some grammar errors and
to clarify the language between an invalid email and an email that
is not allowed by the Zulip organization in question.

Finally, makes any references to the `realm_name` also link to
the Zulip organization with the `realm_uri`.
2023-04-26 15:41:20 -07:00
Lauryn Menard 601d8101f5 templates: Add CSS formatting to no available licenses page.
Adds CSS formatting for `no_spare_licenses.html`.

Uses the `white-box` style because this page is a redirect when
a user tries to register for a Zulip Cloud organization that does
not have any available licenses for new users.

Updates reference to `realm_name` to be a link to the `realm_uri`.
2023-04-26 15:41:20 -07:00
Alya Abbott 977bec25ba portico: Add Atolio case study. 2023-04-26 14:33:30 -07:00
Tim Abbott 10b893e1e3 api_docs: Update API docs for presence changes. 2023-04-26 14:26:47 -07:00
Mateusz Mandera a9f40a64fd presence: Support null values in UserPresence. 2023-04-26 14:26:47 -07:00
Mateusz Mandera 0d79f6dd27 presence: Deduplicate code formatting legacy presence info.
This also removes the error in one of these functions that was using a
different constant instead of
PRESENCE_LEGACY_EVENT_OFFSET_FOR_ACTIVITY_SECONDS.
2023-04-26 14:26:47 -07:00
Mateusz Mandera ed91e72c5c presence: Rewrite the backend data model - final migration. 2023-04-26 14:26:47 -07:00
Tim Abbott 027b67be80 presence: Rewrite the backend data model.
This implements the core of the rewrite described in:

For the backend data model for UserPresence to one that supports much
more efficient queries and is more correct around handling of multiple
clients.  The main loss of functionality is that we no longer track
which Client sent presence data (so we will no longer be able to say
using UserPresence "the user was last online on their desktop 15
minutes ago, but was online with their phone 3 minutes ago").  If we
consider that information important for the occasional investigation
query, we have can construct that answer data via UserActivity
already.  It's not worth making Presence much more expensive/complex
to support it.

For slim_presence clients, this sends the same data format we sent
before, albeit with less complexity involved in constructing it.  Note
that we at present will always send both last_active_time and
last_connected_time; we may revisit that in the future.

This commit doesn't include the finalizing migration, which drops the
UserPresenceOld table.
The way to deploy is to start the backfill migration with the server
down and then start the server *without* the user_presence queue worker,
to let the migration finish without having new data interfering with it.
Once the migration is done, the queue worker can be started, leading to
the presence data catching up to the current state as the queue worker
goes over the queued up events and updating the UserPresence table.

Co-authored-by: Mateusz Mandera <mateusz.mandera@zulip.com>
2023-04-26 14:26:47 -07:00
Tanmay Munjal 80c01e9fc3
auth: Refactor JWT code to push REQ to top level functions.
Fixes #25015.
2023-04-26 10:57:22 -07:00
Anders Kaseorg 03b3c8522d requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-04-25 21:20:33 -07:00
Anders Kaseorg f74cfe0ed5 test_console_output: Fix WrappedIO.writelines with iterator.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-04-25 21:20:33 -07:00
Anders Kaseorg 5ecff88c1b ruff: Fix C418 Unnecessary `dict` literal passed to `dict()`.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-04-25 21:20:33 -07:00
Anders Kaseorg 9db3451333 Remove statsd support.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-04-25 19:58:16 -07:00
Anders Kaseorg ae7a21834e utils: Fix make_safe_digest type safety.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-04-25 19:01:01 -07:00
Anders Kaseorg 408ef63404 utils: Fix process_list_in_batches type safety.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-04-25 19:01:01 -07:00
Daniil Fadeev d79f4d4f46 emails: Fix style regression for dynamically generated content.
The migration to css-inline "fixed" the fact that styles from this
file previously were never applied to the internals of missed-message
emails.

Rewrite much of the CSS to more correctly scope to apply to the
appropriate elements, and document with comments the purpose of most
blocks.

Co-authored-by: Tim Abbott <tabbott@zulip.com>
2023-04-25 17:53:14 -07:00
Alya Abbott dd4b391229 help: Cover unmuting topics in /help/mute-a-topic. 2023-04-25 15:04:24 -07:00
Lauryn Menard 9880fcb936 narrow: Document new filters `is:dm`, `dm` and `dm-including`.
Documents narrows now have support for new filters for direct messages:
`is:dm`, `dm`, and `dm-including`. Also documents that `is:private`,
`pm-with` and `group-pm-with` are now legacy aliases for these three
new filters respectively.

Note that API documentation references the help center documentation
for search/narrow filters.

Fixes #24806.
2023-04-25 09:07:08 -07:00
Lauryn Menard 243216cb60 url-encoding: Update URLs for direct messages.
In a previous commit, the frontend of the web app was updated for
URLs with `#narrow/dm/...` for direct messages (group and 1-on-1).

Updates the URLs generated for email notifications and outgoing
webhook notification messages to use the new `/dm/...` format.
2023-04-25 09:07:08 -07:00
Lauryn Menard 33886575b2 narrow: Add backend support for `dm-including` operator.
Adds backend support for `dm-including` operator. This will
deprecate the `group-pm-with` operator, but we keep support
for backwards-compatibility.

For testing updates, because the messages returned by these
two operators are different, most of the tests for `group-pm-with`
remain unchanged, but added comments about deprecated state.

Also, cleans up remaining instance of "PM" in `narrow.py` to
be "DM".

The general API changelog and documentation updates will be done
in a final commit in the series of commits that adds support for
the various new direct message narrows.
2023-04-25 09:07:08 -07:00
Lauryn Menard 29832de5f6 narrow: Extract function for calculating huddle recipients.
Extracts the code from `by_group_pm_with` that calculates the
huddle recipients. Prep commit for implementing new `by_*`
method for "dm-including".
2023-04-25 09:07:08 -07:00
Lauryn Menard 665e435b58 narrow: Add backend support for `dm` operator.
Adds backend support for `dm` operator. This will deprecate the
`pm-with` operator, but we keep support for backwards-compatibility.

For testing updates, updates the existing tests for `pm-with` to
use `dm`, and adds one basic test for `pm-with` in the `add_term`
tests as the two operators refer to the same `by_*` method.

The general API changelog and documentation updates will be done
in a final commit in the series of commits that adds support for
the various new direct message narrows.
2023-04-25 09:07:08 -07:00
Lauryn Menard ece752014c narrow: Add backend support for `is:dm` narrow.
Adds backend support for `is` operator with the `dm` operand. This
will deprecate the `is` operator with the `private` operand, but we
keep support for backwards-compatibility.

Note that there is some clean up of references to private messages
in the updated backend test. In commit 43ec7ed, the documentation
for `build_narrow_filter` wasn't updated for the rename of
`BuildNarrowFilterTest` to `NarrowLibraryTest`, so that's also
corrected in these changes.

The general API changelog and documentation updates will be done
in a final commit in the series of commits that adds support for
the various new direct message narrows.
2023-04-25 09:07:08 -07:00
sbansal1999 77ae3176ca webhooks: Rephrase message for Issue Assigned and Unassigned Events.
This commit rephrases the message for Issue assigned and unassigned
events for GitHub Integration.
2023-04-24 23:16:48 -07:00
sbansal1999 31a9d0e69e webhooks: Change assignee info for Issue events in Github Integration.
This commit changes the assignee info to the correct one.

Earlier, the current assignee info of the issue was used
to create the message but when an issue was unassigned
this would give incorrect information if there are still
assignees left on the issue. Now, it uses the details of
the user who has been unassigned from the issue.
2023-04-24 23:16:48 -07:00
sbansal1999 e15df084ad webhooks: Remove description from Issue Unassigned event message.
This commit removes the description from the Issue Unassigned
event message that is sent by the Github Integration.
2023-04-24 23:16:48 -07:00
sbansal1999 85689a5343 webhooks: Remove description from Issue Assigned event message.
This commit removes the description from the Issue Assigned
event message that is sent by the Github Integration.

Previously when an issue was assigned, the message also had the
description of the issue, which is redundant as the description
has been already sent through the issue created event. After
this commit, the description no longer appears in the message.

Fixes #24554
2023-04-24 23:16:48 -07:00
sbansal1999 566e59f0c2 webhooks: Add issue unassigned fixture to GitHub Integration.
This commit adds the fixture that is received when an issue
is unassigned. This fixture helps in testing the integration
for that event.
2023-04-24 23:16:48 -07:00
sbansal1999 f2e820d103 webhooks: Add issue assigned fixture to GitHub Integration.
This commit adds the fixture that is received when an issue
is assigned. This fixture helps in testing the integration
for that event.
2023-04-24 23:16:48 -07:00
Alex Vandiver e0eb074b23 export: Skip PreregistrationRealm data.
Much like PreregistrationUser rows, these do not make sense to export.
2023-04-24 09:48:25 -07:00
Lauryn Menard e68a69e91f api-docs: Update docs to reference max lengths in register response.
Updates areas in the API documentation that reference the maximum
length of a stream message topic to note the `max_topic_length`.

Updates areas in the API documentation that reference the maximum
length of a stream name to note the `max_stream_name_length` and
areas that reference the maximum length of a stream description to
note the `max_stream_description_length`.

All of these maximum values are sent by the `POST /register`
response.
2023-04-21 08:33:16 -07:00
Zixuan James Li 6f42542320 migrations: Remove filter_format_validator.
This removes the validator argument for 0423_realmfilter_url_template,
which do not really alter the database schema. It otherwise fails
the migration because the filter_format_validator function is removed.

Migration 0094_realm_filter_url_validator is modified because we can no
longer refer to filter_format_validator.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2023-04-19 12:20:49 -07:00
Zixuan James Li 032e6f8cb4 linkifier: Update edit_linkifier command to use url_template.
This is mainly updating the variable names and relevant docstrings
without actual change to the behavior of the command.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2023-04-19 12:20:49 -07:00
Zixuan James Li 268f858f39 linkifier: Support URL templates for linkifiers.
This swaps out url_format_string from all of our APIs and replaces it
with url_template. Note that the documentation changes in the following
commits  will be squashed with this commit.

We change the "url_format" key to "url_template" for the
realm_linkifiers events in event_schema, along with updating
LinkifierDict. "url_template" is the name chosen to normalize
mixed usages of "url_format_string" and "url_format" throughout
the backend.

The markdown processor is updated to stop handling the format string
interpolation and delegate the task template expansion to the uri_template
library instead.

This change affects many test cases. We mostly just replace "%(name)s"
with "{name}", "url_format_string" with "url_template" to make sure that
they still pass. There are some test cases dedicated for testing "%"
escaping, which aren't relevant anymore and are subject to removal.
But for now we keep most of them as-is, and make sure that "%" is always
escaped since we do not use it for variable substitution any more.

Since url_format_string is not populated anymore, a migration is created
to remove this field entirely, and make url_template non-nullable since
we will always populate it. Note that it is possible to have
url_template being null after migration 0422 and before 0424, but
in practice, url_template will not be None after backfilling and the
backend now is always setting url_template.

With the removal of url_format_string, RealmFilter model will now be cleaned
with URL template checks, and the old checks for escapes are removed.

We also modified RealmFilter.clean to skip the validation when the
url_template is invalid. This avoids raising mulitple ValidationError's
when calling full_clean on a linkifier. But we might eventually want to
have a more centric approach to data validation instead of having
the same validation in both the clean method and the validator.

Fixes #23124.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2023-04-19 12:20:49 -07:00
Zixuan James Li ab53e8d3e6 migrations: Backfill url_template from url format string.
This is implemented by replacing all matches of "%(var_name)s" in a URL
format string with "{var_name}". Since we do want to ensure that the
templates aren't broken after this migration, a RuntimeError is raised
to let the maintainer know that certain linkifier cannot be converted
automatically if it does not pass the uri_template.validate check.

Also, we need to escape "%%", which is used to represent "%" in the old
format string syntax, as well as "{" and "}", which is a part of the
URL template syntax.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2023-04-19 12:20:49 -07:00
Zixuan James Li a19c80df43 linkifiers: Add url_template field.
This will later be used to expand matching linkifier patterns.
Making it nullable for now, but we will make it required in
the APIs.

As a part of this transition, we temporarily make url_format_string
nullable as well, which will be later removed. This allows us to
switch to populating url_template without caring about passing
url_format_string.

Note that the validators are imported in the migration because Django
otherwise diffs it and considers the schema to be different, generating
a migration, failing the "tools/test-migrations" test.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2023-04-19 12:20:49 -07:00
Alex Vandiver ed4de6da4a migrations: Add a migration to fix invalid deleted user emails.
208c0c3034 fixed this for new deleted users, but left existing users
with potentially invalid email addresses.  This is problematic if the
realm is ever exported and re-imported, as the addresses will not
validate.

Add a migration which attempts to fix these invalid email addresses.
2023-04-19 10:01:03 -07:00
Lauryn Menard b910bb107a api-docs: Clean up documentation of API events with `message_type`.
Cleans up the API documentation for get events with a `message_type`
value: `delete_message`, `typing op:start` and `typing op:stop`.
2023-04-19 09:33:58 -07:00
Joelute afbd0a9b43 user_settings: Add web_mark_read_on_scroll_policy field.
This is a backend change that will help support the new
`Mark messages as read on scroll` user setting.
2023-04-18 18:32:02 -07:00
Prakhar Pratyush 83bbd8c767 user_topics: Update 'do_update_message' to handle 'merge userTopic states'.
This commit updates the logic for migrating user_topic rows
during the move-messages operation when the target topic
already has messages.

Previously, the target_topic's visibility_policy was simply
set to the original_topic's visibility_policy,
and the original_topic's visibility_policy was set to INHERIT.

This commit updates the move-messages code path to determine
the new visibility_policy depending on the visibility policies
of the original and target topics.
The target_topic's visibility_policy is then updated.

The number of db queries has increased by two:
One query corresponds to determining if 'target_topic_has_messages'.
Another query corresponds to 'get_users_with_user_topic_visibility_policy'
to determine 'target_topic_user_profile_to_visibility_policy'.
2023-04-18 16:40:57 -07:00
Prakhar Pratyush d5f148aa36 user_topics: Update 'topic_has_visibility_policy' to support INHERIT.
This prep commit updates the lib function
'topic_has_visibility_policy' to add support for the case
when visibility_policy=INHERIT.

Previously, it had support for all the visibility policies
except INHERIT.
2023-04-18 16:40:57 -07:00
Prakhar Pratyush 18eea3fc4a message_edit: Refactor the move user_topic records code block.
This commit refactors the move user_topic records
code block in 'do_update_message', resulting in
clean code.

We directly iterate over the dictionary items
instead of looping over the keys and fetching
values if the key exists.
2023-04-18 16:40:57 -07:00
Ruchir Harbhajanka e08256ce88 urls: Move jwt_fetch_api_key endpoint to v1_api_mobile_patterns.
Moves jwt_fetch_api_key endpoint to v1_api_mobile_patterns so
that tools/test-api detects it as an API endpoint that is pending
documentation.

Fixes #24982.
2023-04-18 15:44:31 -07:00
Lauryn Menard 2c043c6242 message-type: Add support for "direct" as value for type parameter.
For endpoints with a `type` parameter to indicate whether the message
is a stream or direct message, `POST /typing` and `POST /messages`,
adds support for passing "direct" as the preferred value for direct
messages, group and 1-on-1.

Maintains support for "private" as a deprecated value to indicate
direct messages.

Fixes #24960.
2023-04-18 12:29:33 -07:00
Lauryn Menard 42d9560413 message: Use `recipient_type_name` for API message type references.
Refactors instances of `message_type_name` and `message_type`
that are referring to API message type value ("stream" or
"private") to use `recipient_type_name` instead.

Prep commit for adding "direct" as a value for endpoints with a
`type` parameter to indicate whether the message is a stream or
direct  message.
2023-04-18 12:29:33 -07:00
Lauryn Menard c87fbacaed message: Use more accurate `recipient_type` when applying unread event. 2023-04-18 12:29:33 -07:00
Mateusz Mandera eb4fc7568c auth_enabled_helper: Add realm_authentication_methods argument.
This allows removing pointless db queries when calling
*_auth_enabled(realm) repeatedly.
2023-04-18 09:22:56 -07:00
Mateusz Mandera 72d56d5d59 auth: Remove Realm.AUTHENTICATION_FLAGS class attribute.
With the removal of the authentication_methods bitfield this is now
useless and just duplicates AUTH_BACKEND_NAME_MAP keys.
2023-04-18 09:22:56 -07:00
Mateusz Mandera ffa3aa8487 auth: Rewrite data model for tracking enabled auth backends.
So far, we've used the BitField .authentication_methods on Realm
for tracking which backends are enabled for an organization. This
however made it a pain to add new backends (requiring altering the
column and a migration - particularly troublesome if someone wanted to
create their own custom auth backend for their server).

Instead this will be tracked through the existence of the appropriate
rows in the RealmAuthenticationMethods table.
2023-04-18 09:22:56 -07:00
AcKindle3 e08535ab3e typo: Fix a typo in block comment
Changed `one` to `want` so that the sentence make sense.
2023-04-17 09:59:41 -07:00
Aman Agrawal a06f3d26d0 scheduled_messages: Add endpoints to fetch and delete them. 2023-04-14 17:38:37 -07:00
Aman Agrawal c0ef1c360a message_send: Edit scheduled message if its ID is present.
If the ID of the scheduled message is passed by the client, we
edit the existing scheduled message instead of creating a new one.

However, this will soon be moved into its own API endpoint.
2023-04-14 17:38:37 -07:00
Aman Agrawal 555041c081 models: models: Extract get_recipient_ids method.
This will be used by scheduled messages to as it is used by drafts.
2023-04-14 17:38:37 -07:00
Aman Agrawal a0eac4aaf4 models: Rename _type to recipient_type_str. 2023-04-14 17:38:37 -07:00
Aman Agrawal b63f440fb1 models: Allow scheduled msgs to store rendered content.
This is required by the client to display a list of currently
scheduled messages.
2023-04-14 17:38:37 -07:00
Mateusz Mandera 2a45429a51 zilencer: Delete duplicate remote push registrations.
This fixes existing instances of the bug fixed in the previous commit.

Fixes #24969.
2023-04-13 15:17:20 -07:00
Mateusz Mandera ade2225f08 zilencer: Avoid creating duplicate remote push registrations.
Servers that had upgraded from a Zulip server version that did not yet
support the user_uuid field to one that did could end up with some
mobile devices having two push notifications registrations, one with a
user_id and the other with a user_uuid.

Fix this issue by sending both user_id and user_uuid, and clearing
2023-04-13 15:17:20 -07:00
Alex Vandiver d888bb3df2 error-bot: Remove ERROR_BOT support.
This isn't sufficiently useful to keep the added complexity.  Users
should use the email error reporting, or set up Sentry error
reporting.
2023-04-13 14:59:58 -07:00
Alex Vandiver db07b8668f error_notify: Consolidate email and zulip codepaths. 2023-04-13 14:59:58 -07:00
Alex Vandiver daba72c116 error_notify: Drop any remaining browser-side errors in RabbitMQ queue. 2023-04-13 14:59:58 -07:00
Alex Vandiver e536a14b61 report_error: Remove API endpoint for client error reporting. 2023-04-13 14:59:58 -07:00
Alex Vandiver cb7bc1b7b9 report_error: Remove reference to old non-existant path. 2023-04-13 14:59:58 -07:00
Alex Vandiver 52c4cae239 blueslip: Remove unused ui_message / show_ui_msg codepath.
This was last used in 71e14674aa, a decade ago.
2023-04-13 14:59:58 -07:00
Anders Kaseorg 0a3dc8a944 ruff: Fix DJ012 Order of model's inner classes, methods, and fields.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-04-12 17:32:38 -07:00
Sahil Batra 6e4c844907 tests: Fix comment about number of database queries.
This commit fixes the comment about number of database queries
when moving message from muted topic to mention clearly about
the number of queries added due to original topic being muted.

We do not include the queries that is executed to check whether
the topic is muted or not, as they will be executed in all cases.
2023-04-11 11:07:23 -07:00
Sahil Batra bd7f728796 message: Don't allow moving messages that have passed the time limit.
We previously allowed moving messages that have passed the time limit
using "change_all" value for "propagate_mode" parameter. This commit
changes the behavior to not allow moving messages (both stream and
topic edit) that have passed the time limit for non-admin and
non-moderator users.
2023-04-11 11:07:23 -07:00
Sahil Batra 440f9e397a message_edit: Apply topic edit restrictions to "(no topic)" messages.
Previously, editing topic of "(no topic)" messages was allowed
irrespective of time limit or the "edit_topic_policy" setting.
Since we are working in the direction of having "no topic" messages
feel reasonable, this commit changes the code to not consider them
as a special case and topic editing restrictions apply to them as
well now like all other messages.

We still highlight the topic edit icon in recipient bar without
hovering for "no topic" messages, but it is only shown when user
has permission to edit topics.
2023-04-11 11:07:23 -07:00
Sahil Batra 66693f2101 user_groups: Add allow_nobody_group to access_user_group_for_setting.
This commit adds allow_nobody_group parameter to
access_user_group_for_setting with a default value of True.
2023-04-11 09:02:09 -07:00
Sahil Batra 5237e9008f types: Add allow_nobody_group field to GroupPermissionSetting. 2023-04-11 09:02:09 -07:00
Alya Abbott aa65813294 email: Rewrite followup_day2 email.
Keep the focus of the email the same (topics), but improve how
the information is presented.
2023-04-10 17:44:18 -07:00
Lauryn Menard 2654af3675 welcome-emails: Separate context for followup day1 and day2.
Separates the context dictionary that is used for `send_email` for
the `followup_day1` and `followup_day2` emails.

Prep commit for updates to `followup_day2` email.
2023-04-10 17:44:18 -07:00
Alex Vandiver 3efc0c9af3 workers: Rewrite missedmessage_emails with a worker thread.
The previous implementation leaked database connections, as a new
thread (and thus a new thread-local database connection) was made for
each timer execution.  While these connections were relatively
lightweight in Python, they also incur memory overhead in the
PostgreSQL server itself.  The logic for managing the timer was also
unclear, and the unavoidable deadlock in the stopping logic was rather
unfortunate.

Rewrite with one explicit worker thread which handles the delayed
message sending.  The RabbitMQ consumer creates the database rows, and
notifies the worker to start its 5s timeout.  Because it is controlled
by a condition variable, it does not hold the lock while waiting, and
can be notified to exit.
2023-04-10 17:38:08 -07:00
AcKindle3 ff48f3c489 templates: Use `root_domain_url` Django templates.
Per the issue #25045, this commit changes some occurences of `uri`
appeared in variable `root_domain_uri`. Files affected are some
html files that used this variables and a backend file
`context_processors.py` that set it as a key.
2023-04-10 10:26:48 -07:00
Lauryn Menard f391bfeec6 emails: Add new onboarding email with guide for organization type.
Adds a new welcome email, `onboarding_zulip_guide`, to be sent four
days after a new user registers with a Zulip organization if the
organization has specified a particular organization type that has
a guide in the corporate `/for/.../` pages. If there is no guide,
then no email is scheduled or sent.

The current `for/communities/` page is not very useful for users
who are not organization administrators, so these onboarding guide
emails are further restricted for those organization types to
only go to new users who are invited/registered as admins for the
organzation.

Adds two database queries for new user registrations: one to get
the organization's type and one to create the scheduled email.

Adds two email logs because the email is sent both to a new user
who registers with an existing organization and to the organization
owner when they register a new organization.

Co-authored by: Alya Abbott <alya@zulip.com>
2023-04-10 08:38:09 -07:00
Lauryn Menard ab00648e3e email: Refactor calculation of delay for scheduled emails.
Refactors the logic for adjusting the delay for sending an email
to not land on a weekend so that it can be used to schedule any
number of onboarding emails we decide to send.

Consolidates duplicate testing into
`zerver/tests/test_email_notifications.py`. The initial test and
function were introduced in commit 610f2cbacf with the test
located in `zerver/tests/test_signup.py`.

Prep commit for adding new welcome / follow up email.
2023-04-10 08:38:09 -07:00
AcKindle3 b0ef8f0822 test: Replace occurences of `uri` with `url`.
In all the tests files, replaced all occurences of `uri` with `url`
appeared in comments, local variablles, function names and their callers.
2023-04-08 16:27:55 -07:00
Joelute 946b4e73ca integration: Remove branch names from Gitea edited PR event.
This is a follow up to #24673, we want to modify every webhook events to
follow the same pattern and consistency where branch name should only
show on opened and merged events.
2023-04-07 15:17:23 -07:00
Satyam Bansal 338436dfbd
integrations: Add Rundeck webhook integration.
Co-authored-by: sbansal1999 <sbansal1999@gmail.com>
2023-04-07 10:13:21 -07:00
Prakhar Pratyush d96048b0af test_classes: Rename and refactor 'tornado_redirected_to_list'.
This commit renames the 'tornado_redirected_to_list' context
manager to 'capture_send_event_calls' to improve readability.

It also refactors the function to yield a list of events
instead of passing in a list data structure as a parameter
and appending events to it.
2023-04-07 09:45:26 -07:00
Alex Vandiver e3f6e810dc export-search: Support writing out attachments as well. 2023-04-07 09:13:48 -07:00
Alex Vandiver e408f069fe uploads: Add a method to copy attachment contents out. 2023-04-07 09:13:48 -07:00
Alex Vandiver 885334a3ad export-search: Support CSV output, trim down output fields. 2023-04-07 09:13:48 -07:00
Alex Vandiver bde978d5b1 export-search: Move topic into recipient field. 2023-04-07 09:13:48 -07:00
Alex Vandiver 26bc112adc export-search: Add a --recipient filter. 2023-04-07 09:13:48 -07:00
Alex Vandiver e44ba1bef1 export-search: Write output size before beginning export. 2023-04-07 09:13:48 -07:00
Alex Vandiver 21db267202 export-search: Non-time-limited is fine with a sender. 2023-04-07 09:13:48 -07:00
Alex Vandiver ae20af84a9 export-search: Realm is required. 2023-04-07 09:13:48 -07:00
Anders Kaseorg d2af06f4df middleware: Remove ZulipCommonMiddleware patch.
My fix for the relevant performance bug was upstreamed in Django 4.2.

https://code.djangoproject.com/ticket/33700

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-04-07 09:13:20 -07:00
Akarsh Jain f122516e7d notifications: Add link to new location of single moved messages.
Improve the Notification Bot by adding a hyperlink to the new location
of a moved single message. The link will make it easier for users to
find the message in its new context.

Fixes #24604.
2023-04-06 19:04:40 -07:00
Ujjawal Modi d0dbdfa52d user_groups: Send a message on changing user-groups subscribers.
After this commit a notification message is sent to users if they are
added to user_groups by someone else or they are removed from user_groups
by someone else.

Fixes #23642.
2023-04-06 19:03:26 -07:00
Ujjawal Modi e163e3ced5 user_groups: Refactor the view function used for adding members.
This commit changes the variable names inside the view function
used for adding members to user_groups. This was done for better
readability of code.
2023-04-06 19:03:26 -07:00
Anders Kaseorg 52266cbd41 user_groups: Work around django-cte bug with Django 4.2.
https://github.com/dimagi/django-cte/issues/66

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-04-06 17:58:44 -07:00
Anders Kaseorg f1925487e8 db: Force use of TimeTrackingCursor to work around Django 4.2 bug.
Effectively revert commit b4cf9ad777 to
work around https://code.djangoproject.com/ticket/34466.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-04-06 17:58:44 -07:00
sbansal1999 f47a1c67a6 webhooks: Add body to PR Review Event message for GitHub Integration.
This commit passes the body of the PR Review as the message to
the helper function that generates the message to be sent by the
GitHub Integration.

Previously when a PR Review was done the message sent would just
include the link of the review but the message didn't include the
body the review. After this commit, the message also includes the
body of the review.

Fixes #24676
2023-04-05 15:48:26 -07:00
Daniil Fadeev 2f203f4de1 emails: Inline CSS in emails in build_email.
Previously, we had an architecture where CSS inlining for emails was
done at provision time in inline_email_css.py. This was necessary
because the library we were using for this, Premailer, was extremely
slow, and doing the inlining for every outgoing email would have been
prohibitively expensive.

Now that we've migrated to a more modern library that inlines the
small amount of CSS we have into emails nearly instantly, we are able
to remove the complex architecture built to work around Premailer
being slow and just do the CSS inlining as the final step in sending
each individual email.

This has several significant benefits:

* Removes a fiddly provisioning step that made the edit/refresh cycle
  for modifying email templates confusing; there's no longer a CSS
  inlining step that, if you forget to do it, results in your testing a
  stale variant of the email templates.
* Fixes internationalization problems related to translators working
  with pre-CSS-inlined emails, and then Django trying to apply the
  translators to the post-CSS-inlined version.
* Makes the send_custom_email pipeline simpler and easier to improve.

Signed-off-by: Daniil Fadeev <fadeevd@zulip.com>
2023-04-05 12:22:29 -07:00
Lauryn Menard e95b784f6e backend-auth: Add user ID to fetch api key responses.
Adds the user ID to the return values for the `/fetch_api_key` and
`/dev_fetch_api_key` endpoints. This saves clients like mobile a
round trip to the server to get the user's unique ID as it is now
returned as part of the log in flow.

Fixes #24980.
2023-04-04 10:54:49 -07:00
Anders Kaseorg a881918a05 requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-04-03 22:39:21 -07:00
Anders Kaseorg 2ce8357009 codespell: Fix newly found typos.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-04-03 22:39:21 -07:00
Prakhar Pratyush ab7cd3c40a user_topics: Update test to verify end-to-end behaviour.
This commit updates 'test_user_ids_unmuting_topic' to make
an api_post call to '/api/v1/user_topics' instead of
calling the internal function 'do_set_user_topic_visibility_policy'
to verify the logic.
2023-04-03 22:31:49 -07:00
Prakhar Pratyush f012d079c3 user_topics: Add a new endpoint to update visibility_policy.
This commit adds a new endpoint, 'POST /user_topics' which
is used to update the personal preferences for a topic.

Currently, it is used to update the visibility policy of
a user-topic row.
2023-04-03 22:31:49 -07:00
Prakhar Pratyush 0b2fe5b163 user_topics: Rename the 'zerver/lib/user_topics' functions.
This is a prep commit that renames lib functions
so that they can be used while implementing view
for the new endpoint 'POST /user_topics'.

We use a more generic name when removing the visibility_policy of
a topic, i.e., 'access_stream_to_remove_visibility_policy_by_id/name'
instead of 'access_stream_for_unmute_topic_by_id/name' which focused
on removing MUTE from a topic.
2023-04-03 22:31:49 -07:00
Prakhar Pratyush e33eb45507 user_topics: Update tests to use the 'time_machine' module.
This commit updates the tests to use the 'time_machine'
module for mocking datetime.
2023-04-03 22:31:49 -07:00
Prakhar Pratyush 5c86d3ca9e user_topics: Update tests to verify end-to-end behaviour.
This commit updates the existing tests so that they call
API endpoints instead of internal functions to verify logic.
2023-04-03 22:31:49 -07:00
Alex Vandiver 113a8c4782 export: Make --deactivate-realm exports be imported as active. 2023-04-03 16:08:43 -07:00
Alex Vandiver 4b0b8abc6c import_realm: Keep the realm deactivated until import is complete. 2023-04-03 16:08:43 -07:00
Daniil Fadeev 5c6f842b28 emails: Replace Premailer with CSS-inline.
Primary goal of library replacement is improving execution speed.
This commit should not affect the functionality of the system
or make any changes to it.
2023-04-03 15:15:05 -07:00
Joelute 78a9d716ec integration: Remove branch names to Gogs edited PR event.
This is a follow up to #24673, we want to modify every webhook events to
follow the same pattern and consistency where branch name should only
show on opened and merged events.
2023-04-02 23:06:15 -07:00
Josh Klar ef0c281070 integrations: Add documentation for Mastodon via public RSS feeds.
This uses the Mastodon logo SVG as pulled from
https://blog.joinmastodon.org/logo.svg.
2023-03-30 09:36:16 -07:00
Mateusz Mandera e3f754fb9b help center: Document SAML configuration steps with Auth0.
Auth0 is another SAML IdP that we haven't detailed the instructions for
yet.
2023-03-30 09:11:55 -07:00
David Rosa 2b9374875d templates: Update text on /accounts/find.
- Being more specific about what the user will get.
- Putting less emphasis on entering multiple emails, since most
  people probably just have one email they need to check.
- Using more intuitive wording and hint that deactivated or
  deleted accounts won't be included.

Fixes: #24890.
2023-03-30 09:06:14 -07:00
Joelute 75611c6cee integration: Modify branch names for Bitbucket 3 events.
This is a follow up to #24673, we want to modify every webhook events to
follow the same pattern and consistency where branch name should only
show on opened and merged events.
2023-03-29 18:13:36 -07:00
Joelute 1ddfbc94c3 integration: Support reviewer in `get_pull_request_event_message`.
This is a prep commit to help make the changes to make changes to pull
event message easier. Our Bitbucket has been using a custom template to
render the reviewers. This means that values are fixed to how the templates
like it. These changes will allow `get_pull_request_event_message` to
support reviewer and allow for a easier and flexible adjustment to these
messages if needed.
2023-03-29 18:13:36 -07:00
Joelute 4dc85dddc1 integration: Move assignee message to the end for git integrations.
Previously, the assignee message would stick around in the middle of the
event message. This doesn't look as good as if we put it to the end of
the event message. These changes does just that and move the assignee
messages towards the end of the event message to make it look better
and cleaner for the readers.
2023-03-29 18:13:36 -07:00
Mateusz Mandera f3db618cb5 actions: Implement do_delete_user_preserving_messages function.
With an extensive docstring by tabbott documenting its limitations.

Co-authored-by: Tim Abbott <tabbott@zulip.com>
2023-03-29 12:51:21 -07:00
Prakhar Pratyush d8a5ca5ddc user_topics: Update test_edit_muted_topic to verify events sent.
Previously, there was a stale code that didn't verify
if 'muted_topics' and 'user_topic' events are sent correctly.

This commit updates the test to verify if the expected
users are notified via 'muted_topics' and 'user_topic'
events.
2023-03-29 09:50:06 -07:00
Prakhar Pratyush 117264fb37 user_topics: Update move-topic codepath to utilise bulk database operations.
This commit updates the move-topic codepath to perform
bulk database operations on the UserTopic record using
user_profiles for each visibility_policy instead of
previously looping over each user_profile one by one.
2023-03-29 09:50:06 -07:00
Prakhar Pratyush 60762a750b user_topics: Update 'set visibility policy' codepath for bulk db operations.
This commit refactors 'set_user_topic_visibility_policy_in_database'
to perform bulk database operations and the related changes.

There is an increase in database query count because requests
to delete user_topic rows now take two queries instead of one.
This is required for logging the info for a request to delete
a non-existent user_topic row while performing bulk operations
at the same time.

The overall query count will be lower while performing
bulk operations (multiple user_profiles instead of one).
2023-03-29 09:50:06 -07:00
Prakhar Pratyush a890aaf34d user_topics: Update UserTopic records regardless of the visibility_policy.
This commit updates the 'do_update_message' codepath to
update the UserTopic records regardless of visibility policy
during the "move-topic" operation.

This is required before offering new visibility policies
in the UI.

Previously, UserTopic records were moved or deleted only
for objects with a MUTED visibility policy.

Fixes: #24574
2023-03-29 09:50:06 -07:00
Prakhar Pratyush 0377085f15 user_topics: Update a comment regarding the database query count.
This commit updates a stale comment regarding query count
to its correct value.
2023-03-29 09:50:06 -07:00
Prakhar Pratyush e1ec814507 user_topics: Add a test helper to improve readability.
This commit adds a test helper function that helps
to remove line wrappings and improve readability.
2023-03-29 09:50:06 -07:00
Prakhar Pratyush 0ec9ff17ce user_topics: Update lib functions to work with any visibility_policy.
This is a prep commit that renames 'set_topic_mutes' and
'topic_is_muted' to 'set_topic_visibility_policy' and
'topic_has_visibility_policy' respectively, and refactors
them to work with any visibility_policy, not only MUTED.
2023-03-29 09:50:06 -07:00
Joelute a2a9e53423 integration: Refactor `get_issue_event_message` to use kwarg.
Previously, some call sites for the function provided optional
arguments as positional arguments. These changes will allow the
arguments to be passed as keyword arguments to the function and
fix up the call sites of the function to pass keyword arguments
instead.
2023-03-28 15:51:43 -07:00
Joelute a2c578e84d integration: Refactor `get_pull_request_event_message` to use kwarg.
Previously, some call sites for the function provided optional
arguments as positional arguments. These changes will allow the
arguments to be passed as keyword arguments to the function and
fix up the call sites of the function to pass keyword arguments
instead.
2023-03-28 15:51:43 -07:00
Lauryn Menard 7b225245c0 tests: Update ZulipTestCase.tearDown to remove local uploads.
Previously, tests that exercised code paths that added local
uploads did not always clean up `settings.LOCAL_UPLOADS_DIR`
after the test was complete.

Updates the `ZulipTestCase` class to remove any local uploads
in the unique `settings.LOCAL_UPLOADS_DIR` in `tearDown` for
all tests.
2023-03-28 14:38:06 -07:00
Sahil Batra a6fd41e012 migrations: Fix comment in 0403.
This commit fixes the file name in comment in
0403_create_role_based_groups_for_internal_realms.py.
2023-03-28 14:26:22 -07:00
Sahil Batra bed2bf64c4 user_groups: Add "Nobody" system user group.
This commit adds code to create a "Nobody" system user group
to realms which will be used in settings to represent "Nobody"
option.

We also add a migration to add this group to existing realms.
2023-03-28 14:26:22 -07:00
Prakhar Pratyush e45623fccc python: Update tuple handling pattern; returned by a delete() query.
This commit updates the pattern for dealing with tuples
returned by the delete() query.

The '(num_deleted, ignored) = ModelName.objects.filter().delete()'
pattern is preferred due to better readability.

We avoid the pattern '(num_deleted, _)' because Django uses _
for translation, which may lead to future bugs.
2023-03-27 16:18:23 -07:00
Sahil Batra f684d36710 test_classes: Add submit_realm_creation_form helper.
This commit adds a new helper submit_realm_creation_form,
similar to existing submit_reg_form_for_user, to avoid
duplicate code for creating realms in tests.
2023-03-27 15:44:42 -07:00
Sahil Batra 4d3c5fdc91 forms: Use RealmDetailsForm as subclass for RegistrationForm.
We now use RealmDetailsForm as subclass for RegistrationForm
such that we can avoid duplication of the checks for
realm_subdomain field.
2023-03-27 15:44:42 -07:00
Sahil Batra 96979e73c6 registration: Use get_realm_create_form_context for register.html.
This commit adds the fields related to realm creation form using
get_realm_create_form_context in the context passed to register.html
template to avoid duplication.
2023-03-27 15:44:42 -07:00
Sahil Batra 00ffa3e870 registration: Remove redundant code.
Since we have updated the registration code to use
PreregistrationRealm objects for realm creation in
previous commits, some of the code has become
redundant and this commit removes it.

We remove the following code -
- The modification to PreregistrationUser objects in
process_new_human_user can now be done unconditionally
because prereg_user is passed only during user creation
and not realm creation. And we anyway do not expect
any PreregistrationUser objects inside the realm
during the creation.
- There is no need of "realm_creation" parameter in
create_preregistration_user function, since we now
use create_preregistration_realm during realm creation.

Fixes part of #24307.
2023-03-27 15:44:42 -07:00
Sahil Batra cf8d8db132 register: Update the user-registration form for realm creation.
In previous commits, we updated the realm creation flow to show
the realm name, type and subdomain fields in the first form
when asking for the email of the user. This commit updates the
user registration form to show the already filled realm details
as non-editable text and there is also a button to edit the
realm details before registration.

We also update the sub-heading for user registration form as
mentioned in the issue.

Fixes part of #24307.
2023-03-27 15:44:42 -07:00
Sahil Batra c225de789e development: Use PreregistrationRealm object for creating realms. 2023-03-27 15:44:42 -07:00
Sahil Batra 791c3030eb backends: Update ZulipLDAPAuthBackend to use PreregistrationRealm.
This commit updates ZulipLDAPAuthBackend to use the recently added
PreregistrationRealm when creating realms with LDAP used as
authentication method.
2023-03-27 15:44:42 -07:00
Sahil Batra e60b6b68ea registration: Modify PreregistrationRealm objects after registration. 2023-03-27 15:44:42 -07:00
Sahil Batra 765e65f954 registration: Use PreregistrationRealm object for realm creation.
We now use PreregistrationRealm objects in registration_helper
function when creating new realms instead of PreregistrationUser
objects.

Fixes part of #24307.
2023-03-27 15:44:42 -07:00
Sahil Batra 54771cfe94 registration: Create PreregistrationRealm object when creating realm.
This commit adds code to create PreregistrationRealm object when
creating realm and set it to the content_object field of
Confirmation object.
2023-03-27 15:44:42 -07:00
Sahil Batra 80b00933b1 templates: Add realm information fields in create_realm.html.
We now show inputs for realm details like name, type and URL
in the create_realm.html template opened for "/new" url and
these information will be stored in PreregistrationRealm
objects in further commits.

We add a new class RealmDetailsForm in forms.py for this
such that it is used as a base class for RealmCreationForm
and we define RealmDetailsForm such that we can use it as
a subclass for RegistrationForm as well to avoid duplication.
2023-03-27 15:44:42 -07:00
Sahil Batra 7f1bf9d6ab models: Add PreregistrationRealm class.
This commit adds PreregistrationRealm class which will be
similar to PreregistrationUser and will store initial
information of the realm before its creation as we are
changing the organization creation flow as per #24307.

Fixes part of #24307.
2023-03-27 15:44:42 -07:00
Sahil Batra 8c6bf72c1f registration: Rename prereg_user to prereg_object.
This commit renames prereg_user variable in
check_prereg_key and get_prereg_key_and_redirect
functions in zerver/views/registration.py to
prereg_object as in further commits the
preregistration object could also be
PreregistrationRealm object as part of changes
for #24307.
2023-03-27 15:44:42 -07:00
Joelute 8747aa6fa1 integration: Modify branch names for Bitbucket 2 events.
This is a follow up to #24673, we want to modify every webhook events to
follow the same pattern and consistency where branch name should only
show on opened and merged events.
2023-03-27 10:09:25 -07:00
Joelute 505c217db5 bots: Change logic for notification bot after moved message.
Previously, when a user moves a message to another topic, the Notification
bot will post a message saying "This topic was moved here from..." This is
confusing when the topic already contains messages. The changes aims to make
the messages more clear by changing the logic for the Notification bot. When
there is already messages in the topic, the bot will post "A message was
moved here from..." or "N messages were moved here from...". The bot will
post "This topic was moved here from (somewhere) by (someone)." when the
topic is empty.

Fixes #23267.
2023-03-27 10:00:34 -07:00
Joelute c8bce6f8d3 bots: `maybe_send_resolve_topic_notiffications` return message_id or None 2023-03-27 10:00:34 -07:00
Zixuan James Li cf9b95b95a user_groups: rename create_user_group to create_user_group_in_database.
To avoid people calling "create_user_group" instead of
"check_add_user_group", we rename it to make its purpose clearer.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2023-03-27 09:05:00 -07:00
Zixuan James Li e331c356e4 user_groups: Use check_add_user_group instead in test cases.
"check_add_user_group" is a safer helper function than
"create_user_group" to use when creating user_groups. It does
error handling and notify the client with the appropriate event.

Note that the populate_db command still uses "create_user_group"
because we do not need to enqueue events at that point.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2023-03-27 09:05:00 -07:00
Zixuan James Li 0f5d6432a4 user_groups: Move create_user_group to zerver.actions.user_groups.
Since this function creates a new user group into the database,
it is more appropriate to have it not as a generic "lib" function
but as an "action".

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2023-03-27 09:05:00 -07:00
Alex Vandiver 93bfd3fb5f sentry: Provide a server-side tunnel.
Some well-intentioned adblockers also block Sentry client-side error
reporting.  Provide an endpoint on the Zulip server which forwards to
the Sentry server, so that these requests are not blocked.
2023-03-24 15:16:38 -07:00
Alex Vandiver 164d58bec9 web: Save a 301 redirect from /help to /help/. 2023-03-24 14:51:01 -07:00
Alex Vandiver ab7ff6b582 web: Save a 301 redirect from /billing to /billing/. 2023-03-24 14:51:01 -07:00
Alex Vandiver b5731c55b7 web: Save two 301 redirects from /privacy to /privacy/ to /policies/privacy. 2023-03-24 14:51:01 -07:00
Alex Vandiver 822683249d web: Save a 301 redirect from /terms/ to /policies/terms. 2023-03-24 14:51:01 -07:00
Alex Vandiver ce847fd09c web: Save two 301 redirects from /terms to /terms/ to /policies/terms. 2023-03-24 14:51:01 -07:00
Alex Vandiver d9daeb13e0 web: Save a needless 301 redirect from /apps to /apps/. 2023-03-24 14:51:01 -07:00
Alex Vandiver 61adc9cfcc web: Save a needless 301 redirect from /team to /team/. 2023-03-24 14:51:01 -07:00
Alex Vandiver c686c5ed0f web: Save a needless 301 redirect from /plans to /plans/. 2023-03-24 14:51:01 -07:00
Alex Vandiver eb81e62eaa web: Save a needless 301 redirect from /hello to /hello/. 2023-03-24 14:51:01 -07:00
Alex Vandiver a1570ff3dd web: Save a needless 301 redirect from /integrations to /integrations/. 2023-03-24 14:51:01 -07:00