Commit Graph

15206 Commits

Author SHA1 Message Date
Lauryn Menard 9f8022de5e help_docs: Add 'About Zulip' to relative help links.
Adds the 'About Zulip' gear menu option to the available
relative link patterns used in help center documentation.
2022-04-12 15:22:58 -07:00
Alex Vandiver a26d109e7a auth: Reset failed authentication attempts on password reset.
It's natural that someone might try a wrong password 5 times, and then
go through a successful password reset; forcing such users to wait
half an hour before typing in the password they just changed the
account to seems unnecessarily punitive.

Clear the rate-limit upon successful password change.
2022-04-11 16:25:32 -07:00
Lauryn Menard 2615eacca5 tests: Remove ignored parameters from push notifications tests.
Removes `token_kind` parameter being passed to
`remove_apns_device_token` and `remove_android_reg_id` code
paths / endpoints. Possibly missed in a refactor of this
function as the tests for adding these tokens do not pass
a `token_kind` parameter.

Removes `zulip_org_id` and `zulip_org_kay` from code testing
`deactivate_remote_server`. These parameters are passed when
a remote server is added, so possibly a copy and paste error
when these tests were written / last refactored.
2022-04-08 11:39:06 -07:00
Lauryn Menard 195a86f242 tests: Remove ignored parameters from custom profile field tests.
`update_realm_custom_profile_field` does not take `field_type`
as a parameter, so this removes it from any related tests.
Possibly these test parameters were missed in a refactor of this
endpoint / code.
2022-04-08 11:39:06 -07:00
Lauryn Menard c8d8e82118 tests: Remove ignored parameters from bots tests.
`service_interface` is not a parameter of `add_bot_backend`, but
`interface_type` is, and that has the same default value as what
was being provided by the test, so updated for the parameter name
change, which was possibly missed in a previous code refactor.
2022-04-08 11:39:06 -07:00
Lauryn Menard 260012a87e tests: Remove ignored parameters from stream groups tests.
`update_default_stream_group_info` was being passed `op` and
`group_name` in various tests, which are not implemented as
parameters for that endpoint / code path. So this removes those
from the existing tests. This is not a documented API endpoint,
so perhaps these were just overlooked when these tests were
written / last refactored.
2022-04-08 11:39:06 -07:00
Lauryn Menard 8154b4a9af middleware: Add `client` REQ parameter to `parse_client`.
If an API request specified a `client` parameter, we were
already prioritizing that value over parsing the UserAgent.

In order to have these parameters logged in the `RequestNotes`
as processed parameters instead of ignored parameters, we add
the `has_request_variables` decorator to `parse_client` and
then process the potential `client` parameter through the REQ
framework.

Co-authored by: Tim Abbott <tabbott@zulip.com>
2022-04-08 11:29:33 -07:00
Alex Vandiver e6e4b7b3ef timeout: Add test coverage. 2022-04-07 17:26:01 -07:00
Alex Vandiver 04159a674c timeout: Minor comment cleanups.
We remove the StackOverflow link because it is now so dated as to be
irrelevant -- it does not use `self.ident`, and cargo-cults the return
value of PyThreadState_SetAsyncExc.
2022-04-07 17:26:01 -07:00
Alex Vandiver 3af2c8d9a3 timeout: Warn if the thread did not exit.
As noted in the docstring for this function, the timeout is
best-effort only -- if the thread is blocked in a syscall, it will not
service the exception until it returns.  It can also choose to catch
and ignore the TimeoutExpired; in either case it will still be running
even after the `timeout()` function returns.

Raising a vare TimeoutExpired it still somewhat accurate, but obscures
that the backend thread may still be running along merrily.  Notice
such cases, and log a warning about them.
2022-04-07 17:26:01 -07:00
Alex Vandiver e714264756 timeout: Re-raise from where the TimeoutExpired hit the thread.
Having just thrown an exception into the thread, it is often useful to
know _what_ was the slow code that we interrupted.  Raising a bare
TimeoutExpired here obscures that information, as any `exc_info` will
end there.

Examine the thread for any exception information, and use that to
re-raise.  This exception information is not guaranteed to exist -- if
the thread didn't respond to the exception in time, or caught it, for
instance.
2022-04-07 17:26:01 -07:00
Alex Vandiver 85eeaf5f18 timeout: Remove cargo-culted and impossible-to-reach code block.
The quote in question originates in python/cpython@b8b6d0c2c6, when
the code was added.  However, the code stopped having that comment,
and was no longer able to return anything but 1 or 0, starting in
python/cpython@4643c2fda1 -- Python 2.5.

Remove the block.
2022-04-07 17:26:01 -07:00
Lauryn Menard 450245a455 api_docs: Clean uses of shared emoji schemas in OpenAPI.
Reformats two events (`reaction op: add` and `reaction op:remove`)
to follow the general format of events in the OpenAPI that are
returned by the `/get-events` endpoint.

Removes unneeded reference to `EmojiBase` schema in `user_status`
return value for the `/register-queue` endpoint. Also, clarifies
the text about the `user_status` object and fields being returned.
2022-04-07 12:44:28 -07:00
Lauryn Menard 460968179c api_docs: Add roles and permissions page to API documentation.
Adds a non-endpoint specific page to the API documentation about
organization-level roles and permissions for users in order to
highlight important and useful information for clients and API
users.

Also, adds links to new documentation page in related areas
of the API documentation.
2022-04-04 17:09:05 -07:00
Aman Agrawal dcdf071751 message_edit: Fix false sub/unsub bookend on using a `near` link.
We were not setting the `historical` flag correctly for
messages fetched via `json_fetch_raw_message` when used didn't
have any UserMessage.

Extended relevant tests to fetch check message flags too.
2022-04-04 11:51:12 -07:00
Lauryn Menard 0b03275329 message_send: Move `sender` parameter to REQ.
Instead of using request.POST to get any potential `sender`
parameters for `send_message_backend`, moves it to the REQ
framework parameters as `req_sender`.

Also, updates `create_mirrored_message_users` to take specific
parameters instead of an HTTP request parameter, which then
accessed parameters via request.POST. And updates existing tests
for those changes.
2022-04-04 11:47:52 -07:00
Alex Vandiver 35e27aef4a migrations: Remove the possibly-duplicated emoji re-uploading.
In 85e531e377, we duplicated this block
of migration code to fix a bug, but moving it (aka deleting the
original copy) is a cleaner solution.
2022-04-01 17:51:00 -07:00
Tim Abbott 85e531e377 migrations: Repeat part of migration 0376.
The blockquote explains the motivation for this change in detail.

Fixes #21608.
2022-04-01 15:20:43 -07:00
Alex Vandiver 2e50ead9d1 data_import: Fix bot email address de-duplication.
4815f6e28b tried to de-duplicate bot
email addresses, but instead caused duplicates to crash:

```
Traceback (most recent call last):
  File "./manage.py", line 157, in <module>
    execute_from_command_line(sys.argv)
  File "./manage.py", line 122, in execute_from_command_line
    utility.execute()
  File "/srv/zulip-venv-cache/56ac6adf406011a100282dd526d03537be84d23e/zulip-py3-venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/srv/zulip-venv-cache/56ac6adf406011a100282dd526d03537be84d23e/zulip-py3-venv/lib/python3.8/site-packages/django/core/management/base.py", line 354, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/srv/zulip-venv-cache/56ac6adf406011a100282dd526d03537be84d23e/zulip-py3-venv/lib/python3.8/site-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/home/zulip/deployments/2022-03-16-22-25-42/zerver/management/commands/convert_slack_data.py", line 59, in handle
    do_convert_data(path, output_dir, token, threads=num_threads)
  File "/home/zulip/deployments/2022-03-16-22-25-42/zerver/data_import/slack.py", line 1320, in do_convert_data
    ) = slack_workspace_to_realm(
  File "/home/zulip/deployments/2022-03-16-22-25-42/zerver/data_import/slack.py", line 141, in slack_workspace_to_realm
    ) = users_to_zerver_userprofile(slack_data_dir, user_list, realm_id, int(NOW), domain_name)
  File "/home/zulip/deployments/2022-03-16-22-25-42/zerver/data_import/slack.py", line 248, in users_to_zerver_userprofile
    email = get_user_email(user, domain_name)
  File "/home/zulip/deployments/2022-03-16-22-25-42/zerver/data_import/slack.py", line 406, in get_user_email
    return SlackBotEmail.get_email(user["profile"], domain_name)
  File "/home/zulip/deployments/2022-03-16-22-25-42/zerver/data_import/slack.py", line 85, in get_email
    email_prefix += cls.duplicate_email_count[email]
TypeError: can only concatenate str (not "int") to str
```

Fix the stringification, make it case-insensitive, append with a dash
for readability, and add tests for all of the above.
2022-03-31 11:10:18 -07:00
Lauryn Menard 0008a76703 tests: Remove ignored `stream_name` parameter from test.
Removes unnecessary `stream_name` parameter from
`test_stream_permission_changes_updates_updates_attachments`.
2022-03-30 11:30:31 -07:00
jai2201 20e6315705 streams: Fix empty description in stream change notifications.
Send 'No description' text when either the old or new stream
description is empty, to avoid odd looking notifications.
2022-03-29 01:39:55 -07:00
Tim Abbott c789097184 streams: Fix empty description handling in new stream notifications.
"No description." is conveniniently an existing translated string, in
addition to being the right content for this setting.
2022-03-29 01:39:55 -07:00
jai2201 52cf557d2b openapi: Allow empty value of stream's description.
This is important because otherwise tests that change a stream
description to the empty string don't work.
2022-03-29 01:39:55 -07:00
Alya Abbott f04fb51ecc help: Rewrite pages on inviting users and related permissions.
Fixes #21520.

Co-authored-by: Lauryn Menard <lauryn@zulip.com>
2022-03-28 17:00:10 -07:00
Lauryn Menard cbfe2707f4 help_docs: Clarify relative link text for Subscribed streams tab. 2022-03-28 15:13:39 -07:00
Alex Vandiver ca506f71dc push_notifications: Increase severity of APNs ConnectionError.
This has only happened when our APNs certificate expired; logging at
the error level ensures that this shows up in Sentry.
2022-03-25 18:12:14 -07:00
Alya Abbott 2a1e08759b portico: Add Asciidoctor case study. 2022-03-25 17:51:15 -07:00
Lauryn Menard 8d242f3467 help_docs: Update edit history documentation for messages.
`disable-message-edit-history`: Remove text about EDITED label
and link to `view-a-messages-edit-history` instead.

`edit-or-delete-a-message`: Reformat 'EDITED' and '(deleted)'
to be bold instead of using backticks. Make link to view
edit history clearer.

Note that the text used in the `OpenGraphTest` in
`test_middleware.py` had to be updated for the changes to
`disable-message-edit-history`.
2022-03-25 17:03:53 -07:00
Anders Kaseorg 55882fb343 python: Use modern set comprehension syntax.
Generated by pyupgrade.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-03-25 10:45:12 -07:00
Alex Vandiver eae4643cb4 message_edit: Ignore duplicates when re-muting new topic name.
This avoids an error when a user has already muted the new topic name.
We do this by ignoring duplicates, rather than catching the
IntegrityError, because this edit happens in a transaction, and that
would abort the transaction.
2022-03-24 21:27:11 -07:00
Alex Vandiver 141b0c4cec muting: Handle the case of a race muting the same user twice. 2022-03-24 21:27:11 -07:00
Alex Vandiver 781107308d muting: Add a flag to allow user duplicate mutes to silently succeed. 2022-03-24 21:27:11 -07:00
Anders Kaseorg 83c90c53df db: Fix types.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-03-24 19:16:52 -07:00
Tim Abbott e3600900c0 migrations: Adjust related_name settings for ArchivedAttachment.
This is necessary for the migration 0386_fix_attachment_caches to run,
and likely makes more convenient any future parallel code interacting
with both Attachment and ArchivedAttachment.
2022-03-24 19:15:27 -07:00
Tim Abbott f9f111f950 message_edit: Only move muted topic records when moving whole topics.
Our original implementation of moving muted topic records when a topic
is moved took a shortcut of treating all change_later usage as
something with intent to move the whole topic.

This works OK when moving the whole topic via this interface, but not
when moving a last off-topic message in the topic.

Address this by changing the rule to match the existing
moved_all_visible_messages variable.
2022-03-24 17:48:52 -07:00
Tim Abbott b493224cda management: Document logout_all_users interaction with API keys.
Fixes part of #19397, but I'm leaving that open since we'll want to
add an option to rotate all API keys for the target users.
2022-03-24 14:58:31 -07:00
Lauryn Menard 182c00248d help_docs: Update various docs related to permissions.
Adds tab for web-public streams in documentation for setting
who can create new streams, as well as some text about why
this is limited to certain roles.

Removes list of actions that can be restricted to full members
due to maintainability concerns for that type of list in the
documentation and replaces it with a short descriptive text
explaining that many settings in Zulip support this restriction.
2022-03-24 11:37:35 -07:00
Aman Agrawal b799ec32b0 upload: Allow rate limited access to spectators for uploaded files.
We allow spectators access to uploaded files in web public streams
but rate limit the daily requests to 1000 per file by default.
2022-03-24 10:50:00 -07:00
Tim Abbott abea1f4598 migrations: Add migration to fix Attachment cache.
This migration needs to be run after the previous commit is deployed
to a given Zulip installation, to fix any stale values of
is_realm_public and is_web_public.
2022-03-24 10:50:00 -07:00
Tim Abbott d149af936d models: Rework Attachment.is_*_public to be a cache.
Previously, Attachment.is_realm_public and its cousin,
Attachment.is_web_public, were properties that began as False and
transitioned to True only when a message containing a link to the
attachment was sent to the appropriate class of stream, or such a link
was added as part of editing a message.

This pattern meant that neither field was updated in situations where
the access permissions for a message changed:

* Moving the message to a different stream.
* Changing the permissions for a stream containing links to the message.

This correctness issue has limited security impact, because uploaded
files are secured both by a random URL and by these access checks.

To fix this, we reformulate these fields as a cache, with code paths
that change the permissions affecting an attachment responsible for
setting these values to the `None` (uncached) state. We prefer setting
this `None` state over computing the correct permissions, because the
correct post-edit permissions are a function of all messages
containing the attachment, and we don't want to be responsible for
fetching all of those messages in the edit code paths.
2022-03-24 10:49:59 -07:00
Alex Vandiver 4f93b4b6e4 uploads: Skip the outgoing proxy if S3_KEY is unset.
When the credentials are provided by dint of being run on an EC2
instance with an assigned Role, we must be able to fetch the instance
metadata from IMDS -- which is precisely the type of internal-IP
request that Smokescreen denies.

While botocore supports a `proxies` argument to the `Config` object,
this is not actually respected when making the IMDS queries; only the
environment variables are read from.  See
https://github.com/boto/botocore/issues/2644

As such, implement S3_SKIP_PROXY by monkey-patching the
`botocore.utils.should_bypass_proxies` function, to allow requests to
IMDS to be made without Smokescreen impeding them.

Fixes #20715.
2022-03-24 10:21:35 -07:00
Anders Kaseorg 24070c7ad8 beeminder: Accept float for payload["goal"]["pledge"].
The documentation at https://api.beeminder.com/#goal says this is
“number”; empirically, we do in fact get decimal points.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-03-22 21:08:48 -04:00
Alex Vandiver 1ac0035f8c markdown: Allow whitespace overlaps in topic linkifiers.
`prepare_linkifier_pattern`, as of db934be064, adds a match to the
end of the regex, of either the end of string, or a non-word character
-- this is in place of a negative look-ahead, which is no longer
possible in re2.  This causes the regex to consume trailing
whitespace, and thus not be able to match twice in succession with
`pattern.finditer` -- "#1234 #5678" fails to match because the space
is consumed by the first match of the regex.

Rather than use `pattern.finditer`, write own own version, which
rewinds over the non-word character consumed after the match, if any.
This allows the same "after" non-word character to also satisfy the
"before" of the next match.

Fixes #21502.
2022-03-22 15:40:03 -07:00
Alex Vandiver abed174b12 uploads: Add an endpoint which forces a download.
This is most useful for images hosted in S3, which are otherwise
always displayed in the browser.
2022-03-22 15:05:02 -07:00
Lauryn Menard b67288db67 help_docs: Extend options for getting links to Zulip content.
Extends the linking to Zulip documentation to cover:

- Getting URLs to messages via the message timestamp.
- Getting links to topics via the three-dots menu.
- Getting links to streams via right-click context menu.

Creates a new tabbed section for using the browser
address bar to copy URLs.
2022-03-22 12:38:09 -07:00
Tim Abbott ceb9dd5854 migrations: Fix confusing output for migration 0383.
* Don't print the empty list for the vast majority of realms where
  this is a noop.

* Make output a little more clear that this isn't revoking all
  Confirmations, just those associated with deactivated users.
2022-03-21 23:25:06 -07:00
Aman Agrawal 7a25a80242 message_edit: Move muted status of the topic when moved between streams.
Add support for moving MutedTopic entries to another stream where
the user has access to shared history in both streams and
`propagate_mode != "change_one"`.

Also, we delete them the current user does not have access to the
target stream.
2022-03-21 12:42:39 -07:00
Tim Abbott 4da2f154e0 create_user: Improve --help output.
This includes documenting this as not the primary way to
programatically create users in Zulip.
2022-03-21 12:05:59 -07:00
Tim Abbott aebd81c440 management: Remove unnecessary default=SUPPRESS logic. 2022-03-21 12:05:59 -07:00
Tim Abbott e16043547b management: Add new create_realm management command.
This is intended for rare situations where one is creating multiple
realms via a script.

After all the preparatory refactoring in this last several commits, we
can now provide a working implementation of a create_realm management
command.
2022-03-21 12:05:59 -07:00
Tim Abbott 2be2393d3e create_user: Extract get_create_user_params.
We set nocoverage for the new function. Ideally it'd eventually get an
automated test, but we don't want to block this helpful refactoring on
doing so.
2022-03-21 12:05:59 -07:00
Tim Abbott be6ab93b37 create_user: Rename pw => password for readability. 2022-03-21 12:05:59 -07:00
Tim Abbott 2bc1cd6ab4 create_user: Fix overly large try/except block.
Only the do_create_user call can throw IntegrityError, and it's a lot
more readable to thus scope the try/except where it belongs.
2022-03-21 12:05:59 -07:00
Tim Abbott ed3569a470 create_user: Simplify parameter processing.
We remove a bit of error handling for cases where someone provided
only one of the email and full name parameters, with the benefit of
this being a lot cleaner.
2022-03-21 12:05:59 -07:00
Tim Abbott 279b99ab23 create_user: Fix unnecessary nesting of input logic. 2022-03-21 12:05:59 -07:00
Tim Abbott 57fa62ae4b initial_password: Add explicit development environment assertion.
The construction of INITIAL_PASSWORD_SALT is such that it should only
be set in development environments, but we should enforce this rule.
2022-03-21 12:05:59 -07:00
Tim Abbott 5393ce11c7 management: Clean up create_user password logic.
* Assert that we're in a development environment when appropriate.
* Add useful logging messages, including print_initial_password details.
2022-03-21 12:05:59 -07:00
Tim Abbott 6b00c748fd onboarding: Deduplicate realm creation initial user logic.
We now call this function inside do_create_user(...,
realm_creation=True), which generally improves readability and
robustness of the codebase.

This fixes a bug where this onboarding content was not correctly done
when creating a realm via LDAP, and also will be important as we add
new code paths that might let you create a realm.
2022-03-21 12:05:59 -07:00
Tim Abbott a117b224a7 onboarding: Refactor setup_realm_internal_bots call.
This improves robustness of any code paths calling do_create_realm,
which previously needed to call this correctly to achieve the same
results as creating a user via the UI.

This also fixes a bug where this code was not called if a realm were
created using the LDAP code path.
2022-03-21 12:05:59 -07:00
Tim Abbott b4507df8fa create_user: Remove tos parameter from management command.
This parameter was introduced in
ea11ce4ae6, and no longer serves a
purpose. Zulip will already correctly record that the user has not
agreed to ToS, and either prompt them on first login or not depending
whether the server is configured to require ToS.
2022-03-21 12:05:59 -07:00
Tim Abbott fddd83394e create_user: Specify tos_version=None explicitly in automation.
This is an important design detail, so we document this aspect of
creating users via both the management command and API code paths with
an explicit parameter value and comment.
2022-03-21 12:05:59 -07:00
Tim Abbott 4f3894f9f1 management: Improve help text for create_user. 2022-03-21 12:05:59 -07:00
Tim Abbott 9761711351 management: Extract add_create_user_args.
This will avoid code duplication when adding a create_realm management command.
2022-03-21 12:05:59 -07:00
Mateusz Mandera af5d0d6f5e bots: Don't allow admins to change owner of bot with can_create_users.
Ordinary organization administrators shouldn't be allowed to change
ownership of a bot with the can_create_users permission.

This is a special permission that is granted manually by server
administrators to an organization (to a UserProfile of the org owners'
choice) after approval by a server administator. The code comments
provide more detail about why this is sensitive.
2022-03-20 17:18:21 -07:00
Simmo Saan 307a7d8104 bitbucket3: Fix docs typo Zuilp -> Zulip 2022-03-20 16:10:03 -07:00
Felix 780bda0d40
integrations: Fix BigBlueButton password length.
The BigBlueButton integration had a problem with generating
the random password with only 12 characters. This would
cause the attendeePW to be the same as the moderatorPW,
which might be fine but seems like something that could be an
error in a future version of BigBlueButton.
2022-03-20 16:09:36 -07:00
strifel a967a86b10 integration: Generate dynamic name for BigBlueButton video calls.
The name for a BigBlueButton meeting is now generated from the stream
name and topic name.

The createTime option is used to have the user redirected to a link
that is only valid for this meeting.

Even if the same link in Zulip is used again, a new createTime
parameter will be created, as the Meeting on the BigBlueButton server
has to be recreated.

Fixes #16498.
Fixes #20509.
Fixes #20804.
2022-03-18 17:27:39 -07:00
Nikhil Maske 5e8c8bfc0f settings_config: Rename "Unread count summary" to "Unread count badge". 2022-03-18 15:29:53 -07:00
Tim Abbott f8146cfaa1 help: Fix label for Subscribed tab in stream settings. 2022-03-18 14:28:20 -07:00
Tim Abbott e45cebd636 message_edit: Fix unmute of topic when topic name is edited.
Previously, when a topic was edited (including being resolved), it
would become unmuted for any users who had muted it, which was
annoying.

While it's not possible to determine the user's intent completely,
this is clearly incorrect behavior in the `change_all` case, such as
resolving a topic.

The comments discuss some scenarios where we might want to enhance
this further, but this is the best we can do without large increases
in complexity.

Fixes #15210.

Co-authored-by: akshatdalton <akshat.dak@students.iiit.ac.in>
2022-03-18 12:32:46 -07:00
byshen-dev 26d97ce7e3 migrations: Change realm field to be not null in Attachment.
he possibility for it being null was likely an oversight -- it should
have been removed after the early migrations to backfill the field
when it was added.

We've confirmed there are no existing violations of this invariant in
Zulip Cloud.
2022-03-18 12:01:15 -07:00
Mateusz Mandera 995cbc69b4 migrations: Add migration to revoke invites from old deactivated users.
This is a natural follow-up to
93e8740218 - invitations sent by users
deactivated before the commit still need to be revoked, via a
migration.

The logic for finding the Confirmations to deactivated is based on
get_valid_invite_confirmations_generated_by_user in actions.py.
2022-03-17 15:16:05 -07:00
Anders Kaseorg a3d89e049a models: Add missing type annotations.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-03-17 13:07:09 -07:00
Anders Kaseorg 62e049b25a models: Type nullable message_content_delete_limit_seconds as Optional.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-03-17 13:07:09 -07:00
Anders Kaseorg f21842e920 requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-03-16 10:43:23 -07:00
Suyash Vardhan Mathur 20a97bdb05 events: Add functionality to mark messages as unread.
Co-authored-by: Steve Howell <showell@zulip.com>
Co-authored-by: Tim Abbott <tabbott@zulip.com>

This commit adds the backend functionality to
mark messages as unread through update_message_flags
with `unread` flag and `remove` operation.

We also manage incoming events in the webapp.

Tweaked by tabbott to simplify the implementation and add an API
feature level update to the documentation.

This commit was originally drafted by showell, and showell
also finalized the changes.  Many thanks to Suyash here for
the main work here, which was to get all the tests and
documentation work moving forward.
2022-03-15 18:00:35 -07:00
Sahil Batra 6f93f07844 types: Create a new TypedDict RealmPlaygroundDict for playground objects.
This commit creates a new TypedDict RealmPlaygroundDict for realm
playground objects. Now the list of playgrounds in the events sent
to clients and the "added_playground" field of RealmAuditLog entry
use RealmPlaygroundDict instead of Dict.
2022-03-15 15:58:36 -07:00
Sahil Batra 4bbb5f18e4 actions: Use transaction.atomic for do_remove_realm_playground. 2022-03-15 15:58:36 -07:00
Sahil Batra 578b752766 actions: Use transaction.atomic for do_add_realm_playground. 2022-03-15 15:58:36 -07:00
Sahil Batra b86b9bdc02 realm: Create RealmAuditLog entry when removing realm playgrounds.
This commit also adds 'acting_user' parameter to
do_remove_realm_playground function.

Fixes a part of #21268.
2022-03-15 15:58:36 -07:00
Sahil Batra dea3389045 realm: Create RealmAuditLog entry when adding realm playgrounds.
This commit also adds 'acting_user' parameter to
do_add_realm_playground function.

Fixes a part of #21268.
2022-03-15 15:58:36 -07:00
Sahil Batra 9fd9c5f1d2 actions: Pass realm_playgrounds as argument to notify_realm_playgrounds.
This commit modifies the notify_realm_playgrounds function to accept
realm_playgrounds as argument from the caller instead of computing it
in the function to avoid duplicate queries since the realm playgrounds
list will be required in its caller functions as well in further commits.
2022-03-15 15:58:36 -07:00
Alex Vandiver 62ba8e455d CVE-2022-24751: Clear sessions outside of the transaction.
Clearing the sessions inside the transaction makes Zulip vulnerable to
a narrow window where the deleted session has not yet been committed,
but has been removed from the memcached cache.  During this window, a
request with the session-id which has just been deleted can
successfully re-fill the memcached cache, as the in-database delete is
not yet committed, and thus not yet visible.  After the delete
transaction commits, the cache will be left with a cached session,
which allows further site access until it expires (after
SESSION_COOKIE_AGE seconds), is ejected from the cache due to memory
pressure, or the server is upgraded.

Move the session deletion outside of the transaction.

Because the testsuite runs inside of a transaction, it is impossible
to test this is CI; the testsuite uses the non-caching
`django.contrib.sessions.backends.db` backend, regardless.  The test
added in this commit thus does not fail before this commit; it is
merely a base expression that the session should be deleted somehow,
and does not exercise the assert added in the previous commit.
2022-03-15 13:52:15 -07:00
Alex Vandiver 7650b5a972 session: Enforce that changes cannot happen in a transaction. 2022-03-15 13:52:15 -07:00
Anders Kaseorg b70a071124 report: Strengthen report_csp_violations type using WildValue.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-03-15 13:02:02 -07:00
Anders Kaseorg 2a0bc2d1c9 crashlytics: Strengthen types using WildValue.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-03-15 13:02:02 -07:00
Anders Kaseorg be781f19e6 codeship: Strengthen types using WildValue.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-03-15 13:02:02 -07:00
Anders Kaseorg d5a8e040da clubhouse: Strengthen types using WildValue.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-03-15 13:02:02 -07:00
Anders Kaseorg 573d264759 circleci: Strengthen types using WildValue.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-03-15 13:02:02 -07:00
Anders Kaseorg 02393ecad7 canarytoken: Strengthen types using WildValue.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-03-15 13:02:02 -07:00
Anders Kaseorg d86fb95ce6 buildbot: Strengthen types using WildValue.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-03-15 13:02:02 -07:00
Anders Kaseorg 42662f22c8 bitbucket3: Strengthen types using WildValue.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-03-15 13:02:02 -07:00
Anders Kaseorg 6528538188 bitbucket2: Strengthen types using WildValue.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-03-15 13:02:02 -07:00
Anders Kaseorg b637ba5f0f bitbucket: Strengthen types using WildValue.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-03-15 13:02:02 -07:00
Anders Kaseorg af565ef32f beeminder: Strengthen types using WildValue.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-03-15 13:02:02 -07:00
Anders Kaseorg 950e3cfcaf beanstalk: Strengthen types using WildValue.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-03-15 13:02:02 -07:00
Anders Kaseorg 2551320a3b beanstalk: Remove pointless payload mutation.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-03-15 13:02:02 -07:00
Anders Kaseorg 4689678546 basecamp: Strengthen types using WildValue.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-03-15 13:02:02 -07:00
Anders Kaseorg ff187411c0 appveyor: Strengthen types using WildValue.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-03-15 13:02:02 -07:00
Anders Kaseorg 04e09d21a3 appfollow: Strengthen types using WildValue.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-03-15 13:02:02 -07:00
Anders Kaseorg 07ae5f54ee ansibletower: Strengthen types using WildValue.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-03-15 13:02:02 -07:00
Anders Kaseorg d436f8098e alertmanager: Strengthen types using WildValue.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-03-15 13:02:02 -07:00
Anders Kaseorg ac9a5ba894 airbrake: Strengthen types using WildValue.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-03-15 13:02:02 -07:00
Anders Kaseorg af67eee3ea bitbucket2: Cast append_punctuation argument to str.
Commit ab8aae6d0c (#12161) incorrectly
assumed that ‘new’ is a string.  In the case of change == "links",
it’s a dict.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-03-15 13:02:02 -07:00
Anders Kaseorg 82081ddb95 bitbucket3: Remove function signature inspection nonsense.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-03-15 13:02:02 -07:00
Anders Kaseorg a34e09215d bitbucket2: Remove function signature inspection nonsense.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-03-15 13:02:02 -07:00
Anders Kaseorg cd2fc198a9 bitbucket2: Untangle totally different control flow for push events.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-03-15 13:02:02 -07:00
Anders Kaseorg 29ecf415fc validator: Add WildValue class for enforcing JSON type checking.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-03-15 13:02:02 -07:00
Anders Kaseorg 04d772b582 request: Support converter or json_validator with argument_type="body".
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-03-15 13:02:02 -07:00
Anders Kaseorg 970f22380a request: Replace default_assigned flag with continue.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-03-15 13:02:02 -07:00
Anders Kaseorg 5f92078d07 request: Add a var_name parameter to converter.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-03-15 13:02:02 -07:00
Anders Kaseorg 5caeb8444b validator: Replace converter=orjson.loads with json_validator.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-03-15 13:02:02 -07:00
Anders Kaseorg c3788ab847 validator: Replace converter=int with json_validator=check_int.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-03-15 13:02:02 -07:00
Alex Vandiver 95892a5ed3 emoji: Support animated PNGs. 2022-03-15 12:47:21 -07:00
Alex Vandiver fc793c10fa tests: Refactor tests of resizing animated images. 2022-03-15 12:47:21 -07:00
Mateusz Mandera b3f1f1675b forms: Mark a bunch of error strings for translation.
These error messages weren't marked for translation.
DEACTIVATED_ACCOUNT_ERROR and PASSWORD_TOO_WEAK_ERROR are used in
several places and imported, so we can't move them to be in-line errors
and we keep them at top-level, marked with gettext_lazy.
2022-03-15 12:43:10 -07:00
Mateusz Mandera 3822ce6d35 forms: Remove incorrect use of mark_safe on some errors.
Using mark_safe on errors with content in them taken from user-input is
a clearly bad idea. With that said, this code
was not exploitable in the current state, given that username is a value
you have to POST to /login/, and the endpoint is CSRF-protected.

We also remove use of mark_safe from the errors without user input them,
but that are just plaintext and thus don't need it.
2022-03-15 12:43:10 -07:00
Lauryn Menard 6cc818b672 help_docs: Document user management via user profile modal.
Adds documentation for admins to manage users via the user profile
modal for these actions:
- Deactivating a user
- Changing a user's role
- Changing a user's name

Creates two new tab sections because we still want to document
the ability to do these actions through the users section in
the organizational settings modal.

Also cleans up some text in the help center article for changing
a user's role.

Fixes #21318.
Fixes #21415.
2022-03-15 12:35:03 -07:00
Lauryn Menard d327ecaa65 help_docs: Include permissions setting info in user groups article.
Adds content on user group permissions / management to the general
help center article for user groups (`/help/user-groups`) and
removes the then redundant `/help/restrict-user-group-management`
article.

Redirects links in help center and api documentation from deleted
article to the new configure user group settings section of
`/help/user-groups`.

Fixes #21383.
2022-03-15 12:30:16 -07:00
Sahil Batra f0606b34ad user_groups: Add cron job for adding users to full members system group.
This commit adds a cron job which runs every hour to add the users to
full members system group if user is promoted to a full member.

This should ensure that full member status is available no more than
an hour after configuration suggests it should be.
2022-03-14 18:53:47 -07:00
Sahil Batra 565d0928dd import: Create system groups and add users to it while importing.
There can be cases when system groups data is not present while
importing, like when importing from other products, so this
commit adds code to create system user groups and add users to
it according to their role.
2022-03-14 18:53:47 -07:00
Sahil Batra bbf4c25553 user_groups: Update user groups on changing waiting_period_threshold.
The members of "Full members" user group are updated with change in
waiting_period_threshold setting.
2022-03-14 18:53:47 -07:00
Sahil Batra 9345c344d7 user_groups: Add members to the System groups created.
This commit adds users to the appropriate system user group
based on their role. We also change the user groups when
changing role of the user.

We also add migration to add existing users to the appropriate
user groups.

This commit adds update_users_in_full_members_system_group which
is currently used to update the full members group on changing
role of a user. This function will be modified in next commit such
that it can be used to update full members group on changing
waiting_period_threshold setting of realm.
2022-03-14 18:53:46 -07:00
Sahil Batra c33ba4ed6e realm: Create role-based system user groups on creating realm.
We create system user groups for following roles - owners,
admins, moderators, members and guests. Full members user
group will be handled separately.
2022-03-14 18:52:47 -07:00
Sahil Batra f31bb6754b user_groups: Pass list of user ids to remove_members_from_user_group.
We pass list of user ids instead of user profile objects to
remove_members_from_user_group. We still need to call user_id_to_users
in the views function instead of directly passing the ids to
remove_members_from_user_group to make sure we check whether all
ids are valid or not.
2022-03-14 18:52:47 -07:00
Sahil Batra 303fca2ce7 user_groups: Pass list of user ids to bulk_add_members_to_user_group.
We pass list of user ids instead of user profile objects to
bulk_add_members_to_user_group. We still need to call user_id_to_users
in the views function instead of directly passing the ids to
bulk_add_members_to_user_group to make sure we check whether all
ids are valid or not.
2022-03-14 18:52:47 -07:00
Sahil Batra b48b98f1d9 user_groups: Use atomic transaction for adding/removing members.
We execute code for bulk adding and removing members from a user
group inside a transaction.
2022-03-14 18:52:47 -07:00
Steve Howell f9ddd80122 subscriptions: Improve types for subscription info.
The newly extracted module here has significant changes
to all the functions, but they achieve the same thing
as before.
2022-03-14 18:01:39 -07:00
Steve Howell 8f99894302 streams: Extract stream_color library.
This is a pure code move.
2022-03-14 18:01:36 -07:00
Steve Howell 200a89cef0 streams: Extract stream_traffic library.
This is a pure code move.
2022-03-14 18:01:36 -07:00
Steve Howell c44d782305 schema: Sort schema fields. 2022-03-14 18:01:36 -07:00
Steve Howell d86184be34 streams: Sort API fields. 2022-03-14 18:01:36 -07:00
Mateusz Mandera 76ff9b30b1 push_notifs: Log both user id and user uuid if we have them.
Previous behavior was logging only the uuid if it was provided by the
remote server, but that's insufficient, because the user may actually
have no devices registered with uuis and we (at the bouncer) end up
sending notifications to id-based registrations. Not having that id
logged makes it impossible to figure out what's going on.
2022-03-14 17:47:30 -07:00
Mateusz Mandera d800ac33a0 push_notifications: Send user_uuid to the push bouncer.
Fixes #18017.

In previous commits, the change to the bouncer API was introduced to
support this and then a series of migrations added .uuid to
UserProfiles.

Now the code for self-hosted servers that makes requests
to the bouncer is changed to make use of it.
2022-03-14 17:47:30 -07:00
Mateusz Mandera c3209d379c models: Make UserProfile.uuid not nullable.
This is in a separate commit to make deployment easier. It ensures that
this is only marked non-null after the backfill migration (backfilling
.uuid for all old UserProfiles) runs - which was added in the previous
commit.
2022-03-14 17:47:30 -07:00
Mateusz Mandera 3e2ad84bbe models: Add UserProfile.uuid field and backfill migrations. 2022-03-14 17:47:30 -07:00
Mateusz Mandera 0677c90170 zilencer: Change push bouncer API to accept uuids as user identifier.
This is the first step to making the full switch to self-hosted servers
use user uuids, per issue #18017. The old id format is still supported
of course, for backward compatibility.

This commit is separate in order to allow deploying *just* the bouncer
API change to production first.
2022-03-14 17:47:30 -07:00
Anders Kaseorg 560cf06b72 makemessages: Ignore compiled and custom email templates.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-03-14 11:19:35 -07:00
Kartik Srivastava eefaa9120f user_topic: Rename topic_mutes.py to user_topics.py. 2022-03-11 14:26:55 -08:00
Kartik Srivastava ce38eda54d test_events: Fix 'normalize' assuming subscription data is present.
This avoids a crash in normalize for tests that don't include these in
fetch_event_types.
2022-03-11 14:26:14 -08:00
Aman Agrawal 82837304ec api: Send full message in GET /messages/{message_id} response.
Previously, this URL just returned the `raw_content` field. It seems
cleanest to just make it a single-message variant of GET /messages,
deprecating the only format.
2022-03-11 10:25:22 -08:00
Tim Abbott 20368a936c settings: Add web-public streams beta subdomain list.
This will make it convenient to add a handful of organizations to the
beta of this feature during its first few weeks to try to catch bugs,
before we open it to everyone in Zulip Cloud.
2022-03-10 18:37:01 -08:00
Sahil Batra 57f01e0727 actions: Use transaction.atomic for do_remove_realm_domain. 2022-03-10 17:48:02 -08:00
Sahil Batra 5999dcd316 actions: Use transaction.atomic for do_change_realm_domain. 2022-03-10 17:48:02 -08:00
Sahil Batra 9b9931df7f actions: Use transaction.atomic for do_add_realm_domain. 2022-03-10 17:48:02 -08:00
Sahil Batra 07352271b9 realm: Create RealmAuditLog entry for removing a realm domain.
Fixes a part of #21268.
2022-03-10 17:48:02 -08:00
Sahil Batra ab5567e8c5 realm: Create RealmAuditLog entry while changing an allowed domain.
This commit also adds 'acting_user' argument for do_change_realm_domain
function.

Fixes a part of #21268.
2022-03-10 17:48:02 -08:00
Sahil Batra 5ef8da40a9 realm: Create RealmAuditLog entry while adding a new allowed domain.
This commit also adds 'acting_user' argument for do_add_realm_domain
function.

Fixes a part of #21268.
2022-03-10 17:48:02 -08:00
Sahil Batra ee11a68f7a models: Fix return type of get_realm_domains.
The correct return type of get_realm_domains should
be List[Dict[str, Union[bool, str]]] instead of
List[Dict[str, str]] because allowed_subdomains is
a bool field not str.
2022-03-10 17:48:02 -08:00
Alya Abbott a80e470c9e portico: Create use cases index page. 2022-03-10 16:34:34 -08:00
Steve Howell 8e05a9fcf7 unread: Replace sender_id with other_user_id.
Note that we still send sender_id for legacy mobile
clients.
2022-03-10 13:33:21 -08:00
Steve Howell 822c232e37 message flags: Extract create_historical_user_messages.
A user can subscribe to a stream and sometimes (depending
on stream permissions) see messages from the stream
that were sent before they subscribed, and that user
won't have a UserMessage row for that message.

In order to do things like star a message, we need
to create UserMessage records on the fly.

In the past we wisely constrained this logic to the
specific use cases. But I think we can generalize
the logic now.  For example, we are now building a
feature to mark messages as unread, and it motivates
the same need to auto-create UserMessage rows.

So now we handle this in a more generalized fashion.
2022-03-10 13:21:02 -08:00
Lauryn Menard 05a548f5a3 api_docs: Refactor of `MessagesBase` schema.
Refactors and cleans up the shared `MessagesBase` schema in
the OpenAPI so that it accurately reflects the general base
for message objects for endpoints that use it as a reference.

A follow-up to adding `edit_history` as a property of message
objects. And a prepartory commit for `GET /messages/{msg_id}`
to return not only the raw content of the message but also
the message object.
2022-03-10 13:10:14 -08:00
Steve Howell a90d9ef536 unread: Remove unused client parameter. 2022-03-10 10:04:51 -05:00
Alex Vandiver c2f2863d37 push_notifications: Remove access control on "remove" notifications.
When removing notifications, we skip the access control on if the user
still can read them -- they should not have a notification of them,
both because they currently cannot read the message, as well as
because they have already done so.
2022-03-09 16:33:51 -08:00
Alex Vandiver 19dfd8e6a7 push_notifications: Ensure notifications are on for the remove codepath.
This causes it to mirror the handle_push_notification codepath.
2022-03-09 16:33:51 -08:00
Tim Abbott d693a6717c i18n: Remove quote syntax from stream description notification.
Translators found it confusing, since it's not at all obvious that the
word "quote" should not be translated.

I'm not altogether happy with the code formatting for this.

While we're changing this, also standardize on the "```` quote" style
of quote blocks to ensure code/quote blocks in stream descriptions are
unlikely to conflict with this syntax.
2022-03-09 15:22:57 -08:00
Alex Vandiver 3264361f63 push_notifications: Set a timeout on FCM requests.
In steady-state, requests to FCM take about a second; however, in
cases where the remote FCM server is unstable, the request has been
observed to block for more than a minute.

As noted in the previous commit, pushes must complete within 30s;
fail fast, and let the retries and exponential backoff handle errors.

The worst-case total time taken with timeouts and errors for an FCM
notification is 19.5s.  Unfortunately, `aioapns` does not appear to
have any timeouts, and thus this commit cannot guarantee a total of
fewer than 30s.
2022-03-08 12:52:58 -08:00
Alex Vandiver f531f3a27f push_notifications: Drop FCM retries to 2, not 10.
This reverts bc15085098 (which provided
not justification for its change) and moves further, down to 2 retries
from the default of 5.

10 retries, with exponential backoff, is equivalent to sleeping 2^11
seconds, or just about 34 minutes (though the code uses a jitter which
may make this up to 51 minutes).  This is an unreasonable amount of
time to spend in this codepath -- as only one worker is used, and it
is single-threaded, this could effectively block all missed message
notifications for half an hour or longer.

This is also necessary because messages sent through the push bouncer
are sent synchronously; the sending server uses a 30-second timeout,
set in PushBouncerSession.  Having retries which linger longer than
this can cause duplicate messages; the sending server will time out
and re-queue the message in RabbitMQ, while the push bouncer's request
will continue, and may succeed.

Limit to 2 retries (APNS currently uses 3), and results expected max
of 4 seconds of sleep, potentially up to 6.  If this fails, there
exists another retry loop above it, at the RabbitMQ layer (either
locally, or via the remote server's queue), which will result in up to
3 additional retries -- all told, the request will me made to FCM up
to 12 times.
2022-03-08 12:52:58 -08:00
byshen-dev 73bc5480f3 models: Add unique constraint on RealmUserDefault.realm.
This model is by designed intended to exist on a 1:1 relationship with
Realms, and we attempt to ensure that with application code, but we
should have a unique constraint too, since a database with duplicate
such entries would be corrupted.

We do this via the standard Django OneToOneField.
2022-03-07 21:43:07 -08:00
Anders Kaseorg 43ee1f7b93 tests: Avoid use of Python internal __unittest_skip__ flag.
It was there to work around https://bugs.python.org/issue17519.  This
workaround with del seems like a partial improvement.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-03-07 16:26:37 -08:00
Steve Howell df47e4312b unread: Refactor aggregated message info.
We want TypedDicts that have actual teeth.

In order to make type checks meaningful, we want
to avoid Any, object, or crazy Union types when
we aggregate each type of message, so we replaced
a generic function with three concrete functions.
2022-03-07 15:34:56 -08:00
Mateusz Mandera dec092751c tests: Mark test_migrations tests to be automatically skipped.
This fixes an issue where these tests will fail when adding new
migrations on top of them.
2022-03-07 15:33:29 -08:00
Dinesh 9f8d60cd5a actions: Mention that stream can also be changed with do_update_message(). 2022-03-07 12:44:36 -08:00
NerdyLucifer 4b9770e270 stream_settings: Show stream privacy & description in stream events.
Provide stream privacy and description in stream notification events
when stream is created.
In function "send_messages_for_new_subscribers" for when stream is
created, put policy name and description of the stream.

Fixes #21004
2022-03-07 11:53:49 -08:00
Tim Abbott 8376acb3b8 edit_history: Add comments detailing when user_id can be null.
This is already in the API documentation, but I think it's worth
having here.
2022-03-04 11:54:03 -08:00
Lauryn Menard 58f21fc748 edit_history: Remove `LEGACY_PREV_TOPIC` constant from code base.
Removes `LEGACY_PREV_TOPIC` which is no longer needed due to the
message edit history migration.

Also remove additions to the linter exclude list that were added
earlier in this commit series.
2022-03-04 10:25:48 -08:00
Lauryn Menard 072051f81e api: Add additional fields to `edit_history` entries.
Since we've changed the database to contain these new fields, we just
need to stop dropping them in the API code.

This also changes the public API to match the database format again
by removing `prev_subject` from edit history API.

Adds an API changelog feature update for the renamed `prev_subject`
field (to `prev_topic`) and new fields (`topic` and `stream`)
in the message `edit_history`.

Also, documents said `edit_history` in the `MessagesBase` schema
in the api documentation, which is used by the `/get-messages`,
`/get-events` and `/zulip-outgoing-webhooks` endpoints.

Fixes #21076.

Co-authored-by: Lauryn Menard <lauryn.menard@gmail.com>
2022-03-04 10:25:48 -08:00
Tim Abbott 4e91d03d56 edit_history: Remove prev_subject from EditHistoryEvent type.
It is no longer possible for this field to appear in actual entries.
2022-03-04 10:25:48 -08:00
Tim Abbott 7e298365dc edit_history: Migrate database to modernized edit history format.
This migration can only be safely run after the previous commit has
been deployed.
2022-03-04 10:25:48 -08:00
Tim Abbott a3a9adbfa9 edit_history: Stop writing prev_subject to the database.
Now that we have code to support reading prev_topic, this is no longer
necessary.

We'll want to deploy this change to production before running the
migration to remove prev_subject from edit history entries, so that
prev_subject can be fully purged from the database.
2022-03-04 10:25:48 -08:00
Tim Abbott 5e819cdebc edit_history: Support prev_subject not being present in database.
This commit serves as a checkpoint that can be deployed to production
before running the big Message format migration in the next commit.
2022-03-04 10:25:48 -08:00
Tim Abbott a410eee2e1 edit_history: Store additional fields on edit history events.
We modify the message_edit_history marshalling code so that this
commit does not change the API, since we haven't backfilled the data
yet.

FormattedEditHistoryEvent, introduced in the previous commit, doesn't
directly inherit fields from EditHistoryEvent, so no changes are
required there.
2022-03-04 10:25:48 -08:00
Tim Abbott 85222b790d edit_history: Refactor to use FormattedEditHistoryEvent type.
We fix the mutation of caller and other bad patterns, as well as
adding explicit typing to make the code readable.

We also update the OpenAPI documentation for previously
undocumented `prev_strem` field in the `/get-message-history`
endpoint for API validation testing.

Co-authored-by: Lauryn Menard <lauryn.menard@gmail.com>
2022-03-04 10:25:48 -08:00
Tim Abbott f1e5ed91a1 types: Add EditHistoryEvent and APIEditHistoryEvent types.
These types will help make iteration on this code easier.

Note that `user_id` can be null due to the fact that
edit history entries before March 2017 did not log
the user that made the edit, which was years after
supporting topic edits (discovered in test deployment
of migration on chat.zulip.org).

Co-authored-by: Lauryn Menard <lauryn.menard@gmail.com>
2022-03-04 10:25:48 -08:00
Lauryn Menard 44e08c9289 message: Rename edit_history JSON string variable.
Co-authored by: Tim Abbott <tabbott@zulip.com>
2022-03-04 10:25:48 -08:00
Austin Riba c0828f118b hash_util: Move enc/decodeHashComponent to shared.
Moves the encodeHashComponent and decodeHashComponent functions out of
hash_util and into internal_url which belongs to shared. This is to
accommodate sharing of this code with mobile or any other codebases that
do not wish to duplicate logic.
2022-03-01 18:14:31 -08:00
Mateusz Mandera 3c5ea830a9 create_preregistration_user: Add additional hardening assertion.
TestMaybeSendToRegistration needs tweaking here, because it wasn't
setting the subdomain for the dummy request, so
maybe_send_to_registration was actually running with realm=None, which
is not right for these tests.

Also, test_sso_only_when_preregistration_user_exists was creating
PreregistrationUser without setting the realm, which was also incorrect.
2022-02-25 14:02:24 -08:00
Mateusz Mandera 7b795b6338 registration: Change create_preregistration_user to take realm as arg.
create_preregistration_user is a footgun, because it takes the realm
from the request. The calling code is supposed to validate that
registration for the realm is allowed
first, but can sometimes do that on "realm" taken from something else
than the request - and later on calls create_preregistration_user, thus
leading to prereg user creation on unvalidated request.realm.

It's safer, and makes more sense, for this function to take the intended
realm as argument, instead of taking the entire request. It follows that
the same should be done for prepare_activation_url.
2022-02-25 14:02:24 -08:00
Mateusz Mandera ec5e12ef4e tests: Fix some instances of logged in session polluting test state.
In these tests, the code ends up with a logged in session when it's
undesired - later on these tests make requests to a different subdomain
- where a logged in session is not supposed to exist. This leads to an
unintended, strange situation where request.user is a user from the old
subdomain but the request itself is to a *different* subdomain. This
throws off get_realm_from_request, which will return the realm from
request.user.realm - which is not what these tests want and can lead to
these tests failing when some of the production code being tested
switches to using get_realm_from_request instead of
get_realm(get_subdomain).
2022-02-25 14:02:24 -08:00
Mateusz Mandera 88917019f0 CVE-2022-21706: Prevent use of multiuse invites to join other orgs.
The codepaths for joining an organization via a multi-use invitation
(accounts_home_from_multiuse_invite and maybe_send_to_registration)
weren't validating whether
the organization the invite was generated for matches the organization
the user attempts to join - potentially allowing an attacker with access
to organization A to generate a multi-use invite and use it to join
organization B within the same deployment, that they shouldn't have
access to.
2022-02-25 14:02:24 -08:00
Mateusz Mandera d5db254ca8 CVE-2021-3967: Only regenerate the API key by authing with the old key. 2022-02-25 14:00:52 -08:00
Sahil Batra 392b17da5f invite: Add backend support for "Never expires" option.
The database value for expiry_date is None for the invite
that will never expire and the clients send -1 as value
in the API similar to the message retention setting.

Also, when passing invite_expire_in_days as an argument
in various functions, invite_expire_in_days is passed as
-1 for "Never expires" option since invite_expire_in_days
is an optional argument in some functions and thus we cannot
pass "None" value.
2022-02-24 16:32:19 -08:00
Anders Kaseorg 1682a6a50d migrations: Lazily import zerver.lib.streams from 0206.
A migration should not import zerver.lib.streams at all, but this
solves the immediate problem with check-database-compatibility.py.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-24 14:31:24 -08:00
Anders Kaseorg 9a80847366 signals: Lazily import zerver.actions.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-24 14:31:24 -08:00
Anders Kaseorg 21cd1c10b3 docs: Add missing space in “time zone”.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-24 14:05:12 -08:00
Lauryn Menard 47056ef06f tests: Remove `client` parameter if test can use default `User-Agent`.
Removes `client` parameter from backend tests using the
`POST /messages` endpoint when the test can use the default
`User-Agent` as the client, which is set to `ZulipMobile` for API
requests and a browser user agent string for web app requests.
2022-02-24 12:57:18 -08:00
Lauryn Menard bd0ddbe20e tests: Remove unused `stream_name` from tests.
Various tests use the `PATCH /stream/{stream_id} endpoint in
`test_subs.py`. Because the stream id is in the URL path, it
does not also need to be passed as a query parameter.

Removes instances of `stream_name` being passed as a query
parameter to tests.
2022-02-24 12:57:18 -08:00
Lauryn Menard acb7abb57c tests: Remove unused `topic_name` parameter in tests.
Removes `topic_name` parameter in `test_message_flags.py`
where is being passed to a test for marking a stream as
read because it is an ignored parameter for that endpoint.
2022-02-24 12:57:18 -08:00
Anders Kaseorg 80def8d2c2 models: Manage indexes from migration 0001 with Django.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-23 11:59:45 -08:00
Anders Kaseorg 5703b581fd models: Manage index from migration 0082 with Django.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-23 11:59:45 -08:00
Anders Kaseorg 0e7ee68bcd models: Manage index from migration 0083 with Django.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-23 11:59:45 -08:00
Anders Kaseorg 001a310d75 models: Manage index from migration 0095 with Django.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-23 11:59:45 -08:00
Anders Kaseorg ddd217efdf models: Manage index from migration 0098 with Django.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-23 11:59:45 -08:00
Anders Kaseorg 09bd08ba0d models: Manage index from migration 0099 with Django.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-23 11:59:45 -08:00
Anders Kaseorg 903e219654 models: Manage index from migration 0112 with Django.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-23 11:59:45 -08:00
Anders Kaseorg 71f2b43336 models: Manage index from migration 0177 with Django.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-23 11:59:45 -08:00
Anders Kaseorg 23c8a98157 models: Manage index from migration 0180 with Django.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-23 11:59:45 -08:00
Anders Kaseorg cf0eb46afc models: Manage index from migration 0244 with Django.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-23 11:59:45 -08:00
Anders Kaseorg 0799acf3b6 models: Manage indexes from migration 0279 with Django.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-23 11:59:45 -08:00