Commit Graph

15038 Commits

Author SHA1 Message Date
Tim Abbott 4e91d03d56 edit_history: Remove prev_subject from EditHistoryEvent type.
It is no longer possible for this field to appear in actual entries.
2022-03-04 10:25:48 -08:00
Tim Abbott 7e298365dc edit_history: Migrate database to modernized edit history format.
This migration can only be safely run after the previous commit has
been deployed.
2022-03-04 10:25:48 -08:00
Tim Abbott a3a9adbfa9 edit_history: Stop writing prev_subject to the database.
Now that we have code to support reading prev_topic, this is no longer
necessary.

We'll want to deploy this change to production before running the
migration to remove prev_subject from edit history entries, so that
prev_subject can be fully purged from the database.
2022-03-04 10:25:48 -08:00
Tim Abbott 5e819cdebc edit_history: Support prev_subject not being present in database.
This commit serves as a checkpoint that can be deployed to production
before running the big Message format migration in the next commit.
2022-03-04 10:25:48 -08:00
Tim Abbott a410eee2e1 edit_history: Store additional fields on edit history events.
We modify the message_edit_history marshalling code so that this
commit does not change the API, since we haven't backfilled the data
yet.

FormattedEditHistoryEvent, introduced in the previous commit, doesn't
directly inherit fields from EditHistoryEvent, so no changes are
required there.
2022-03-04 10:25:48 -08:00
Tim Abbott 85222b790d edit_history: Refactor to use FormattedEditHistoryEvent type.
We fix the mutation of caller and other bad patterns, as well as
adding explicit typing to make the code readable.

We also update the OpenAPI documentation for previously
undocumented `prev_strem` field in the `/get-message-history`
endpoint for API validation testing.

Co-authored-by: Lauryn Menard <lauryn.menard@gmail.com>
2022-03-04 10:25:48 -08:00
Tim Abbott f1e5ed91a1 types: Add EditHistoryEvent and APIEditHistoryEvent types.
These types will help make iteration on this code easier.

Note that `user_id` can be null due to the fact that
edit history entries before March 2017 did not log
the user that made the edit, which was years after
supporting topic edits (discovered in test deployment
of migration on chat.zulip.org).

Co-authored-by: Lauryn Menard <lauryn.menard@gmail.com>
2022-03-04 10:25:48 -08:00
Lauryn Menard 44e08c9289 message: Rename edit_history JSON string variable.
Co-authored by: Tim Abbott <tabbott@zulip.com>
2022-03-04 10:25:48 -08:00
Austin Riba c0828f118b hash_util: Move enc/decodeHashComponent to shared.
Moves the encodeHashComponent and decodeHashComponent functions out of
hash_util and into internal_url which belongs to shared. This is to
accommodate sharing of this code with mobile or any other codebases that
do not wish to duplicate logic.
2022-03-01 18:14:31 -08:00
Mateusz Mandera 3c5ea830a9 create_preregistration_user: Add additional hardening assertion.
TestMaybeSendToRegistration needs tweaking here, because it wasn't
setting the subdomain for the dummy request, so
maybe_send_to_registration was actually running with realm=None, which
is not right for these tests.

Also, test_sso_only_when_preregistration_user_exists was creating
PreregistrationUser without setting the realm, which was also incorrect.
2022-02-25 14:02:24 -08:00
Mateusz Mandera 7b795b6338 registration: Change create_preregistration_user to take realm as arg.
create_preregistration_user is a footgun, because it takes the realm
from the request. The calling code is supposed to validate that
registration for the realm is allowed
first, but can sometimes do that on "realm" taken from something else
than the request - and later on calls create_preregistration_user, thus
leading to prereg user creation on unvalidated request.realm.

It's safer, and makes more sense, for this function to take the intended
realm as argument, instead of taking the entire request. It follows that
the same should be done for prepare_activation_url.
2022-02-25 14:02:24 -08:00
Mateusz Mandera ec5e12ef4e tests: Fix some instances of logged in session polluting test state.
In these tests, the code ends up with a logged in session when it's
undesired - later on these tests make requests to a different subdomain
- where a logged in session is not supposed to exist. This leads to an
unintended, strange situation where request.user is a user from the old
subdomain but the request itself is to a *different* subdomain. This
throws off get_realm_from_request, which will return the realm from
request.user.realm - which is not what these tests want and can lead to
these tests failing when some of the production code being tested
switches to using get_realm_from_request instead of
get_realm(get_subdomain).
2022-02-25 14:02:24 -08:00
Mateusz Mandera 88917019f0 CVE-2022-21706: Prevent use of multiuse invites to join other orgs.
The codepaths for joining an organization via a multi-use invitation
(accounts_home_from_multiuse_invite and maybe_send_to_registration)
weren't validating whether
the organization the invite was generated for matches the organization
the user attempts to join - potentially allowing an attacker with access
to organization A to generate a multi-use invite and use it to join
organization B within the same deployment, that they shouldn't have
access to.
2022-02-25 14:02:24 -08:00
Mateusz Mandera d5db254ca8 CVE-2021-3967: Only regenerate the API key by authing with the old key. 2022-02-25 14:00:52 -08:00
Sahil Batra 392b17da5f invite: Add backend support for "Never expires" option.
The database value for expiry_date is None for the invite
that will never expire and the clients send -1 as value
in the API similar to the message retention setting.

Also, when passing invite_expire_in_days as an argument
in various functions, invite_expire_in_days is passed as
-1 for "Never expires" option since invite_expire_in_days
is an optional argument in some functions and thus we cannot
pass "None" value.
2022-02-24 16:32:19 -08:00
Anders Kaseorg 1682a6a50d migrations: Lazily import zerver.lib.streams from 0206.
A migration should not import zerver.lib.streams at all, but this
solves the immediate problem with check-database-compatibility.py.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-24 14:31:24 -08:00
Anders Kaseorg 9a80847366 signals: Lazily import zerver.actions.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-24 14:31:24 -08:00
Anders Kaseorg 21cd1c10b3 docs: Add missing space in “time zone”.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-24 14:05:12 -08:00
Lauryn Menard 47056ef06f tests: Remove `client` parameter if test can use default `User-Agent`.
Removes `client` parameter from backend tests using the
`POST /messages` endpoint when the test can use the default
`User-Agent` as the client, which is set to `ZulipMobile` for API
requests and a browser user agent string for web app requests.
2022-02-24 12:57:18 -08:00
Lauryn Menard bd0ddbe20e tests: Remove unused `stream_name` from tests.
Various tests use the `PATCH /stream/{stream_id} endpoint in
`test_subs.py`. Because the stream id is in the URL path, it
does not also need to be passed as a query parameter.

Removes instances of `stream_name` being passed as a query
parameter to tests.
2022-02-24 12:57:18 -08:00
Lauryn Menard acb7abb57c tests: Remove unused `topic_name` parameter in tests.
Removes `topic_name` parameter in `test_message_flags.py`
where is being passed to a test for marking a stream as
read because it is an ignored parameter for that endpoint.
2022-02-24 12:57:18 -08:00
Anders Kaseorg 80def8d2c2 models: Manage indexes from migration 0001 with Django.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-23 11:59:45 -08:00
Anders Kaseorg 5703b581fd models: Manage index from migration 0082 with Django.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-23 11:59:45 -08:00
Anders Kaseorg 0e7ee68bcd models: Manage index from migration 0083 with Django.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-23 11:59:45 -08:00
Anders Kaseorg 001a310d75 models: Manage index from migration 0095 with Django.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-23 11:59:45 -08:00
Anders Kaseorg ddd217efdf models: Manage index from migration 0098 with Django.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-23 11:59:45 -08:00
Anders Kaseorg 09bd08ba0d models: Manage index from migration 0099 with Django.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-23 11:59:45 -08:00
Anders Kaseorg 903e219654 models: Manage index from migration 0112 with Django.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-23 11:59:45 -08:00
Anders Kaseorg 71f2b43336 models: Manage index from migration 0177 with Django.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-23 11:59:45 -08:00
Anders Kaseorg 23c8a98157 models: Manage index from migration 0180 with Django.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-23 11:59:45 -08:00
Anders Kaseorg cf0eb46afc models: Manage index from migration 0244 with Django.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-23 11:59:45 -08:00
Anders Kaseorg 0799acf3b6 models: Manage indexes from migration 0279 with Django.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-23 11:59:45 -08:00
Anders Kaseorg bbca3d048c models: Register custom __andz, __andnz lookups.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-23 11:59:45 -08:00
Anders Kaseorg 1fa2761790 upgrade-zulip-stage-2: Remove create_large_indexes optimization.
This was only used for upgrading from Zulip < 1.9.0, which is no
longer possible because Zulip < 2.1.0 had no common supported
platforms with current main.

If we ever want this optimization for a future migration, it would be
better implemented using Django merge migrations.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-23 11:59:45 -08:00
YashRE42 f6458db183 openapi: Fix typo in description of emojiset parameter. 2022-02-22 16:20:44 -08:00
evykassirer 8c3ff92964 populate_db: Add timezones for some test users.
After failing to notice a place where we wanted to hide timezone
information, we decided to add timezones to some of the test
users, so that we can better consider the effects of timezones
when manually testing.

Testing:

* ran populate_db and confirmed users had timezones in the UI
* updated test_populate_db.py
2022-02-22 11:14:58 -08:00
Aman Agrawal 762cf72305 events: Remove some properties of user sent to spectators. 2022-02-22 10:36:08 -08:00
Lauryn Menard 8e1f62af89 api: Include realm_web_public_access_enabled in get-server-settings.
Adds `realm_web_public_access_enabled` as a realm-specific server
setting potentially returned by the `/get-server-settings` endpoint
so that clients that support browsing of web-public stream content
without an account can generate a login page that supports that
type of access.
2022-02-22 10:11:11 -08:00
Lauryn Menard 7a7f3337c1 tests: Fix unused `message_id` parameter in tests.
Various backend tests use the `PATCH /messages/{msg_id}` endpoint.
For that endpoint, the message ID is encoded in the URL path and
ignored if provided as a parameter in the the query.

Verified that the tests were providing the same message ID to both
the path and then removed the ignored parameter in the query.
2022-02-21 08:52:33 -08:00
Sahil Batra 630b37f2d8 users: Refactor get_user_by_email.
This commit refactors get_user_by_email function
to use access_user_by_email which is similar to
already existing access_user_by_id and thus using
get_user_data function added recently.

We also remove the unnecessary check for email as
email will always be passed to this endpoint.

Preparatory commit for #10970.
2022-02-21 06:47:54 -08:00
Sahil Batra 5471584fa7 users: Add get_user_data function.
This commit adds get_user_data which is called by
get_members_backend to compute the client_gravatar
value and then return the data of a single user or
all accessible users.

This function will also be used by get_user_by_email
in further commtis.
2022-02-21 06:47:54 -08:00
Sahil Batra a46501353d users: Extract access_user_common function.
This function is currently used by access_user_by_id
and would be used by access_user_by_email to be added
in further commits.
2022-02-21 06:47:54 -08:00
Anders Kaseorg 4d858b874e python: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-18 18:03:13 -08:00
Anders Kaseorg 1629d6bfb3 python: Reformat with Black 22 (stable).
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-18 18:03:13 -08:00
Alex Vandiver e53f9fad29 url_preview: Only return image URLs that validate as URLs. 2022-02-18 15:32:27 -08:00
Alex Vandiver dfe3727b6d deliver_scheduled_emails: Add a try/except KeyboardInterrupt.
This makes explicit ^C's to exit not spew a backtrace.
2022-02-18 14:04:56 -08:00
Alex Vandiver c541276ab3 deliver_scheduled_emails: Ue a 10s sleep instead of 2s. 2022-02-18 14:04:56 -08:00
Alex Vandiver 335ef26126 deliver_scheduled_emails: Lock and send emails one-at-a-time.
When pulling batches out of the ScheduledEmail list in a single
transaction, an unexpected failure to send an email will result in the
whole batch getting retried.  This will result in infinite email
sending loops.

Pull a single row off at a time and send it.  We continue without
retries to the next email on EmailNotDeliveredException, but will
retry infinitely on other exceptions.

Fixes: #20943.
2022-02-18 11:13:14 -08:00
Shantanu 4371705a4c streams: Update formatting of retention change notification.
This changes the notifications to be more readable like our other
similar stream events notifications.

Fixes #20969.
2022-02-18 11:11:38 -08:00
Alya Abbott e93320c40d portico: Add a self-hosting page.
Co-authored-by: Aman Agrawal <amanagr@zulip.com>
2022-02-17 12:43:13 -08:00
Alex Vandiver a40b3e1118 realm_emoji: Stop swallowing all exceptions from upload_emoji_image.
Putting all of the logic in a `finally` block is equivalent to a bare
`except` block, which silently consumes all exceptions.

Move only the most-necessary parts into the except; this lets
`BadImageError` exceptions from `zerver/lib/upload.py` to escape,
allowing better the generic "Image file upload failed" to be replaced
with a more specific message.

It also allows unexpected exceptions, as the previous commit resolved,
to escape and 500.  This lets them be detected and resolved, rather
than give users a silently bad experience.
2022-02-17 12:19:47 -08:00
Alex Vandiver 96a5fa9d78 upload: Fix resizing non-animated images.
5dab6e9d31 began honoring the list of disposals for every frame.
Unfortunately, passing a list of disposals for a non-animated image
raises an exception:
```
  File "zerver/lib/upload.py", line 212, in resize_emoji
    image_data = resize_gif(im, size)
  File "zerver/lib/upload.py", line 165, in resize_gif
    frames[0].save(
  File "[...]/PIL/Image.py", line 2212, in save
    save_handler(self, fp, filename)
  File "[...]/PIL/GifImagePlugin.py", line 605, in _save
    _write_single_frame(im, fp, palette)
  File "[...]/PIL/GifImagePlugin.py", line 506, in _write_single_frame
    _write_local_header(fp, im, (0, 0), flags)
  File "[...]/PIL/GifImagePlugin.py", line 647, in _write_local_header
    disposal = int(im.encoderinfo.get("disposal", 0))
TypeError: int() argument must be a string, a bytes-like object or a
number, not 'list'
```

`check_add_realm_emoji` calls this as:

```
    try:
        is_animated = upload_emoji_image(image_file, emoji_file_name, a
uthor)
        emoji_uploaded_successfully = True
    finally:
        if not emoji_uploaded_successfully:
            realm_emoji.delete()
            return None
        # ...
```

This is equivalent to dropping _all_ exceptions silently.  As such,
Zulip has silently rejected all non-animated images larger than 64x64
since 5dab6e9d31.

Adjust to only pass a single disposal if there are no additional
frames.  Add a test for non-animated images, which requires also
fixing the incidental bug that all GIF images were being recorded as
animated, regardless of if they had more than 1 frame or not.
2022-02-17 12:19:47 -08:00
Aman Agrawal c396c2f7a4 home: Send rendered realm description for spectators.
Display rendered markdown after passing it through our local
rendered_markdown JS library.
2022-02-17 16:17:15 +00:00
Aman Agrawal 7c9e8a5071 decorator: Enable web-public view for production. 2022-02-16 13:25:53 -08:00
Anders Kaseorg 2b3a821807 docs: Remove highlight parameters from links.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-16 13:15:39 -08:00
Anders Kaseorg 2a5adf1453 auth: Remove unused kwargs for saml_sp_metadata.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-16 13:01:34 -08:00
Anders Kaseorg ce4261cbfb auth: Fix incorrect kwargs forwarding for logout_then_login.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-16 13:01:34 -08:00
Mateusz Mandera 26f9f3cc4b migrations: Don't push events to deferred_work queue if TEST_SUITE.
The deferred_work events can't be processed in the test suite
environment, since it tries to make requests to the host <realm.uri>
which is "zulip.testserver" and obviously not going to work. Since the
test suite base data set has no animated emoji, there's nothing to do,
and we can skip this step.

This code only runs when applying the migration to an already
provisioned test db - because during an initial db set up there are no
realms yet, so no events get pushed by the migration.
2022-02-14 16:05:25 -08:00
Anders Kaseorg 12da3ac0ad pgroonga: Re-enable PGroonga in development.
This partially reverts commit fdabf0b357
(#21104).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-13 19:11:49 -08:00
Anders Kaseorg 56518e094c test_new_users: Select test emails deterministically.
The previous random strategy for picking 5 emails would result in
collisions in 1 out of 10000.35 tests, leading to
psycopg2.errors.UniqueViolation.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-11 18:09:43 -08:00
Anders Kaseorg 961633fcec timezone: List only canonical timezone identifiers.
For aliases that will no longer be listed, see the third column of

grep '^L ' zulip-py3-venv/lib/python3.*/site-packages/pytz/zoneinfo/tzdata.zi

Time zones previously set to an alias will be canonicalized on demand.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-11 17:38:57 -08:00
Tim Abbott fdabf0b357 pgroonga: Disable PGroonga in development to fix CI.
A recent Postgres upstream release appears to have broken PGroonga.
While we wait for https://github.com/pgroonga/pgroonga/issues/203 to
be resolved, disable PGroonga in our automated tests so that Zulip
CI passes.
2022-02-11 11:35:43 -08:00
Tim Abbott edf16cb861 upload: Mark migration code as nocoverage.
We want to add tests here, but it's more important to fix main failing
CI.
2022-02-11 10:37:58 -08:00
Mateusz Mandera c6be15908c import: Ensure that .author gets set when importing RealmEmoji.
Sometimes we may get data to import, due to export bugs, malformed data
etc., which doesn't have the invariant of RealmEmoji.author always being
set. The import code should fix that, by choosing a reasonable default
and setting it.
2022-02-11 18:21:38 +01:00
Mateusz Mandera 30ac291eba emoji: Add migration to reupload all RealmEmoji and ensure .author.
Fixes #19732.
2022-02-10 17:45:31 -08:00
Anders Kaseorg ad232c9b65 sqlalchemy_utils: Remove NonClosingPool.recreate override.
There doesn’t seem to be a reason to override this, and the upstream
method it was based on has diverged since this was written.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-10 11:59:41 -08:00
Anders Kaseorg 29330c180a sqlalchemy_utils: Make get_sqlalchemy_connection a context manager.
Although our NonClosingPool prevents the SQLAlchemy connection from
closing the underlying Django connection, we still want to properly
dispose of the associated SQLAlchemy structures.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-10 11:59:41 -08:00
Anders Kaseorg 8e5ae4e829 message_fetch: Use Row._mapping.
Fixes these warnings with SQLALCHEMY_WARN_20=1:

RemovedIn20Warning: Using non-integer/slice indices on Row is
deprecated and will be removed in version 2.0; please use
row._mapping[<key>], or the mappings() accessor on the Result
object. (Background on SQLAlchemy 2.0 at: https://sqlalche.me/e/b8d9)

RemovedIn20Warning: Using the 'in' operator to test for string or
column keys, or integer indexes, in a :class:`.Row` object is
deprecated and will be removed in a future release. Use the
`Row._fields` or `Row._mapping` attribute, i.e. 'key in
row._fields' (Background on SQLAlchemy 2.0 at:
https://sqlalche.me/e/b8d9)

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-10 11:59:41 -08:00
Anders Kaseorg fbb8f54d80 message_fetch: Use modern select() calling style.
Fixes this warning with SQLALCHEMY_WARN_20=1:

RemovedIn20Warning: The legacy calling style of select() is deprecated
and will be removed in SQLAlchemy 2.0.  Please use the new calling
style described at select(). (Background on SQLAlchemy 2.0 at:
https://sqlalche.me/e/b8d9)

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-10 11:59:41 -08:00
Anders Kaseorg 8769c0c485 message_fetch: Replace Select.column() with Select.add_columns().
Fixes “SADeprecationWarning: The Select.column() method is deprecated
and will be removed in a future release.  Please use
Select.add_columns() (deprecated since: 1.4)”.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-10 11:59:41 -08:00
Anders Kaseorg d5a5b0b5ac message_fetch: Replace SELECT coercion with .subquery().
Fixes “SADeprecationWarning: Implicit coercion of SELECT and textual
SELECT constructs into FROM clauses is deprecated; please call
.subquery() on any Core select or ORM Query object in order to produce
a subquery object.”

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-10 11:59:41 -08:00
Anders Kaseorg e5b24b1723 test_auth_backends: Remove deprecated match_querystring argument.
match_querystring is irrelevant in these cases.  Fixes this warning:

/srv/zulip-py3-venv/lib/python3.7/site-packages/responses/__init__.py:340:
DeprecationWarning: Argument 'match_querystring' is deprecated. Use
'responses.matchers.query_param_matcher' or
'responses.matchers.query_string_matcher'

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-10 11:40:34 -08:00
Mateusz Mandera b8ba63e792 management: Run log_email_config_errors in send_test_email command.
The tool needs to run this function, since it uses django's send_email
directly instead of going through our zerver.lib.send_email.send_email
codepath.
2022-02-10 10:55:49 -08:00
Tim Abbott fc02849865 message_edit: Fix unused parameter in most tests.
The message ID is encoded in the URL, not the PATCH parameters, so
this argument was ignored.  I verified that it appears to have always
matched the value present in the URL.
2022-02-09 15:13:04 -08:00
Tim Abbott 12ed0c3104 message edit: Improve logic for detecting whole topic moves.
The new logic better matches reasonable user expectations, that if you
move all the messages, that's a whole-topic move, regardless of which
propagation mode you selected.
2022-02-09 15:13:00 -08:00
Shubh Gupta 79069b5dec message edit: Indicate how many messages were moved in notificactions.
When moving only part of a topic, it's useful to display that
information to users in these notifications so that it's clear what's
happening.

The most important consequence is actually just increasing confidence
that when you see that the whole topic was moved, that's accurate.

Substantially modified by tabbott.

Fixes #20575.
2022-02-09 15:08:56 -08:00
madrix01 a1e71e8639 topic: Return JsonableError for race condition in topic mute.
To avoid an uncaught IntegrityError causing a 500 HTTP response in a
race between two processes trying to mute a topic, we catch the
integrity error and raise the error exception with status 400 we'd
have gotten if the second request had been a bit later.

Fixes #21011.
2022-02-09 14:35:12 -08:00
madrix01 386de83d74 typo: Remove duplicate assert statement.
We remove duplicate assert statement in
`zerver/tests/test_muting_topic.py`.
2022-02-09 14:31:25 -08:00
Mateusz Mandera fe61243cfe upload: Don't access emoji_file.name attribute upload_emoji_image.
The S3 backend implementation of upload_emoji_image was accessing
emoji_file.name - which is redundant because emoji_file_name already
gets passed in and can be used, and an object of type IO[bytes] may not
have the .name attribute. Spotted by @Fingel.
2022-02-09 11:26:39 -08:00
Mateusz Mandera e3080f2db4 send_email: Log error if EMAIL_HOST_USER is set without password.
Fixes #20132.

EMAIL_HOST_USER without EMAIL_HOST_PASSWORD is not going to be a valid
configuration, and may result from making mistake in correctly setting
it in the secrets file and end up being a non-obvious cause of failure
to send email. Logging an error will be useful for detecting it. Further
conditions can be added to the function in the future.
2022-02-09 11:04:40 -08:00
Aman Agrawal 7614f2203a pricing: Replace "Zulip Standard" with "Zulip Cloud Standard".
Case sensitive replace.
2022-02-09 11:00:24 -08:00
Aman Agrawal 86a78065b4 test_message_fetch: Reorganize web-public tests.
With some extensions by tabbott to cover adjacent corner cases.

Fixes #20288
2022-02-08 17:09:15 -08:00
Alex Vandiver a80840aa4d push_notifications: Don't open unnecessary APNs connections.
Calling `get_apns_context` opens (and caches) an open connection to
the APNs servers.  Since `apns_enabled` is called from Django
codepaths, this means that the Django processes hold unnecessary
connections open to the APNs servers.

Switch `apns_enabled` to checking what `get_apns_context` checks when
we're just returning True/False.
2022-02-08 15:20:23 -08:00
Anders Kaseorg 0ba0620000 push_notifications: Fix for aioapns 2.1.
aioapns 2.1 removed the loop parameter from the aioapns.APNs
constructor, because Python 3.10 removed the loop parameter from the
asyncio.Lock constructor.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-08 15:16:31 -08:00
Anders Kaseorg e1f42c1ac5 docs: Add missing space to compound verbs “back up”, “log in”, etc.
Noun: backup, login, logout, lookup, setup.

Verb: back up, log in, log out, look up, set up.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-07 19:20:54 -08:00
Anders Kaseorg b0ce4f1bce docs: Fix many spelling mistakes.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-07 18:51:06 -08:00
Anders Kaseorg d8f4e0ffb9 webhooks: Correct OpsGenie to Opsgenie.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-07 18:51:06 -08:00
Anders Kaseorg cd9b9675cd migrations: Fix typo in 0099 reverse_sql.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-07 18:47:33 -08:00
Alya Abbott 5eda383833 portico: Add iDrift AS case study. 2022-02-07 14:40:31 -08:00
Tim Abbott de411be576 home: Simplify and comment terms of service notice implementation. 2022-02-04 15:48:38 -08:00
Aman Agrawal 37d7d6ccd4 tos: Adjust migration tos to look same as our new standard tos. 2022-02-04 15:48:38 -08:00
Aman Agrawal ca71e28cd6 accounts_accept_terms: Make elements looks similar to other pages.
Add even vertical space between elements.
2022-02-04 15:48:38 -08:00
Mateusz Mandera a1fa2a8cf5 scim: Upgrade to django-scim2 0.17.1.
The new release adds the commit:
20ac22b96d

Which allows us to get rid of the entire ugly override that was needed
to do this commit's job in our code. What we do here in this commit:
* Use django-scim2 0.17.1
* Revert the relevant parts of f5a65846a8
* Adjust the expected error message in test_exception_details_not_revealed_to_client
  since the message thrown by django-scim2 in this release is slightly
  different.

We do not have to add anything to set EXPOSE_SCIM_EXCEPTIONS, since
django-scim2 uses False as the default, which is what we want - and we
have the aforementioned test verifying that indeed information doesn't
get revealed to the SCIM client.
2022-02-04 15:43:45 -08:00
Tim Abbott 1045737be6 test_subs: Use HostRequestMock for request object.
I incorrectly removed this when simplifying
dbddbee5a115b9352862cb13d4c66820865c30b6; while that commit did not
require the hunk re-added here, the later commit
3be622ffa7 added a call that did require it.
2022-02-04 15:36:57 -08:00
Lauryn Menard 3be622ffa7 backend: Add request as parameter to json_success.
Adds request as a parameter to json_success as a refactor towards
making `ignored_parameters_unsupported` functionality available
for all API endpoints.

Also, removes any data parameters that are an empty dict or
a dict with the generic success response values.
2022-02-04 15:16:56 -08:00
Lauryn Menard c532829c35 backend: Change `do_report_error` return value.
As a preparatory step to refactoring json_success to accept
request as a parameter, change `do_report_error`, which is
called from the events queue for "error_reports", to return
None instead of json_success.

Adds an assertion error to `ErrorReporter` queue processor
and removes `JsonableError` from `do_report_error`.

It is likely that `do_error_report` was moved from a view in a
previous refactor, but was not updated to no longer return an
HttpReponse.
2022-02-04 15:16:55 -08:00
Lauryn Menard dbddbee5a1 backend: Change return value of `compose_views`.
As a preparatory step to refactoring json_success to accept
request as a parameter, update helper function `compose_views`
in `views.streams.py` to return the response data and call
json_success from view functions that utilize `compose_views`.

Also, updates related test in `zerver.tests.test_subs.py`.
2022-02-04 15:16:55 -08:00
Lauryn Menard a4b347bb5f backend: Move json_success to callers vs helper functions.
As a preparatory step to refactoring json_success to accept
request as a parameter, change interface of helper functions:
`handle_deferred_message` in `views.message_send.py` and
`mute_topic` and `unmute_topic` in `views.muting.py`, so
that they return None or data for json_success.

Instead call json_sucess in the caller function, which already
has the HttpRequest as a parameter.
2022-02-04 15:12:22 -08:00
Anders Kaseorg df304c40da markdown: Use built-in hex formatting for unicode_emoji_to_codepoint.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-03 11:00:04 -08:00
Lauryn Menard cc712dbade api_docs: Add check for extra `additionalProperties` in return vals.
Adds a check for `additionalProperties: true` when there are no
properties listed in the schema.

This currently only happens in one place, but will be helpful for
deduplicating text between the `register-queue` and `get-events`
endpoints.
2022-02-02 17:25:58 -08:00
Puneeth Chaganti d55c137277 emoji: Add yellow_large_square and green_large_square emojis.
Wordle has recently become a thing and it uses green, yellow and white (or
black in dark mode) large square unicode characters to let people share their
gameplay. Zulip converts the white and black large square unicode characters to
emojis, but not the green and yellow ones. This causes the Wordle grid to be
misaligned when shared on Zulip.

This commit adds green and yellow large square emojis to our emoji list to fix
the problem.
2022-02-02 16:26:31 -08:00
Puneeth Chaganti 6beb84b553 emoji: Use str.rjust to pad codepoint strings instead of a loop. 2022-02-02 16:26:30 -08:00
Puneeth Chaganti 0eeb74b3c2 emoji: Fix minor typo in unicode_emoji_to_codepoint comment. 2022-02-02 16:26:28 -08:00
Sahil Batra 9f0d498994 message: Check mandatory_topics setting in backend as well.
Previously, we only checked mandatory_topics setting before
sending message in frontend and there was no restriction in
backend. This commit adds the check in backend also making
sure messages without topic cannot be sent through API as
well if mandatory_topics setting is set to True.
2022-02-02 10:11:35 -08:00
Alex Vandiver d1234ef18c string_validation: Prohibit non-printable characters in stream names.
Fixes part of #20128.

Co-authored-by: Shlok Patel <shlokcpatel2001@gmail.com>
2022-02-01 16:04:34 -08:00
Mateusz Mandera 8affb8e4a3 migrations: Remove disallowed characters from stream names.
character_is_printable logic is taken from similar work by @madrix01
2022-02-01 15:14:57 -08:00
Tim Abbott 438af33b19 create_user_messages: Rename mark_as_read_user_ids.
This should use a name that follows the naming pattern of adjacent
fields.
2022-02-01 12:01:03 -08:00
Tim Abbott 1a46df40c3 resolve topic: Limit unread flag on automated notifications.
Previously, users found it annoying that the automated "Resolve topic"
notifications triggered an unread for everyone in the stream; this
discouraged some users from using the feature on older threads for
fear of being annoying. We change this to a better default, of only
users who participated in the topic (via either messages or reactions)
being eligible for the new message being unread.

We will likely want to create global and stream-level notifications
settings to control this behavior as a follow-up -- some users, like
me, might prefer the simpler "Always unread" behavior in some streams.

Note that the automated notifications that a topic was resolved will
still result in the topic being moved to the top of the left sidebar.
This would be somewhat difficult to change, since the left sidebar
algorithm just looks at the highest message ID in the topic.

Fixes #19709.

Tests added by Aman Agrawal (amanagr@zulip.com).
2022-02-01 11:35:50 -08:00
Aman Agrawal f00db27b64 message_feed_errors: Convert to handlebars. 2022-01-31 17:10:05 -08:00
Mateusz Mandera d5a784a1ca oidc: Don't raise AssertionError if no name is provided.
Closes #20821.
Just like we did this for SAML in
cee4da64fa, so should we for oidc, as some
providers like Keycloak may not send the name by default.
2022-01-31 10:15:24 -08:00
Mateusz Mandera c0f7158378 push_notifications: Include stream_id in the notification data.
Closes #18067.
Previous only the stream name was sent, which is an unstable stream
identifier.
2022-01-29 17:37:48 -08:00
Anders Kaseorg 90e202cd38 docs: Consistently hyphenate “web-public”.
In English, compound adjectives should essentially always be
hyphenated.  This makes them easier to parse, especially for users who
might not recognize that the words “web public” go together as a
phrase.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-01-28 17:45:45 -08:00
jai2201 c14b3a8844 settings: Don't mention username in fetch API key error message.
There isn't a username prompt in the form for it, so the only
possibility for triggering this error is a wrong password.

Fixes #20924.
2022-01-28 14:10:15 -08:00
Mateusz Mandera 364139feec actions: Add appropriate argument to transaction.atomic decorators.
These were missing an argument to the decorator - and we generally want
to use either savepoint=False or durable=True.
2022-01-28 13:03:40 -08:00
Mateusz Mandera 120de1db19 do_deactivate_stream: Use transaction.atomic. 2022-01-28 13:03:39 -08:00
Mateusz Mandera e025e85b77 do_remove_default_stream: Use transaction.atomic.
on_commit on the event will make it easier to use transaction.atomic in
do_deactivate_stream, which calls this function.
2022-01-28 13:03:39 -08:00
Mateusz Mandera 0dd97eeaab do_set_realm_property: Use transaction.atomic. 2022-01-28 13:03:39 -08:00
Lauryn Menard 0499af583d api_docs: Have user settings with enum integers use schema keyword.
There are three user settings that are integer enums:
`color_scheme`, `demote_inactive_streams` and
`desktop_icon_count_displays`.

Unlike the other user settings, these were using the `content`
keyword instead of the `schema` keyword in their definitions,
which caused them not to be rendered correctly in the api
documentation.

Changes the keyword to `schema` and fixes the indentation
for these three user settings in the two endpoints using
them.
2022-01-28 08:33:49 -08:00
Lauryn Menard 44c86d823a api_docs: Remove unnecessary quotation marks in openapi arrays.
Removes various instances of quotation marks that are not needed,
specifically looking at instances of arrays, e.g. `- "`, in the
OpenAPI documentation.
2022-01-28 08:31:24 -08:00
Mateusz Mandera 4102816240 upload: Pass the target realm to create_attachment.
The target realm was not being passed to create_attachment in
upload_message_file implementations. This was a bug in the edge-case of
cross-realm messages - in particular, causing a bug in the email
gateway:
When an email with an attachment is sent, the message is mirrored to
Zulip with Email Gateway Bot as the message sender and uploader of the
attachment. Due to the realm not being passed to create_attachment, the
Attachment would get created with .realm being the system bot realm,
making the attachment inaccessible under some conditions due to failing
the following condition check (that's expected to pass, provided that
the .realm is set correctly):
```
    if (
        attachment.is_realm_public
        and attachment.realm == user_profile.realm
        and user_profile.can_access_public_streams()
    ):
        # Any user in the realm can access realm-public files
        return True
```
2022-01-27 17:23:44 -08:00
Lauryn Menard a832a8a3af api_docs: Fix enum strings in parameters to have quotes.
Fixes the rendering of enums to show strings with quotation marks,
while integers will continue to be rendered without quotation marks.

This allows for an empty string to be passed as an enum value and be
rendered as such in the documentation. Null will be rendered without
quotation marks, like integer values.
2022-01-27 13:03:20 -08:00
Sahil Batra 5e506a833f actions: Use transaction.atomic for do_change_avatar_fields. 2022-01-27 10:33:55 -08:00
Sahil Batra a2df7470d5 actions: Use transaction.atomic for do_change_realm_plan_type. 2022-01-27 10:33:55 -08:00
Sahil Batra f5ea13eea8 actions: Use transaction.atomic for do_change_user_delivery_email. 2022-01-27 10:33:55 -08:00
Lauryn Menard aaa627229e api: Update `update_message` event required fields.
Makes `edit_timestamp` and `user_id` required fields for all
`update_message` events.

Adds `rendering_only` as another required field to signal if
events are only updating the rendered content of the message,
which is currently the case for adding inline url previews.

Updates `test_event.py` so that `do_update_message` and
`do_update_embedded_data` refer to the same testing schema
for `update_message` events, and therefore reflect the same
required fields for the `update_message` event.

The OpenAPI definition for `update_message` events is also
updated to reflect the required field and descriptions of
various properties are updated for the addition of the
`rendering_only` property.
2022-01-26 13:11:26 -08:00
Lauryn Menard 7077871111 api_documentation: Update rate limit error description.
Moves details about the rate limit error object and handling to
the OpenAPI documentation description for that common error.

Previously, this information was on the general rest error
handling documentation page without clear connection to the
specific rate limit error.

Fixes a typo in the changelog (feature 36) for that same error
and also fixes a misplaced colon in the description of the error
for missing request parameters.
2022-01-26 12:51:02 -08:00
Lauryn Menard fee7c5712f api_docs: Fix small typos in organizational role description.
Fixes two small typos in the description of the `role` value
returned from the `/get-own-user` endpoint.
2022-01-26 10:36:51 -08:00
Sharif Naas d560d124a3 python: Replace string concatenations with f-strings. 2022-01-25 17:32:59 -08:00
Mateusz Mandera 40aa156b28 migrations: Fix RealmAuditLog creation in migration 0374. 2022-01-25 17:20:13 -08:00
Lauryn Menard ce49c4ca9e api: Add `subscription_data` details to `/update-subscription-settings`.
Adds detailed definition of objects in the `subscription_data` parameter
array for the `/update-subscription-settings` endpoint.

Fixes #20825. Follow-up to #20409.
2022-01-25 17:19:58 -08:00
Lauryn Menard 7b1597d4cf api_docs: Format required or optional detail of param object field.
Formats and moves whether a field of an object in a request
parameter is required or optional to be in the same location
and have the same formatting as the general api parameter
documentation.

Also formats any examples within the object detailed
description to be the same as the general api parameter
documentation.

Follow up to #20409.
2022-01-25 11:20:18 -08:00
Lauryn Menard e479acc809 api_docs: Add line break before return value description text.
Adds a line break before the descriptive text for return
values and events in the api documentation in order to
help with readability of descriptions with multiple
paragraphs of descriptive text.

Adjustments made to the CSS of list items in unordered
lists to visually group the first paragraph of text
to any following paragraphs or unordered lists.
2022-01-24 10:02:02 -08:00
Anders Kaseorg 4922632601 mypy: Add types-beautifulsoup4.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-01-23 23:39:40 -08:00
Anders Kaseorg 2caeb38e9e python: Replace IOError with OSError.
IOError is an alias for OSError in Python ≥ 3.3.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-01-23 22:17:02 -08:00
Anders Kaseorg 3437e38612 string_validation: Use set comprehension syntax.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-01-23 22:17:02 -08:00
Anders Kaseorg 97e4e9886c python: Replace universal_newlines with text.
This is supported in Python ≥ 3.7.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-01-23 22:16:01 -08:00
Anders Kaseorg b729f00fc2 test_upload: Uncomment subTest contexts.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-01-23 22:14:43 -08:00
Anders Kaseorg 5fb935bf60 python: Replace deprecated urllib3 BACKOFF_MAX.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-01-23 22:14:17 -08:00
Anders Kaseorg 2612f57d51 requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-01-23 22:14:17 -08:00
Anders Kaseorg ee71650908 python: Replace requests.packages.urllib3 alias with urllib3.
requests stopped vendoring urllib3 in 2.16.0 (2017-05-26).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-01-23 22:14:17 -08:00
Anders Kaseorg 18aa4f4df4 tornado: Replace deprecated urllib3 method_whitelist.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-01-23 22:14:17 -08:00
Anders Kaseorg a58a71ef43 Remove Ubuntu 18.04 support.
As a consequence:

• Bump minimum supported Python version to 3.7.
• Move Vagrant environment to Debian 10, which has Python 3.7.
• Move CI frontend tests to Debian 10.
• Move production build test to Debian 10.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-01-21 17:26:14 -08:00
Eeshan Garg 93329c2d70 send_custom_email: Only send emails to active remote servers. 2022-01-21 15:03:08 -08:00
Tim Abbott 367fed89f2 register_server: Improve HTTP error reporting. 2022-01-21 15:02:30 -08:00
Eeshan Garg fb09c18462 management/commands: Refactor error handling in register_server. 2022-01-21 14:57:09 -08:00
Eeshan Garg bd197886fe management/commands: Refactor logging code in register_server. 2022-01-21 14:57:09 -08:00
Eeshan Garg 3bc0f8c6f9 zilencer: Add endpoint for deactivating remote server registration. 2022-01-21 14:57:04 -08:00
Eeshan Garg 94d00ca942 zilencer: Stop serving requests from deactivated remote servers. 2022-01-21 14:56:04 -08:00
Tim Abbott afd8d5ef0c openapi: Improve stream description details. 2022-01-21 14:10:48 -08:00
Palash 61e87b0ac8 api_documentation: Update subscriptions parameter in OpenAPI doc.
regarding -
POST https://yourZulipDomain.zulipchat.com/api/v1/users/me/subscriptions

The definition of the "subscription" parameter didn't include full
information about the parameter. It only said that an array of objects
is passed as a parameter, and relied on description of the parameter
to explain what the object contained. I edited the definition to contain
the full information about the object.

Fixes #20824.
2022-01-21 14:04:46 -08:00
Eeshan Garg aa8b3f9729 streams: Add RealmAuditLog entries for permission changes. 2022-01-21 13:59:35 -08:00
Eeshan Garg 0d99809fd3 streams: Add notifications for permission policy changes.
The change to curl_param_value_generators.py warrants a brief
explanation. Stream permission changes now generate a notification
message. Our curl example test for removing a reaction comes after
the two tests for updating the stream permission changes, thus the
hardcoded message ID in that test needs to be incremented by 2 to
account for the two notification messages that now come before it.

This is a part of #20289.
2022-01-21 13:59:34 -08:00
Eeshan Garg fab1b7f5d5 actions: Refactor functions for stream permission changes.
do_make_stream_web_public and do_change_stream_invite_only seem
to contain very similar logic that could just live inside the
do_change_stream_permission function that handles all permission
changes in one place.
2022-01-21 13:59:34 -08:00
Eeshan Garg f0ee065292 streams: Use bulleted format for description change notifications.
We want the format for our description change notifications to be
consistent with the format of our stream posting policy change
notifications.
2022-01-21 13:59:34 -08:00
Anders Kaseorg 1f3e87b2a3 test_queue_error_json: Acknowledge the received message.
Otherwise it stays in the queue.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-01-21 13:38:13 -08:00
Anders Kaseorg 773c909d7f test_queue: Ensure the test_suite queue exists in setUp.
queue_client.queues does not list all the queues that exist on the
server (you can’t do that over AMQP); the condition "test_suite" in
queue_client.queues was always false.  So the test_suite queue could
accumulate extra messages that broke test_queue_error_json.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-01-21 13:38:13 -08:00
Anders Kaseorg db24eca832 test_auth_backends: Verify the avatar contents.
Consuming response.streaming_content fixes this warning from
TestLDAP.test_login_success_when_user_does_not_exist_with_valid_subdomain:
“ResourceWarning: unclosed file <_io.FileIO
name='/srv/zulip/var/…/3cc4b5a15b6f4f06b3f9a6ecb179b08702329716.png'
mode='rb' closefd=True>”.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-01-21 13:37:26 -08:00
Anders Kaseorg ba7ea7cc80 test_classes: Extract assert_streaming_content helper.
This also fixes a warning from
RealmExportTest.test_endpoint_local_uploads: “ResourceWarning:
unclosed file <_io.BufferedReader
name='/srv/zulip/var/…/test-export.tar.gz'>”.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-01-21 13:37:26 -08:00
Lauryn Menard 7993aed530 api_documentation: Render object details in parameter description.
Adds a check for object in parameter type that will render the details
of the object in the parameter description if they are in the object
definition in the OpenAPI documentation.

Fixes #19424.
2022-01-19 13:04:04 -08:00
Alex Vandiver 915c2b2fd9 muting: Fix a race in topic unmuting.
Rather than check if the topic exists and then try to delete it, just
try to delete it, and catch the lack of matching rows.
2022-01-18 14:15:06 -08:00
Mateusz Mandera f8b06ed952 events: Send invites_changed event if user deactivation revokes invites.
revoke_invites_generated_by_user should send invites_changed event if it
actually revokes some invitations. This is called in the user
deactivatoin codepath.
2022-01-18 14:12:55 -08:00
Mateusz Mandera 74d2aea76a apply_events: Update state["subscribers"] upon "remove user" event.
Event of type "realm_user", op "remove", emitted by do_deactivate_user
should remove the user id from subscriptions in the state. We weren't
catching this bug, because test_do_deactivate_bot uses a newly created
bot, so no stream subscriptions are affected. The bug shows up if
deactivating e.g. cordelia - thus we want to have two tests instead,
one for testing bot deactivation and one for user deactivation.
2022-01-18 14:12:55 -08:00
Steve Howell dd1c9c45c7 stream colors: Try harder to avoid collisions.
We now use recipient_id % 24 for new stream colors
when users have already used all 24 of our canned
colors.

This fix doesn't address the scenario that somebody
dislikes one of our current canned colors, so if a
user continually changes canned color N to some other
color for new streams, their new streams will continue
to include color N (and the user will still need to
change them).

This fix doesn't address the fact that it can be expensive
during bulk-add situations to query for all the colors
that users have already used up.

See https://chat.zulip.org/#narrow/stream/3-backend/topic/assigning.20stream.20colors
for more discussion.
2022-01-18 13:56:54 -08:00
Sahil Batra 06cba4ae1f actions: Use transaction.atomic in do_change_bot_owner. 2022-01-18 12:43:04 -08:00
Sahil Batra 7c44151135 actions: Use transaction.atomic in do_change_tos_version. 2022-01-18 12:43:04 -08:00
Sahil Batra 06d715a41d actions: Use transaction.atomic in do_change_icon_source. 2022-01-18 12:43:04 -08:00
Sahil Batra 64d1dc6525 actions: Use transaction.atomic in do_change_logo_source. 2022-01-18 12:43:04 -08:00
Sahil Batra 8945a64024 actions: Use transaction.atomic in do_change_realm_org_type. 2022-01-18 12:43:04 -08:00
Sahil Batra c8f81ded4e actions: Use transaction.atomic in do_change_default_sending_stream. 2022-01-18 12:43:04 -08:00
Sahil Batra cb43bdab93 actions: Use transaction.atomic for do_change_default_all_public_streams. 2022-01-18 12:43:04 -08:00
Sahil Batra 4a7461361e actions: Use transaction.atomic for do_change_default_events_register_stream. 2022-01-18 12:43:04 -08:00
Sahil Batra 5c758af3b4 actions: Use transaction.atomic for do_change_user_setting. 2022-01-18 12:43:04 -08:00
Alex Vandiver 19f891968d markdown: Increase the maximum number of image previews per message.
The limit here is purely to prevent breakage in case of a pathological
number of images in a single message; 5 images is entirely possible in
a reasonable message, and causes user confusion when they are not
expended.

Increase the limit to 10 per message.
2022-01-14 11:30:07 -08:00
Anders Kaseorg 6e00d6f97a change_password: Set requires_system_checks to a list.
Django 3.2 expects a list, and Django 4.1 will require one.  Fixes
“RemovedInDjango41Warning: Using a boolean value for
requires_system_checks is deprecated. Use '__all__' instead of True,
and [] (an empty list) instead of False.”

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-01-13 21:02:41 -08:00
Anders Kaseorg 9e70a47f93 test_push_notifications: Close event loops.
Fixes “ResourceWarning: unclosed event loop <_UnixSelectorEventLoop
running=False closed=False debug=False>”.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-01-13 20:40:46 -08:00
Anders Kaseorg 87b4e9259f documentation: Replace deprecated request.is_ajax() method.
This was deprecated in Django 3.1 for being jQuery-specific, and
removed in Django 4.0.  Replicate the jQuery-specific check.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-01-13 15:40:22 -08:00
Anders Kaseorg 4147da24dd tests: Use read_test_image_file.
Fixes a ResourceWarning from the unclosed file at test_upload.py:1954.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-01-13 14:59:46 -08:00
Anders Kaseorg 031f4596ab openapi: Use openapi_core ResponseValidator to validate responses.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-01-13 14:34:11 -08:00
Anders Kaseorg 4cd5e0e578 openapi: Fix display_brief_error not to rely on naively_merge_allOf.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-01-13 14:34:11 -08:00
Anders Kaseorg 86c39e5792 test_openapi: Make testing.yml a full conformant specification.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-01-13 14:34:11 -08:00
Anders Kaseorg 465ea4ac51 openapi: Validate Python example responses against the entire schema.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-01-13 14:34:11 -08:00
Anders Kaseorg 02c1196e3e openapi: Merge success examples for POST users/me/subscriptions.
oneOf with two identical branches (modulo example) is a bug because
oneOf means exclusive or.  It’s also a totally inappropriate kludge
for encoding multiple examples.  The OpenAPI specification provides a
perfectly good standard way to do that:

https://spec.openapis.org/oas/v3.0.3#example-object

However, we don’t handle that in our OpenAPI documentation generator
yet, so for now just merge the examples.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-01-13 14:34:11 -08:00
Anders Kaseorg e9fd52da01 openapi: Mark subscribe error as a JsonError.
This was a oneOf with two identical branches modulo example, which is
always a bug because oneOf means exclusive or.  But the example for
the first branch did not fit the schema for AddSubscriptionsResponse,
which is a subset of JsonSuccessBase.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-01-13 14:34:11 -08:00
Anders Kaseorg a503d19eae openapi: Declare items: {} for “inherited” array properties.
This should not be needed, but works around an openapi-core bug:
https://github.com/p1c2u/openapi-core/issues/380

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-01-13 14:34:11 -08:00
Anders Kaseorg 6c25c628b1 openapi: Mark “inherited” nullable properties as nullable.
Although allOf is often used to indicate inheritance, its semantics
are that of a plain set intersection.  The intersection of a nullable
property with a non-nullable property is a non-nullable property.
Therefore, if we want an inherited property to remain nullable, we
need to mark it as such.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-01-13 14:34:11 -08:00
Anders Kaseorg 1cb397c1ed openapi: Remove additionalProperties: false from BasicStreamBase.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-01-13 14:34:11 -08:00
Anders Kaseorg 834542d097 openapi: Don’t forbid result, msg with additionalProperties: false.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-01-13 14:34:11 -08:00
Anders Kaseorg bea64f8608 python: Convert deprecated Django ugettext alias to gettext again.
django.utils.translation.ugettext is a deprecated alias of
django.utils.translation.gettext as of Django 3.0, and will be removed
in Django 4.0.

Commit e7ed907cf6 (#18174) fixed this
before, but new instances have been added.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-01-13 14:28:12 -08:00
Anders Kaseorg 19b470c260 openapi: Fix principals array type.
We do not accept heterogeneous arrays containing both user ids and
email addresses.

This also happens to disallow an empty array, which is fine since the
principals parameter should be omitted if the default to the calling
user is desired.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-01-13 14:25:18 -08:00
Anders Kaseorg 78e54a0d7a python: Replace deprecated jinja2.utils.Markup with markupsafe.Markup.
Fixes “DeprecationWarning: 'jinja2.Markup' is deprecated and will be
removed in Jinja 3.1. Import 'markupsafe.Markup' instead.”

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-01-13 14:22:48 -08:00
Lauryn Menard cf9483e6a9 api_docs: Fix typos in UserBase schema descriptions.
Fixes two small typos and adds backticks to a reference to an
object field.
2022-01-13 14:21:05 -08:00
Lauryn Menard 662fc64a65 api_docs: Update regex for generating code example fixture.
Updates regex in the openapi markdown extension to match api
endpoint names that contain dashes, which is the case for
`zulip-outgoing-webhook` and `rest-error-handling`.
2022-01-13 14:20:46 -08:00
Sahil Batra 9b8713fc1e users: Send peer_add subscription events on reactivating users.
The subscriber list was not updating without a refresh on
reactivating user, because the subscriptions data with the
client was not updated on reactivation.

This commit adds code to send peer_add subscription events
on reactivating the user.

We do not send peer_remove events on deactivating the user,
but the subscriber list is still live-updated because we
have the data of the streams which the deactivated user is
susbcribed to and the clients itself updates the data and UI
on receiving event of deactivation of user, which it is not
possible when reactivating the user.

Fixes #20383.
2022-01-12 14:30:21 -08:00
Mateusz Mandera 93e8740218 do_deactivate_user: Revoke invitations generated by the user.
Leaving old invitations valid, potentially for a very long time, is
clearly unexpected and undesired behavior under normal circumstances. A
user shouldn't be able to e.g. generate a multiuse invite link, get
banned from the organization by being deactivated and then just re-join
using the link they've created for themselves.
2022-01-12 13:53:34 -08:00
Mateusz Mandera 76f1e902a6 notify_invites_changed: Fix passing of deleted objects to the function.
do_revoke_user_invite and do_revoke_multi_use_invite were using objects
after their deletion to pass the argument to notify_invites_changed. We
should avoid that. The function was only using the .realm attribute of
the received objects, so it's simpler to make it just take realm as its
argument.
2022-01-12 13:53:34 -08:00
Mateusz Mandera ff688c3a8d actions: Give do_get_user_invites a more specific name.
The added docstrings elaborates on why the new name is more appropriate.
2022-01-12 13:53:34 -08:00
BIKI DAS fb1ff71b2e
python: Remove duplicates of global imports inside functions.
Neither of these appears to have been intentional.
2022-01-12 13:50:12 -08:00
Anders Kaseorg 82f2707278 test_openapi: Fix incorrect use of assertRaises msg parameter.
The msg parameter is a string to be displayed when the expected
exception wasn’t raised, not a pattern to match against the raised
exception’s message.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-01-11 21:01:45 -08:00
Anders Kaseorg 96159688c2 test_helpers: Fix exception safety of stdout_suppressed.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-01-11 19:38:36 -08:00
Alex Vandiver 1f9373585a string_validation: Use enumerate rather than `range(len(...))`. 2022-01-11 15:59:50 -08:00