Commit Graph

6849 Commits

Author SHA1 Message Date
Anders Kaseorg 961633fcec timezone: List only canonical timezone identifiers.
For aliases that will no longer be listed, see the third column of

grep '^L ' zulip-py3-venv/lib/python3.*/site-packages/pytz/zoneinfo/tzdata.zi

Time zones previously set to an alias will be canonicalized on demand.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-11 17:38:57 -08:00
Tim Abbott edf16cb861 upload: Mark migration code as nocoverage.
We want to add tests here, but it's more important to fix main failing
CI.
2022-02-11 10:37:58 -08:00
Mateusz Mandera c6be15908c import: Ensure that .author gets set when importing RealmEmoji.
Sometimes we may get data to import, due to export bugs, malformed data
etc., which doesn't have the invariant of RealmEmoji.author always being
set. The import code should fix that, by choosing a reasonable default
and setting it.
2022-02-11 18:21:38 +01:00
Mateusz Mandera 30ac291eba emoji: Add migration to reupload all RealmEmoji and ensure .author.
Fixes #19732.
2022-02-10 17:45:31 -08:00
Anders Kaseorg ad232c9b65 sqlalchemy_utils: Remove NonClosingPool.recreate override.
There doesn’t seem to be a reason to override this, and the upstream
method it was based on has diverged since this was written.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-10 11:59:41 -08:00
Anders Kaseorg 29330c180a sqlalchemy_utils: Make get_sqlalchemy_connection a context manager.
Although our NonClosingPool prevents the SQLAlchemy connection from
closing the underlying Django connection, we still want to properly
dispose of the associated SQLAlchemy structures.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-10 11:59:41 -08:00
Tim Abbott 12ed0c3104 message edit: Improve logic for detecting whole topic moves.
The new logic better matches reasonable user expectations, that if you
move all the messages, that's a whole-topic move, regardless of which
propagation mode you selected.
2022-02-09 15:13:00 -08:00
Shubh Gupta 79069b5dec message edit: Indicate how many messages were moved in notificactions.
When moving only part of a topic, it's useful to display that
information to users in these notifications so that it's clear what's
happening.

The most important consequence is actually just increasing confidence
that when you see that the whole topic was moved, that's accurate.

Substantially modified by tabbott.

Fixes #20575.
2022-02-09 15:08:56 -08:00
Mateusz Mandera fe61243cfe upload: Don't access emoji_file.name attribute upload_emoji_image.
The S3 backend implementation of upload_emoji_image was accessing
emoji_file.name - which is redundant because emoji_file_name already
gets passed in and can be used, and an object of type IO[bytes] may not
have the .name attribute. Spotted by @Fingel.
2022-02-09 11:26:39 -08:00
Mateusz Mandera e3080f2db4 send_email: Log error if EMAIL_HOST_USER is set without password.
Fixes #20132.

EMAIL_HOST_USER without EMAIL_HOST_PASSWORD is not going to be a valid
configuration, and may result from making mistake in correctly setting
it in the secrets file and end up being a non-obvious cause of failure
to send email. Logging an error will be useful for detecting it. Further
conditions can be added to the function in the future.
2022-02-09 11:04:40 -08:00
Alex Vandiver a80840aa4d push_notifications: Don't open unnecessary APNs connections.
Calling `get_apns_context` opens (and caches) an open connection to
the APNs servers.  Since `apns_enabled` is called from Django
codepaths, this means that the Django processes hold unnecessary
connections open to the APNs servers.

Switch `apns_enabled` to checking what `get_apns_context` checks when
we're just returning True/False.
2022-02-08 15:20:23 -08:00
Anders Kaseorg 0ba0620000 push_notifications: Fix for aioapns 2.1.
aioapns 2.1 removed the loop parameter from the aioapns.APNs
constructor, because Python 3.10 removed the loop parameter from the
asyncio.Lock constructor.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-08 15:16:31 -08:00
Anders Kaseorg b0ce4f1bce docs: Fix many spelling mistakes.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-07 18:51:06 -08:00
Mateusz Mandera a1fa2a8cf5 scim: Upgrade to django-scim2 0.17.1.
The new release adds the commit:
20ac22b96d

Which allows us to get rid of the entire ugly override that was needed
to do this commit's job in our code. What we do here in this commit:
* Use django-scim2 0.17.1
* Revert the relevant parts of f5a65846a8
* Adjust the expected error message in test_exception_details_not_revealed_to_client
  since the message thrown by django-scim2 in this release is slightly
  different.

We do not have to add anything to set EXPOSE_SCIM_EXCEPTIONS, since
django-scim2 uses False as the default, which is what we want - and we
have the aforementioned test verifying that indeed information doesn't
get revealed to the SCIM client.
2022-02-04 15:43:45 -08:00
Lauryn Menard 3be622ffa7 backend: Add request as parameter to json_success.
Adds request as a parameter to json_success as a refactor towards
making `ignored_parameters_unsupported` functionality available
for all API endpoints.

Also, removes any data parameters that are an empty dict or
a dict with the generic success response values.
2022-02-04 15:16:56 -08:00
Lauryn Menard c532829c35 backend: Change `do_report_error` return value.
As a preparatory step to refactoring json_success to accept
request as a parameter, change `do_report_error`, which is
called from the events queue for "error_reports", to return
None instead of json_success.

Adds an assertion error to `ErrorReporter` queue processor
and removes `JsonableError` from `do_report_error`.

It is likely that `do_error_report` was moved from a view in a
previous refactor, but was not updated to no longer return an
HttpReponse.
2022-02-04 15:16:55 -08:00
Anders Kaseorg df304c40da markdown: Use built-in hex formatting for unicode_emoji_to_codepoint.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-03 11:00:04 -08:00
Lauryn Menard cc712dbade api_docs: Add check for extra `additionalProperties` in return vals.
Adds a check for `additionalProperties: true` when there are no
properties listed in the schema.

This currently only happens in one place, but will be helpful for
deduplicating text between the `register-queue` and `get-events`
endpoints.
2022-02-02 17:25:58 -08:00
Puneeth Chaganti d55c137277 emoji: Add yellow_large_square and green_large_square emojis.
Wordle has recently become a thing and it uses green, yellow and white (or
black in dark mode) large square unicode characters to let people share their
gameplay. Zulip converts the white and black large square unicode characters to
emojis, but not the green and yellow ones. This causes the Wordle grid to be
misaligned when shared on Zulip.

This commit adds green and yellow large square emojis to our emoji list to fix
the problem.
2022-02-02 16:26:31 -08:00
Puneeth Chaganti 6beb84b553 emoji: Use str.rjust to pad codepoint strings instead of a loop. 2022-02-02 16:26:30 -08:00
Puneeth Chaganti 0eeb74b3c2 emoji: Fix minor typo in unicode_emoji_to_codepoint comment. 2022-02-02 16:26:28 -08:00
Sahil Batra 9f0d498994 message: Check mandatory_topics setting in backend as well.
Previously, we only checked mandatory_topics setting before
sending message in frontend and there was no restriction in
backend. This commit adds the check in backend also making
sure messages without topic cannot be sent through API as
well if mandatory_topics setting is set to True.
2022-02-02 10:11:35 -08:00
Alex Vandiver d1234ef18c string_validation: Prohibit non-printable characters in stream names.
Fixes part of #20128.

Co-authored-by: Shlok Patel <shlokcpatel2001@gmail.com>
2022-02-01 16:04:34 -08:00
Tim Abbott 438af33b19 create_user_messages: Rename mark_as_read_user_ids.
This should use a name that follows the naming pattern of adjacent
fields.
2022-02-01 12:01:03 -08:00
Tim Abbott 1a46df40c3 resolve topic: Limit unread flag on automated notifications.
Previously, users found it annoying that the automated "Resolve topic"
notifications triggered an unread for everyone in the stream; this
discouraged some users from using the feature on older threads for
fear of being annoying. We change this to a better default, of only
users who participated in the topic (via either messages or reactions)
being eligible for the new message being unread.

We will likely want to create global and stream-level notifications
settings to control this behavior as a follow-up -- some users, like
me, might prefer the simpler "Always unread" behavior in some streams.

Note that the automated notifications that a topic was resolved will
still result in the topic being moved to the top of the left sidebar.
This would be somewhat difficult to change, since the left sidebar
algorithm just looks at the highest message ID in the topic.

Fixes #19709.

Tests added by Aman Agrawal (amanagr@zulip.com).
2022-02-01 11:35:50 -08:00
Mateusz Mandera c0f7158378 push_notifications: Include stream_id in the notification data.
Closes #18067.
Previous only the stream name was sent, which is an unstable stream
identifier.
2022-01-29 17:37:48 -08:00
Anders Kaseorg 90e202cd38 docs: Consistently hyphenate “web-public”.
In English, compound adjectives should essentially always be
hyphenated.  This makes them easier to parse, especially for users who
might not recognize that the words “web public” go together as a
phrase.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-01-28 17:45:45 -08:00
Mateusz Mandera 364139feec actions: Add appropriate argument to transaction.atomic decorators.
These were missing an argument to the decorator - and we generally want
to use either savepoint=False or durable=True.
2022-01-28 13:03:40 -08:00
Mateusz Mandera 120de1db19 do_deactivate_stream: Use transaction.atomic. 2022-01-28 13:03:39 -08:00
Mateusz Mandera e025e85b77 do_remove_default_stream: Use transaction.atomic.
on_commit on the event will make it easier to use transaction.atomic in
do_deactivate_stream, which calls this function.
2022-01-28 13:03:39 -08:00
Mateusz Mandera 0dd97eeaab do_set_realm_property: Use transaction.atomic. 2022-01-28 13:03:39 -08:00
Mateusz Mandera 4102816240 upload: Pass the target realm to create_attachment.
The target realm was not being passed to create_attachment in
upload_message_file implementations. This was a bug in the edge-case of
cross-realm messages - in particular, causing a bug in the email
gateway:
When an email with an attachment is sent, the message is mirrored to
Zulip with Email Gateway Bot as the message sender and uploader of the
attachment. Due to the realm not being passed to create_attachment, the
Attachment would get created with .realm being the system bot realm,
making the attachment inaccessible under some conditions due to failing
the following condition check (that's expected to pass, provided that
the .realm is set correctly):
```
    if (
        attachment.is_realm_public
        and attachment.realm == user_profile.realm
        and user_profile.can_access_public_streams()
    ):
        # Any user in the realm can access realm-public files
        return True
```
2022-01-27 17:23:44 -08:00
Lauryn Menard a832a8a3af api_docs: Fix enum strings in parameters to have quotes.
Fixes the rendering of enums to show strings with quotation marks,
while integers will continue to be rendered without quotation marks.

This allows for an empty string to be passed as an enum value and be
rendered as such in the documentation. Null will be rendered without
quotation marks, like integer values.
2022-01-27 13:03:20 -08:00
Sahil Batra 5e506a833f actions: Use transaction.atomic for do_change_avatar_fields. 2022-01-27 10:33:55 -08:00
Sahil Batra a2df7470d5 actions: Use transaction.atomic for do_change_realm_plan_type. 2022-01-27 10:33:55 -08:00
Sahil Batra f5ea13eea8 actions: Use transaction.atomic for do_change_user_delivery_email. 2022-01-27 10:33:55 -08:00
Lauryn Menard aaa627229e api: Update `update_message` event required fields.
Makes `edit_timestamp` and `user_id` required fields for all
`update_message` events.

Adds `rendering_only` as another required field to signal if
events are only updating the rendered content of the message,
which is currently the case for adding inline url previews.

Updates `test_event.py` so that `do_update_message` and
`do_update_embedded_data` refer to the same testing schema
for `update_message` events, and therefore reflect the same
required fields for the `update_message` event.

The OpenAPI definition for `update_message` events is also
updated to reflect the required field and descriptions of
various properties are updated for the addition of the
`rendering_only` property.
2022-01-26 13:11:26 -08:00
Lauryn Menard 7b1597d4cf api_docs: Format required or optional detail of param object field.
Formats and moves whether a field of an object in a request
parameter is required or optional to be in the same location
and have the same formatting as the general api parameter
documentation.

Also formats any examples within the object detailed
description to be the same as the general api parameter
documentation.

Follow up to #20409.
2022-01-25 11:20:18 -08:00
Lauryn Menard e479acc809 api_docs: Add line break before return value description text.
Adds a line break before the descriptive text for return
values and events in the api documentation in order to
help with readability of descriptions with multiple
paragraphs of descriptive text.

Adjustments made to the CSS of list items in unordered
lists to visually group the first paragraph of text
to any following paragraphs or unordered lists.
2022-01-24 10:02:02 -08:00
Anders Kaseorg 4922632601 mypy: Add types-beautifulsoup4.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-01-23 23:39:40 -08:00
Anders Kaseorg 3437e38612 string_validation: Use set comprehension syntax.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-01-23 22:17:02 -08:00
Anders Kaseorg 97e4e9886c python: Replace universal_newlines with text.
This is supported in Python ≥ 3.7.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-01-23 22:16:01 -08:00
Anders Kaseorg ee71650908 python: Replace requests.packages.urllib3 alias with urllib3.
requests stopped vendoring urllib3 in 2.16.0 (2017-05-26).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-01-23 22:14:17 -08:00
Eeshan Garg 94d00ca942 zilencer: Stop serving requests from deactivated remote servers. 2022-01-21 14:56:04 -08:00
Eeshan Garg aa8b3f9729 streams: Add RealmAuditLog entries for permission changes. 2022-01-21 13:59:35 -08:00
Eeshan Garg 0d99809fd3 streams: Add notifications for permission policy changes.
The change to curl_param_value_generators.py warrants a brief
explanation. Stream permission changes now generate a notification
message. Our curl example test for removing a reaction comes after
the two tests for updating the stream permission changes, thus the
hardcoded message ID in that test needs to be incremented by 2 to
account for the two notification messages that now come before it.

This is a part of #20289.
2022-01-21 13:59:34 -08:00
Eeshan Garg fab1b7f5d5 actions: Refactor functions for stream permission changes.
do_make_stream_web_public and do_change_stream_invite_only seem
to contain very similar logic that could just live inside the
do_change_stream_permission function that handles all permission
changes in one place.
2022-01-21 13:59:34 -08:00
Eeshan Garg f0ee065292 streams: Use bulleted format for description change notifications.
We want the format for our description change notifications to be
consistent with the format of our stream posting policy change
notifications.
2022-01-21 13:59:34 -08:00
Anders Kaseorg ba7ea7cc80 test_classes: Extract assert_streaming_content helper.
This also fixes a warning from
RealmExportTest.test_endpoint_local_uploads: “ResourceWarning:
unclosed file <_io.BufferedReader
name='/srv/zulip/var/…/test-export.tar.gz'>”.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-01-21 13:37:26 -08:00
Lauryn Menard 7993aed530 api_documentation: Render object details in parameter description.
Adds a check for object in parameter type that will render the details
of the object in the parameter description if they are in the object
definition in the OpenAPI documentation.

Fixes #19424.
2022-01-19 13:04:04 -08:00