Commit Graph

16053 Commits

Author SHA1 Message Date
Tim Abbott 0ba63826dd tornado: Disable event compression for mark-as-unread events.
Doing rapid pace mark-as-unread in the Zulip web application, one
observed assertion failures showing that the server would send an
event containing multiple message IDs but only one of the messages
present in the message_details side data structure.

The cause of this was the "virtual events" compression system; two
flags/remove/read events were being combined by simply concatenating
the lists of events, without any attempt to merge the
`message_details` field on those objects.

The immediate fix is to disable virtual events compression for this
event class, but it's not unlikely we'll need to just eliminate the
virtual_events system entirely, because it seems difficult to make it
soundly handle a message whose state for a given flag changes back and
forth while the client is offline.

But we'll leave that for later, since removing that optimization
deserves more discussion than fixing this event corruption bug.
2022-11-17 00:46:26 -08:00
Anders Kaseorg 842a5bb54b message_flags: Allow updating flags by narrows and anchors.
Fixes #22893.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-16 21:13:44 -08:00
Anders Kaseorg 5d0711df6d message_flags: Don’t send events for no-op flag updates.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-16 21:13:44 -08:00
Anders Kaseorg 54878a1826 message_flags: Convert operation string to boolean.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-16 21:13:44 -08:00
Anders Kaseorg 5b5be1fd82 message_flags: Use bulk_access_messages in do_update_message_flags.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-16 21:13:44 -08:00
Anders Kaseorg 342cba7b4d message: Fetch streams in bulk in bulk_access_messages.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-16 21:13:44 -08:00
Anders Kaseorg fef365248b create_user: Create historical UserMessage rows with bulk_create.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-16 21:13:44 -08:00
Anders Kaseorg ee2cb855f0 message_fetch: Add include_anchor parameter.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-16 21:13:44 -08:00
Anders Kaseorg dae4633745 message_fetch: Extract fetch_messages helper to zerver.lib.narrow.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-16 21:13:44 -08:00
Anders Kaseorg bd5d200722 narrow: Return a dataclass from post_process_limited_query.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-16 21:13:44 -08:00
Anders Kaseorg 0a0a70b33d message_fetch: Move limit_query_to_range to zerver.lib.narrow.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-16 21:13:44 -08:00
Anders Kaseorg 1095efeb52 message_fetch: Move parse_anchor_value to zerver.lib.narrow.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-16 21:13:44 -08:00
Anders Kaseorg 4eea5e102e message_fetch: Move ok_to_include_history to zerver.lib.narrow.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-16 21:13:44 -08:00
M@ 280523ad48
slack import: Merge and dedupe same-base emoji reactions and userlists.
The naive solution #23465 creates situations where the same user can have
multiple reactions as the base emojis are not unique, e.g. +1::skin2
and +1::skin4 would both reduce to +1 but the userlists are separate.
This solution handles the reduction, merges the same-base reactions,
and deduplicates the userlist.

Co-authored-by: Alex Vandiver <alexmv@zulip.com>
Co-authored-by: rht <rhtbot@protonmail.com>
2022-11-16 11:11:43 -08:00
Anders Kaseorg 9a8a2bd345 ruff: Enable import sorting, replacing isort.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-16 09:29:11 -08:00
Anders Kaseorg 1b016bef73 ruff: Enable naming errors.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-16 09:29:11 -08:00
Anders Kaseorg 924d530292 ruff: Fix N813 camelcase imported as lowercase.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-16 09:29:11 -08:00
Anders Kaseorg 8f17093be2 ruff: Fix N812 lowercase imported as non-lowercase.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-16 09:29:11 -08:00
Anders Kaseorg 0258fba345 ruff: Fix N811 constant imported as non-constant.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-16 09:29:11 -08:00
Anders Kaseorg 2876ae8e48 ruff: Fix N803 argument name should be lowercase.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-16 09:29:11 -08:00
Anders Kaseorg 46955da3a0 ruff: Fix ANN204 missing return type annotation for __init__.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-16 09:29:11 -08:00
Anders Kaseorg 3442bf2843 requirements: Upgrade Ruff.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-16 09:29:11 -08:00
Anders Kaseorg 1735b8863e ruff: Fix B012 return inside finally blocks.
return inside finally blocks causes exceptions to be silenced.
Although these blocks follow blanket ‘except Exception’ handlers, they
do not seem to have a goal of silencing BaseException and exceptions
thrown by the exception handler, so rewrite them to avoid it.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-16 09:29:11 -08:00
Anders Kaseorg 2cc3fa4fba scim: Check SCIM tokens using constant-time comparison.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-16 10:52:48 -05:00
Anders Kaseorg e5c26eeb86 tornado: Support sharding by user ID.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-15 17:27:01 -08:00
Tim Abbott 8010d06f9e compatiblity: Delete obsolete compatibility code.
Both of these compatibility blocks can be deleted, since you can't
upgrade directly to any supported release from the versions where the
old event formats would be used.
2022-11-15 15:39:38 -08:00
Hari Prashant Bhimaraju f374498f38 papertrail: Strengthen types using WildValue. 2022-11-15 15:05:02 -08:00
Hari Prashant Bhimaraju 4210ab6c41 ifttt: Strengthen types using WildValue. 2022-11-15 15:05:02 -08:00
Zixuan James Li 71abbbdd7a message_edit: Handle truncated topic resolution.
This solves the problem that resolving a topic with a long name (>60
characters) will cause the topic name to be truncated, and thus the edit
message code path thinks that the topic is being moved in addition to
being resolved.

We store the pre-truncation topic and use it to check against the
original topic when determining whether a topic is being moved while
getting (un)resovled or not.

Fixes #23482

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-11-15 14:33:39 -08:00
Zixuan James Li c821127131 message_edit: Fix resolution notification not being sent twice.
We intended to send both the "topic was resolved" and the "topic was
moved here" notification when resolving and moving a topic at the same
time in #22312.

The previous implementation did not work as expected and it was only
sending the "topic was moved here" notification.

This removes the check for old_topic and new_topic that have
RESOLVED_TOPIC_PREFIX stripped in maybe_send_resolve_notifications, so
that the notification will be sent regardless if the topic name without
the prefix stays the same or not.

Note that weird topic handling ("✔ ✔✔ some topic") in the comments
was added in e231a03eff is unaffected. In case of confusion, the lstrip
check is not essential to detecting topic being unresolved/resolved.
As we mainly have that handled in the latter part of the helper.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-11-15 14:33:39 -08:00
Hari Prashant Bhimaraju 5346de9164 circleci: Rewrite integration to support the new webhook format.
CircleCI has updated its webhook format[1] for CircleCI Cloud,
Server version 3.x and 4.x. This commit rewrites the CircleCI
integration to parse the new webhook structure. The tests have also
been rewritten for the new format.

With this commit, we support webhooks from projects that use GitHub,
BitBucket and GitLab as VCS providers. The CircleCI integration doc
has been updated to mention the same. The doc has also been updated
with the latest instructions for configuring a webhook on the CircleCI
interface, and the new output screenshots.

References:
[1]: https://circleci.com/docs/webhooks
2022-11-15 10:45:05 -08:00
rht 5ce2103b87 Slack import: Cache emoji.json in static/generated/emoji.
Previously, emoji.json was read from
"$ZULIP_PATH/node_modules/emoji-datasource-google/emoji.json".
This path doesn't exist in production when installing from scratch from
a release tarball. And so, we ensure emoji.json exists by copying it to
`static/generated/emoji`.

With tweaks to comments by tabbott.

Fixes: #23469
2022-11-15 10:43:11 -08:00
Mateusz Mandera f80a999828 invites: Add defensive assert in do_get_invites_controlled_by_user.
This is a follow-up to d201229df8.
do_get_invites_controlled_by_user queries for Confirmations when finding
multiuse invites controlled by a user. This means that a revoked
multiuse invite cannot really be fetched here, because
do_revoke_multi_use_invite deletes the Confirmation object when revoking
the invitations. However, having a defensive assert here should be
useful to make this doesn't secretly break in the future if the query
used changes or if there are unexpected revoked multiuse invites with an
existing Confirmations for any (buggy) reason.
2022-11-15 09:27:38 -08:00
Mateusz Mandera d201229df8 signup: Implement use of MultiUseInvite.status attribute.
This allows us to revoke MultiUseInvites by changing their .status
instead of deleting them (which has been deleting the helpful tracking
information on PreregistrationUsers about which MultiUseInvite they came
from).
2022-11-14 17:13:16 -08:00
Hari Prashant Bhimaraju 977a043d03 grafana: Support notifications from Grafana Alerting.
This commit adds support for Grafana's new alerting system, Grafana
Alerting. The existing Grafana integration has been modified to
detect the version of the notification through the structure of the
payload body, since the the structure varies by version. Support for
legacy alerting is been continued. Example fixtures have been added
for Grafana Alerting's webhooks.

Tests updated.
2022-11-14 16:28:59 -08:00
Hari Prashant Bhimaraju 804eb15aa5 grafana: Strengthen types using WildValue. 2022-11-14 16:28:59 -08:00
Hari Prashant Bhimaraju 8ab47bbcc4 opbeat: Strengthen types using WildValue. 2022-11-14 13:49:38 -08:00
Hari Prashant Bhimaraju 645e464f98 jira: Strengthen types using WildValue. 2022-11-14 13:49:38 -08:00
Hari Prashant Bhimaraju d8732e6769 intercom: Strengthen types using WildValue. 2022-11-14 13:49:38 -08:00
Hari Prashant Bhimaraju ac1a38db75 gocd: Strengthen types using WildValue. 2022-11-14 13:49:38 -08:00
Hari Prashant Bhimaraju 2bd1093c38 gosquared: Strengthen types using WildValue. 2022-11-14 13:49:38 -08:00
Hari Prashant Bhimaraju 08eb34d0d6 greenhouse: Strengthen types using WildValue. 2022-11-14 13:49:38 -08:00
Hari Prashant Bhimaraju a38b1390ac groove: Strengthen types using WildValue. 2022-11-14 13:49:38 -08:00
Hari Prashant Bhimaraju 8f4133b63e harbor: Strengthen types using WildValue. 2022-11-14 13:49:38 -08:00
Hari Prashant Bhimaraju c692fc400d hellosign: Strengthen types using WildValue. 2022-11-14 13:49:38 -08:00
Hari Prashant Bhimaraju f0fd083e88 homeassistant: Strengthen types using WildVale. 2022-11-14 13:49:38 -08:00
Hari Prashant Bhimaraju 6f374b666e insping: Strengthen types using WildValue. 2022-11-14 13:49:38 -08:00
Hari Prashant Bhimaraju 1ac9573a67 jotform: Strengthen types using WildValue. 2022-11-14 13:49:38 -08:00
Hari Prashant Bhimaraju 537632d58c wekan: Strengthen types using WildValue. 2022-11-14 13:49:38 -08:00
Matt Keller 958b58f174 slack: Skip reactions for deleted users.
Fixes #23552.
2022-11-14 13:08:15 -08:00