Commit Graph

16224 Commits

Author SHA1 Message Date
Zixuan James Li 0f2d4a3ab9 webhooks: Update Zabbix integration documentation.
Since the removal of `CurlHttpRequest` in Rabbix 6.2, the old script for
setting up the Zabbix integration no longer works.

https://www.zabbix.com/documentation/6.2/en/manual/installation/upgrade_notes_620?hl=CurlHttpRequest#curlhttprequest-removed

This updates the documentation to use `HttpRequest` instead and keep it
up-to-date with the latest Zabbix server. We raise the minimum supported
version from 5.2 to 5.4 because `HttpRequest` was introduced in 5.4.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-12-08 12:08:07 -08:00
Anders Kaseorg 8a96c68780 javascript_examples: Fix unicorn/no-useless-spread.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-12-07 09:55:05 -08:00
Anders Kaseorg 872f4b41c1 ci: Check that non-scripts aren’t marked executable.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-12-07 09:54:01 -08:00
Anders Kaseorg 5f2e895c5e scrub_realm: Exit unsuccessfully when asked to scrub an active realm.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-12-05 11:07:48 -08:00
Anders Kaseorg 4051435c2a ruff: Fix PLR0402 Consider using from … import.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-12-04 22:11:24 -08:00
Anders Kaseorg 4e9cced32b ruff: Fix PLR1701 Consider merging these isinstance calls.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-12-04 22:11:24 -08:00
Anders Kaseorg e3d57c9586 ruff: Fix B006 Do not use mutable data structures for argument defaults.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-12-04 22:11:24 -08:00
Anders Kaseorg e634e3276a ruff: Fix PLC0414 Import alias does not rename original package.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-12-04 22:11:24 -08:00
Anders Kaseorg 9a89a52086 ruff: Fix UP015 Unnecessary open mode parameters.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-12-04 22:11:24 -08:00
Anders Kaseorg f3f5dfb5aa ruff: Fix RUF004 exit() is only available in the interpreter.
‘exit’ is pulled in for the interactive interpreter as a side effect
of the site module; this can be disabled with python -S and shouldn’t
be relied on.

Also, use the NoReturn type where appropriate.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-12-04 22:11:24 -08:00
Lauryn Menard bfcaaf976b exceptions: Update error message for BAD_EVENT_QUEUE_ID.
Updates BAD_EVENT_QUEUE_ID error message to use "ID" instead
of "id".
2022-11-30 17:16:12 -08:00
Lauryn Menard 332d417324 api-docs: Replace description instances of "id" with "ID". 2022-11-30 17:16:12 -08:00
Alex Vandiver 8f6f38c97c cache: Decline to store querysets, with an error.
As we have seen no further cases of this in production since #23215,
increase the severity to an error, and switch from returning a
list (which is not type-safe if the function declares a QuerySet
return) to returning the QuerySet without caching.

Failing to store the result in the cache, with an error, seems
superior to raising an exception; in both cases the next request will
redo the work, but we are guaranteed a worse user experience if we 500
the request.

Ref https://github.com/zulip/zulip/pull/23215#discussion_r994186493
2022-11-29 16:45:11 -08:00
Alex Vandiver eb3674362a slack_incoming: Verify that keys exist before checking their value. 2022-11-29 16:34:16 -08:00
Alex Vandiver f1a8209705 slack_incoming: Fix formatting fixups with overlaps, or at start/end.
The previous regular expression required a `[^\w]` at the start and
end of the match.  This had two unintended effects -- it meant that it
could never match at the start or end of a string, and it meant
that *adjacent* words required *two* non-word characters between them,
as the pattern matches cannot overlap.

Switch to allowing string start/end to anchor the matches, and make
the trailing `[^\w]` be a zero-width look-ahead, to allow the patterns
to overlap.  Also remove the spurious `^` within the inner character
classes, which prevented `*foo^bar*` from matching.  Finally, add
tests to cover the functionality, which was previously untested.
2022-11-29 16:34:16 -08:00
Anders Kaseorg 79e859427a migrations: Fix Python-looped SQL in 0376; don’t crash if no user found.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-28 20:23:30 -08:00
Lauryn Menard 8bad751b82 docs: Remove `settings_html` and `subscriptions_html` from api url context.
Removes the now unused `settings_html` and `subscriptions_html`
links / text when adding a documentation pages' API URL context
information.
2022-11-28 12:21:36 -08:00
Lauryn Menard c1d0ffe099 integrations-docs: Update Hello World for current macros and style.
Updates the Hello World integration documentation and the section
of the related tutorial on documenting the example integration
for the currently used shared macro `create-bot-construct-url.md`.

Also, updates them to use the numbered style currently used in
the majority of the integrations documentation pages.
2022-11-28 12:21:36 -08:00
Zixuan James Li 1698778145 user_groups: Remove unused remove_user_from_user_group.
remove_user_from_user_group's only caller has been removed in 271333301d.
Its usage has been superseded by remove_members_from_user_group.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-11-21 13:41:06 -08:00
Anders Kaseorg 73c4da7974 ruff: Fix N818 exception name should be named with an Error suffix.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-17 16:52:00 -08:00
Alex Vandiver e41ee10966 export_search: Add a sender limit. 2022-11-17 11:19:59 -08:00
Tim Abbott 0ba63826dd tornado: Disable event compression for mark-as-unread events.
Doing rapid pace mark-as-unread in the Zulip web application, one
observed assertion failures showing that the server would send an
event containing multiple message IDs but only one of the messages
present in the message_details side data structure.

The cause of this was the "virtual events" compression system; two
flags/remove/read events were being combined by simply concatenating
the lists of events, without any attempt to merge the
`message_details` field on those objects.

The immediate fix is to disable virtual events compression for this
event class, but it's not unlikely we'll need to just eliminate the
virtual_events system entirely, because it seems difficult to make it
soundly handle a message whose state for a given flag changes back and
forth while the client is offline.

But we'll leave that for later, since removing that optimization
deserves more discussion than fixing this event corruption bug.
2022-11-17 00:46:26 -08:00
Anders Kaseorg 842a5bb54b message_flags: Allow updating flags by narrows and anchors.
Fixes #22893.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-16 21:13:44 -08:00
Anders Kaseorg 5d0711df6d message_flags: Don’t send events for no-op flag updates.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-16 21:13:44 -08:00
Anders Kaseorg 54878a1826 message_flags: Convert operation string to boolean.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-16 21:13:44 -08:00
Anders Kaseorg 5b5be1fd82 message_flags: Use bulk_access_messages in do_update_message_flags.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-16 21:13:44 -08:00
Anders Kaseorg 342cba7b4d message: Fetch streams in bulk in bulk_access_messages.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-16 21:13:44 -08:00
Anders Kaseorg fef365248b create_user: Create historical UserMessage rows with bulk_create.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-16 21:13:44 -08:00
Anders Kaseorg ee2cb855f0 message_fetch: Add include_anchor parameter.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-16 21:13:44 -08:00
Anders Kaseorg dae4633745 message_fetch: Extract fetch_messages helper to zerver.lib.narrow.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-16 21:13:44 -08:00
Anders Kaseorg bd5d200722 narrow: Return a dataclass from post_process_limited_query.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-16 21:13:44 -08:00
Anders Kaseorg 0a0a70b33d message_fetch: Move limit_query_to_range to zerver.lib.narrow.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-16 21:13:44 -08:00
Anders Kaseorg 1095efeb52 message_fetch: Move parse_anchor_value to zerver.lib.narrow.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-16 21:13:44 -08:00
Anders Kaseorg 4eea5e102e message_fetch: Move ok_to_include_history to zerver.lib.narrow.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-16 21:13:44 -08:00
M@ 280523ad48
slack import: Merge and dedupe same-base emoji reactions and userlists.
The naive solution #23465 creates situations where the same user can have
multiple reactions as the base emojis are not unique, e.g. +1::skin2
and +1::skin4 would both reduce to +1 but the userlists are separate.
This solution handles the reduction, merges the same-base reactions,
and deduplicates the userlist.

Co-authored-by: Alex Vandiver <alexmv@zulip.com>
Co-authored-by: rht <rhtbot@protonmail.com>
2022-11-16 11:11:43 -08:00
Anders Kaseorg 9a8a2bd345 ruff: Enable import sorting, replacing isort.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-16 09:29:11 -08:00
Anders Kaseorg 1b016bef73 ruff: Enable naming errors.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-16 09:29:11 -08:00
Anders Kaseorg 924d530292 ruff: Fix N813 camelcase imported as lowercase.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-16 09:29:11 -08:00
Anders Kaseorg 8f17093be2 ruff: Fix N812 lowercase imported as non-lowercase.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-16 09:29:11 -08:00
Anders Kaseorg 0258fba345 ruff: Fix N811 constant imported as non-constant.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-16 09:29:11 -08:00
Anders Kaseorg 2876ae8e48 ruff: Fix N803 argument name should be lowercase.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-16 09:29:11 -08:00
Anders Kaseorg 46955da3a0 ruff: Fix ANN204 missing return type annotation for __init__.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-16 09:29:11 -08:00
Anders Kaseorg 3442bf2843 requirements: Upgrade Ruff.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-16 09:29:11 -08:00
Anders Kaseorg 1735b8863e ruff: Fix B012 return inside finally blocks.
return inside finally blocks causes exceptions to be silenced.
Although these blocks follow blanket ‘except Exception’ handlers, they
do not seem to have a goal of silencing BaseException and exceptions
thrown by the exception handler, so rewrite them to avoid it.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-16 09:29:11 -08:00
Anders Kaseorg 2cc3fa4fba scim: Check SCIM tokens using constant-time comparison.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-16 10:52:48 -05:00
Anders Kaseorg e5c26eeb86 tornado: Support sharding by user ID.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-15 17:27:01 -08:00
Tim Abbott 8010d06f9e compatiblity: Delete obsolete compatibility code.
Both of these compatibility blocks can be deleted, since you can't
upgrade directly to any supported release from the versions where the
old event formats would be used.
2022-11-15 15:39:38 -08:00
Hari Prashant Bhimaraju f374498f38 papertrail: Strengthen types using WildValue. 2022-11-15 15:05:02 -08:00
Hari Prashant Bhimaraju 4210ab6c41 ifttt: Strengthen types using WildValue. 2022-11-15 15:05:02 -08:00
Zixuan James Li 71abbbdd7a message_edit: Handle truncated topic resolution.
This solves the problem that resolving a topic with a long name (>60
characters) will cause the topic name to be truncated, and thus the edit
message code path thinks that the topic is being moved in addition to
being resolved.

We store the pre-truncation topic and use it to check against the
original topic when determining whether a topic is being moved while
getting (un)resovled or not.

Fixes #23482

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-11-15 14:33:39 -08:00
Zixuan James Li c821127131 message_edit: Fix resolution notification not being sent twice.
We intended to send both the "topic was resolved" and the "topic was
moved here" notification when resolving and moving a topic at the same
time in #22312.

The previous implementation did not work as expected and it was only
sending the "topic was moved here" notification.

This removes the check for old_topic and new_topic that have
RESOLVED_TOPIC_PREFIX stripped in maybe_send_resolve_notifications, so
that the notification will be sent regardless if the topic name without
the prefix stays the same or not.

Note that weird topic handling ("✔ ✔✔ some topic") in the comments
was added in e231a03eff is unaffected. In case of confusion, the lstrip
check is not essential to detecting topic being unresolved/resolved.
As we mainly have that handled in the latter part of the helper.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-11-15 14:33:39 -08:00
Hari Prashant Bhimaraju 5346de9164 circleci: Rewrite integration to support the new webhook format.
CircleCI has updated its webhook format[1] for CircleCI Cloud,
Server version 3.x and 4.x. This commit rewrites the CircleCI
integration to parse the new webhook structure. The tests have also
been rewritten for the new format.

With this commit, we support webhooks from projects that use GitHub,
BitBucket and GitLab as VCS providers. The CircleCI integration doc
has been updated to mention the same. The doc has also been updated
with the latest instructions for configuring a webhook on the CircleCI
interface, and the new output screenshots.

References:
[1]: https://circleci.com/docs/webhooks
2022-11-15 10:45:05 -08:00
rht 5ce2103b87 Slack import: Cache emoji.json in static/generated/emoji.
Previously, emoji.json was read from
"$ZULIP_PATH/node_modules/emoji-datasource-google/emoji.json".
This path doesn't exist in production when installing from scratch from
a release tarball. And so, we ensure emoji.json exists by copying it to
`static/generated/emoji`.

With tweaks to comments by tabbott.

Fixes: #23469
2022-11-15 10:43:11 -08:00
Mateusz Mandera f80a999828 invites: Add defensive assert in do_get_invites_controlled_by_user.
This is a follow-up to d201229df8.
do_get_invites_controlled_by_user queries for Confirmations when finding
multiuse invites controlled by a user. This means that a revoked
multiuse invite cannot really be fetched here, because
do_revoke_multi_use_invite deletes the Confirmation object when revoking
the invitations. However, having a defensive assert here should be
useful to make this doesn't secretly break in the future if the query
used changes or if there are unexpected revoked multiuse invites with an
existing Confirmations for any (buggy) reason.
2022-11-15 09:27:38 -08:00
Mateusz Mandera d201229df8 signup: Implement use of MultiUseInvite.status attribute.
This allows us to revoke MultiUseInvites by changing their .status
instead of deleting them (which has been deleting the helpful tracking
information on PreregistrationUsers about which MultiUseInvite they came
from).
2022-11-14 17:13:16 -08:00
Hari Prashant Bhimaraju 977a043d03 grafana: Support notifications from Grafana Alerting.
This commit adds support for Grafana's new alerting system, Grafana
Alerting. The existing Grafana integration has been modified to
detect the version of the notification through the structure of the
payload body, since the the structure varies by version. Support for
legacy alerting is been continued. Example fixtures have been added
for Grafana Alerting's webhooks.

Tests updated.
2022-11-14 16:28:59 -08:00
Hari Prashant Bhimaraju 804eb15aa5 grafana: Strengthen types using WildValue. 2022-11-14 16:28:59 -08:00
Hari Prashant Bhimaraju 8ab47bbcc4 opbeat: Strengthen types using WildValue. 2022-11-14 13:49:38 -08:00
Hari Prashant Bhimaraju 645e464f98 jira: Strengthen types using WildValue. 2022-11-14 13:49:38 -08:00
Hari Prashant Bhimaraju d8732e6769 intercom: Strengthen types using WildValue. 2022-11-14 13:49:38 -08:00
Hari Prashant Bhimaraju ac1a38db75 gocd: Strengthen types using WildValue. 2022-11-14 13:49:38 -08:00
Hari Prashant Bhimaraju 2bd1093c38 gosquared: Strengthen types using WildValue. 2022-11-14 13:49:38 -08:00
Hari Prashant Bhimaraju 08eb34d0d6 greenhouse: Strengthen types using WildValue. 2022-11-14 13:49:38 -08:00
Hari Prashant Bhimaraju a38b1390ac groove: Strengthen types using WildValue. 2022-11-14 13:49:38 -08:00
Hari Prashant Bhimaraju 8f4133b63e harbor: Strengthen types using WildValue. 2022-11-14 13:49:38 -08:00
Hari Prashant Bhimaraju c692fc400d hellosign: Strengthen types using WildValue. 2022-11-14 13:49:38 -08:00
Hari Prashant Bhimaraju f0fd083e88 homeassistant: Strengthen types using WildVale. 2022-11-14 13:49:38 -08:00
Hari Prashant Bhimaraju 6f374b666e insping: Strengthen types using WildValue. 2022-11-14 13:49:38 -08:00
Hari Prashant Bhimaraju 1ac9573a67 jotform: Strengthen types using WildValue. 2022-11-14 13:49:38 -08:00
Hari Prashant Bhimaraju 537632d58c wekan: Strengthen types using WildValue. 2022-11-14 13:49:38 -08:00
Matt Keller 958b58f174 slack: Skip reactions for deleted users.
Fixes #23552.
2022-11-14 13:08:15 -08:00
Aman Agrawal fbe9a9e539 left_side_userlist: Remove feature from frontend.
Fixes #23517.

While this feature was added to Zulip very early, it has been troubled
for most of that time; it never looked great visually, had a lot of
implementation complexity around resize.js, and has a weird model (a
setting that changes the UI only in certain window sizes).

This option is not commonly used; while a significant portion of users
have it enabled, many of them just don't use window sizes where it
actually has an effect. So it's not clear that it will be missed if
removed; we got very few bug reports when it was completely broken for
a few days after we first integrated the new left sidebar private
messages design.

Even with it no longer being broken, it does not work very well with
the addition of the new PMs section in the left sidebar. (Having two
scrollbars in the sidebar looks quite awkward.) The new private
messages section in the left sidebar also addresses some of the use
cases for always keeping the Users list always visible, even in narrow
windows.

This option is only removed from frontend for now. To make this
decision easily reversible, the backend code of this feature
is still kept.
2022-11-14 12:23:55 -08:00
Lauryn Menard de12500f68 popovers: Rename "full profile" to "profile".
Now that we now longer have another thing called a "user profile", we
can simplify the name for this part of the UI.

Fixes #23530.
2022-11-14 12:16:22 -08:00
Lauryn Menard 9aacb15530 tabbed-sections: Update "via-user-profile" to be "via-user-card". 2022-11-14 12:16:22 -08:00
Lauryn Menard 48e2482bbd popovers: Rename "profile summary" to "user card".
This is a briefer and more intuitive name for this frequently accessed
part of the Zulip UI.
2022-11-14 12:15:54 -08:00
Mateusz Mandera 3bb2096a7f do_reactivate_realm: Noop if realm is already active.
If the realm doesn't actually need re-activation, this should be a noop
rather than creating a confusing RealmAuditLog entry.
2022-11-13 14:28:33 -08:00
Mateusz Mandera 5c20e2af78 test_do_send_realm_reactivation_email: Deactivate the realm first.
This doesn't make sense if the realm is active and will fail as soon as
do_reactivate_realm is fixed in the next commit to be a noop and not
create confused RealmAuditLog entries when the realm is active.
2022-11-13 14:28:33 -08:00
Mateusz Mandera 23a776c144 maybe_send_to_registration: Don't reuse pre-existing PreregistraionUser.
There was the following bug here:
1. Send an email invite to a user.
2. Have the user sign up via social auth without going through that
   invite, meaning either going via a multiuse invite link or just
   straight-up Sign up if the org permissions allow.

That resulted in the PreregistrationUser that got generated in step (1)
having 2 Confirmations tied to it - because maybe_send_to_registration
grabbed the object and created a new confirmation link for it. That is a
corrupted state, Confirmation is supposed to be unique.

One could try to do fancy things with checking whether a
PreregistrationUser already have a Confirmation link, but to avoid races
between ConfirmationEmailWorker and maybe_send_to_registration, this
would require taking locks and so on - which gets needlessly
complicated. It's simpler to not have them compete for the same object.

The point of the PreregistrationUser re-use in
maybe_send_to_registration is that if an admin invites a user, setting
their initial streams and role, it'd be an annoying experience if the
user ends up signing up not via the invite and those initial streams
streams etc. don't get set up. But to handle this, we can just copy the
relevant values from the pre-existing prereg_user, rather than re-using
the object itself.
2022-11-10 17:46:53 -08:00
Mateusz Mandera b5b8cc06dc auth: Clean up redundant request.session.modified change.
This line was added in 94e099eaab,
presumably because of the
del request.session["multiuse_object_key"]
line that was just above it.
Looks like it should have been removed in
868a763cec, which eliminated that `del`
operation.
2022-11-10 17:46:53 -08:00
Alex Vandiver f3135eb414 export_search: Add ability to limit by date sent. 2022-11-10 16:49:19 -08:00
Alex Vandiver 2d5366f455 export_search: Add a compliance tool to export a subset of messages.
This is primarily for administrators needing to provide message
history for compliance or auditing purposes.  Search terms can be
pulled from a file, one per line, or from arguments provided on the
command line.
2022-11-10 16:49:19 -08:00
Alex Vandiver 1f18fa6580
register: Not all clients send a User-Agent header.
In 1fce1c3c73, we added logic to parse
the User-Agent in /register requests; this logic crashed if an HTTP request
was missing that header.

Includes a test for `/register` with no user agent passed; this should catch
similar regressions in the future.

Co-authored-by: Mateusz Mandera <mateusz.mandera@zulip.com>
2022-11-10 16:21:04 -08:00
Anders Kaseorg 70dbeb197f middleware: Set the correct options on the django_language cookie.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-09 14:24:22 -08:00
Lauryn Menard 62c9bb728e tests: Remove ignored `field_type` param in custom profile data test.
For PATCH requests to `/realm/profile_fields`, the field_type is
determined via the field.id, which is in the URL. Therefore,
`field_type` does not need to be passed as part of the request
data in tests.
2022-11-08 11:08:14 -08:00
Sahil Batra 7c4da60b53 message: Don't create historical UserMessage rows when marking as read.
We do not create historical UserMessage rows, for messages that didn't
have one, while marking messages as read and simply ignore those messages.
We do so because there is no user of creating UserMessage rows and it just
wastes storage.

Note that we still allow to mark messages from unsubscribed streams as
read but only those which have UserMessage rows for them to handle the
case when the unread messages were not marked as read while unsubscribing
from the stream due to some race condition. In such cases, messages
will not be included in the unread count shown in "All messages" menu
(and stream is anyways not present in the left sidebar), but the message
border on the left is green if viewing the stream after unsusbcribing it.
So, to avoid the confusion for users, the messages will be marked as read
when user scrolls down.
2022-11-08 08:50:17 -08:00
Zixuan James Li 390e2a12e5 webhooks: Use get_short_sha to get shortened sha.
This unifies the length of the shortened SHA our integrations generate,
and ensures that they are long enough for projects of various sizes with
a chosen value defined in get_short_sha.

Fixes #23475

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-11-08 08:20:51 -08:00
Zixuan James Li 4e46899494 webhooks: Pick a more reasonable length for short sha.
7 characters are not enough for large projects, so we change
it to reasonably longer. As an example, The Linux kernel needs
at least 11 characters of sha in its shortened form to identify
a revision. We pick 11 so it should work for most of the projects.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-11-08 08:20:51 -08:00
Lauryn Menard 2a1ceb58d9 api-docs: Return 404 for direct `/api-doc-template` call.
We use `templates/zerver/api/api-doc-template.md` as a base template
for the documented API endpoints in `zerver/openapi/zulip.yaml`.

Previously, if this template was called as an endpoint, then it
would fail an assertion check and send server error. Now we check
for specifically for that potential path and return a 404 error
response for no existing article.

Fixes #21876.
2022-11-08 08:10:15 -08:00
Matt Keller 8aa7ff4bbb slack: Parse emoji skin tone variants.
Fixes part of #23276.
2022-11-07 14:25:49 -08:00
Lauryn Menard fbe2ef9431 api-docs: Remove unused description in get-subscription-status. 2022-11-07 12:31:55 -08:00
Lauryn Menard 566a6823af api-docs: Use default example response description for get-messages. 2022-11-07 12:31:55 -08:00
Lauryn Menard 6f44681354 api-docs: Use default response example description for create-drafts. 2022-11-07 12:31:55 -08:00
Alya Abbott 9ce484e267 portico: Add a page about our values. 2022-11-07 12:27:09 -08:00
Alya Abbott 1ef94cc2b4 portico: Update /history page. 2022-11-07 12:27:09 -08:00
Lauryn Menard 19a4c3907f decorator: Put back check for dev env in `ignore_unhashable_lru_cache`.
Prior to 53231aa, the `ignore_unhashable_lru_cache` decorator had
a check for the development environment so that changes could be
seen on refresh.

Puts that check back in IgnoreUnhashableLruCacheWrapper class.
2022-11-07 12:17:59 -08:00
Lauryn Menard 66e607cf4c tests: Remove ignored `message_id` parameter in message edit tests.
For PATCH requests to edit messages, the message ID is in the URL
and does not need to be passed as part of the data in tests.
2022-11-07 12:16:28 -08:00
Anders Kaseorg 55342efd33 scim: Upgrade django-scim2; remove request.user monkey patching.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-05 15:13:50 -07:00
Anders Kaseorg 7e8ebd18b6 test_scim: Add test for missing or wrong authentication.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-05 15:13:46 -07:00
Alex Vandiver d8ebbedbbb outgoing_webhook: Ignore the exception on failure if the stream is gone.
In the outgoing webhook handler, there is potentially several seconds
of trying between when a message triggering an outgoing webhook
arrives, and when it fails.  In the meantime, the stream the
triggering message was on may have been deleted, causing the
"Failure!" message to have no valid stream to be sent to.

Rather than raise an exception in the outgoing webhook worker, ignore
the exception and move on.
2022-11-04 14:46:17 -07:00
Alex Vandiver eb7a2f2c38 queue: Do test retries in tests.
The lambda passed to `queue_json_publish` is used if
`settings.USING_RABBITMQ` is unset -- which is only true in tests.  As
such, this pattern causes failures to never actually retry within
tests.

This behaviour has existed ever since the outgoing webhook code was
introduced in 53a8b2ac87, with no explanation.  Not passing that
argument allows tests to verify the retry behaviour when webhooks
fail.
2022-11-04 14:46:17 -07:00
Zixuan James Li 7dbd98d25c rest: Restrict access to json views via basic auth.
Previously, test cases or clients accessing /json/ views using HTTP
Basic Auth would be accepted, while we intended to only allow clients
authenticated with a session cookie to access these views.

This adds a check on the accessed path to avoid this possibility.

It seems unlikely that any API clients clients were taking advantage
of this unintended quirk; so we're not going to bother documenting
this bug fix as an API change. In any case, it should be trivial for
anyone affected to consult the documentation and then switch their
/json/foo URL to a correct /api/v1/foo URL.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-11-04 14:44:07 -07:00
Sahil Batra 1fce1c3c73 register: Handle "Pronouns" type fields for older mobile clients.
Mobile clients older than v27.192 do not support PRONOUNS type
custom profile fields, so we instead change the type of it to
SHORT_TEXT in the data sent with register response and also in
the events sent to those clients.
2022-11-04 14:31:18 -07:00
Sahil Batra b2737b0878 migrations: Add migration to migrate pronouns custom profile fields.
This commit adds a migration to migrate SHORT_TEXT type profile
fields for pronouns to recently added PRONOUNS type.
2022-11-04 14:31:18 -07:00
Anders Kaseorg b9dfa23666 requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-03 14:08:58 -07:00
Anders Kaseorg c6f6df986c openapi: Fix “contatining” typo.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-03 14:08:58 -07:00
M@ 47ecd33d3a
import: Narrow the id-window to just the current realm.
On multi-realm systems this results in traversal of all messages in
all realms and returns a massive payload of 1 row per stream on 
the server, not the intended one row per realm.
2022-11-03 12:13:43 -07:00
Anders Kaseorg 3d853caf16 ruff: Fix C417 Unnecessary `map` usage.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-03 12:10:15 -07:00
Anders Kaseorg ab864be904 ruff: Fix C416 Unnecessary `set` comprehension (rewrite using `set()`).
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-03 12:10:15 -07:00
Anders Kaseorg 77c15547e6 ruff: Fix C414 Unnecessary `list` call within `sorted()`.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-03 12:10:15 -07:00
Anders Kaseorg 69e94b5991 ruff: Fix C413 Unnecessary `list` call around `sorted()`.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-03 12:10:15 -07:00
Anders Kaseorg 9e53f0c756 ruff: Fix C405 Unnecessary `list` literal (rewrite as a `set` literal).
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-03 12:10:15 -07:00
Anders Kaseorg 033d2615f6 ruff: Fix B017 `assertRaises(Exception):` should be considered evil.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-03 12:10:15 -07:00
Anders Kaseorg e1a2894af9 ruff: Fix B011 Do not `assert False` (`python -O` removes these calls).
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-03 12:10:15 -07:00
Anders Kaseorg adffad384c lint: Replace pycodestyle and pyflakes with ruff.
https://github.com/charliermarsh/ruff

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-03 12:10:15 -07:00
Tim Abbott 873ab01c7a message_flags: Fix accidentally quadratic behavior. 2022-11-02 21:05:24 -07:00
Sahil Batra aeb9a27d0e message: Ignore messages in unsubcribed streams when marking them unread.
Zulip's unread messages design has an invariant that all unread stream
messages must be in streams the user is subscribed to. For example, We
do not include the unread messages from unsubscribed streams in the
"unread_msgs" data structure in "/register" response and we mark all
unread messages as read when unsubscribing a user from a stream.

Previously, the mark as unread endpoint allowed violating that
invariant, allowing you to mark messages in any stream as unread.

Doing so caused the "message_details" data structures sent with
"update_message_flags" events to not contain messages from
unsubscribed streams, even though those messages were present in the
set of message IDs. These malformed events, in turn, caused exceptions
in the frontend's processing of such an event.

This change is paired with a separate UI change to not offer the "Mark
as unread" feature in such streams; with just this commit, that will
silently fail.

With some additions to the tests by tabbott.
2022-11-02 14:49:23 -07:00
Mateusz Mandera 6e336ef6f6 delete_topic: Use the same timeout pattern as /mark_all_as_read.
We don't want to use the nginx 502 timeout as an API pattern. In
/mark_all_as_read we've already figured out how we want to handle this
instead.
2022-11-02 16:50:06 -04:00
Zixuan James Li da9ad7c3a7 markdown: Pass realm down into render_stream_description.
This guarantees that the Realm is always non-None when we hit the
codepath is_static_or_current_realm_url via
do_change_stream_description, so that we can properly skip rewritting
some images.

Fixes #19405

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-11-01 08:20:47 -07:00
Mateusz Mandera 582d5b0aa3 realm_creation: Rework error pages.
The previous error page was inadequate for serving the two different
scenarios where we show errors in realm_creations, in particular
containing a misleading sentence about realm creation being disabled
(even in the case where it was actually enabled and the user simply had
an expired link).
2022-10-31 17:35:06 -07:00
Tim Abbott b057d8ea17 invites: Delete ScheduledEmails in revoke invites transactions.
This prevents likely rare race conditions where a reminder about a
revoked invitation might get sent.
2022-10-31 15:06:42 -07:00
Mateusz Mandera 3dd0545118 invites: Add atomic() around some invite-revoking blocks.
These should only ever happen together, so should be wrapped in
atomic().
2022-10-31 14:37:46 -07:00
Lauryn Menard 0c947985bd tests: Update tests to use example profile picture.
So that we can stop using Tim's photo for tests, adds an open
license profile picture to use instead.

Updates tests that used `tim.png` to use the new example profile
picture, which is located in `static/images/test-images/avatars/`.
2022-10-31 14:36:54 -07:00
Lauryn Menard 1eda42f509 images: Create `/authentication_backends` directory for icons.
Creates `static/images/authentication_backends` directory for icons
of backend authentication methods, which are used on the log-in page.

And updates the example documentation in the API `/server_settings`
endpoint.
2022-10-31 14:36:54 -07:00
Lauryn Menard 89713bab76 tests: Use Zulip icon for markdown test cases instead of octopus.
Updates the markdown test case that used `zulip-octopus.png` to
instead use an zulip logo that's also referenced in a frontend
puppeteer test, `static/images/logo/zulip-icon-128x128.png`.
2022-10-31 14:36:54 -07:00
Sahil Batra 888e4f9bf4 custom_profile_fields: Fix query for display_in_profile_summary limit.
The query in display_in_profile_summary_limit_reached should check
realm also since there is per-realm limit of 2 fields, otherwise
this will cause issues where multiple realms are hosted on the
same server.

Fixes #23368.
2022-10-31 14:10:47 -07:00
Mateusz Mandera 760587450e retention: Remove two redundant comments.
These two identical comments don't contribute anything useful and seem
just out of place at this point.
2022-10-31 10:23:57 -07:00
Mateusz Mandera 7b13204e8f retention: Use Message.realm to simplify private message query.
We no longer need to do the inner joins to figure out the message's
realm and split up the cross-realm and regular case - now we just look
at zerver_message.realm directly.
2022-10-31 10:23:57 -07:00
Mateusz Mandera 78a325ac58 internal_send_private_message: Fix edge case of message to system bot.
I don't think this is used anywhere outside of tests, but we should have
this logic correct. If this function is used to send a message from a
user to a cross-realm bot, the message.realm should be the realm of the
user.

In the normal case, where a user send a message to a cross-realm bot
through the API is already handled correctly, this bug is unrelated.
2022-10-31 10:23:57 -07:00
Mateusz Mandera ebc144896f delete_realm: Use Message.realm column for message count. 2022-10-31 09:57:42 -07:00
Mateusz Mandera 601f0677f3 update_first_visible_message_id: Use Message.realm column.
With the Message.realm column added, this query can now be improved.
2022-10-31 09:57:42 -07:00
Lauryn Menard bcd1763dca api-docs: Clarify narrow parameters in API documentation.
Clarifies most of the narrow parameter descriptions by adding
information about what a user's message history includes, about
new bot users not generally being subscribed to streams, and
about the specific `streams:public` narrow.

Updates the main descriptions for the `/get-messages` and
`/check-messages-match-narrow` endpoints.

Fixes #19477.
2022-10-28 17:32:44 -07:00
Sahil Batra 2a8b7412ff url_encoding: Use user name instead of email in personal_narrow_url.
This commit updates the urls for personal narrow sent in email
notifications to be of form "{user_id}-{encoded_full_name}" to
make it consistent with the urls that we use for such narrows
in webapp which were recently updated in b4eddad for improving
performance. We encode the full name in the same way that we do in
webapp by replacing the url characters encoded by browser with "-".
2022-10-28 09:47:43 -07:00
Mateusz Mandera a410f6b241 do_mark_all_as_read: Split up the work into batches.
Fixes #15403.
2022-10-27 16:59:54 -07:00
Mateusz Mandera ef468322f1 test_openapi: Fix status code passed to validate_... in test_attributes.
It doesn't seem to make sense to append _{number} to the status code in
that arg, because the resulting string stops looking like a status code
and actually makes this test fail in the follow-up commit with the
confusing error message of

Unknown response http status: 2000

So this just seems like a bug.
2022-10-27 16:59:54 -07:00
Alya Abbott 8f6cf65f28 help center: Reorganize and extend the "Logging in" help page.
Also move some instructions from related pages into shared /include files,
with minor tweaks.
2022-10-26 12:28:29 -07:00
Matt Keller 4d87bf291c slack: Skip files where file_access: file_not_found. 2022-10-25 12:18:20 -07:00
Lauryn Menard e66f125ee6 recent-topics: Update hash for view to be "#recent".
Updates the hash used for the recent conversations view to be
"#recent" instead of "#recent_topics".

We will need to keep the logic for handling "#recent_topics"
permanently because users potentially have messages from
Welcome Bot with links to that hash.

Including "recent_topics" as a web_public_allowed_hash in
hash_util.js can be changed once self-hosted servers cannot
upgrade directly to Zulip 5.x from the current version.

Fixes #23132.
2022-10-25 11:03:44 -07:00
Anders Kaseorg d41afb2b25 tools: Add missing spaces in concatenated string literals.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-10-24 14:52:49 -07:00
Prakhar Pratyush 38aa1ee1ff test_classes: Improve assert_database_query_count output message.
Output message should talk about both the cases:
actual_count > expected_count and actual_count < expected_count.

The message now includes information for the case where
actual_query_count < expected_query_count.

Fixes: #23325
2022-10-24 10:18:58 -07:00
Lauryn Menard c9044f7acd recent-topics: Rename to "Recent conversations" in web-app and docs.
Replaces instances of "recent topics" in the web-app and documentation
to be "recent conversations".

Renames both `recent-topics.md` files in the help center to be
`recent-conversations.md` and updates/redirects links to new URL.

Does not update instances of "recent topics" in frontend code comments
and does not update the main overview changelog, for now.

Does not change case study text where "recent topics" was referenced
in a quote, but does change generic text references to be "recent
conversations".
2022-10-21 10:28:19 -07:00
Anders Kaseorg c5c180fda3 email_validation: Restore case-insensitive domain validation.
This was broken by commit b945aa3443
(#22604), because email_to_domain implicitly lowercased the result.

No adjustment is needed for is_disposable_domain, which already
lowercases its argument.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-10-20 08:19:07 -07:00
Anders Kaseorg 89e4233b68 timeout: Correct thread id type passed to PyThreadState_SetAsyncExc.
This type changed in Python 3.7:
https://github.com/python/cpython/pull/781

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-10-19 14:49:24 -07:00
David Rosa 64c31b7cb9 help: Rename "Add custom profile fields" -> "Custom profile fields".
Renames the help article on custom profile fields to reflect that
its content is not just about adding fields.

Adds a redirect from the old URL to the new URL and updates internal
links, linking to #add-a-custom-profile-field where appropriate.

Fixes #23170.
2022-10-19 11:58:00 -07:00
Mateusz Mandera f6f7e4c53f migrations: Remove bugged Recipients in 0419. 2022-10-18 20:50:10 -07:00
David Rosa d9dd272a78 help: Update documentation on bots.
Adds new tab to `zerver/lib/markdown/tabbed_sections.py` to document
managing bots from both personal settings and organization settings.

Documents adding bots from the organization settings Bots panel.

Separates instructions for deactivating and reactivating a bot from
both personal settings and organization settings.

Fixes a few formatting issues such as missing bold formatting and
heading level.

Fixes: #23066.
2022-10-18 10:16:42 -07:00
Florian Pritz f37ac80384 import_realm: Speed up first_message_id calculation.
On my data (about 10 million messages in 1600 streams) this used to take
about 40 hours, while the improved statement completes in roughly 30
seconds.

The old solution had postgres go through the entire table until the
first match for each stream. Thus, the time spent scanning the table
got longer and longer for each stream because postgres always started at
the beginning (and somehow it did not use any indices) and had to skip
over all rows until it found the first message from the stream that is
was looking for each time.

This new statement just performans a bulk operation, scanning the table
only once and then inserts the results directly into the destination
table.

Slightly more verbose inforation about this change can be found in:
https://chat.zulip.org/#narrow/stream/31-production-help/topic/Import.20Rocketchat.20data/near/1408867

Signed-off-by: Florian Pritz <bluewind@xinu.at>
2022-10-17 11:43:21 -07:00
Zixuan James Li 46329a2710 test_classes: Create a dedicate helper for query count check.
This adds a helper based on testing patterns of using the "queries_captured"
context manager with "assert_length" to check the number of queries
executed for preventing performance regression.

It explains the rationale of checking the query count through an
"AssertionError" and prints the queries captured as assert_length does,
but with a format optimized for displaying the queries in a more
readable manner.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-10-17 11:32:52 -07:00
Zixuan James Li 9117db8887 test_helpers: Tighten type annotation for queries_captured.
This adds CapturedQueryDict to provide a more accurate type annotation
for the return value of queries_captured. We also replace "Generator"
with "Iterator" because the latter two type parameters were unused.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-10-17 11:32:52 -07:00
PIG208 7db221c8dc
migrations: Remove noop realm filters operations.
We followed a same approach as in #22611 to mark migrations as noops. We
might eventually squash them.

Migration operations that only change the validators should be removed
as they are essentially noops that do not affect the database schema.

However, ./manage.py makemigration still generates a new migration for
validators change regardless. So we still have to keep one migration
that updates the validators to the latest state. We prefer to keep the
earliest one for this purpose.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-10-14 17:52:28 -07:00
Mateusz Mandera 61de767967 login page: Show form-independent errors even if email auth disabled.
These used to only be shown conditional on the
{% if password_auth_enabled %} in the template. Meaning that if you had
an org with email auth disabled and a deactivated user tried to log in,
they wouldn't see the error shown and get confused.

This switches the position of where these error will be shown (above the
login+password form instead of below it), but it looks fine.
2022-10-14 13:07:58 -07:00
Lauryn Menard 171da34546 events: Send empty list for custom_profile_fields in spectator view.
Send an empty list of `custom_profile_fields` in `page_params` for
spectators, rather than not sending the field at all.

Also, updates the user info popover to not show the manage user
three-dot menu when in a spectator view.
2022-10-14 13:05:35 -07:00
Tim Abbott 966e63cc14 migrations: Add noop reverse_code for migration 0419.
This migration is naturally one that should be a noop when
downgrading, so declare it as such.
2022-10-13 23:16:01 -07:00
Lauryn Menard 98074951ef api-docs: Update examples of queue_id for uuid format. 2022-10-13 10:08:42 -07:00
Alex Vandiver c328de3372 cache: Log a warning when attempting to store a whole QuerySet.
As noted in the previous commit, this causes bloat in memcached, for
no purpose.  Log a warning when `cache_with_key` sees a QuerySet
returned from the function it is decorating.
2022-10-12 22:25:48 -07:00
Alex Vandiver 55c0e670d9 cache: Only cache list results of QuerySets, not the QuerySet itself.
Storing a QuerySet rather than the list version of the result in it
has a large overhead -- and, as noted by the type annotations, the
result is only ever used as a list.  This difference is particularly
important because the cached `get_realm_user_dicts` can get extremely
large for realms with large numbers of users, potentially overflowing
the 1MB default object limit in memcached.

Switch all cases of `cache_with_key` which return QuerySets to
returning the list values of them.
2022-10-12 22:25:48 -07:00
Alex Vandiver 204f1b58e8 cache: Drop realm_id from `realm_user_dict_fields`.
Storing this key is superfluous, as it will be the same for all users,
and definitionally already known to fetch the cache for the realm.  It
is also not currently used by the callsites that read rows from the
cache.
2022-10-12 22:25:48 -07:00
Alex Vandiver 5d42a0cb00 linkifiers: Support %20 in URLs for topic links.
9381a3bd45 added support for linkifier pattern URLs containing
`%20`-style escapes, but only did so for the codepath which is used in
the message body -- topic links did not understand them.

Expand the support to include when they are substituted into topics.
2022-10-11 14:31:13 -07:00
Anders Kaseorg f28f30085a openapi: Remove documentation for unstable timezones.json path.
The location of files in /static is not part of our stable API.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-10-11 14:26:33 -07:00
Julia Bichler 4bb381fc80 message_edit: Support sending notifications with topic changes.
Previously we did not send notification for topic-only edits.
Now, we add backend support for sending notification to topic-only
edits as well.

We would add support for this in webapp in further commits since
message edit UI will be updated as well. We just make sure that no
notifications are sent when editing topic using pencil icon in
message header.

We also change the API default for moving a topic to only notify the
new location, not the old one; this matches the current defaults in
the web UI.

Includes many tests.

We also update the puppeteer tests to test only content edit as
we are going to change the UI to not allow topic editing from
message edit UI. Also fixing the existing tests to pass while
doing topic edits is somewhat complex as notification message
is also sent to new topic by default.

Fixes #21712.

Co-authored-by: Aman Agrawal <amanagr@zulip.com>
Co-authored-by: Tim Abbott <tabbott@zulip.com>
2022-10-11 11:35:41 -07:00
Matt Keller c5f106ce1b slack: Skip files where file_access: access_denied.
These stubs are incomplete and should be treated akin to tombstones.
2022-10-11 10:53:16 -07:00
Sahil Batra 3712d9c3f5 realm: Remove redundant code.
This code is not needed as we have changed the message
edit settings to be handled by "do_set_realm_property"
function in 04693b6ac1.
2022-10-11 10:51:43 -07:00
Mateusz Mandera 470c0458e6 do_deactivate_user: Use .on_commit around send_event calls.
The previous commit did this for revoking sessions. send_events should
be handled similarly too, to correctly handle calling do_deactivate_user
inside a transaction.
2022-10-11 10:49:07 -07:00
Mateusz Mandera a94b2572be do_deactivate_user: Revoke sessions in transaction.on_commit().
Fixes #21709.
2022-10-11 10:49:07 -07:00
Anders Kaseorg 6d19256877 notes: Separate __notes_map per-subclass.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-10-10 08:42:13 -07:00
Anders Kaseorg 1385a827c2 python: Clean up getattr, setattr, delattr calls with literal names.
These were useful as a transitional workaround to ignore type errors
that only show up with django-stubs, while avoiding errors about
unused type: ignore comments without django-stubs.  Now that the
django-stubs transition is complete, switch to type: ignore comments
so that mypy will tell us if they become unnecessary.  Many already
have.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-10-10 08:40:28 -07:00
Anders Kaseorg 5e1ebf2861 python: Fix __dict__ mutation abuse.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-10-10 08:33:08 -07:00
Anders Kaseorg b45484573e python: Use format string for logging str(obj).
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-10-10 08:32:29 -07:00
Anders Kaseorg fcd81a8473 python: Replace avoidable uses of __special__ attributes.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-10-10 08:32:29 -07:00
Mateusz Mandera cefed552f6 test_slack_importer: Add assertion about message count.
This will help catch any future regression that might lead the import
tool to fail to import messages into the correct realm.
2022-10-07 10:10:01 -07:00
Mateusz Mandera 42b3575183 models: Make Message.realm and ArchivedMessage.realm non-null.
In the previous commits, these have been backfilled, so now we can
enforce non-nullable.
2022-10-07 10:10:01 -07:00
Mateusz Mandera c1e701ec22 migrations: Add migration to backfill Message.realm. 2022-10-07 10:10:01 -07:00
Mateusz Mandera 00b3546c9f models: Add denormalized .realm column to Message.
This commit adds the OPTIONAL .realm attribute to Message
(and ArchivedMessage), with the server changes for making new Messages
have this set. Old Messages still have to be migrated to backfill this,
before it can be non-nullable.

Appropriate test changes to correctly set .realm for Messages the tests
manually create are included here as well.
2022-10-07 10:09:38 -07:00
Mateusz Mandera a4008d938a test_soft_deactivation: Add docstring explaining send_fake_message. 2022-10-07 10:09:38 -07:00
Anders Kaseorg 133c8b16ed requirements: Upgrade openapi-core.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-10-07 10:03:51 -07:00
Anders Kaseorg ba3aefc3d1 openapi: Set x-model: dict for objects used as associative arrays.
This works around an openapi-core bug.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-10-07 10:03:51 -07:00
Sahil Batra 2bf70fe4db custom_profile_field: Add "Pronouns" custom field type.
This commit adds "Pronouns" custom profile field type. We also
add "Pronouns" type field in the development environment
2022-10-06 17:56:26 -07:00
Anders Kaseorg 92ad4455ed requirements: Upgrade Django to 4.1.
zerver/migrations/0240_usermessage_migrate_bigint_id_into_id.py needs
to be updated to account for Django 4.1 creating AutoField as an
identity column rather than a serial column.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-10-06 15:59:07 -07:00
Anders Kaseorg 67cbb21267 migrations: Fix UserMessage.id sequence calculation in 0240.
The sequence value should reflect the last id, not the next id, to
avoid leaving a gap of 1.  Also, it should take ArchivedUserMessage.id
into account to avoid collisions during future archiving.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-10-06 15:59:07 -07:00
Anders Kaseorg 8230324068 markdown: Store ZulipMarkdown in members with the right type.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-10-06 15:15:10 -07:00
Anders Kaseorg 2bd81dd5c9 fenced_code: Avoid sloppy AttributeError handler.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-10-06 15:15:10 -07:00
Anders Kaseorg 3cf91e9e45 markdown: Rename our Markdown subclass to ZulipMarkdown.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-10-06 15:15:10 -07:00
Anders Kaseorg 97be895cf0 markdown: Remove Optional from zulip_rendering_result type.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-10-06 15:15:10 -07:00
Anders Kaseorg d01c99d2ee markdown: Add missing None check in InlineInterestingLinkProcessor.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-10-06 15:15:10 -07:00
Anders Kaseorg eb9957aa35 subdomains: Fix realm=None case for is_static_or_current_realm_url.
Fixes #22636.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-10-06 15:15:10 -07:00
Anders Kaseorg ad2795698b models: Remove explicit id fields.
With django-stubs, these explicit copies of Django’s implicit id
fields are no longer needed for type checking.  An exception is the
BigAutoField AbstractUserMessage.id, which is left alone.

This reverts commit c08ee904d8 (#15641).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-10-06 13:49:11 -07:00
Anders Kaseorg 47c5deeccd python: Mark dict parameters with defaults as read-only.
Found by semgrep 0.115 more accurately applying the rule added in
commit 0d6c771baf (#15349).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-10-06 13:48:28 -07:00
Zixuan James Li 7fd8d77ce0 typing: Import ValuesQuerySet alias from django_stubs_ext.
This saves us from using a conditional import.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-10-05 16:15:56 -07:00
Zixuan James Li a4eaa770f0 typing: Import StrPromise alias from django_stubs_ext.
This saves us from using a conditional import.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-10-05 16:15:56 -07:00
PIG208 df18bbbd48 requirements: Add django-stubs and configure plugin.
Note that django_stubs_ext is required to be placed within common.in
because we need the monkeypatched types in runtime; django-stubs
itself is for type checking only.

In the future, we would like to pin to a release instead of a git
revision, but several patches we've contributed upstream have not
appeared in a release yet.

We also remove the type annotation for RealmAuditLog.event_last_message_id
here instead of earlier because type checking fails otherwise.

Fixes #11560.
2022-10-05 16:15:56 -07:00
Zixuan James Li 4c3c976174 models: Implicitly type model fields with django-stubs.
Previously, we type the model fields with explicit type annotations
manually with the approximate types. This was because the lack of types
for Django.

django-stubs provides more specific types for all these fields that
incompatible with our previous approximate annotations. So now we can
remove the inline type annotations and rely on the types defined in the
stubs. This allows mypy to infer the types of the model fields for us.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-10-05 16:15:56 -07:00
Zixuan James Li b28949b9e7 typing: Use django-stubs' type annotations for QuerySet.
We no longer need to annotate the type of objects returned
from queries since django-stubs plugin infers that already.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-10-05 16:15:56 -07:00
evykassirer 6a90a4b27c emoji: Switch users with blob emoji to use Google style. 2022-10-04 12:29:35 -07:00
evykassirer c321f57785 settings: Explain that Google blob emojis are deprecated.
Part of fixing #19371.
2022-10-04 12:29:35 -07:00
evykassirer 8a9e68e026 emoji: Finish script to generate emoji_names.py with CLDR data.
This script pulls from our previously custom-written emoji strings
and fills in the rest from CLDR. It also removes 4 custom emoji which
collide with some of the new CLDR names (they will now just be called
by their CLDR name).
2022-10-04 12:29:35 -07:00
Anders Kaseorg 7dcffca50e tornado: Construct Django BaseHandler once, not per-request.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-09-30 11:20:45 -07:00
Anders Kaseorg 4c808026b2 tests: Disable lazy loading of moto.
This works around some regression in moto 1.3.15 that I bisected to
b8820009e8
where ‘tools/test-backend test_transfer’ fails when run by itself.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-09-30 11:19:39 -07:00
Zixuan James Li e16de8d9e7 scim: Further slim down SCIMClient removing unused attributes.
This removes everything from SCIMClient except the "is_authenticated`
method. Previously, "realm" and "name" were only needed for logging
purposes. It is the best to keep SCIMClient as minimal as possible, as
it is only intended to be used for authenticating requests to SCIM
views.

This change also gurantees that the "LogRequests" middleware will not
rely on the type unsafe access of the format_requestor_for_logs method
on SCIMClient.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-09-30 15:34:50 +02:00
Zixuan James Li a9273c9aac scim: Remove unused test case attribute.
The scim_client attribute on SCIMTestCase is currently unused since
9198fe4fac.

The creation of the SCIMClient instance was previously needed because
zerver.middleware.validate_scim_bearer_token returns SCIMClient from the db.
The attribute itself on the was never really used in the test case.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-09-30 15:34:50 +02:00
Anders Kaseorg e00f883263 test_runner: Remove empty ParallelTestSuite.__init__ override.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-09-29 18:43:24 -07:00
khanhdq ee2141b0f1 email_notifications: Change missed message for wildcard mention.
Add #stream_name to wildcard mention because it is important
information for interpreting the wildcard mention (larger streams may
mean something very different to you than small ones).

Fixes #22885.
2022-09-29 15:55:14 -07:00