Commit Graph

7045 Commits

Author SHA1 Message Date
Alex Vandiver 608c787c52 email_mirror: Create attachments as the message sender.
When the email mirror gateway is sending messages "as" a user (as
triggered by having access to the missed-message email address),
attachments were still created as the Email Gateway bot.  Since the
sender (the end-user) was not the owner of those attachments (the
gateway bot), nor were they referenced yet anywhere, this resulted in
the attachments being "orphaned" and not allowed to be accessed by
anyone -- despite the attachment links being embedded in the message.
This was accompanied by the error:

```
WARN [] User 12345 tried to share upload 123/3LkSA4OcoG6OpAknS2I0SFAQ/example.jpf in message 123456, but lacks permission
INFO [zerver.lib.email_mirror] Successfully processed email from user 12345 to example-stream
```

We solve this by creating attachment objects as the users the message
will be sent from.
2023-01-18 15:42:40 -08:00
Anders Kaseorg 8f7a7877fe python: Clean up janky URL matching code with urlsplit.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-18 17:25:46 -05:00
Trident Pancake c6ea673cc9 markdown: Update max inline preview from 10 to 24.
The max inline preview limit was previously increased to 10 by #20789.
However, as issue #23624 shows, it's still causing confusion for users
when they include more than 10 links.

Bump this limit up to 24, which is a multiple of the 4 image preview
per line logic.
2023-01-18 14:58:00 -05:00
Lauryn Menard ba443cac03 templates: Set robots noindex for attribution corporate template.
Overrides the default context `allow_search_engine_indexing` to
always be `False` for `templates/corporate/attribution.html` so
that it does not appear in Google / search engine indexes.

Updates test of documentation pages in `test_docs.py` to have an
option for corporate pages to set this value in the template and
verifies that the meta tag for robots noindex, nofollow is
always in the response.
2023-01-17 15:00:39 -08:00
Mateusz Mandera 89d1f1f385 messages: Eliminate redundant realm fetch in has_message_access.
Accessing .realm will cause a fetch query from the database if the
attribute hasn't been fetched already earlier in the codepath. That's
completely redundant if we're just comparing realms, and we should only
access .realm_id attribute. This seems to eliminate a query in some
codepaths, which is nice in this performance-sensitive function.
2023-01-10 15:27:55 -08:00
Alex Vandiver 04cf68b45e uploads: Serve S3 uploads directly from nginx.
When file uploads are stored in S3, this means that Zulip serves as a
302 to S3.  Because browsers do not cache redirects, this means that
no image contents can be cached -- and upon every page load or reload,
every recently-posted image must be re-fetched.  This incurs extra
load on the Zulip server, as well as potentially excessive bandwidth
usage from S3, and on the client's connection.

Switch to fetching the content from S3 in nginx, and serving the
content from nginx.  These have `Cache-control: private, immutable`
headers set on the response, allowing browsers to cache them locally.

Because nginx fetching from S3 can be slow, and requests for uploads
will generally be bunched around when a message containing them are
first posted, we instruct nginx to cache the contents locally.  This
is safe because uploaded file contents are immutable; access control
is still mediated by Django.  The nginx cache key is the URL without
query parameters, as those parameters include a time-limited signed
authentication parameter which lets nginx fetch the non-public file.

This adds a number of nginx-level configuration parameters to control
the caching which nginx performs, including the amount of in-memory
index for he cache, the maximum storage of the cache on disk, and how
long data is retained in the cache.  The currently-chosen figures are
reasonable for small to medium deployments.

The most notable effect of this change is in allowing browsers to
cache uploaded image content; however, while there will be many fewer
requests, it also has an improvement on request latency.  The
following tests were done with a non-AWS client in SFO, a server and
S3 storage in us-east-1, and with 100 requests after 10 requests of
warm-up (to fill the nginx cache).  The mean and standard deviation
are shown.

|                   | Redirect to S3      | Caching proxy, hot  | Caching proxy, cold |
| ----------------- | ------------------- | ------------------- | ------------------- |
| Time in Django    | 263.0 ms ±  28.3 ms | 258.0 ms ±  12.3 ms | 258.0 ms ±  12.3 ms |
| Small file (842b) | 586.1 ms ±  21.1 ms | 266.1 ms ±  67.4 ms | 288.6 ms ±  17.7 ms |
| Large file (660k) | 959.6 ms ± 137.9 ms | 609.5 ms ±  13.0 ms | 648.1 ms ±  43.2 ms |

The hot-cache performance is faster for both large and small files,
since it saves the client the time having to make a second request to
a separate host.  This performance improvement remains at least 100ms
even if the client is on the same coast as the server.

Cold nginx caches are only slightly slower than hot caches, because
VPC access to S3 endpoints is extremely fast (assuming it is in the
same region as the host), and nginx can pool connections to S3 and
reuse them.

However, all of the 648ms taken to serve a cold-cache large file is
occupied in nginx, as opposed to the only 263ms which was spent in
nginx when using redirects to S3.  This means that to overall spend
less time responding to uploaded-file requests in nginx, clients will
need to find files in their local cache, and skip making an
uploaded-file request, at least 60% of the time.  Modeling shows a
reduction in the number of client requests by about 70% - 80%.

The `Content-Disposition` header logic can now also be entirely shared
with the local-file codepath, as can the `url_only` path used by
mobile clients.  While we could provide the direct-to-S3 temporary
signed URL to mobile clients, we choose to provide the
served-from-Zulip signed URL, to better control caching headers on it,
and greater consistency.  In doing so, we adjust the salt used for the
URL; since these URLs are only valid for 60s, the effect of this salt
change is minimal.
2023-01-09 18:23:58 -05:00
Alex Vandiver ed6d62a9e7 avatars: Serve /user_avatars/ through Django, which offloads to nginx.
Moving `/user_avatars/` to being served partially through Django
removes the need for the `no_serve_uploads` nginx reconfiguring when
switching between S3 and local backends.  This is important because a
subsequent commit will move S3 attachments to being served through
nginx, which would make `no_serve_uploads` entirely nonsensical of a
name.

Serve the files through Django, with an offload for the actual image
response to an internal nginx route.  In development, serve the files
directly in Django.

We do _not_ mark the contents as immutable for caching purposes, since
the path for avatar images is hashed only by their user-id and a salt,
and as such are reused when a user's avatar is updated.
2023-01-09 18:23:58 -05:00
Alex Vandiver f0f4aa66e0 uploads: Inline the one callsite of get_local_file_path.
This helps make more explicit the assert_is_local_storage_path which
makes using local_path safe.
2023-01-09 18:23:58 -05:00
Alex Vandiver 7ad06473b6 uploads: Add LOCAL_AVATARS_DIR / LOCAL_FILES_DIR computed settings.
This avoids strewing "avatars" and "files" constants throughout.
2023-01-09 18:23:58 -05:00
Alex Vandiver 24f95a3788 uploads: Move internal upload serving path to under /internal/. 2023-01-09 18:23:58 -05:00
Alex Vandiver cc9b028312 uploads: Set X-Accel-Redirect manually, without using django-sendfile2.
The `django-sendfile2` module unfortunately only supports a single
`SENDFILE` root path -- an invariant which subsequent commits need to
break.  Especially as Zulip only runs with a single webserver, and
thus sendfile backend, the functionality is simple to inline.

It is worth noting that the following headers from the initial Django
response are _preserved_, if present, and sent unmodified to the
client; all other headers are overridden by those supplied by the
internal redirect[^1]:
 - Content-Type
 - Content-Disposition
 - Accept-Ranges
 - Set-Cookie
 - Cache-Control
 - Expires

As such, we explicitly unset the Content-type header to allow nginx to
set it from the static file, but set Content-Disposition and
Cache-Control as we want them to be.

[^1]: https://www.nginx.com/resources/wiki/start/topics/examples/xsendfile/
2023-01-09 18:23:58 -05:00
Alex Vandiver 8e68d68f32 uploads: Be consistent about first arguments to write_local_file.
Enforcing a consistent `type` helps us double-check that we're not
playing fast-and-loose with any file paths for local files.  As noted
in the comment, this is purely for defense-in-depth.

Passing `write_local_file` a consistent `type` requires removing the
"avatars" out of `realm_avatar_and_logo_path` -- which makes it
consistent across upload backends.

This, in turn, requires a compensatory change to zerver.lib.export, to
be explicit that the realm icons are exported from the avatars
directory.  This clarity is likely an improvement.
2023-01-09 18:23:58 -05:00
Alex Vandiver 679fb76acf uploads: Provide our own Content-Disposition header.
sendfile already applied a Content-Disposition header, but the
algorithm may provide both `filename=` and `filename*=` values (which
is potentially confusing to clients) and incorrectly slash-escapes
quotes in Unicode strings.

Django provides a correct implementation, but it is only accessible to
FileResponse objects.  Since the entire point is to offload the
filehandle handling, we cannot use a FileResponse.

Django 4.2 will make the function available outside of FileResponse.
Until then, extract our own Content-Disposition handling, based on
Django's.

We remove the very verbose comment added in d4360e2287, describing
Content-Disposition headers, as it does not add much.
2023-01-09 18:23:58 -05:00
Alex Vandiver 7c0d414aff uploads: Split out S3 and local file backends into separate files.
The uploads file is large, and conceptually the S3 and local-file
backends are separable.
2023-01-09 18:23:58 -05:00
Abhijeet Prasad Bodas 932ce6cee4 MissedMessageHookTest: Add more tests.
Add more tests analogous to existing ones but for different scenarios.

This is mostly boring text, but is important for completeness, since the
notificability logic underneath is subtle.
2023-01-06 17:08:20 -08:00
Abhijeet Prasad Bodas 6288d7ae3e test_event_queue: Split missed message hook tests.
Split the one giant `test_end_to_end_missedmessage_hook` into many
smaller tests.

This allows us to not worry about resetting database state after each
test case and also allows extracting a lot of common stuff into setUp
and tearDown.

There is probably even more scope of deduplication here (for example,
the mock and the `assert_maybe_enqueue_notifications_call_args` call are
same for all test cases) but that might not be worth the added
complexity.

We also change a few
```
user_profile.<setting> = <value>
user_profile.save()
```
expressions to instead use the `do_change_user_setting` function.
2023-01-06 17:08:20 -08:00
Abhijeet Prasad Bodas 04e6b04213 message send: Ignore alert words for presence idle users.
For alert words, we currently don't send email/push notifications --
only desktop notifications. Thus, we don't need to consider alert words
here, since desktop notifications do not utilize the presence status
calculated at this stage.

Tested manually that alert word desktop notifications work as expected.

When we implement email/push notifications for alert words (issues #5137
and #13127), we can add new fields like
`notifications_data.alert_word_email_notify`, similar to the existing
`notifications_data.wildcard_mention_email_notify`, which will allow us
to keep the alert word notifiability check inside the dataclass, similar
to how the mentions checks are done currently. So, even when that
feature is implemented, the code which this commit removes would be
unnecessary.
2023-01-06 17:08:20 -08:00
Abhijeet Prasad Bodas 75cb401f0b test_messages: Rename helper for clarity. 2023-01-06 17:08:20 -08:00
Zixuan James Li 33716f6156 decorator: Do not send HEAD response with non-empty body.
An HTTP HEAD response with a non-empty message body is not compliant
with the standard.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2023-01-06 13:32:47 -08:00
Anders Kaseorg 17300f196c ruff: Fix ISC003 Explicitly concatenated string.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-04 16:25:07 -08:00
Anders Kaseorg 2c5e114f8b ruff: Fix ISC001 Implicitly concatenated string literals on one line.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-04 16:25:07 -08:00
Anders Kaseorg 46cdcd3f33 ruff: Fix PIE790 Unnecessary `pass` statement.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-04 16:25:07 -08:00
Anders Kaseorg 7216ba4813 ruff: Fix DTZ001 `datetime.datetime()` without `tzinfo` argument.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-04 16:25:07 -08:00
Anders Kaseorg e1ed44907b ruff: Fix SIM118 Use `key in dict` instead of `key in dict.keys()`.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-04 16:25:07 -08:00
Josh Klar ebc6ab4744 invites: Validation error instead of DB exception on overflowed SMALLINT.
If `invite_as` is passed as a number outside the range of a PostgreSQL
`SMALLINT` field, the database throws an exception. Move this exception
to the glass as a validation error to allow better client-side error
handling and reduce database round-trips.
2023-01-04 09:44:26 -08:00
Josh Klar ea9b05d88a invites: Use check_int_in to validate invite_as. 2023-01-04 09:44:26 -08:00
Anders Kaseorg bd884c88ed Fix typos caught by typos.
https://github.com/crate-ci/typos

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-03 11:09:50 -08:00
Alex Vandiver 7613928e8a apps: Fix redirect from /apps -> https://zulip.com/apps/.
When this code was moved from being in zerver in 21a2fd482e, it kept
the `if ZILENCER_ENABLED` blocks.  Since ZILENCER and CORPORATE are
generally either both on or both off, the if statement became
mostly-unnecessary.

However, because tests cannot easily remove elements from
INSTALLED_APPS and re-determine URL resolution, we switch to checking
`if CORPORATE_ENABLED` as a guard, and leave these in-place.

The other side effect of this is that with e54ded49c4, most Zulip
deployments started to 404 requests for `/apps` instead of redirecting
them to `https://zulip.com/apps/` since they no longer had any path
configured for `/apps`.  Unfortunately, this URL is in widespread use
in the app (e.g. in links from the Welcome Bot), so we should ensure
that it does successfully redirect.

Add the `/apps` path to `zerver`, but only if not CORPORATE_ENABLED,
so the URLs do not overlap.
2022-12-30 17:47:16 -08:00
Anders Kaseorg edab4ec997 rocketchat: Import timezone-aware datetimes.
The bson library creates naive datetime objects by default.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-12-27 10:34:30 -08:00
Josh Klar c15d066bf5 email-notifs: Use bracketed prefix to indicate a resolved topic.
Some email clients (notably, Gmail Web) support automatically threading
emails together if recipients and subjects match[1]. Manual testing
indicated that prefixing a subject with "[bracketed content]" does not
break this threading behavior, but the added checkmark in a resolved
topic's title does. Before sending an email notification, determine
whether the topic is resolved, and pass this information to the Jinja
template to properly format a threadable email subject.

Fixes: #22538

[1]: https://support.google.com/mail/answer/5900
2022-12-15 23:56:48 -08:00
Josh Klar 5a9b33fa94 tests: Add helper method to mark a message's topic as resolved.
Additionally, migrate existing EditMessageTest to use this helper
method, with the side effect of migrating the tested flow from a
/json/messages URL to a /api/v1/messages URL.
2022-12-15 23:56:48 -08:00
Sahil Batra cbdb4e98e5 message_edit: Topic editing permission should not depend on message sender.
This commit changes the topic edit permssions to not depend whether the user
editing the message had sent the message or it was sent by someone else.
We only do backend changes in this commit and frontend changes will be done
in further commits.

Previously, we always allowed topic edits when the user themseleves had
sent the message not considering the edit_topic_policy and the 3-day time
limit. But now we consider all messages as same and editing is allowed only
according to edit_topic_policy setting and the time limit of 3 days in
addition for users who are not admins or moderators.
2022-12-13 23:11:50 -08:00
Sahil Batra 815bf609fa message_edit: Topic and stream editing do not depend on allow_message_editing.
We change the topic and stream edit permssions to not depend on
allow_message_editing setting in the API and are allowed even
if allow_message_editing is set to False based on other settings
like edit_topic_policy and can_move_message_between_streams.

Fixes a part of #21739.
2022-12-13 23:11:50 -08:00
Sahil Batra ad9a7d2e06 message_edit: Add "Nobody" option for move_messages_between_streams_policy. 2022-12-13 23:11:50 -08:00
Sahil Batra 02eee3a04f message_edit: Add "Nobody" option for edit_topic_policy setting. 2022-12-13 23:11:50 -08:00
Ujjawal Modi 35f05631dd css: Fix spilling out of long email on email change confirmation.
In the email change confirmation page, now long emails break to
multiple lines instead of spilling out.

Fixes #23654.
2022-12-13 16:29:10 -08:00
Zixuan James Li a3a0103d86 markdown: Calculate linkifier precedence in topics.
This uses the linkifier index among the list of linkifiers in the
replacement as the priority to order the replacement order for
patterns in the topic. This avoids having multiple overlapping matches
that each produce a link.

The linkifier with the lowest id will be prioritized when its pattern
overlaps with another. Linkifiers are prioritized over raw URLs.

Note that the same algorithm is used for local echoing and the
backend markdown processor.

Fixes #23715.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-12-13 15:16:20 -08:00
Zixuan James Li 5f4d857d3c linkifier: Order linkifiers by id on query.
This explicitly enforces ordering on the linkifiers. This is useful when
there are overlapping linkifier patterns that matches the same text. In
our current linkifier implementation, this order affects how the
patterns are handled in the markdown processor, with the earlier ones
being prioritized.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-12-13 15:16:20 -08:00
Zixuan James Li 4602c34108 markdown: Correctly retrieve indices for repeated matches.
The same pattern being matched multiple times in a topic cannot be
properly ordered using topic_name.find(match_text) and etc. when there
are multiple matches of the same pattern in the topic.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-12-13 15:16:20 -08:00
Zixuan James Li 5b37306c27 user_groups: Track acting user for check_delete_user_group.
This is a prep-commit for populating RealmAuditLogs for changes made to
UserGroup.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-12-13 14:58:58 -08:00
Zixuan James Li e78dfc2868 user_groups: Track acting user for remove_subgroups_from_user_group.
This is a prep-commit for populating RealmAuditLogs for changes made to
UserGroup.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-12-13 14:58:58 -08:00
Zixuan James Li 32f8e0324d user_groups: Track acting user for add_subgroups_to_user_group.
This is a prep-commit for populating RealmAuditLogs for changes made to
UserGroup.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-12-13 14:58:58 -08:00
Zixuan James Li e4cf3611cf user_groups: Track acting user for remove_members_from_user_group.
This is a prep-commit for populating RealmAuditLogs for changes made to
UserGroup.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-12-13 14:58:58 -08:00
Zixuan James Li 7ae4549b3f user_groups: Track acting user for bulk_add_members_to_user_group.
This is a prep-commit for populating RealmAuditLogs for changes made to
UserGroup.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-12-13 14:58:58 -08:00
Zixuan James Li d5ae88b4a0 user_groups: Track acting user for do_update_user_group_description.
This is a prep-commit for populating RealmAuditLogs for changes made to
UserGroup.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-12-13 14:58:58 -08:00
Zixuan James Li eae0ef4c0e user_groups: Track acting user for do_update_user_group_name.
This is a prep-commit for populating RealmAuditLogs for changes made to
UserGroup.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-12-13 14:58:58 -08:00
Zixuan James Li b3aba796f1 user_groups: Track acting user for user group creation.
This is a prep-commit for populating RealmAuditLogs for changes made to
UserGroup.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-12-13 14:58:58 -08:00
Lauryn Menard 8bad751b82 docs: Remove `settings_html` and `subscriptions_html` from api url context.
Removes the now unused `settings_html` and `subscriptions_html`
links / text when adding a documentation pages' API URL context
information.
2022-11-28 12:21:36 -08:00
Zixuan James Li 1698778145 user_groups: Remove unused remove_user_from_user_group.
remove_user_from_user_group's only caller has been removed in 271333301d.
Its usage has been superseded by remove_members_from_user_group.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-11-21 13:41:06 -08:00
Anders Kaseorg 73c4da7974 ruff: Fix N818 exception name should be named with an Error suffix.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-17 16:52:00 -08:00
Anders Kaseorg 842a5bb54b message_flags: Allow updating flags by narrows and anchors.
Fixes #22893.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-16 21:13:44 -08:00
Anders Kaseorg 5d0711df6d message_flags: Don’t send events for no-op flag updates.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-16 21:13:44 -08:00
Anders Kaseorg ee2cb855f0 message_fetch: Add include_anchor parameter.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-16 21:13:44 -08:00
Anders Kaseorg dae4633745 message_fetch: Extract fetch_messages helper to zerver.lib.narrow.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-16 21:13:44 -08:00
Anders Kaseorg bd5d200722 narrow: Return a dataclass from post_process_limited_query.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-16 21:13:44 -08:00
Anders Kaseorg 0a0a70b33d message_fetch: Move limit_query_to_range to zerver.lib.narrow.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-16 21:13:44 -08:00
Anders Kaseorg 1095efeb52 message_fetch: Move parse_anchor_value to zerver.lib.narrow.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-16 21:13:44 -08:00
Anders Kaseorg 4eea5e102e message_fetch: Move ok_to_include_history to zerver.lib.narrow.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-16 21:13:44 -08:00
Anders Kaseorg 3442bf2843 requirements: Upgrade Ruff.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-16 09:29:11 -08:00
Tim Abbott 8010d06f9e compatiblity: Delete obsolete compatibility code.
Both of these compatibility blocks can be deleted, since you can't
upgrade directly to any supported release from the versions where the
old event formats would be used.
2022-11-15 15:39:38 -08:00
Zixuan James Li 71abbbdd7a message_edit: Handle truncated topic resolution.
This solves the problem that resolving a topic with a long name (>60
characters) will cause the topic name to be truncated, and thus the edit
message code path thinks that the topic is being moved in addition to
being resolved.

We store the pre-truncation topic and use it to check against the
original topic when determining whether a topic is being moved while
getting (un)resovled or not.

Fixes #23482

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-11-15 14:33:39 -08:00
Zixuan James Li c821127131 message_edit: Fix resolution notification not being sent twice.
We intended to send both the "topic was resolved" and the "topic was
moved here" notification when resolving and moving a topic at the same
time in #22312.

The previous implementation did not work as expected and it was only
sending the "topic was moved here" notification.

This removes the check for old_topic and new_topic that have
RESOLVED_TOPIC_PREFIX stripped in maybe_send_resolve_notifications, so
that the notification will be sent regardless if the topic name without
the prefix stays the same or not.

Note that weird topic handling ("✔ ✔✔ some topic") in the comments
was added in e231a03eff is unaffected. In case of confusion, the lstrip
check is not essential to detecting topic being unresolved/resolved.
As we mainly have that handled in the latter part of the helper.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-11-15 14:33:39 -08:00
Mateusz Mandera d201229df8 signup: Implement use of MultiUseInvite.status attribute.
This allows us to revoke MultiUseInvites by changing their .status
instead of deleting them (which has been deleting the helpful tracking
information on PreregistrationUsers about which MultiUseInvite they came
from).
2022-11-14 17:13:16 -08:00
Mateusz Mandera 3bb2096a7f do_reactivate_realm: Noop if realm is already active.
If the realm doesn't actually need re-activation, this should be a noop
rather than creating a confusing RealmAuditLog entry.
2022-11-13 14:28:33 -08:00
Mateusz Mandera 5c20e2af78 test_do_send_realm_reactivation_email: Deactivate the realm first.
This doesn't make sense if the realm is active and will fail as soon as
do_reactivate_realm is fixed in the next commit to be a noop and not
create confused RealmAuditLog entries when the realm is active.
2022-11-13 14:28:33 -08:00
Mateusz Mandera 23a776c144 maybe_send_to_registration: Don't reuse pre-existing PreregistraionUser.
There was the following bug here:
1. Send an email invite to a user.
2. Have the user sign up via social auth without going through that
   invite, meaning either going via a multiuse invite link or just
   straight-up Sign up if the org permissions allow.

That resulted in the PreregistrationUser that got generated in step (1)
having 2 Confirmations tied to it - because maybe_send_to_registration
grabbed the object and created a new confirmation link for it. That is a
corrupted state, Confirmation is supposed to be unique.

One could try to do fancy things with checking whether a
PreregistrationUser already have a Confirmation link, but to avoid races
between ConfirmationEmailWorker and maybe_send_to_registration, this
would require taking locks and so on - which gets needlessly
complicated. It's simpler to not have them compete for the same object.

The point of the PreregistrationUser re-use in
maybe_send_to_registration is that if an admin invites a user, setting
their initial streams and role, it'd be an annoying experience if the
user ends up signing up not via the invite and those initial streams
streams etc. don't get set up. But to handle this, we can just copy the
relevant values from the pre-existing prereg_user, rather than re-using
the object itself.
2022-11-10 17:46:53 -08:00
Alex Vandiver 1f18fa6580
register: Not all clients send a User-Agent header.
In 1fce1c3c73, we added logic to parse
the User-Agent in /register requests; this logic crashed if an HTTP request
was missing that header.

Includes a test for `/register` with no user agent passed; this should catch
similar regressions in the future.

Co-authored-by: Mateusz Mandera <mateusz.mandera@zulip.com>
2022-11-10 16:21:04 -08:00
Lauryn Menard 62c9bb728e tests: Remove ignored `field_type` param in custom profile data test.
For PATCH requests to `/realm/profile_fields`, the field_type is
determined via the field.id, which is in the URL. Therefore,
`field_type` does not need to be passed as part of the request
data in tests.
2022-11-08 11:08:14 -08:00
Sahil Batra 7c4da60b53 message: Don't create historical UserMessage rows when marking as read.
We do not create historical UserMessage rows, for messages that didn't
have one, while marking messages as read and simply ignore those messages.
We do so because there is no user of creating UserMessage rows and it just
wastes storage.

Note that we still allow to mark messages from unsubscribed streams as
read but only those which have UserMessage rows for them to handle the
case when the unread messages were not marked as read while unsubscribing
from the stream due to some race condition. In such cases, messages
will not be included in the unread count shown in "All messages" menu
(and stream is anyways not present in the left sidebar), but the message
border on the left is green if viewing the stream after unsusbcribing it.
So, to avoid the confusion for users, the messages will be marked as read
when user scrolls down.
2022-11-08 08:50:17 -08:00
Lauryn Menard 2a1ceb58d9 api-docs: Return 404 for direct `/api-doc-template` call.
We use `templates/zerver/api/api-doc-template.md` as a base template
for the documented API endpoints in `zerver/openapi/zulip.yaml`.

Previously, if this template was called as an endpoint, then it
would fail an assertion check and send server error. Now we check
for specifically for that potential path and return a 404 error
response for no existing article.

Fixes #21876.
2022-11-08 08:10:15 -08:00
Alya Abbott 9ce484e267 portico: Add a page about our values. 2022-11-07 12:27:09 -08:00
Alya Abbott 1ef94cc2b4 portico: Update /history page. 2022-11-07 12:27:09 -08:00
Lauryn Menard 66e607cf4c tests: Remove ignored `message_id` parameter in message edit tests.
For PATCH requests to edit messages, the message ID is in the URL
and does not need to be passed as part of the data in tests.
2022-11-07 12:16:28 -08:00
Anders Kaseorg 7e8ebd18b6 test_scim: Add test for missing or wrong authentication.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-05 15:13:46 -07:00
Alex Vandiver d8ebbedbbb outgoing_webhook: Ignore the exception on failure if the stream is gone.
In the outgoing webhook handler, there is potentially several seconds
of trying between when a message triggering an outgoing webhook
arrives, and when it fails.  In the meantime, the stream the
triggering message was on may have been deleted, causing the
"Failure!" message to have no valid stream to be sent to.

Rather than raise an exception in the outgoing webhook worker, ignore
the exception and move on.
2022-11-04 14:46:17 -07:00
Alex Vandiver eb7a2f2c38 queue: Do test retries in tests.
The lambda passed to `queue_json_publish` is used if
`settings.USING_RABBITMQ` is unset -- which is only true in tests.  As
such, this pattern causes failures to never actually retry within
tests.

This behaviour has existed ever since the outgoing webhook code was
introduced in 53a8b2ac87, with no explanation.  Not passing that
argument allows tests to verify the retry behaviour when webhooks
fail.
2022-11-04 14:46:17 -07:00
Zixuan James Li 7dbd98d25c rest: Restrict access to json views via basic auth.
Previously, test cases or clients accessing /json/ views using HTTP
Basic Auth would be accepted, while we intended to only allow clients
authenticated with a session cookie to access these views.

This adds a check on the accessed path to avoid this possibility.

It seems unlikely that any API clients clients were taking advantage
of this unintended quirk; so we're not going to bother documenting
this bug fix as an API change. In any case, it should be trivial for
anyone affected to consult the documentation and then switch their
/json/foo URL to a correct /api/v1/foo URL.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-11-04 14:44:07 -07:00
Sahil Batra 1fce1c3c73 register: Handle "Pronouns" type fields for older mobile clients.
Mobile clients older than v27.192 do not support PRONOUNS type
custom profile fields, so we instead change the type of it to
SHORT_TEXT in the data sent with register response and also in
the events sent to those clients.
2022-11-04 14:31:18 -07:00
Anders Kaseorg 3d853caf16 ruff: Fix C417 Unnecessary `map` usage.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-03 12:10:15 -07:00
Anders Kaseorg 9e53f0c756 ruff: Fix C405 Unnecessary `list` literal (rewrite as a `set` literal).
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-03 12:10:15 -07:00
Anders Kaseorg 033d2615f6 ruff: Fix B017 `assertRaises(Exception):` should be considered evil.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-03 12:10:15 -07:00
Anders Kaseorg e1a2894af9 ruff: Fix B011 Do not `assert False` (`python -O` removes these calls).
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-03 12:10:15 -07:00
Sahil Batra aeb9a27d0e message: Ignore messages in unsubcribed streams when marking them unread.
Zulip's unread messages design has an invariant that all unread stream
messages must be in streams the user is subscribed to. For example, We
do not include the unread messages from unsubscribed streams in the
"unread_msgs" data structure in "/register" response and we mark all
unread messages as read when unsubscribing a user from a stream.

Previously, the mark as unread endpoint allowed violating that
invariant, allowing you to mark messages in any stream as unread.

Doing so caused the "message_details" data structures sent with
"update_message_flags" events to not contain messages from
unsubscribed streams, even though those messages were present in the
set of message IDs. These malformed events, in turn, caused exceptions
in the frontend's processing of such an event.

This change is paired with a separate UI change to not offer the "Mark
as unread" feature in such streams; with just this commit, that will
silently fail.

With some additions to the tests by tabbott.
2022-11-02 14:49:23 -07:00
Mateusz Mandera 6e336ef6f6 delete_topic: Use the same timeout pattern as /mark_all_as_read.
We don't want to use the nginx 502 timeout as an API pattern. In
/mark_all_as_read we've already figured out how we want to handle this
instead.
2022-11-02 16:50:06 -04:00
Zixuan James Li da9ad7c3a7 markdown: Pass realm down into render_stream_description.
This guarantees that the Realm is always non-None when we hit the
codepath is_static_or_current_realm_url via
do_change_stream_description, so that we can properly skip rewritting
some images.

Fixes #19405

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-11-01 08:20:47 -07:00
Mateusz Mandera 582d5b0aa3 realm_creation: Rework error pages.
The previous error page was inadequate for serving the two different
scenarios where we show errors in realm_creations, in particular
containing a misleading sentence about realm creation being disabled
(even in the case where it was actually enabled and the user simply had
an expired link).
2022-10-31 17:35:06 -07:00
Lauryn Menard 0c947985bd tests: Update tests to use example profile picture.
So that we can stop using Tim's photo for tests, adds an open
license profile picture to use instead.

Updates tests that used `tim.png` to use the new example profile
picture, which is located in `static/images/test-images/avatars/`.
2022-10-31 14:36:54 -07:00
Lauryn Menard 89713bab76 tests: Use Zulip icon for markdown test cases instead of octopus.
Updates the markdown test case that used `zulip-octopus.png` to
instead use an zulip logo that's also referenced in a frontend
puppeteer test, `static/images/logo/zulip-icon-128x128.png`.
2022-10-31 14:36:54 -07:00
Mateusz Mandera 7b13204e8f retention: Use Message.realm to simplify private message query.
We no longer need to do the inner joins to figure out the message's
realm and split up the cross-realm and regular case - now we just look
at zerver_message.realm directly.
2022-10-31 10:23:57 -07:00
Sahil Batra 2a8b7412ff url_encoding: Use user name instead of email in personal_narrow_url.
This commit updates the urls for personal narrow sent in email
notifications to be of form "{user_id}-{encoded_full_name}" to
make it consistent with the urls that we use for such narrows
in webapp which were recently updated in b4eddad for improving
performance. We encode the full name in the same way that we do in
webapp by replacing the url characters encoded by browser with "-".
2022-10-28 09:47:43 -07:00
Mateusz Mandera a410f6b241 do_mark_all_as_read: Split up the work into batches.
Fixes #15403.
2022-10-27 16:59:54 -07:00
Mateusz Mandera ef468322f1 test_openapi: Fix status code passed to validate_... in test_attributes.
It doesn't seem to make sense to append _{number} to the status code in
that arg, because the resulting string stops looking like a status code
and actually makes this test fail in the follow-up commit with the
confusing error message of

Unknown response http status: 2000

So this just seems like a bug.
2022-10-27 16:59:54 -07:00
Lauryn Menard e66f125ee6 recent-topics: Update hash for view to be "#recent".
Updates the hash used for the recent conversations view to be
"#recent" instead of "#recent_topics".

We will need to keep the logic for handling "#recent_topics"
permanently because users potentially have messages from
Welcome Bot with links to that hash.

Including "recent_topics" as a web_public_allowed_hash in
hash_util.js can be changed once self-hosted servers cannot
upgrade directly to Zulip 5.x from the current version.

Fixes #23132.
2022-10-25 11:03:44 -07:00
Anders Kaseorg d41afb2b25 tools: Add missing spaces in concatenated string literals.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-10-24 14:52:49 -07:00
Lauryn Menard c9044f7acd recent-topics: Rename to "Recent conversations" in web-app and docs.
Replaces instances of "recent topics" in the web-app and documentation
to be "recent conversations".

Renames both `recent-topics.md` files in the help center to be
`recent-conversations.md` and updates/redirects links to new URL.

Does not update instances of "recent topics" in frontend code comments
and does not update the main overview changelog, for now.

Does not change case study text where "recent topics" was referenced
in a quote, but does change generic text references to be "recent
conversations".
2022-10-21 10:28:19 -07:00
Zixuan James Li 46329a2710 test_classes: Create a dedicate helper for query count check.
This adds a helper based on testing patterns of using the "queries_captured"
context manager with "assert_length" to check the number of queries
executed for preventing performance regression.

It explains the rationale of checking the query count through an
"AssertionError" and prints the queries captured as assert_length does,
but with a format optimized for displaying the queries in a more
readable manner.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-10-17 11:32:52 -07:00
Zixuan James Li 9117db8887 test_helpers: Tighten type annotation for queries_captured.
This adds CapturedQueryDict to provide a more accurate type annotation
for the return value of queries_captured. We also replace "Generator"
with "Iterator" because the latter two type parameters were unused.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-10-17 11:32:52 -07:00
Mateusz Mandera 61de767967 login page: Show form-independent errors even if email auth disabled.
These used to only be shown conditional on the
{% if password_auth_enabled %} in the template. Meaning that if you had
an org with email auth disabled and a deactivated user tried to log in,
they wouldn't see the error shown and get confused.

This switches the position of where these error will be shown (above the
login+password form instead of below it), but it looks fine.
2022-10-14 13:07:58 -07:00
Lauryn Menard 98074951ef api-docs: Update examples of queue_id for uuid format. 2022-10-13 10:08:42 -07:00
Alex Vandiver 5d42a0cb00 linkifiers: Support %20 in URLs for topic links.
9381a3bd45 added support for linkifier pattern URLs containing
`%20`-style escapes, but only did so for the codepath which is used in
the message body -- topic links did not understand them.

Expand the support to include when they are substituted into topics.
2022-10-11 14:31:13 -07:00
Julia Bichler 4bb381fc80 message_edit: Support sending notifications with topic changes.
Previously we did not send notification for topic-only edits.
Now, we add backend support for sending notification to topic-only
edits as well.

We would add support for this in webapp in further commits since
message edit UI will be updated as well. We just make sure that no
notifications are sent when editing topic using pencil icon in
message header.

We also change the API default for moving a topic to only notify the
new location, not the old one; this matches the current defaults in
the web UI.

Includes many tests.

We also update the puppeteer tests to test only content edit as
we are going to change the UI to not allow topic editing from
message edit UI. Also fixing the existing tests to pass while
doing topic edits is somewhat complex as notification message
is also sent to new topic by default.

Fixes #21712.

Co-authored-by: Aman Agrawal <amanagr@zulip.com>
Co-authored-by: Tim Abbott <tabbott@zulip.com>
2022-10-11 11:35:41 -07:00
Mateusz Mandera a94b2572be do_deactivate_user: Revoke sessions in transaction.on_commit().
Fixes #21709.
2022-10-11 10:49:07 -07:00
Anders Kaseorg 1385a827c2 python: Clean up getattr, setattr, delattr calls with literal names.
These were useful as a transitional workaround to ignore type errors
that only show up with django-stubs, while avoiding errors about
unused type: ignore comments without django-stubs.  Now that the
django-stubs transition is complete, switch to type: ignore comments
so that mypy will tell us if they become unnecessary.  Many already
have.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-10-10 08:40:28 -07:00
Anders Kaseorg fcd81a8473 python: Replace avoidable uses of __special__ attributes.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-10-10 08:32:29 -07:00
Mateusz Mandera cefed552f6 test_slack_importer: Add assertion about message count.
This will help catch any future regression that might lead the import
tool to fail to import messages into the correct realm.
2022-10-07 10:10:01 -07:00
Mateusz Mandera 00b3546c9f models: Add denormalized .realm column to Message.
This commit adds the OPTIONAL .realm attribute to Message
(and ArchivedMessage), with the server changes for making new Messages
have this set. Old Messages still have to be migrated to backfill this,
before it can be non-nullable.

Appropriate test changes to correctly set .realm for Messages the tests
manually create are included here as well.
2022-10-07 10:09:38 -07:00
Mateusz Mandera a4008d938a test_soft_deactivation: Add docstring explaining send_fake_message. 2022-10-07 10:09:38 -07:00
Anders Kaseorg 133c8b16ed requirements: Upgrade openapi-core.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-10-07 10:03:51 -07:00
Sahil Batra 2bf70fe4db custom_profile_field: Add "Pronouns" custom field type.
This commit adds "Pronouns" custom profile field type. We also
add "Pronouns" type field in the development environment
2022-10-06 17:56:26 -07:00
Anders Kaseorg 92ad4455ed requirements: Upgrade Django to 4.1.
zerver/migrations/0240_usermessage_migrate_bigint_id_into_id.py needs
to be updated to account for Django 4.1 creating AutoField as an
identity column rather than a serial column.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-10-06 15:59:07 -07:00
Anders Kaseorg 47c5deeccd python: Mark dict parameters with defaults as read-only.
Found by semgrep 0.115 more accurately applying the rule added in
commit 0d6c771baf (#15349).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-10-06 13:48:28 -07:00
evykassirer 8a9e68e026 emoji: Finish script to generate emoji_names.py with CLDR data.
This script pulls from our previously custom-written emoji strings
and fills in the rest from CLDR. It also removes 4 custom emoji which
collide with some of the new CLDR names (they will now just be called
by their CLDR name).
2022-10-04 12:29:35 -07:00
Anders Kaseorg 4c808026b2 tests: Disable lazy loading of moto.
This works around some regression in moto 1.3.15 that I bisected to
b8820009e8
where ‘tools/test-backend test_transfer’ fails when run by itself.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-09-30 11:19:39 -07:00
Zixuan James Li e16de8d9e7 scim: Further slim down SCIMClient removing unused attributes.
This removes everything from SCIMClient except the "is_authenticated`
method. Previously, "realm" and "name" were only needed for logging
purposes. It is the best to keep SCIMClient as minimal as possible, as
it is only intended to be used for authenticating requests to SCIM
views.

This change also gurantees that the "LogRequests" middleware will not
rely on the type unsafe access of the format_requestor_for_logs method
on SCIMClient.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-09-30 15:34:50 +02:00
Zixuan James Li a9273c9aac scim: Remove unused test case attribute.
The scim_client attribute on SCIMTestCase is currently unused since
9198fe4fac.

The creation of the SCIMClient instance was previously needed because
zerver.middleware.validate_scim_bearer_token returns SCIMClient from the db.
The attribute itself on the was never really used in the test case.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-09-30 15:34:50 +02:00
khanhdq ee2141b0f1 email_notifications: Change missed message for wildcard mention.
Add #stream_name to wildcard mention because it is important
information for interpreting the wildcard mention (larger streams may
mean something very different to you than small ones).

Fixes #22885.
2022-09-29 15:55:14 -07:00
khanhdq bb8365ab36 email_notifications: Add ``realm_name`` to email notifications.
Add {{ realm_name }} to the "Reply to this email directly ..." line.

This ensures the realm name is always present in the email
notification footer area, in a consistent location.
2022-09-29 15:54:35 -07:00
khanhdq 59d0f9b301 email_notifications: Highlight personal mentions in explanations.
Change the explanations for why notifications were received, with
different text for personal and wildcard mentions.
2022-09-29 15:54:21 -07:00
Sahil Batra 04693b6ac1 message_edit: Send only changed settings in event data and api response.
Previously, we included all three message edit related settings
("allow_message_editing", "message_content_edit_limit_seconds" and
"edit_topic_policy") in the event data and api response irrespective
of which of these settings were changed. Now, we only include changed
settings and separate events are sent for each setting if more than
one of them is changed.

Note that the previous typed in event_schema.py for
`message_content_edit_limit_seconds` incorrectly did not allow `None`
as a value, which is used to encode no limit.
2022-09-28 11:47:40 -07:00
Mateusz Mandera 522c159441 test_push_notifs: Change type_id arg of get_message to no default.
This isn't used anywhere, nor would type_id 100 make much sense.
2022-09-28 16:45:34 +02:00
Mateusz Mandera b35ad99035 test_push_notifications: Use proper user.id for Recipient type_id.
Recipient with type PERSONAL type_id 1 is a Recipient for a system bot,
since those get created first. Even if it doesn't break tests, it's
still bad, because it is not the intention of those tests to simulate a
cross-realm private message to a system bot.
2022-09-28 16:45:34 +02:00
Mateusz Mandera 5850c38f4e test_digest: Use proper stream.id in test_get_hot_topics.
Just using values 1 and 2 as stream ids is not good, because there's no
idea in which realm these streams are (or hypothetically if they exist).
This can create weird Messages with sender being a user of "zulip" realm
and the stream being in another realm - which would be a corrupted
state.
2022-09-28 16:45:25 +02:00
Kartik Srivastava 409ef040bb stream_topic: Refactor user_ids_muting_topic.
This refactors and renames user_ids_muting_topic to accept a parameter
'visibility_policy' and fetch user IDs that have a specific
visibility_policy(provided as the parameter) set for a topic.
2022-09-27 17:18:48 -07:00
Anders Kaseorg 8eec4bf171 message_fetch: Move narrowing query builder to zerver.lib.narrow.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-09-27 17:02:10 -07:00
Aman Agrawal b84722de01 events: Clean up logic for spectator events_register parameters.
Unfortunately, doing so requires forking common API documentation
text, since we're not making any changes to other endpoints that don't
allow unauthenticated requests at all.

Follow-up on #21995.
2022-09-27 16:12:59 -07:00
Mateusz Mandera 318d7fd4cd export: Only export messages that a consenting user can access.
As mentioned in the TODO this commit deletes, the export with member
consent system was failing to account for the fact that if consenting
users only have access to a subset of messages of a stream with
protected history, only that subset should be exported - rather than all
the stream's messages.
2022-09-27 11:56:27 -07:00
Lauryn Menard 569863ffa6 templates: Improve footer content on documentation pages.
Makes the footer content on doc pages more contextually appropriate
for self-hosted organizations, when `settings.CORPORATE_ENABLED` is
false.

When `settings.CORPORATE_ENABLED` is true, there is specific footer
content for the policy documentation pages, and for the help center
and API documentation pages.

Fixes #23068.
2022-09-27 11:13:55 -07:00
Anders Kaseorg 9198fe4fac scim: Downgrade SCIMClient from a model to an ephemeral dataclass.
SCIMClient is a type-unsafe workaround for django-scim2’s conflation
of SCIM users with Django users.  Given that a SCIMClient is not a
UserProfile, it might as well not be a model at all, since it’s only
used to satisfy django-scim2’s request.user.is_authenticated queries.

This doesn’t solve the type safety issue with assigning a SCIMClient
to request.user, nor the performance issue with running the SCIM
middleware on non-SCIM requests.  But it reduces the risk of potential
consequences worse than crashing, since there’s no longer a
request.user.id for Django to confuse with the ID of an actual
UserProfile.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-09-26 11:36:48 -07:00
Lauryn Menard 37aca4ac67 user-status: Stop updating the UserStatus model for `away` updates.
Fourth step in making user status `away` a deprecated way to access
`presence_enabled` for clients supporting older servers, and
checkpoint commit prior to deleting the `status` field from the
UserStatus model.

Part of transitioning from 'unavailable' user status feature to
'invisible mode' user presence feature.
2022-09-23 12:27:54 -07:00
Lauryn Menard e36cfdb0a7 user-status: Send `away=True` if `!presence_enabled` for user status.
We stop sending the `away=True` based on the user's `UserStatus`
object having `status=AWAY`, and instead send that value if
`!presence_enabled` for the user.

Third step in making user status `away` a deprecated way to access
`presence_enabled` for clients supporting older servers.

Part of transitioning from 'unavailable' user status feature to
'invisible mode' user presence feature.
2022-09-23 12:27:54 -07:00
Lauryn Menard 843eb4e4fc user-status: Update `presence_enabled` with changes to user status `away`.
When a user toggles a status update for `away=True|False`, we now update
their `presence_enabled` setting to match (`away!=presence_enabled`).

First step of making user status `away` updates a deprecated way to
access presence_enabled for clients supporting older servers, and
checkpoint commit before migrating users with a current UserStatus
of `status=AWAY` to have their `presence_enabled` set to `False`.

Note that when user status `away` is updated, we now send 4 events:
user_status, user_settings, presence, and update_global_notifications.

Also, this means that these updates change the UserPresence.status
value, which impacts the test for importing and exporting user
information.

Part of transitioning from 'unavailable' user status feature to
'invisible mode' user presence feature.
2022-09-23 12:27:54 -07:00
Lauryn Menard 3428fe86d6 user-status: Move `do_update_user_status` to separate actions file.
We need to move this function to a separate actions file specifically
for `user_status` because otherwise we will have a circular import
between `actions/user_settings.py` and `actions/presence.py` in an
upcoming commit.

Prep commit for migrating "unavailable" user status feature to
"invisible" user presence feature.
2022-09-23 12:27:54 -07:00
Lauryn Menard 47c1dbaa7d user-status: Refactor function names with "user_info" for clarity.
Rename functions that refer to "user_info" without a reference to
"status" to help clarify in the backend between UserPresence
and UserStatus models.

Prep commit for migrating "unavailable" user status feature to
"invisible" user presence feature.
2022-09-23 12:27:54 -07:00
Lauryn Menard 32381f8678 user-presence: Refactor function names with "status" for clarity.
Rename functions that refer to "status" without a reference to
"presence" to help clarify in the backend between UserPresence
and UserStatus models.

Prep commit for migrating "unavailable" user status feature to
"invisible" user presence feature.
2022-09-23 12:27:54 -07:00
yogesh sirsat 543f36b7da custom_profile_fields: Add "display_in_profile_summary" field in model.
To allow `custom_profile_field` to display in user profile popover,
added new boolean field "display_in_profile_summary" in its model class.

In `custom_profile_fields.py`, functions are edited as per conditions,
like currently we can display max 2 `custom_profile_fields` except
`LONG_TEXT` and `USER` type fields.

Default external account custom profile fields made updatable for only
this new field, as previous they were not updatable.

Fixes part of: #21215
2022-09-20 17:03:57 -07:00
Anders Kaseorg 7222f3fe2b tornado: Raise the same error for nonexistent and unauthorized queues.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-09-20 15:20:36 -07:00
Anders Kaseorg f929050230 external_accounts: Make ExternalAccount dataclass to fix typing.
Commit 1a426fa6be (#22977) changed name
to be a StrPromise rather than a str.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-09-20 15:14:33 -07:00
Lauryn Menard 34f5218cc8 templates: Replace default HTML title and meta-description elements.
Removes the default title element of "Zulip" from `base.html` and
the default meta-description sentence from `meta_tags.html`. Also
removes default open graph metadata.

For portico templates that would would fail tests, set both
`PAGE_TITLE` and `PAGE_DESCRIPTION` variables with appropriate
content.

Co-authored-by: Alya Abbott <alya@zulip.com>
2022-09-19 20:17:17 -07:00
Anders Kaseorg 92f9a9ba41 forms: Fix another 500 error on realm creation with invalid email.
Followup to commit c0287473b9 (#22723).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-09-19 14:12:32 -07:00
rht a7cff0f091 Slack import: Translate to emoji name to codepoint using iamcal data.
Because Slack emoji naming is different from Zulip's.
According to https://emojipedia.org/slack/, Slack's emoji shortcodes are
derived from https://github.com/iamcal/emoji-data.
There are probably some deviations from that dataset, but this PR should
at least catch the ones that are identical to iamcal's.
2022-09-17 12:04:07 -07:00
Jason Hildebrand 7efd59b6d7 email_mirror: Remove limits (expiry, max uses) to improve usability.
These limits don't appear to provide useful security benefits, and
they do impact usability because they prevented email-based users from
replying more than once, or from replying to message more than 5 days
old.

Fixes #2755.
Fixes #19994.
2022-09-16 18:07:28 -07:00
yogesh sirsat 180a9cbdcb stream_bots: Allow bot owners to unsubscribe their bots from streams.
Users who owns bots can unsubscribe their bots from streams.

Fixes part of: #21402
2022-09-16 17:51:34 -07:00
Lauryn Menard eb377a8872 read_receipts: Exclude muted users from read receipts.
Removes IDs of users who have muted or been muted by the current
user from the list of user IDs returned by the read receipts
endpoint.

Fixes #22909.
2022-09-16 16:19:54 -07:00
Sahil Batra b9e0575829 realm: Allow only owners to change waiting_period_threshold setting.
We allow only owners to change the waiting period setting to become
full member. This commit contains only backend changes, frontend
changes will be done separately.
2022-09-16 15:27:52 -07:00
Sahil Batra f23bfbd462 realm_domains: Allow only owners to add, edit or delete domains.
We allow only owners to add, edit or delete the allowed domains.
This commit only contains backend changes, frontend changes will
be done in a separate commit.
2022-09-16 15:27:52 -07:00
Sahil Batra 1396fbab6e realm: Allow only owners to change restricted-login related settings.
We allow only owners to change disallow_disposable_email_addresses
and emails_restricted_to_domains settings. This commit only contains
change in backend part, frontend changes will be done separately.
2022-09-16 15:27:52 -07:00
Sahil Batra 63fa2a9338 realm: Allow only owners to change invite_required in backend.
We allow only owners to change the invite_required setting.
This commit only adds the restriction in backend, frontend
changes will be done separately.

We also add a helper function in test_realm.py to avoid
writing same code repeatedly and this helper will also
be used in tests for other settings to be added in
further commits.
2022-09-16 15:04:54 -07:00
Tim Abbott 76bcb96414 realm_export: Improve estimate of data export size.
As suggested by the new comments, the cost for a Zulip data export
scales with messages actually included in the export, so an
organizations with 1M private messages but only 50K public stream
messages should not be modeled the same as one with 1M public stream
messages for the purpose of the limits here.

Also improve the comments and variable names more generally.
2022-09-16 14:54:45 -07:00
Sahil Batra c3fe8420fd streams: Allow unsubscribing others based on can_remove_subscribers_group.
Current value of can_remove_subscribers_group field is admins system group
only so behavior is not changed. We would provide support to change this
setting from API and UI in further commits.
2022-09-14 16:14:15 -07:00
Sahil Batra b9248c75f4 stream: Add do_change_can_remove_subscribers_group and field to objects.
This commit adds do_change_can_remove_subscriber_group function for
changing can_remove_subscribers_group field of a stream. We also add
can_remove_subscribers_group_id field to stream and subscription
objects.

This function will be helpful for writing tests in next commit.
We would add API and UI support to change this setting in further
commits.
2022-09-14 16:03:11 -07:00
Sahil Batra b0de5c0f36 streams: Set can_remove_subscribers_group while creating streams.
This commit sets can_remove_subscribers_group to admins system
group while creating streams as it will be the default value
of this setting. In further we would provide an option to set
value of this setting to any user group while creating streams
using API or UI.
2022-09-14 16:03:11 -07:00
Zixuan James Li e2bf8c49af test_decorators: Remove cachify test cases.
cachify has been removed in 9d448e73d2.
We don't need to keep its tests.

TODO: functools.lru_cache can be replaced by functools.cache when we
drop Python 3.8.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-09-14 09:40:46 -07:00
Lauryn Menard f0dac7beef docs: Update base titles for documentation pages.
Updates the base titles so that they begin with the page content,
and end with "| Zulip" +  information about the type of doc: "help
center", "API documentation", "terms and policies", or "integrations".
2022-09-13 11:36:52 -07:00
Sahil Batra 1e55e7641e export: Do not export direct_members and direct_subgroups field.
We do not need direct_members and direct_subgroups field of
UserGroup objects in the export data since we already have
UserGroupMembership and GroupGroupMembership object data.

While importing we keep these fields empty when creating
UserGroup objects and direct_members and direct_subgroups
fields will get set when UserGroupMembership and
GroupGroupMembership objects are created.

This change will also help us in further changes when we
will change the order of importing to import UserGroup
objects just after Realm objects.
2022-09-13 11:07:09 -07:00
Sahil Batra c714ad993b test_import_export: Add tests for GroupGroupMembership objects. 2022-09-13 11:07:09 -07:00
Anders Kaseorg bd9a1dc971 tests: Consistently JSON-encode ‘to’ parameter
Although our POST /messages handler accepts the ‘to’ parameter with or
without JSON encoding, there are two problems with passing it as an
unencoded string.

Firstly, you’d fail to send a message to a stream named ‘true’ or
‘false’ or ‘null’ or ‘2022’, as the JSON interpretation is prioritized
over the plain string interpretation.

Secondly, and more importantly for our tests, it violates our OpenAPI
schema, which requires the parameter to be JSON-encoded.  This is
because OpenAPI has no concept of a parameter that’s “optionally
JSON-encoded”, nor should it: such a parameter cannot be unambiguously
decoded for the reason above.

Our version of openapi-core doesn’t currently detect this schema
violation, but after the next upgrade it will.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-09-13 11:05:37 -07:00
Yogesh Sirsat 50c13f9db0 custom_profile_fields: Backend tests for default external account type. 2022-09-12 14:29:11 -07:00
Yogesh Sirsat c355934179 custom_profile_fields: Change data of default external account type.
change the names of "github" and "twitter" external account fields to
"GitHub username" and "Twitter username" respectively and remove the
hints of them.
2022-09-12 14:29:11 -07:00
Florian Pritz a276603766 rocketchat: Deduplicate and ignore huddle rooms with same users.
If there are more than 1 room with the same set of users, the import
will fail due to a unique constraint on the huddle_hash. Figuring out
why and which room is causing this database error is kinda difficult.

We deduplicate those cases here and simply merge the rooms together.
Note however, that the deduplication does not work as expected so we
simply ignore them all together for now and only raise an exception
along some logging output. At least this way, it is pretty clear what is
wrong and you do not have to wait to get a database error during the
actual import.

We also ignore empty huddle rooms since those are the duplicates that
caused problems for me and if they are empty, ignoring them is easier
than trying to get the merge to work.

Not sure where those channels come from since we discovered this with
production data.

Signed-off-by: Florian Pritz <bluewind@xinu.at>
2022-09-09 16:57:24 -07:00
Florian Pritz 26fe028534 rocketchat: Truncate long stream names.
These will lead to an error during import otherwise.

Signed-off-by: Florian Pritz <bluewind@xinu.at>
2022-09-09 16:57:24 -07:00
Raghav Luthra 4dad9fa158 user_settings: Add user setting to control the user list style.
Added a user_list_style personal user setting to the bottom of
Settings > Display settings > Theme section which controls the look
of the right sidebar user list.

The radio button UI includes a preview of what the styles look like.

The setting is intended to eventually have 3 possible values: COMPACT,
WITH_STATUS and WITH_AVATAR; the final value is not yet implemented.

Co-authored-by: Tim Abbott <tabbott@zulip.com>
2022-09-09 16:30:54 -07:00
Anders Kaseorg 7d485aa58f docs: Correct more “webapp” to “web app”.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-09-08 20:23:28 -07:00
Anders Kaseorg 4c38b1371b test_auth_backends: Fix SOCIAL_AUTH_GITHUB_TEAM_ID mock.
SOCIAL_AUTH_GITHUB_TEAM_ID is expected to be the string of an integer.

The requests mock for the bogus /None URL is unused because the
function that would request it is itself mocked.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-09-08 20:23:28 -07:00
yogesh sirsat e4b74c2099 streams: Fix misleading comment in "test_subs". 2022-09-08 16:56:07 -07:00
Christopher Chong 28173cafc8 message_flags: Fix deadlocks when updating message flags.
Previously, an active production Zulip server would experience a class
of deadlocks caused by two or more concurrent bulk update operations
on the UserMessage table.

This is because UPDATE ... SET ... WHERE statements that execute in
parallel take row-level UPDATE locks as they get results; since the
query plans may result in getting rows in different orders between two
queries, this can result in deadlocks.

Some databases allow ORDER BY on their UPDATE ... WHERE statements;
PostgreSQL does not. In PostgreSQL, the answer is to do a sub-select
with an ORDER BY ... FOR UPDATE to ensure consistent ordering on row
locks.

We do this all code paths using bitand or bitor as part of bulk
editing message flags, which should ensure that these concurrent
operations obtain row level locks on the table in the same order.

Fixes #19054.
2022-09-06 16:06:58 -07:00
Mateusz Mandera 01d659b672 internal_prep_private_message: Remove redundant realm arg.
This is redundant and is taken based on recipient_user anyway.
2022-09-06 15:03:17 -07:00
Lauryn Menard 18b401f476 tests: Update DocPageTest for HTML metatags for landing pages.
Updates `_test` in DocPageTest so that the generic test boolean
parameters will confirm that there is an HTML title element as well
as a meta-description for SEO and meta tags for open graph data.

Sets tests for error pages and dev env pages to `landing_page=False`
since these pages will not have the metadata added in subsequent
commits.

API and integration docs are automated to have this metadata. There
is a specific test for the integrations open graph data. The list of
API endpoints with specific content are tested for this now.

The specific test for portico pages open graph data is removed in
favor of the more generic test style, which will not fail if/when
the template data and text is changed or updated.
2022-09-06 14:57:06 -07:00
Lauryn Menard cde096ffa5 tests: Split `test_doc_endpoints` into more specific endpoint tests.
In `zerver/tests/test_docs.py`, we split the catch all test for doc
endpoints into more specific test groups: dev environment pages,
error pages, corporate pages. Also, moves the api endpoints being
tested to the specific test for api endpoints.
2022-09-06 14:57:06 -07:00
Lauryn Menard 13852c54e8 tests: Expand test for new communities directory page.
Expands specific test for new open communities directory page to
test that the zulip dev realm was added to the page. Adds a generic
test for the endpoint to the generic `test_doc_endpoint`
test.
2022-09-06 14:57:06 -07:00
Lauryn Menard 17974cafbc tests: Move test for non-existent integration to integration doc test.
Moves test for a non-existent integration to the specific test for
integration doc endpoints.
2022-09-06 14:57:06 -07:00
Lauryn Menard 9ad6457f32 tests: Move URL redirect tests for portico landing pages.
Moves three URL redirect tests in `zerver/tests/test_docs.py` to
the unit test for URL redirects in `zerver/tests/test_urls.py`.
2022-09-06 14:57:06 -07:00
Tim Abbott 3826fc8af9 test_events: Fix test failures 10 days after last provision.
In bbf4c25553, we added support for
triggering user group changes when the waiting_period_threshold realm
setting was changed.

The test_events test did not expect this, and thus would fail if the
last provision was between 10 and 20 days ago.

The simplest fix is to just increase those numbers, since computing
whether the database was too old would be more complex than it is
worth for this test.
2022-09-06 11:31:13 -07:00
madrix01 4303ba1efc actions: Create a separate message_delete.py file.
This is preparatory commit for #18941.
Importing `do_delete_message` from `message_edit.py` was causing a
circular import error. In order to avoid that, we create a separate
message_delete.py file which has all the functions related to deleting
messages.
The tests for deleting messages are present in
`zerver/tests/test_message_edit.py`.

Fixes a part of #18941
2022-09-01 14:18:38 -07:00
Alya Abbott 87a295677e portico: Fix title and description for /attribution. 2022-09-01 11:33:35 -07:00
Lauryn Menard 5a7aa8228a urls: Extend documentation URL redirects system to corporate landing pages.
Extends the URL redirect system used for documentation pages to corporate
landing pages. This makes it easier and consistent for contributors who
work on both areas to create new URL redirects when needed.
2022-08-31 15:01:29 -07:00
Lauryn Menard 878b46e758 urls: Simplify process to add URL redirects for documentation pages.
Creates `zerver.lib.url_redirects.py` to record old and new URLs
for documentation pages that have been renamed/moved and need URL
redirects.

This file is then used by `zproject.urls.py` to redirect links and
by `zerver.test.test_urls.py` to test that all of the old URLs
return a success response with a common page header/text depending
on the type of redirect (help center, policy, or API).

Adds a section to contributor docs on writing documentation for
how to use this redirect system when renaming a help center or api
documentation page.

Fixes #21946. Fixes #17897.
2022-08-31 15:01:29 -07:00
Mateusz Mandera d21a1fe47f middleware: Log 5xx json_errors in JsonErrorHandler.
django.request logs responses with 5xx response codes (our configuration
of the logger prevents it from logging 4xx as well which it normally
does too). However, it does it without the traceback which results in
quite unhelpful log message that look like
"Bad Gateway:/api/v1/users/me/apns_device_token" - particularly
confusing when sent via email to server admins.

The solution here is to do the logging ourselves, using Django's
log_response() (which is meant for this purpose), and including the
traceback. Django tracks (via response._has_been_logged attribute) that
the response has already been logged, and knows to not duplicate that
action. See log_response() in django's codebase for these details.

Fixes #19596.
2022-08-31 14:43:15 -07:00
Mateusz Mandera 10a1596d96 send_analytics_to_remote_server: Log connection errors with traceback.
It seems helpful for this to get logged with the traceback rather than
just the general
"<exception name>  while trying to connect to push notification bouncer."
2022-08-31 14:43:15 -07:00
Anders Kaseorg 019c9527c1 test_message_flags: Fix “accessibility” spelling.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-08-30 17:36:21 -07:00
Aman Agrawal 379c6acbea portico: Add page to list open organizations.
Fixes #22020
2022-08-30 16:02:06 -07:00
Mateusz Mandera 5bcf78e0cb import: Fix timestamp check in long_term_idle_helper.
This is supposed to be 60 days, but timestamps are in seconds.
2022-08-29 15:18:00 -07:00
Mateusz Mandera d350406991 gitter: Make imported Realm start with only GitHub auth enabled.
Users will only be able to login via GitHub, because imported users
get GitHub's generated noreply email addresses - so this should be the
only auth method enabled at first, to avoid confusion.
2022-08-29 11:10:18 -07:00
Mateusz Mandera d5b450c1e0 import: Fix the import of authentication_methods BitField.
The ordering of bits that fix_realm_authentication_bitfield was making
was incorrect, it needs to be reversed.
2022-08-29 11:10:18 -07:00
Mateusz Mandera 4c7a9816ff gitter: Soft deactivate appropriate imported users.
We want to use the long_term_idle_helper logic for gitter imports just
like we do for slack.
2022-08-29 11:03:27 -07:00
Mateusz Mandera f55ec8a026 tests: Replace some testserver.com dummy domains with zulip.example.com. 2022-08-29 10:39:14 -07:00
Mateusz Mandera 3ff0b64281 test_import_export: Use get_output_dir() helper.
That's much briefer and cleaner than repeating the os.path.join
expression and the helper already exists.
2022-08-29 10:39:14 -07:00
Mateusz Mandera ac9e927989 import_realm: Validate emails of users being imported.
Having users with malformed email addresses in our database can lead to
hard to debug issues and we really should try to prevent that.
2022-08-29 10:39:14 -07:00
Mateusz Mandera 43107fcdc3 scim: Upgrade scim2-filter-parser to 0.4.0 to fix case-sensitivity.
userName lookups are supposed to be case-insensitive per the SCIM2 RFC
and this was fixed upstream in
https://github.com/15five/scim2-filter-parser/pull/31
and included in 0.4.0 release.
2022-08-29 10:35:29 -07:00
evykassirer efee77b41f emoji: Add which emoji are supported to the /register call.
Fixes #21037.

This is part of fixing #19371. To bulk-add new emoji regularly,
mobile needs to know which servers support which emoji.
`staticfiles_storage.url` generates a unique URL with a hash
based on the file content, which lets mobile know if it needs
to update its locally stored data.
2022-08-26 17:58:31 -07:00
sahil839 bd1a27491e tests: Rename missed message email tests in test_email_notifications.py.
This commit changes the name of missed message email tests for
personal and huddle messages to be more clear:

- from *_personal_missed_stream_messages to *_missed_personal_messages

- from *_huddle_missed_stream_messages to *_missed_huddle_messages
2022-08-26 11:48:29 -07:00
sahil839 decc8d41fa emails: Add quote prefix ">" in plain text missed message emails.
We add quote prefix ">" to each line of the message in the plain text
missed message emails, which are then rendered as quotes by email
clients. We also move the message content in the next line after sender.

This helps us in clearly showing the message authors in missed message
emails especially in emails with multiple messages and senders.

Fixes #15836.
2022-08-26 11:48:29 -07:00
Adam Sah c6130ce4b1 email_notifications: Complete test coverage for module.
Following the last commit, there was this WELCOME_EMAIL_SENDER code
without an existing automated test.
2022-08-25 13:04:51 -07:00
Adam Sah 637867dad1 settings: Send presence update event toggling presence_enabled.
Fixes #21180.
2022-08-25 12:30:12 -07:00
Zixuan James Li 059d0e7be8 settings: Make SHARED_SECRET mandatory.
This implements get_mandatory_secret that ensures SHARED_SECRET is
set when we hit zerver.decorator.authenticate_notify. To avoid getting
ZulipSettingsError when setting up the secrets, we set an environment
variable DISABLE_MANDATORY_SECRET_CHECK to skip the check and default
its value to an empty string.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-08-25 12:13:03 -07:00
Anders Kaseorg 4a61e36def CVE-2022-36048: Rewrite only specific local links to relative.
Due to mismatches between the URL parsers in Python and browsers, it
was possible to hoodwink rewrite_local_links_to_relative into
generating links that browsers would interpret as absolute.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-08-24 16:29:09 -07:00
Mateusz Mandera c1c9024af5 email_mirror: Replace disallowed characters in incoming email subject.
These characters are not allowed and trying to create a Zulip message
with those characters throws a JsonableError in check_stream_topic.

We don't want to reject emails with those chars in the subject, so
it's best to just modify it appropriately.
2022-08-22 17:16:20 -07:00
Aman Agrawal ef21f9107c urls: Move /hello files to the corporate folder. 2022-08-22 15:53:43 -07:00
Aman Agrawal 21a2fd482e portico: Move to corporate folder.
Since the URLs that use these function are present in corporate/urls,
this file belongs in the corporate folder.
2022-08-22 15:53:43 -07:00
Zixuan James Li db92033b23 typing: Tighten type annotations for patch_queue_publish.
Since this decorator is only used for methods of
TestServiceBotEventTriggers, we can type the decorated method's
signature accurately without using ParamSpec.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-08-22 15:46:16 -07:00
Zixuan James Li 32a77fc0bf test_helpers: Move patch_queue_publish to test_service_bot_system.
This prepares us to tighten the type annotations for this helper.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-08-22 15:46:16 -07:00
Zixuan James Li 30536caa68 typing: Remove FuncT.
We can express the type of these decorators with Concatenate and ParamSpec
now for tighter type annotations.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-08-22 15:46:16 -07:00
Zixuan James Li 21fd62427d typing: Remove ViewFuncT.
This removes ViewFuncT and all the associated type casts with ParamSpec
and Concatenate. This provides more accurate type annotation for
decorators at the cost of making the concatenated parameters
positional-only. This change does not intend to introduce any other
behavioral difference. Note that we retype args in process_view as
List[object] because the view functions can not only be called with
arguments of type str.

Note that the first argument of rest_dispatch needs to be made
positional-only because of the presence of **kwargs.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-08-22 15:46:16 -07:00
Zixuan James Li 965ed728e1 type_debug: Remove type_debug.
This module was originally introduced in 2016 to assist adding mypy
annotations to the project. Back then static type checking was not that
established throughout the codebase, so it was helpful to be able to
print out the types for type checking purposes.

This workflow is no longer helpful for improving type annotations right
now, and it has been unused for a while.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-08-22 13:19:09 -07:00
Mateusz Mandera 2c693f3bd9 billing: Fix licenses amount check during user signup/invitation.
Our seat count calculation is different for guest user than normal users
(a number of initial guests are free, and additional marginal guests are
worth 1/5 of a seat) - so these checks we apply when a user is being
invited or signing up need to know whether it's a guest or non-guest
being added.
2022-08-18 11:56:54 -07:00
Zixuan James Li de22b72670 rate_limit: Remove rate_limit_ip.
Because rate_limit_request_by_ip is the only caller of it, it is safe
for us to inline RateLimitedIpAddr and remove this helper. This ensures
that we have consistent internals for rate limiting functions, which all
have a should_rate_limit check.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-08-17 12:05:38 -07:00
Zixuan James Li db3fd6947f test_decorators: Remove errors_disallowed helper.
The said hack was removed in c00aab8ede,
and rate_limit itself has also been removed.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-08-17 12:05:38 -07:00
Zixuan James Li 26a518267a rate_limit: Replace rate_limit with inlined rate limit checks.
This change incorporate should_rate_limit into rate_limit_user and
rate_limit_request_by_ip. Note a slight behavior change to other callers
to rate_limit_request_by_ip is made as we now check if the client is
eligible to be exempted from rate limiting now, which was previously
only done as a part of zerver.lib.rate_limiter.rate_limit.

Now we mock zerver.lib.rate_limiter.RateLimitedUser instead of
zerver.decorator.rate_limit_user in
zerver.tests.test_decorators.RateLimitTestCase, because rate_limit_user
will always be called but rate limit only happens the should_rate_limit
check passes;

we can continue to mock zerver.lib.rate_limiter.rate_limit_ip, because the
decorated view functions call rate_limit_request_by_ip that calls
rate_limit_ip when the should_rate_limit check passes.

We need to mock zerver.decorator.rate_limit_user for SkipRateLimitingTest
now because rate_limit has been removed. We don't need to mock
RateLimitedUser in this case because we are only verifying that
the skip_rate_limiting flag works.

To ensure coverage in add_logging_data, a new test case is added to use
a web_public_view (which decorates the view function with
add_logging_data) with a new flag to check_rate_limit_public_or_user_views.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-08-17 12:05:38 -07:00
Zixuan James Li 2aac1dc40a rate_limit: Move rate_limit_remote_server to zilencer.auth.
This allows us to avoid importing from zilencer conditionally in
zerver.lib.rate_limiter, as we make rate limiting self-contained now.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-08-17 12:05:38 -07:00
Zixuan James Li f158c86ae1 test_decorators: Rework RateLimitTestCase.
- RateLimitTestCase.get_ratelimited_view is replaced by a view
function directly decorated by public_json_view.

- the META dict is initialized with "PATH_INFO": "test" because now the
tests cover the process_client codepath;

- HostRequestMock is initialized with host="zulip.testserver" to pass
the validate_account_and_subdomain check;

- check_rate_limit_public_or_user_views replaces both
test_rate_limiting_happens_in_normal_case and
test_rate_limiting_happens_by_ip_if_unauthed.

Overall, we deduplicate the test cases in this change, and make sure
that they also cover the view function decorators for authentication.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-08-17 12:05:38 -07:00
Zixuan James Li 0916f9eca2 test_decorators: Extract check_rate_limit_public_or_user_views.
The test setup for some of the test cases are largely similar, so it
would be cleaner to be able to reuse them.

Note that we use "check" in the name of this helper because later we
will extend it to take a flag to set whether rate limiting is expected.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-08-17 12:05:38 -07:00
Zixuan James Li 29bad25f83 zilencer: Only apply rate limit to remote server.
This refactors the test case alongside, since normal views accessed by
remote server do not get rate limited by remote server anymore.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-08-17 12:05:38 -07:00
Lauryn Menard 885c3d6593 subscriptions: Send update events for `is_muted` property.
In Zulip 2.1.0, the `is_muted` stream subscription property was
added and replaced the `in_home_view` property. But the server has
still only been sending subscription update events with the
`in_home_view` property.

Updates `do_change_subscription_property` to send a subscription
update event for both `is_muted` and `in_home_view`, so that
clients can fully migrate away from using `in_home_view` allowing
us to eventually remove it completely.
2022-08-16 16:50:51 -07:00
Kartik Srivastava 342fd72a10 api: Document /users/me/alert_words API endpoint. 2022-08-16 13:37:52 -07:00
Mateusz Mandera d48075de58 logout_all_users: Add --rotate-api-keys option to the command.
Fixes #19397.
2022-08-15 17:58:05 -07:00
Anders Kaseorg c0287473b9 forms: Fix 500 error on realm creation with invalid email.
Commit b945aa3443 (#22604) incorrectly
assumed that Django would run the extra EmailField validators if basic
email address validation passed.  Actually, it runs all validators
unconditionally and collects all failures.  So email_is_not_disposable
needs to catch email address parsing errors.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-08-15 17:19:37 -07:00
Adam Sah 800de1e682 search: Add test for pgroogna parts-of-URL search.
This test verifies that the PGroonga backend fixes the issues in
backend by default.
2022-08-15 17:14:17 -07:00
Zixuan James Li 5c49e4ba06 rest: Extract remote_server_path from rest_path.
This allows us to separate the zilencer paths from other JSON paths,
with explicit type annotation expecting `RemoteZulipServer` as the
second parameter of the handler using
authenticated_remote_server_view.

The test case is also updated to remove a test for a situation that no
longer occurs anymore, since we don't perform subdomain checks on
remote servers.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-08-13 14:53:52 -07:00
Zixuan James Li af88417847 decorator: Extract validate_remote_server.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-08-13 14:33:59 -07:00
Sahil Batra a1f40ccda5 message_edit: Make zero invalid value for message_content_edit_time_limit_seconds.
This commit changes the code to consider zero as an invalid value for
message_content_edit_time_limit_seconds. Now to represent the setting that
user can edit the message anytime, the setting value will be "None" in
database and "unlimited" will be passed to API from clients.
2022-08-12 18:09:53 -07:00
Zixuan James Li 8fae87e2df zephyr: Check PERSONAL_ZMIRROR_SERVER before updating cache.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-08-12 17:08:04 -07:00
Zixuan James Li a0ccc7b872 test_signup: Replace MagicMock with HostRequestMock.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-08-12 17:08:04 -07:00
Zixuan James Li f54ecad6cd decorator: Extract public_json_view.
This refactoring is necessary to separate the expected type annotation
for view functions with different authentication methods. Currently the
signature aren't actually check against view functions because
`rest_path` does not support type checking parameter types, but it will
become useful once we do.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-08-12 16:51:55 -07:00
Zixuan James Li c9f54766c3 rate_limiter: Extract rate limit related functions.
This refactors rate limit related functions from `zerver.decorator` to
zerver.lib.rate_limiter.

We conditionally import `RemoteZulipServer`, `RequestNotes`, and
`RateLimitedRemoteZulipServer` to avoid circular dependency.

Most instances of importing these functions from `zerver.decorator` got
updated, with a few exceptions in `zerver.tests.test_decorators`, where
we do want to mock the rate limiting functions imported in
`zerver.decorator`. The same goes with the mocking example in the
"testing-with-django" documentation.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-08-12 16:51:55 -07:00
Zixuan James Li 232ba4866a rate_limit: Stop wrapping rate limited functions.
This refactors `rate_limit` so that we no longer use it as a decorator.
This is a workaround to https://github.com/python/mypy/issues/12909 as
`rate_limit` previous expects different parameters than its callers.

Our approach to test logging handlers also needs to be updated because
the view function is not decorated by `rate_limit`.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-08-12 16:51:55 -07:00
Dinesh 48d2783559 read_receipts: Add support for displaying read receipts.
Adds an API endpoint for accessing read receipts for other users, as
well as a modal UI for displaying that information.

Enables the previously merged privacy settings UI for managing whether
a user makes read receipts data available to other users.

Documentation is pending, and we'll likely want to link to the
documentation with help_settings_link once it is complete.

Fixes #3618.

Co-authored-by: Tim Abbott <tabbott@zulip.com>
2022-08-12 13:16:35 -07:00
Sahil Batra 13e0311ef3 realm: Add support to change enable_read_receipts setting.
This commit adds support to change enable_read_receipts
setting through API and also adds the field to response
of "/register" endpoint so that the setting value
is available to clients.
2022-08-12 17:10:03 +05:30
Zixuan James Li 1d7abb3f5d settings: Extract unnecessary test specific settings.
APPLE_ID_TOKEN_GENERATOR_KEY and EXAMPLE_JWK are exclusively used in
test_auth_backends.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-08-11 13:45:28 -07:00
Sahil Batra 544d58a693 user_groups: Add EVERYONE_GROUP_NAME constant.
We now use EVERYONE_GROUP_NAME instead of writing
the actual group name at multiple places, so that we
can have all the group names coded at one place only.
2022-08-11 04:38:36 -07:00
Sahil Batra 8eed801380 user_groups: Add MEMBERS_GROUP_NAME constant.
We now use MEMBERS_GROUP_NAME instead of writing
the actual group name at multiple places, so that we
can have all the group names coded at one place only.
2022-08-11 04:38:36 -07:00
Sahil Batra 9a94d2b762 user_groups: Add MODERATORS_GROUP_NAME constant.
We now use MODERATORS_GROUP_NAME instead of writing
the actual group name at multiple places, so that we
can have all the group names coded at one place only.
2022-08-11 04:38:36 -07:00
Sahil Batra 2f634e6640 user_groups: Add ADMINISTRATORS_GROUP_NAME constant.
We now use ADMINISTRATORS_GROUP_NAME instead of writing
the actual group name at multiple places, so that we can
have all the group names coded at one place only.
2022-08-11 04:38:36 -07:00
Sahil Batra 86b496a981 user_groups: Add OWNERS_GROUP_NAME constant.
We now use OWNERS_GROUP_NAME instead of writing
the actual group name at multiple places, so that
we can have all the group names coded at one place
only.
2022-08-11 04:38:36 -07:00
Sahil Batra 15ef0daaf8 user_groups: Add EVERYONE_ON_INTERNET_GROUP_NAME constant.
We now use EVERYONE_ON_INTERNET_GROUP_NAME instead of
writing the actual group name at multiple places, so
that we can have all the group names coded at one place
only.
2022-08-11 04:38:36 -07:00
Sahil Batra 31d639160f user_groups: Add FULL_MEMBERS_GROUP_NAME constant.
We now use FULL_MEMBERS_GROUP_NAME instead of
writing the actual full members system group
name at multiple places, so that we can have
all the group names coded at one place only.
2022-08-11 04:38:36 -07:00
Sahil Batra 150f77aea2 bulk_create: Add users to system user groups in bulk_create_users.
This commit modifies bulk_create_users to add the users to the
respective system groups. And due to this change, now bots in
development environment are also added to system groups.

Tests are changed accordingly as more UserGroupMembeship objects
are created.
2022-08-11 04:38:36 -07:00
Sahil Batra 393afc9781 realms: Create default system user groups for internal realm.
Since we include internal realms while creating system groups
in "0382_create_role_based_system_groups.py", we should do it
when creating new internal realms as well to be consistent.

Tests are changed accordingly as UserGroup objects are created.
We also change the user group ids used in api docs examples
such that user groups are of correct realm.
2022-08-11 04:38:36 -07:00
Mateusz Mandera 9b8d9d038a test_rate_limiter: Make test_add_remove_rule clean up its rules.
This test was polluting the rules for other tests, due to adding rate
limiting rules and not removing them.
2022-08-10 16:30:39 -07:00
Sahil Batra 3e6463804e streams: Allow changing history access without is_private parameter.
We now allow changing access to history of the stream by only passing
"history_public_to_subscribers" parameter. Previously, "is_private"
parameter was also required to change history_public_to_subscribers
otherwise the request was silently ignored.

We also raise error when only history_public_to_subscribers parameter
is passed with value False without "is_private: True" for a public
or web-public stream since we do not allow public streams with
protected history.
2022-08-09 17:05:38 -07:00
Sahil Batra 6ccfebac56 streams: Raise error when making public stream with private history.
We raise error when we try to change a public stream (except for
zephyr mirror realms) to be public with protected history, as we do
not support such streams yet.

Previously, in such case we changed nothing and a notification was
sent to the "stream events" topic with message being "stream is
changed from public to public" and was weird.

Note that this commit only handles the case when both is_private and
history_public_to_subscribers parameters are passed to API and commit
not covers the case when only "history_public_to_subscribers" with
value False is passed to API, since we currently ignore requests
which has only history_public_to_subscribers parameter with not None
and not is_private and is_web_public.
We would do this in further commits when we add support for accepting
only history_public_to_subscribers parameter.
2022-08-09 17:05:38 -07:00
Sahil Batra 75768decf1 tests: Pass all arguments in do_change_stream_permission.
This is a prep commit for changing do_change_stream_permission
to require passing all (invite_only, history_public_to_subscribers
and is_web_public) arguments in further commits.
2022-08-09 17:05:38 -07:00
Kartik Srivastava 887233a8eb api: Document /default_streams API endpoint. 2022-08-09 14:55:27 -07:00
Zixuan James Li 172a166159 cache: Avoid flushing invalid cache for realm emoji.
In certain cases, we call `RealmEmoji.save()` before the filename
becomes available. This result in getting invalid urls generated and
flushed. Normally we call it again shortly after, making it harder to
trigger this bug.

Fixes #22552.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-08-09 14:44:28 -07:00
Sahil Batra 5a29f4133b realm: Don't allow changing enable_spectator_access to True on limited plan.
We do not allow changing enable_spectator_access to True using API on
limited plan realms. Frontend changes have been done previously.

This is a follow-up of #22179.
2022-08-09 11:47:33 -07:00
Lauryn Menard aa796af0a8 upload: Remove `mimetype` url parameter in `get_file_info`.
This `mimetype` parameter was introduced in c4fa29a and its last
usage removed in 5bab2a3. This parameter was undocumented in the
OpenAPI endpoint documentation for `/user_uploads`, therefore
there shouldn't be client implementations that rely on it's
presence.

Removes the `request.GET` call for the `mimetype` parameter and
replaces it by getting the `content_type` value from the file,
which is an instance of Django's `UploadedFile` class and stores
that file metadata as a property.

If that returns `None` or an empty string, then we try to guess
the `content_type` from the filename, which is the same as the
previous behaviour when `mimetype` was `None` (which we assume
has been true since it's usage was removed; see above).

If unable to guess the `content_type` from the filename, we now
fallback to "application/octet-stream", instead of an empty string
or `None` value.

Also, removes the specific test written for having `mimetype` as
a url parameter in the request, and replaces it with a test that
covers when we try to guess `content_type` from the filename.
2022-08-08 16:06:09 -07:00
Lauryn Menard df3b8c590f user-settings: Make default `None` for name, email and password changes.
Updates `json_change_settings` so that the default value for the `email`,
`full_name`, `new_password` and `old_password` parameters is `None` instead
of an empty string, which also makes the type annotation `Optional[str]`.

Also, updates tests for email and full name changes to include an empty
string as one of the tested invalid values.
2022-08-08 15:37:41 -07:00
Julia Bichler 0a278c39d2 settings: Send email after deactivating user.
This adds a feature where an admin can choose to send an email
with custom content to an user after they deactivated them.

Fixes #18943.
2022-08-06 21:41:53 -07:00
N-Shar-ma ef044b8697 markdown: Update characters allowed before @ and stream mentions.
Now the following characters are allowed before @-mentions and stream
references (starting with #) for proper rendering - {, [, /.

This commit makes the markdown rendering consistent with autocomplete
(anything that is autocompleted is also rendered properly).
2022-08-06 19:29:39 -07:00
Kartik Srivastava 1291e7000b user_topic: Add user_topic event.
We now send a new user_topic event while muting and unmuting topics.
fetch_initial_state_data now returns an additional user_topics array to
the client that will maintain the user-topic relationship data.
This will support any future addition of new features to modify the
relationship between a user-topic pair.

This commit adds the relevent backend code and schema for the new
event.
2022-08-04 17:44:00 -07:00
Kartik Srivastava e2760a2bf2 api: Document /realm/presence API endpoint. 2022-08-04 16:34:13 -07:00
Zixuan James Li ae949bce04 test_auth_backends: Return an Iterable for idp names.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-08-03 15:03:15 -07:00