Commit Graph

6348 Commits

Author SHA1 Message Date
Alex Vandiver eae4643cb4 message_edit: Ignore duplicates when re-muting new topic name.
This avoids an error when a user has already muted the new topic name.
We do this by ignoring duplicates, rather than catching the
IntegrityError, because this edit happens in a transaction, and that
would abort the transaction.
2022-03-24 21:27:11 -07:00
Alex Vandiver 141b0c4cec muting: Handle the case of a race muting the same user twice. 2022-03-24 21:27:11 -07:00
Tim Abbott e3600900c0 migrations: Adjust related_name settings for ArchivedAttachment.
This is necessary for the migration 0386_fix_attachment_caches to run,
and likely makes more convenient any future parallel code interacting
with both Attachment and ArchivedAttachment.
2022-03-24 19:15:27 -07:00
Tim Abbott f9f111f950 message_edit: Only move muted topic records when moving whole topics.
Our original implementation of moving muted topic records when a topic
is moved took a shortcut of treating all change_later usage as
something with intent to move the whole topic.

This works OK when moving the whole topic via this interface, but not
when moving a last off-topic message in the topic.

Address this by changing the rule to match the existing
moved_all_visible_messages variable.
2022-03-24 17:48:52 -07:00
Aman Agrawal b799ec32b0 upload: Allow rate limited access to spectators for uploaded files.
We allow spectators access to uploaded files in web public streams
but rate limit the daily requests to 1000 per file by default.
2022-03-24 10:50:00 -07:00
Tim Abbott d149af936d models: Rework Attachment.is_*_public to be a cache.
Previously, Attachment.is_realm_public and its cousin,
Attachment.is_web_public, were properties that began as False and
transitioned to True only when a message containing a link to the
attachment was sent to the appropriate class of stream, or such a link
was added as part of editing a message.

This pattern meant that neither field was updated in situations where
the access permissions for a message changed:

* Moving the message to a different stream.
* Changing the permissions for a stream containing links to the message.

This correctness issue has limited security impact, because uploaded
files are secured both by a random URL and by these access checks.

To fix this, we reformulate these fields as a cache, with code paths
that change the permissions affecting an attachment responsible for
setting these values to the `None` (uncached) state. We prefer setting
this `None` state over computing the correct permissions, because the
correct post-edit permissions are a function of all messages
containing the attachment, and we don't want to be responsible for
fetching all of those messages in the edit code paths.
2022-03-24 10:49:59 -07:00
Alex Vandiver 1ac0035f8c markdown: Allow whitespace overlaps in topic linkifiers.
`prepare_linkifier_pattern`, as of db934be064, adds a match to the
end of the regex, of either the end of string, or a non-word character
-- this is in place of a negative look-ahead, which is no longer
possible in re2.  This causes the regex to consume trailing
whitespace, and thus not be able to match twice in succession with
`pattern.finditer` -- "#1234 #5678" fails to match because the space
is consumed by the first match of the regex.

Rather than use `pattern.finditer`, write own own version, which
rewinds over the non-word character consumed after the match, if any.
This allows the same "after" non-word character to also satisfy the
"before" of the next match.

Fixes #21502.
2022-03-22 15:40:03 -07:00
Alex Vandiver abed174b12 uploads: Add an endpoint which forces a download.
This is most useful for images hosted in S3, which are otherwise
always displayed in the browser.
2022-03-22 15:05:02 -07:00
Aman Agrawal 7a25a80242 message_edit: Move muted status of the topic when moved between streams.
Add support for moving MutedTopic entries to another stream where
the user has access to shared history in both streams and
`propagate_mode != "change_one"`.

Also, we delete them the current user does not have access to the
target stream.
2022-03-21 12:42:39 -07:00
Mateusz Mandera af5d0d6f5e bots: Don't allow admins to change owner of bot with can_create_users.
Ordinary organization administrators shouldn't be allowed to change
ownership of a bot with the can_create_users permission.

This is a special permission that is granted manually by server
administrators to an organization (to a UserProfile of the org owners'
choice) after approval by a server administator. The code comments
provide more detail about why this is sensitive.
2022-03-20 17:18:21 -07:00
Felix 780bda0d40
integrations: Fix BigBlueButton password length.
The BigBlueButton integration had a problem with generating
the random password with only 12 characters. This would
cause the attendeePW to be the same as the moderatorPW,
which might be fine but seems like something that could be an
error in a future version of BigBlueButton.
2022-03-20 16:09:36 -07:00
strifel a967a86b10 integration: Generate dynamic name for BigBlueButton video calls.
The name for a BigBlueButton meeting is now generated from the stream
name and topic name.

The createTime option is used to have the user redirected to a link
that is only valid for this meeting.

Even if the same link in Zulip is used again, a new createTime
parameter will be created, as the Meeting on the BigBlueButton server
has to be recreated.

Fixes #16498.
Fixes #20509.
Fixes #20804.
2022-03-18 17:27:39 -07:00
Tim Abbott f8146cfaa1 help: Fix label for Subscribed tab in stream settings. 2022-03-18 14:28:20 -07:00
Tim Abbott e45cebd636 message_edit: Fix unmute of topic when topic name is edited.
Previously, when a topic was edited (including being resolved), it
would become unmuted for any users who had muted it, which was
annoying.

While it's not possible to determine the user's intent completely,
this is clearly incorrect behavior in the `change_all` case, such as
resolving a topic.

The comments discuss some scenarios where we might want to enhance
this further, but this is the best we can do without large increases
in complexity.

Fixes #15210.

Co-authored-by: akshatdalton <akshat.dak@students.iiit.ac.in>
2022-03-18 12:32:46 -07:00
byshen-dev 26d97ce7e3 migrations: Change realm field to be not null in Attachment.
he possibility for it being null was likely an oversight -- it should
have been removed after the early migrations to backfill the field
when it was added.

We've confirmed there are no existing violations of this invariant in
Zulip Cloud.
2022-03-18 12:01:15 -07:00
Suyash Vardhan Mathur 20a97bdb05 events: Add functionality to mark messages as unread.
Co-authored-by: Steve Howell <showell@zulip.com>
Co-authored-by: Tim Abbott <tabbott@zulip.com>

This commit adds the backend functionality to
mark messages as unread through update_message_flags
with `unread` flag and `remove` operation.

We also manage incoming events in the webapp.

Tweaked by tabbott to simplify the implementation and add an API
feature level update to the documentation.

This commit was originally drafted by showell, and showell
also finalized the changes.  Many thanks to Suyash here for
the main work here, which was to get all the tests and
documentation work moving forward.
2022-03-15 18:00:35 -07:00
Sahil Batra 6f93f07844 types: Create a new TypedDict RealmPlaygroundDict for playground objects.
This commit creates a new TypedDict RealmPlaygroundDict for realm
playground objects. Now the list of playgrounds in the events sent
to clients and the "added_playground" field of RealmAuditLog entry
use RealmPlaygroundDict instead of Dict.
2022-03-15 15:58:36 -07:00
Sahil Batra b86b9bdc02 realm: Create RealmAuditLog entry when removing realm playgrounds.
This commit also adds 'acting_user' parameter to
do_remove_realm_playground function.

Fixes a part of #21268.
2022-03-15 15:58:36 -07:00
Sahil Batra dea3389045 realm: Create RealmAuditLog entry when adding realm playgrounds.
This commit also adds 'acting_user' parameter to
do_add_realm_playground function.

Fixes a part of #21268.
2022-03-15 15:58:36 -07:00
Alex Vandiver 62ba8e455d CVE-2022-24751: Clear sessions outside of the transaction.
Clearing the sessions inside the transaction makes Zulip vulnerable to
a narrow window where the deleted session has not yet been committed,
but has been removed from the memcached cache.  During this window, a
request with the session-id which has just been deleted can
successfully re-fill the memcached cache, as the in-database delete is
not yet committed, and thus not yet visible.  After the delete
transaction commits, the cache will be left with a cached session,
which allows further site access until it expires (after
SESSION_COOKIE_AGE seconds), is ejected from the cache due to memory
pressure, or the server is upgraded.

Move the session deletion outside of the transaction.

Because the testsuite runs inside of a transaction, it is impossible
to test this is CI; the testsuite uses the non-caching
`django.contrib.sessions.backends.db` backend, regardless.  The test
added in this commit thus does not fail before this commit; it is
merely a base expression that the session should be deleted somehow,
and does not exercise the assert added in the previous commit.
2022-03-15 13:52:15 -07:00
Anders Kaseorg ac9a5ba894 airbrake: Strengthen types using WildValue.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-03-15 13:02:02 -07:00
Anders Kaseorg 29ecf415fc validator: Add WildValue class for enforcing JSON type checking.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-03-15 13:02:02 -07:00
Anders Kaseorg 04d772b582 request: Support converter or json_validator with argument_type="body".
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-03-15 13:02:02 -07:00
Anders Kaseorg 5f92078d07 request: Add a var_name parameter to converter.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-03-15 13:02:02 -07:00
Anders Kaseorg 5caeb8444b validator: Replace converter=orjson.loads with json_validator.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-03-15 13:02:02 -07:00
Anders Kaseorg c3788ab847 validator: Replace converter=int with json_validator=check_int.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-03-15 13:02:02 -07:00
Alex Vandiver 95892a5ed3 emoji: Support animated PNGs. 2022-03-15 12:47:21 -07:00
Alex Vandiver fc793c10fa tests: Refactor tests of resizing animated images. 2022-03-15 12:47:21 -07:00
Sahil Batra f0606b34ad user_groups: Add cron job for adding users to full members system group.
This commit adds a cron job which runs every hour to add the users to
full members system group if user is promoted to a full member.

This should ensure that full member status is available no more than
an hour after configuration suggests it should be.
2022-03-14 18:53:47 -07:00
Sahil Batra bbf4c25553 user_groups: Update user groups on changing waiting_period_threshold.
The members of "Full members" user group are updated with change in
waiting_period_threshold setting.
2022-03-14 18:53:47 -07:00
Sahil Batra 9345c344d7 user_groups: Add members to the System groups created.
This commit adds users to the appropriate system user group
based on their role. We also change the user groups when
changing role of the user.

We also add migration to add existing users to the appropriate
user groups.

This commit adds update_users_in_full_members_system_group which
is currently used to update the full members group on changing
role of a user. This function will be modified in next commit such
that it can be used to update full members group on changing
waiting_period_threshold setting of realm.
2022-03-14 18:53:46 -07:00
Sahil Batra c33ba4ed6e realm: Create role-based system user groups on creating realm.
We create system user groups for following roles - owners,
admins, moderators, members and guests. Full members user
group will be handled separately.
2022-03-14 18:52:47 -07:00
Sahil Batra f31bb6754b user_groups: Pass list of user ids to remove_members_from_user_group.
We pass list of user ids instead of user profile objects to
remove_members_from_user_group. We still need to call user_id_to_users
in the views function instead of directly passing the ids to
remove_members_from_user_group to make sure we check whether all
ids are valid or not.
2022-03-14 18:52:47 -07:00
Sahil Batra 303fca2ce7 user_groups: Pass list of user ids to bulk_add_members_to_user_group.
We pass list of user ids instead of user profile objects to
bulk_add_members_to_user_group. We still need to call user_id_to_users
in the views function instead of directly passing the ids to
bulk_add_members_to_user_group to make sure we check whether all
ids are valid or not.
2022-03-14 18:52:47 -07:00
Steve Howell f9ddd80122 subscriptions: Improve types for subscription info.
The newly extracted module here has significant changes
to all the functions, but they achieve the same thing
as before.
2022-03-14 18:01:39 -07:00
Steve Howell 8f99894302 streams: Extract stream_color library.
This is a pure code move.
2022-03-14 18:01:36 -07:00
Steve Howell 200a89cef0 streams: Extract stream_traffic library.
This is a pure code move.
2022-03-14 18:01:36 -07:00
Mateusz Mandera 76ff9b30b1 push_notifs: Log both user id and user uuid if we have them.
Previous behavior was logging only the uuid if it was provided by the
remote server, but that's insufficient, because the user may actually
have no devices registered with uuis and we (at the bouncer) end up
sending notifications to id-based registrations. Not having that id
logged makes it impossible to figure out what's going on.
2022-03-14 17:47:30 -07:00
Mateusz Mandera d800ac33a0 push_notifications: Send user_uuid to the push bouncer.
Fixes #18017.

In previous commits, the change to the bouncer API was introduced to
support this and then a series of migrations added .uuid to
UserProfiles.

Now the code for self-hosted servers that makes requests
to the bouncer is changed to make use of it.
2022-03-14 17:47:30 -07:00
Mateusz Mandera 0677c90170 zilencer: Change push bouncer API to accept uuids as user identifier.
This is the first step to making the full switch to self-hosted servers
use user uuids, per issue #18017. The old id format is still supported
of course, for backward compatibility.

This commit is separate in order to allow deploying *just* the bouncer
API change to production first.
2022-03-14 17:47:30 -07:00
Kartik Srivastava eefaa9120f user_topic: Rename topic_mutes.py to user_topics.py. 2022-03-11 14:26:55 -08:00
Kartik Srivastava ce38eda54d test_events: Fix 'normalize' assuming subscription data is present.
This avoids a crash in normalize for tests that don't include these in
fetch_event_types.
2022-03-11 14:26:14 -08:00
Aman Agrawal 82837304ec api: Send full message in GET /messages/{message_id} response.
Previously, this URL just returned the `raw_content` field. It seems
cleanest to just make it a single-message variant of GET /messages,
deprecating the only format.
2022-03-11 10:25:22 -08:00
Tim Abbott 20368a936c settings: Add web-public streams beta subdomain list.
This will make it convenient to add a handful of organizations to the
beta of this feature during its first few weeks to try to catch bugs,
before we open it to everyone in Zulip Cloud.
2022-03-10 18:37:01 -08:00
Sahil Batra 07352271b9 realm: Create RealmAuditLog entry for removing a realm domain.
Fixes a part of #21268.
2022-03-10 17:48:02 -08:00
Sahil Batra ab5567e8c5 realm: Create RealmAuditLog entry while changing an allowed domain.
This commit also adds 'acting_user' argument for do_change_realm_domain
function.

Fixes a part of #21268.
2022-03-10 17:48:02 -08:00
Sahil Batra 5ef8da40a9 realm: Create RealmAuditLog entry while adding a new allowed domain.
This commit also adds 'acting_user' argument for do_add_realm_domain
function.

Fixes a part of #21268.
2022-03-10 17:48:02 -08:00
Alya Abbott a80e470c9e portico: Create use cases index page. 2022-03-10 16:34:34 -08:00
Steve Howell 8e05a9fcf7 unread: Replace sender_id with other_user_id.
Note that we still send sender_id for legacy mobile
clients.
2022-03-10 13:33:21 -08:00
Steve Howell 822c232e37 message flags: Extract create_historical_user_messages.
A user can subscribe to a stream and sometimes (depending
on stream permissions) see messages from the stream
that were sent before they subscribed, and that user
won't have a UserMessage row for that message.

In order to do things like star a message, we need
to create UserMessage records on the fly.

In the past we wisely constrained this logic to the
specific use cases. But I think we can generalize
the logic now.  For example, we are now building a
feature to mark messages as unread, and it motivates
the same need to auto-create UserMessage rows.

So now we handle this in a more generalized fashion.
2022-03-10 13:21:02 -08:00