We change the topic and stream edit permssions to not depend on
allow_message_editing setting in the API and are allowed even
if allow_message_editing is set to False based on other settings
like edit_topic_policy and can_move_message_between_streams.
Fixes a part of #21739.
This solves the problem that resolving a topic with a long name (>60
characters) will cause the topic name to be truncated, and thus the edit
message code path thinks that the topic is being moved in addition to
being resolved.
We store the pre-truncation topic and use it to check against the
original topic when determining whether a topic is being moved while
getting (un)resovled or not.
Fixes#23482
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
We intended to send both the "topic was resolved" and the "topic was
moved here" notification when resolving and moving a topic at the same
time in #22312.
The previous implementation did not work as expected and it was only
sending the "topic was moved here" notification.
This removes the check for old_topic and new_topic that have
RESOLVED_TOPIC_PREFIX stripped in maybe_send_resolve_notifications, so
that the notification will be sent regardless if the topic name without
the prefix stays the same or not.
Note that weird topic handling ("✔ ✔✔ some topic") in the comments
was added in e231a03eff is unaffected. In case of confusion, the lstrip
check is not essential to detecting topic being unresolved/resolved.
As we mainly have that handled in the latter part of the helper.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
This is a follow-up to d201229df8.
do_get_invites_controlled_by_user queries for Confirmations when finding
multiuse invites controlled by a user. This means that a revoked
multiuse invite cannot really be fetched here, because
do_revoke_multi_use_invite deletes the Confirmation object when revoking
the invitations. However, having a defensive assert here should be
useful to make this doesn't secretly break in the future if the query
used changes or if there are unexpected revoked multiuse invites with an
existing Confirmations for any (buggy) reason.
This allows us to revoke MultiUseInvites by changing their .status
instead of deleting them (which has been deleting the helpful tracking
information on PreregistrationUsers about which MultiUseInvite they came
from).
We do not create historical UserMessage rows, for messages that didn't
have one, while marking messages as read and simply ignore those messages.
We do so because there is no user of creating UserMessage rows and it just
wastes storage.
Note that we still allow to mark messages from unsubscribed streams as
read but only those which have UserMessage rows for them to handle the
case when the unread messages were not marked as read while unsubscribing
from the stream due to some race condition. In such cases, messages
will not be included in the unread count shown in "All messages" menu
(and stream is anyways not present in the left sidebar), but the message
border on the left is green if viewing the stream after unsusbcribing it.
So, to avoid the confusion for users, the messages will be marked as read
when user scrolls down.
Zulip's unread messages design has an invariant that all unread stream
messages must be in streams the user is subscribed to. For example, We
do not include the unread messages from unsubscribed streams in the
"unread_msgs" data structure in "/register" response and we mark all
unread messages as read when unsubscribing a user from a stream.
Previously, the mark as unread endpoint allowed violating that
invariant, allowing you to mark messages in any stream as unread.
Doing so caused the "message_details" data structures sent with
"update_message_flags" events to not contain messages from
unsubscribed streams, even though those messages were present in the
set of message IDs. These malformed events, in turn, caused exceptions
in the frontend's processing of such an event.
This change is paired with a separate UI change to not offer the "Mark
as unread" feature in such streams; with just this commit, that will
silently fail.
With some additions to the tests by tabbott.
This guarantees that the Realm is always non-None when we hit the
codepath is_static_or_current_realm_url via
do_change_stream_description, so that we can properly skip rewritting
some images.
Fixes#19405
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
I don't think this is used anywhere outside of tests, but we should have
this logic correct. If this function is used to send a message from a
user to a cross-realm bot, the message.realm should be the realm of the
user.
In the normal case, where a user send a message to a cross-realm bot
through the API is already handled correctly, this bug is unrelated.
Previously we did not send notification for topic-only edits.
Now, we add backend support for sending notification to topic-only
edits as well.
We would add support for this in webapp in further commits since
message edit UI will be updated as well. We just make sure that no
notifications are sent when editing topic using pencil icon in
message header.
We also change the API default for moving a topic to only notify the
new location, not the old one; this matches the current defaults in
the web UI.
Includes many tests.
We also update the puppeteer tests to test only content edit as
we are going to change the UI to not allow topic editing from
message edit UI. Also fixing the existing tests to pass while
doing topic edits is somewhat complex as notification message
is also sent to new topic by default.
Fixes#21712.
Co-authored-by: Aman Agrawal <amanagr@zulip.com>
Co-authored-by: Tim Abbott <tabbott@zulip.com>
The previous commit did this for revoking sessions. send_events should
be handled similarly too, to correctly handle calling do_deactivate_user
inside a transaction.
This commit adds the OPTIONAL .realm attribute to Message
(and ArchivedMessage), with the server changes for making new Messages
have this set. Old Messages still have to be migrated to backfill this,
before it can be non-nullable.
Appropriate test changes to correctly set .realm for Messages the tests
manually create are included here as well.
zerver/migrations/0240_usermessage_migrate_bigint_id_into_id.py needs
to be updated to account for Django 4.1 creating AutoField as an
identity column rather than a serial column.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Previously, we included all three message edit related settings
("allow_message_editing", "message_content_edit_limit_seconds" and
"edit_topic_policy") in the event data and api response irrespective
of which of these settings were changed. Now, we only include changed
settings and separate events are sent for each setting if more than
one of them is changed.
Note that the previous typed in event_schema.py for
`message_content_edit_limit_seconds` incorrectly did not allow `None`
as a value, which is used to encode no limit.
This refactors and renames user_ids_muting_topic to accept a parameter
'visibility_policy' and fetch user IDs that have a specific
visibility_policy(provided as the parameter) set for a topic.
Fourth step in making user status `away` a deprecated way to access
`presence_enabled` for clients supporting older servers, and
checkpoint commit prior to deleting the `status` field from the
UserStatus model.
Part of transitioning from 'unavailable' user status feature to
'invisible mode' user presence feature.
When a user toggles a status update for `away=True|False`, we now update
their `presence_enabled` setting to match (`away!=presence_enabled`).
First step of making user status `away` updates a deprecated way to
access presence_enabled for clients supporting older servers, and
checkpoint commit before migrating users with a current UserStatus
of `status=AWAY` to have their `presence_enabled` set to `False`.
Note that when user status `away` is updated, we now send 4 events:
user_status, user_settings, presence, and update_global_notifications.
Also, this means that these updates change the UserPresence.status
value, which impacts the test for importing and exporting user
information.
Part of transitioning from 'unavailable' user status feature to
'invisible mode' user presence feature.
We need to move this function to a separate actions file specifically
for `user_status` because otherwise we will have a circular import
between `actions/user_settings.py` and `actions/presence.py` in an
upcoming commit.
Prep commit for migrating "unavailable" user status feature to
"invisible" user presence feature.
To allow `custom_profile_field` to display in user profile popover,
added new boolean field "display_in_profile_summary" in its model class.
In `custom_profile_fields.py`, functions are edited as per conditions,
like currently we can display max 2 `custom_profile_fields` except
`LONG_TEXT` and `USER` type fields.
Default external account custom profile fields made updatable for only
this new field, as previous they were not updatable.
Fixes part of: #21215
This commit adds do_change_can_remove_subscriber_group function for
changing can_remove_subscribers_group field of a stream. We also add
can_remove_subscribers_group_id field to stream and subscription
objects.
This function will be helpful for writing tests in next commit.
We would add API and UI support to change this setting in further
commits.
Previously, an active production Zulip server would experience a class
of deadlocks caused by two or more concurrent bulk update operations
on the UserMessage table.
This is because UPDATE ... SET ... WHERE statements that execute in
parallel take row-level UPDATE locks as they get results; since the
query plans may result in getting rows in different orders between two
queries, this can result in deadlocks.
Some databases allow ORDER BY on their UPDATE ... WHERE statements;
PostgreSQL does not. In PostgreSQL, the answer is to do a sub-select
with an ORDER BY ... FOR UPDATE to ensure consistent ordering on row
locks.
We do this all code paths using bitand or bitor as part of bulk
editing message flags, which should ensure that these concurrent
operations obtain row level locks on the table in the same order.
Fixes#19054.
This is preparatory commit for #18941.
Importing `do_delete_message` from `message_edit.py` was causing a
circular import error. In order to avoid that, we create a separate
message_delete.py file which has all the functions related to deleting
messages.
The tests for deleting messages are present in
`zerver/tests/test_message_edit.py`.
Fixes a part of #18941
In zerver.management.commands.logout_all_users,
we pass a values queryset containing the ids into
this function, which is not actually a list. This
broadens the type annotation so that the ValuesQuerySet
is accepted.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
This uses a more specific type `_StrPromise` to replace `Promise`
providing typing information for lazy translation strings.
In places where the callee evaluates the `_StrPromise` object in all
cases we simply force the evaluation with `str()`. This includes
`JsonableError` that ends up handled by the error handler middleware,
and `internal_send_stream_message` that depends on `check_stream_topic`,
requiring the `topic` to be evaluated anyway. In other siuations, the
callee is expected to be able to handle `StrPromise` explicitly.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
Whether we sent a resolve topic notification or not may be useful in
the caller. It was originally intended to be used in #21712, but may
only be relevant for future logging.
Part of #21712.
This will have no real effect in most situations. However, a user
moves a topic to another stream while also adding/removing the
resolved-topic checkmark from the topic name, then the "This topic was
resolved" notificaiton will now appear just before the "This topic was
moved" notification rather than just after.
This is likely slightly less confusing to users, since the topic
having been moved from somewhere else is likely the most salient fact
to a reader.
We expect to change things to not send both notifications in an
upcoming commit.
This refactoring helps with #21712.
Our seat count calculation is different for guest user than normal users
(a number of initial guests are free, and additional marginal guests are
worth 1/5 of a seat) - so these checks we apply when a user is being
invited or signing up need to know whether it's a guest or non-guest
being added.
In Zulip 2.1.0, the `is_muted` stream subscription property was
added and replaced the `in_home_view` property. But the server has
still only been sending subscription update events with the
`in_home_view` property.
Updates `do_change_subscription_property` to send a subscription
update event for both `is_muted` and `in_home_view`, so that
clients can fully migrate away from using `in_home_view` allowing
us to eventually remove it completely.
This is a prep commit such that we can avoid duplicate code when we
unsubscribe bots for inaccessible private streams when changing owner
or reactivating them.
This commit changes the code to consider zero as an invalid value for
message_content_edit_time_limit_seconds. Now to represent the setting that
user can edit the message anytime, the setting value will be "None" in
database and "unlimited" will be passed to API from clients.
Adds an API endpoint for accessing read receipts for other users, as
well as a modal UI for displaying that information.
Enables the previously merged privacy settings UI for managing whether
a user makes read receipts data available to other users.
Documentation is pending, and we'll likely want to link to the
documentation with help_settings_link once it is complete.
Fixes#3618.
Co-authored-by: Tim Abbott <tabbott@zulip.com>
We now use MEMBERS_GROUP_NAME instead of writing
the actual group name at multiple places, so that we
can have all the group names coded at one place only.
We now use FULL_MEMBERS_GROUP_NAME instead of
writing the actual full members system group
name at multiple places, so that we can have
all the group names coded at one place only.
I found the previous model for computing what settings to use for
streams increasingly difficult to understand, which is generally a
recipe for future bugs.
Refactor to have a clear computation of what complete permissions
state the client is requesting, validate that state, and then pass
that state to the do_change_stream_permission.
We now allow changing access to history of the stream by only passing
"history_public_to_subscribers" parameter. Previously, "is_private"
parameter was also required to change history_public_to_subscribers
otherwise the request was silently ignored.
We also raise error when only history_public_to_subscribers parameter
is passed with value False without "is_private: True" for a public
or web-public stream since we do not allow public streams with
protected history.
This commit removes the unnecessary assertion statements in
do_change_stream_permission for case when "is_web_public" is
True, since we already check those cases in the view function
update_stream_backend and this is the only place from where
do_change_stream_permission is called.
We aim to remove other assertions also from there as mentioned
in the comment and instead check the values in caller itself.
We now send a new user_topic event while muting and unmuting topics.
fetch_initial_state_data now returns an additional user_topics array to
the client that will maintain the user-topic relationship data.
This will support any future addition of new features to modify the
relationship between a user-topic pair.
This commit adds the relevent backend code and schema for the new
event.
We separate the permission checks for content and topic edits
by changing the can_edit_topic_or_content to can_edit_topic
and use it only for checking topic edits and check content
edits separately in check_update_message itself. There is no
change in behavior as of this PR, there will be more changes
as per #21739.
This is a prep commit for #21739. The permission checks for
them are essentially separate except the one that message
sender is allowed to edit content and topic irresepctive of
edit_topic_policy setting, and this will too be changed in
future commit and so it will be better to have these checks
separate for readability.
We can also probably create a new function for checking content
edits but currently we only check the sender is same as the use
who is editing and it does not make sense to have a separate
function for just one check. We can do so in future in case we
do some more refactoring for #21739.
We pass the realm_emoji dict to notify_realm_emoji instead
of computing it in notify_realm_emoji itself, because its
caller functions will require the dict in further commits
for creating RealmAuditLog entries.
This commit also renames the legacy event variable to
legacy_event from event because same event is sent
twice if we keep only single variable and change its
value. This was happening because the event value was
changed when the first send_event was actually executed.
We pass the realm_linkfiers list to notify_linkifiers instead of
computing it in notify_linkifiers itself, because its caller
functions will require the list in further commits for creating
RealmAuditLog entries.
Since we not allow enabling public access on limited plan realms,
we set the enable_spectator_access setting to False when downgrading
to a limited plan. Setting is still shown in the UI but it is
disabled.
This commit adds code to send stream creation and peer add events
when stream is changed from private to public. These events are
only sent to users who are not susbcribed to the stream and are
not realm admins as subscribers and realm admins already have
the stream data. This will update the stream data with clients
and will remove the need to reload to view the modified stream.
Fixes#22194.
We should not monkey-patch message when unnecessary. Adding
`service_queue_events` to `SendMessageRequests` suits our need to type
safety here.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
Now that we can assume Python 3.6+, we can use the
email.headerregistry module to replace hacky manual email address
parsing.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
`extra_data` as a `TextField` expects a `str`, but we had been passing
`dict` instead. This is a temporary solution before #18391 to fix the
type annotation.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
This reverts commit 40fcf5a633.
This commit triggers bug that we haven't fully tracked down, where web
app clients will continually send `update_message_flags` requests,
that then send out via the events system "0 messages were marked as
read" notices, eventually leading to a load spike.
The Tornado part can likely be fixed by checking if
updated_message_ids is empty, but we need to track down the frontend
bug as well.
Previously, automated stream messages for new user signups were not
being translated into the realm's default language for said messages.
Moves `override_language` context manager so that it wraps the
new user message content in `notify_new_user` and topic string in
`send_message_to_signup_notification_stream`.
Fixes#22510.
The dangling comment was not very helpful and unclear about the pieces
of code it was referring to.
We expand the part about linking the prereg_user to the created user,
while the part about "revoking other preregistration users" is
redundant, because the relevant code block lower down already has
comments on it with better explanations.
Closes#22274.
This assertion was added in 4b903c5dcd
where it may have made sense, because indeed when doing realm creation
there was always a PreregistrationUser (created because realms were
created via going to a generated realm creation link). With the addition
of the create_realm command that's no longer the case.
It would be unnatural to create a PreregistrationUser in the
realm_creation command, because there is no confirmation link for it to
be tied to - and it just doesn't make sense conceptually.
The intended, correct behavior added in
4b903c5dcd is still maintained - the code
lower down correctly handles the
(prereg_user is None and realm_creation) case.
The type safety of a TypeGuard is unchecked by mypy. While this
particular TypeGuard is safe given the current context, one could
imagine future changes that make it unsafe, so it’s preferable to
avoid unchecked constructs whenever possible.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
django-stubs types the return type of query.values(...) as a TypedDict.
This makes Dict[str, Any] that we have been using incompatible with it.
We use TypeGuard to ensure that `service_bot_tuples` is correctly
inferred to be `Tuple[int, int]` instead of `Tuple[int, Optional[int]]`.
Given that `bot_type` is optional for `ActiveUserDict`, we need to
narrow `row` to `ActiveBotUserDict` to make sure that `bot_type` is
non-optional. An advantage of this approach is that no assertions or
type casts are needed.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
This commit removes "role" field from subscription
objects since we are not moving forward with stream
administrator concept and instead working on new
permssions model as per #19525.
As discussed in the new comments results in a better failure mode if
an error occurs while adding subscriptions; running the merge tool
again after fixing whatever caused the error will work just fine.
This ensure that the return type is compatible with the
actual type of `realm.realmdomain_set.values`.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
To explain the rationale of this change, for example, there is
`get_user_activity_summary` which accepts either a `Collection[UserActivity]`,
where `QuerySet[T]` is not strictly `Sequence[T]` because its slicing behavior
is different from the `Protocol`, making `Collection` necessary.
Similarily, we should have `Iterable[T]` instead of `List[T]` so that
`QuerySet[T]` will also be an acceptable subtype, or `Sequence[T]` when we
also expect it to be indexed.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
We were blindly adding / removing flag from UserMessages without
check if they even need to be updated.
This caused server to repeatedly update flags for messages which
already had been updated, creating a confusion for other clients
like mobile.
Fixes#22164
Sometimes (e.g. when moving an old realm out of the way of an import
into that name) we do *not* wish to add a redirect realm. Add a flag
to support that.
When there is no topic/stream being change, `propagate_mode` becomes
unnecessary. We add an assertion to ensure that the previous assumption
that `propagate_mode` is not `None` still holds when either `topic_name`
or `new_stream` is not `None`.
We can possibly improve this by overloading `do_update_message` and
`check_update_message`, but that's beyond the scope of the PR and
feasibility of doing that should also be further discussed.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
While it is possible to have `stream.recipient_id` being `None`,
the code works under the assumption that it is not. Potentially
we will get a runtime error, but it is not quite explicit without
the assertion.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
When editing an old message in a private stream with protected
history, the server would incorrectly send an API event including the
edited message to all of the stream’s current subscribers, including
those who should not have access to the old message. This API event is
ignored by official clients, so it could only be observed by a user
using a modified client or their browser’s developer tools.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This function is oblivious to the existence of ArchivedAttachment, which
is incorrect. A file can be removed if and only if it is not referenced
by any Messages or ArchivedMessages.
Add none-checks, rename variables (to avoid redefinition of
the same variable with different types error), add necessary
type annotations.
This is a part of #18777.
Signed-off-by: Zixuan James Li <359101898@qq.com>
We remove one call to get_occupied_streams to get occupied
streams before unsubscribing because we already know which
streams can become vacant, i.e. the one from which users are
being unsubscribed, and we can directly use the list of streams
from which users are being unsubscribed and get vacant streams
by checking which of these streams are not in get_occupied_streams
called after unsubscribing users.
Previously, we were marking messages of all the streams passed
to bulk_remove_subscriptions even if user was not subscribed
to some of them and those streams would ideally not have
any unread messages. This code was added in 766511e519.
This commit changes the code to only mark messages of actually
unsubscribed streams as read.
cfcbf58cd1 rightly removed the use of `user_ids` in
`render_markdown`, which in turn makes it unnecessary in
`render_incoming_message`.
Remove the unnecessary parameter from `render_incoming_message`.
This commit changes the code to always pass delivery_email
field in the user's own object in 'realm_users'.
This commit also fixes the events sent by notify_created_user.
In the "realm_user/add" event sent when creating the user,
the delivery_email field was set according to the access
for the created user itself as the created user was passed as
acting_user to format_user_row. But now since we have changed
the code to always allow the user themselves to have access
to the email, this bug was caught in tests and we fix the person
object in the event to have delivery_email field based on whether
the user receiving the event has access to email or not.
This commit reads the browser locale during user registration, and
sets it as default language of user if it is supported by Zulip.
Otherwise, it is set to realm's default language.
`org_type` already exists as a field in the Realm model and is
used when organizations are created / updated in Zulip Cloud,
via the `/analytics/support` view.
Extends the `PATCH /realm` view to be able update `org_type` as
other realm / organization settings are updated, but using the
special log / action that was created for the analytics view.
Adds a field to the `realm op: update` / `realm op: update_dict`
events, which also means an event is now sent when and if the
`org_type` is updated via the analytics view. This is similar
to how updates to an organization's `plan_type` trigger events.
Adds `realm_org_type` as a realm setting fetched from the
`POST /register` endpoint.
This commit also adds 'subgroups' field to the user_group present
in the event sent on creating a user group. We do not allow passing
the subgroups while creating a user group as of this commit, but added
the field in the event object to pass tests.
This commit changes the invite API to accept invitation
expiration time in minutes since we are going to add a
custom option in further commits which would allow a user
to set expiration time in minutes, hours and weeks as well.
Based on an audit, this closes out the last core instances in which
acting_user was not being passed explicitly when creating
RealmAuditLog instances.
There are some outstanding issues in the billing system, which we plan
to extract as a separate issue.
Fixes#14808.
The `get_link_embed_data` / `link_embed_data_from_cache` pair as
introduced in c93f1d4eda uses the cache
as a temporary store inside of the `embed_links` worker; this means
that it must be durable storage, or the worker will stall and re-fetch
the same links to preview them.
Switch to plumbing through the fetched URL embed data as an parameter
to the Markdown evaluation which uses them, rather than using the
cache as an intermediary. This frees up the cache to be merely a
non-durable cache.
As a side-effect, this removes get_cache_with_key, and
link_embed_data_from_cache which was its only callsite.
Sahil Batra
Zixuan James Li
Anders Kaseorg
Mateusz Mandera
Tim Abbott
Julia Bichler
Kartik Srivastava
Lauryn Menard
yogesh sirsat
Christopher Chong
madrix01
Adam Sah
Dinesh
Aman Agrawal
Alex Vandiver
Zixuan James Li
Siddharth Asthana