No codepath except tests passes in more than one user_profile -- and
doing so is what makes the deduplication necessary.
Simplify the API by making it only take one user_profile id.
We send a event with type 'user_settings' on updating user's display
and notification settings.
The old event types - 'update_global_notifications' and
'update_display_settings', are still supported for backwards
compatibility.
Previously, we checked for the `enable_offline_email_notifications` and
`enable_offline_push_notifications` settings (which determine whether the
user will receive notifications for PMs and mentions) just before sending
notifications. This has a few problem:
1. We do not have access to all the user settings in the notification
handlers (`handle_missedmessage_emails` and `handle_push_notifications`),
and therefore, we cannot correctly determine whether the notification should
be sent. Checks like the following which existed previously, will, for
example, incorrectly not send notifications even when stream email
notifications are enabled-
```
if not receives_offline_email_notifications(user_profile):
return
```
With this commit, we simply do not enqueue notifications if the "offline"
settings are disabled, which fixes that bug.
Additionally, this also fixes a bug with the "online push notifications"
feature, which was, if someone were to:
* turn off notifications for PMs and mentions (`enable_offline_push_notifications`)
* turn on stream push notifications (`enable_stream_push_notifications`)
* turn on "online push" (`enable_online_push_notifications`)
then, they would still receive notifications for PMs when online.
This isn't how the "online push enabled" feature is supposed to work;
it should only act as a wrapper around the other notification settings.
The buggy code was this in `handle_push_notifications`:
```
if not (
receives_offline_push_notifications(user_profile)
or receives_online_push_notifications(user_profile)
):
return
// send notifications
```
This commit removes that code, and extends our `notification_data.py` logic
to cover this case, along with tests.
2. The name for these settings is slightly misleading. They essentially
talk about "what to send notifications for" (PMs and mentions), and not
"when to send notifications" (offline). This commit improves this condition
by restricting the use of this term only to the database field, and using
clearer names everywhere else. This distinction will be important to have
non-confusing code when we implement multiple options for notifications
in the future as dropdown (never/when offline/when offline or online, etc).
3. We should ideally re-check all notification settings just before the
notifications are sent. This is especially important for email notifications,
which may be sent after a long time after the message was sent. We will
in the future add code to thoroughly re-check settings before sending
notifications in a clean manner, but temporarily not re-checking isn't
a terrible scenario either.
In this commit:
* We update the `UserStatus` model to accept
`AbstractReaction` as a base class so, we can get all the
fields related to store status emoji.
* We update the user status endpoint
(`users/me/status`) to accept status emoji fields.
* We update the user status event to add status emoji
fields.
Co-authored-by: Yash Rathore <33805964+YashRE42@users.noreply.github.com>
This removes a bunch of non-functional duplicate JavaScript, HTML, and
CSS that was interfering with maintenance on the functional originals,
because it was never clear how to update the duplicates or how to
check that you’d updated the duplicates correctly.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This commit moves "enter_sends" setting to property_types dict.
With this change, changing enter_sends setting also sends an
event of type "update_display_settings" and thus enables us
to live-update the UI.
When calling some functions or assigning values to certain attributes,
the arguments/right operand do not match the exact type that the
functions/attributes expect, and thus we fix that by converting types
beforehand.
Sometime in the deep past, Zulip the GET /users/me/subscriptions
endpoint started returning subscribers. We noticed this and made it
optional via the include_subscribers parameter in
1af72a2745, however, we didn't notice
that they were being returned as emails rather than user IDs.
We migrated the core /register code paths to use subscriber IDs years
ago; this change completes that for the endpoints we forgot about.
The documentation allowed this error because we apparently had no
tests for this code path that used the actual API.
We use subs as a common variable name for a collection of stream
data structure used in settings, in lot of modules. So this
rename clears a bunch of related shadowed variables.
This function had a confusing name, which could result in someone
using it unintentionally when they meant do_reactivate_user.
We also add docstrings for both functions.
We remove timezone setting from UserProfile.property_types
so that we can directly use UserProfile.property_types for
implementation of realm-default values of various user
settings.
We will later use this data to include text like:
`<sender> mentioned @<user_group>` instead of the current
`<sender> mentioned you` when someone mentions a user group
the current user is a part of in email/push notification.
Part of #13080.
This adds a new class called MessageRenderingResult to contain the
additional properties we added to the Message object (like alert_words)
as well as the rendered content to ensure typesafe reference. No
behavioral change is made except changes in typing.
This is a preparatory change for adding django-stubs to the backend.
Related: #18777
This is a prep commit for adding realm-level default for various
user settings. We add the language, in which the invite email will
be sent, to the dict added to queue itself to avoid making queries
in a loop when sending multiple emails from queue.
We also handle the case for old events in the queue.
We removed the use of email_body field in 47fcb27e39, but was
still passed in events from do_resend_user_invite_email and
in tests. So this commit removes the email_body field from
these places.
We already have this data in the `flags` for each user, so no need to
send this set/list in the event dictionary.
The `flags` in the event dict represent the after-message-update state,
so we can't avoid sending `prior_mention_user_ids`.
Previously, it was possible for an unusual series of topic-edit
actions to result in Notification Bot reporting that a topic was
marked as resolved that had already been marked as resolved, etc.
A buggy client might send a message_edit request to change the topic
field, sending the current topic as the new value. Previously, we
would treat that as a normal request to edit the topic; now we act as
though the API request had not requested a topic change. In the
common case that only the topic was in the edit request, this now
results in an error that should help client implementations identify
their bug.
This fixes a bad interaction with the "unresolve topic" logic, which
assumed that upstream logic had verified that the topic was actually
changing.
* Have the `get_active_presence_idle_user_ids` function look at all the
user data, not just `private_message` and `mentioned`.
* Fix a couple of incorrect `missedmessage_hook` tests, which did not
catch the earlier behaviour.
* Add some comments to the tests for this function for clarity.
* Add a helper to create `UserMessageNotificationsData` objects from the
user ID lists. This will later help us deduplicate code in the event_queue
logic.
This fixes a bug which earlier existed, that if a user turned on stream
notifications, and received a message in that stream which did not mention
them, they wouldn't be in the `presence_idle_users` list, and hence would
never get notifications for that message.
Note that, after this commit, users might still not get notifications in
the above scenarios in some cases, because the downstream logic in the
notification queue consumers sometimes erroneously skips sending
notifications for stream messages.
Before this commit, we used to pre-calculate flags for user data and send
it to Tornado, like so:
```
{
"id": 10,
"flags": ["mentioned"],
"mentioned": true,
"online_push_enabled": false,
"stream_push_notify": false,
"stream_email_notify": false,
"wildcard_mention_notify": false,
"sender_is_muted": false,
}
```
This has the benefit of simplifying the logic in the event_queue code a bit.
However, because we sent such an object for each user receiving the event,
the string keys (like "stream_email_notify") get duplicated in the JSON
blob that is sent to Tornado.
For 1000 users, this data may take up upto ~190KB of space, which can
cause performance degradation in large organisations.
Hence, as an alternative, we send just the list of user_ids fitting
each notification criteria, and then calculate the flags in Tornado.
This brings down the space to ~60KB for 1000 users.
This commit reverts parts of following commits:
- 2179275
- 40cd6b5
We will in the future, add helpers to create `UserMessageNotificationsData`
objects from these lists, so as to avoid code duplication.
We now encode resolved topics with just:
U+2714 HEAVY CHECK MARK, SPACE
Previously, the encoding was unintentionally this:
U+2714 HEAVY CHECK MARK, U+FE0F VARIATION SELECTOR-16, SPACE
This commit replaces the allow_community_topic_editing boolean with
integer field edit_topic_policy and includes both frontend and
backend changes.
We also update settings_ui.disable_sub_settings_onchange to not
change the color of label as we did previously when the setting
was a checkbox. But now as the setting is dropdown we keep the
label as it is and we don't do anything with label when disabling
dropdowns. Also, this function was used only here so we can safely
change this.
We do not need the 'topic_name is None' check in this function as this is
called only when atleast one of the content and topic_name is not None,
and this condition cannot be true as there is 'content is not None'
check just before it.
Thus, 'if topic_name is None' condition being true means that both content
and topic_name are None which is not possible as this function itself will
not be called in such case. An assert statement is added to check that
topic_name is not None to make sure that it is handled when the function
is called in some other way later.
This is a prep change for calling `get_active_presence_idle_user_ids`
after we have collected all user data variables, so that that function
does not erroneously skip some user IDs from not having the complete
data.
We will in later commits, extend this class to contain methods
to determine if a message is notifiable or not, but for now
we only turn it into a dict and pass it on.
This gives us a single place where all user data for the message
send event is calculated, and is a prep change for introducing
a TypedDict or dataclass to keep this data toghether.
Earlier, the notification-blocking for messages from muted senders
was a side-effect of we never sending notifications for messages
with the "read" flag.
This commit decouples these two things, as a prep for having new
settings which will allow users to **always** receive email
notifications, including when/if they read the message during the
time the notifications is in the queue.
We still mark muted-sender messages as "read" when they are sent,
because that's desirable anyways.
This is a prep change for importing (and using) `dataclasses.field`
elsewhere in the same file, because pyflakes would throw "Import
module shodowed" errors otherwise.
This locks the message row while a reaction is being added/removed,
which will handle race conditions caused by deleting the message
at the same time.
We make sure that events work happens outside the transaction,
so that in case there's some problem with the queue processor, the
locks aren't held for too long.
As a nice side-effect, we also handle race conditions from double
adding reactions, because once the message is locked, a duplicate
request will wait till the earlier transaction commits, and hence
will not throw `IntegrityErrors`s (rather, will be handled in our
safety check in the /views code itself), which earlier had to be
handled explicitly.
This locks the message while creating a submessage, which
will handle race conditions caused by deleting the message
simultaneously.
We make sure that events work happens outside the transaction,
so that in case there's a problem with the queue processor,
the locks aren't held for too long.
Further commits will start locking the message rows while
adding related fields like reactions or submessages,
to handle races caused by deleting the message itself at the
same time.
The message locking implemented then will create a possibility
of deadlocks, where the related field transaction holds a lock
on the message row, and the message-delete transaction holds a
lock on the database row of the related field (which will also
need to be deleted when the message is deleted), and both
transactions wait for each other.
To prevent such a deadlock, we lock the message itself while
it is being deleted, so that the message-delete transaction
will have to wait till the other transaction (which is about
to delete the related field, and also holds a lock on the
message row) commits.
https://chat.zulip.org/#narrow/near/1185943 has more details.
This commit fixes a bug where moving messages between streams was
not allowed for non-admins when allow_community_topic_editing was
set to false and move_messages_between_streams_policy was set to
Realm.POLICY_MEMBERS_ONLY.
The bug is fixed by calling can_edit_content_or_topic only when
topic or content edit is there and not in the case where only
message is moved from one stream to another.
This commit extracts the logic of checking the message edit permissions,
like whether the sender is same as user, whether it is a (no topic)
message or whether community topic editing is allowed, into a separate
function.
This is a prep commit for fixing a bug where permission to move messages
between streams is affected by permission of editing topics.
Previously when enforcing the check to do not allow editing topics
after a certain time, we were checking whether 'content is None' and
considering it as that if content is None then there must be topic
edit.
But after adding support for moving messages between streams it can be
the case that we are neither changing topic nor content and just moving
streams, and the original code raises error if this is done after the
time limit of editing topics, which is wrong.
This commit fixes this by actually checking 'topic_name is not None'.
This will offer users who are self-hosting to adjust
this value. Moreover, this will help to reduce the
overall time taken to test `test_markdown.py` (since
this can be now overridden with `override_settings`
Django decorator).
This is done as a prep commit for #18641.
d66cbd2832 added these mentioning
"always_notify" for some reason, but always_notify clearly isn't a real
thing in this context so the comments need to be fixed to eliminate this
potential source of confusion.
These checks are more related to the API than the editability
or permissions logic, so it makes sense to handle them first
before further processing the request.
Also split the main test class to separate out the tests for
this logic.
This also simplifies some tests by reducing the data setup
required to reach failure.
Tweaked by tabbott to avoid losing the topic_name.strip().
modified_user=sub_info.user and modified_stream=sub_info.stream, added
by commit 6d1f9de7d3 (#16553), were
always coming from the last entry in the loop above, not from the
enclosing list comprehension.
Found by the Pylint rule undefined-loop-variable.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
The old name `push_notify_user_ids` was misleading, because
it does not contain user ids which should be notified for
the current message, but rather user ids who have the online
push notifications setting enabled.
When the Tornado server is restarted during an upgrade, if
server has old events with the `push_notify_user_ids` fields,
the server will throw error after this rename. Hence, we need
to explicitly handle such cases while processing the event.
We should only show the referrer name in subject of invitation emails,
and show only 'Zulip' in the 'From' header. This helps in preventing
the email from being marked as suspicious by the detection systems
when they see an employee's name as sender of an email sent from an
unrelated domain.
The behavior is already the same for reminder invitation emails where
we do not show name and only 'Zulip' in the 'From' header.
Fixes#18256.
Commit 1a7ddd9ea3 “Fix
UserActivityInterval overlap bug” introduced a mathematically
incorrect assertion about how intervals work. There’s a third way two
intervals could overlap: both the start and end of the old interval
could be inside the new interval. This probably can’t happen here
because the old interval should be at least as long as the new
interval. However, a correct overlap test can be formulated in a
simpler way anyway.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Convert this function that absolutely makes a stream web public.
We already have do_change_stream_invite_only to convert
streams to public and private streams.
We also update all the fields that should be set when a stream
is made web public.
After re-assignment, mypy will still think the type of
`widget_content` to be `str`, not `Dict`. So we need to
create a new variable.
This is a prep change for stronger type checking in this
code.
Previously only admins were allowed to move messages between streams
and admins are allowed to post in any stream irresepctive of stream
post policy, so there was no need to check for stream post policy.
But as we now allow other members to also move messages, we need
to check whether the user who is moving the message is allowed
to post to the target stream (i.e. stream to which the messages
are being moved) and thus we allow moving messages only if the
user is allowed to post in target stream.
This new function optimizes how we fetch subscriptions
for streams. Basically, it excludes most long-term-idle
users from the query.
With 8k users, of which all but 400 are long term idle,
this speeds up get_recipient_info from about 150ms
to 50ms.
Overall this change appears to save a factor of 2-3 in the backend
processing time for sending or editing a message in large, public
streams in chat.zulip.org (at 18K users today).
We combine the two loops into one, so that we
can check our flags before creating the
UserMessageList object.
And we lift a few calculations out of the loop.
For 8k users, with 95% long-term-idle, this was
about a 10x speedup for me. (~30ms -> 3ms)
As discussed in the comment, this is a critical scalability
optimization for organizations with thousands of users.
With substantial comment updates by tabbott.
Anders Kaseorg
Alex Vandiver
Sahil Batra
Hemanth V. Alluri
Abhijeet Prasad Bodas
Riken Shah
PIG208
Mateusz Mandera
Tim Abbott
akshatdalton
m-e-l-u-h-a-n
Gaurav Pandey
sahil839
Vishnu KS
Steve Howell
Aman Agrawal