Commit Graph

46 Commits

Author SHA1 Message Date
Sahil Batra 1fdffaec73 message_send: Check group mention permission when sending message.
This commit adds backend code to check whether user has permission
to mention a group while sending message as per the can_mention_group
setting of the group.

Fixes a part of #25927.
2023-06-30 17:28:33 -07:00
Lauryn Menard d3f7cfccbc zerver: Update comments with "private message" or "PM".
Updates comments/doc-strings that use "private message" or "PM" in
files in the `/zerver` directory to instead use "direct message".
2023-06-23 11:24:13 -07:00
Prakhar Pratyush e71d3ada87 settings: Add wildcard mention notifications for the followed topics.
This commit makes it possible for users to control the wildcard
mention notifications for messages sent to followed topics
via a global notification setting.

There is no support for configuring this setting
through the UI yet.
2023-06-13 18:01:41 -07:00
Prakhar Pratyush d73c715dc2 settings: Add push notifications for the followed topics.
This commit makes it possible for users to control
the push notifications for messages sent to followed topics
via a global notification setting.

There is no support for configuring this setting
through the UI yet.
2023-06-13 18:01:41 -07:00
Prakhar Pratyush 5e5538886f settings: Add email notifications for the followed topics.
This commit makes it possible for users to control
the email notifications for messages sent to followed topics
via a global notification setting.
Although there is no support for configuring this setting
through the UI yet.

Add five new fields to the UserBaseSettings class for
the "followed topic notifications" feature, similar to
stream notifications. But this commit consists only of
the implementation of email notifications.
2023-06-13 18:01:41 -07:00
Tim Abbott 2d3d3f6072 message_send: Don't mark scheduled messages to self as read.
The only reasonable intent for such a scheduled message is to remind
oneself of something at that time, which requires it being unread.

Fixes #25523.
2023-05-12 17:55:46 -07:00
Tim Abbott daf6fb17bf do_send_messages: Use mandatory kwargs pattern. 2023-05-12 17:55:46 -07:00
Aman Agrawal 7bf0793c94 scheduled_messages: Move database related function to /actions.
This would match the pattern we use for other modules and also
shrink the giant message_send.py.
2023-04-28 17:25:00 -07:00
Tim Abbott 027b67be80 presence: Rewrite the backend data model.
This implements the core of the rewrite described in:

For the backend data model for UserPresence to one that supports much
more efficient queries and is more correct around handling of multiple
clients.  The main loss of functionality is that we no longer track
which Client sent presence data (so we will no longer be able to say
using UserPresence "the user was last online on their desktop 15
minutes ago, but was online with their phone 3 minutes ago").  If we
consider that information important for the occasional investigation
query, we have can construct that answer data via UserActivity
already.  It's not worth making Presence much more expensive/complex
to support it.

For slim_presence clients, this sends the same data format we sent
before, albeit with less complexity involved in constructing it.  Note
that we at present will always send both last_active_time and
last_connected_time; we may revisit that in the future.

This commit doesn't include the finalizing migration, which drops the
UserPresenceOld table.
The way to deploy is to start the backfill migration with the server
down and then start the server *without* the user_presence queue worker,
to let the migration finish without having new data interfering with it.
Once the migration is done, the queue worker can be started, leading to
the presence data catching up to the current state as the queue worker
goes over the queued up events and updating the UserPresence table.

Co-authored-by: Mateusz Mandera <mateusz.mandera@zulip.com>
2023-04-26 14:26:47 -07:00
Lauryn Menard 42d9560413 message: Use `recipient_type_name` for API message type references.
Refactors instances of `message_type_name` and `message_type`
that are referring to API message type value ("stream" or
"private") to use `recipient_type_name` instead.

Prep commit for adding "direct" as a value for endpoints with a
`type` parameter to indicate whether the message is a stream or
direct  message.
2023-04-18 12:29:33 -07:00
Aman Agrawal c0ef1c360a message_send: Edit scheduled message if its ID is present.
If the ID of the scheduled message is passed by the client, we
edit the existing scheduled message instead of creating a new one.

However, this will soon be moved into its own API endpoint.
2023-04-14 17:38:37 -07:00
Aman Agrawal b63f440fb1 models: Allow scheduled msgs to store rendered content.
This is required by the client to display a list of currently
scheduled messages.
2023-04-14 17:38:37 -07:00
Abhijeet Prasad Bodas ba0dd70904 user_topics: Introduce visibility policy enum subclass in models.py.
This commit gives more readable code than using the `VISIBILITY_POLICY`
prefix.
This commit does not alter the database schema.
2023-03-14 17:50:45 -07:00
Kartik Srivastava ce5d13f9b2 message_send: Handle notifications for UNMUTED topic in a muted stream.
This commit adds 'visibility_policy' as a
parameter to user_allows_notifications_in_StreamTopic
function.

This adds logic inside the user_allows_notifications_in_StreamTopic
function, to not return False when a stream is muted
but the topic is UNMUTED.

Adds a method `user_id_to_visibility_policy_dict`
to 'StreamTopicTarget' class to fetch
(user_id => visibility_policy) in single db query.

Co-authored-by: Kartik Srivastava <kaushiksri0908@gmail.com>
Co-authored-by: Prakhar Pratyush <prakhar841301@gmail.com>
2023-03-06 19:15:45 -08:00
Lauryn Menard a0fd7b2afc private-messages: Update translated backend strings use "direct message".
Updates user-facing translated strings containing "private message" on
the backend to use "direct message" instead.
2023-02-24 11:47:26 -08:00
Alex Vandiver f78c2eab2c message_send: Use stream object for name in "no subscribers" message.
In the case where a stream existed but had no subscribers, the error
message used to send to the owner always used `stream_name`, which
may have been None.

Switch to using `stream.name` rather than `stream_name` for this case.
2023-02-23 11:02:55 -08:00
Mateusz Mandera 4ba4305ff9 presence: Change server-side OFFLINE_THRESHOLD_SECS to 140s.
This old 300s value was meaningfully used in 2 places:
1. In the do_change_user_settings presence_enabled codepath when turning
   a user invisible. It doesn't matter there, 140s is just since the
   point is to make clients see this user as offline. And 140s is the
   threshold used by clients (see the presence.js constant).
2. For calculating whether to set "offline" "status" in
   result["presence"]["aggregated"] in get_presence_backend. It's fine
   for this to become 140s, since clients shouldn't be looking at the
   status value anymore anyway and just do their calculation based on
   the timestamps.
2023-02-21 12:10:43 -08:00
Lauryn Menard 06dd6f8254 message-send: Deduplicate check of `settings.MAX_MESSAGE_LENGTH`.
Removes the initial check in `_internal_prep_message` of the length
of the message content because the `check_message` in the try block
will call `normalize_body` on the message content string, which
does a more robust check of the message content (empty string, null
bytes, length). If the message content length exceeds the value of
`settings.MAX_MESSAGE_LENGTH`, then it is truncated based on that
value. Updates associated backend test for these changes.

The removed length check would truncate the message content with a
hard coded value instead of using the value for
`settings.MAX_MESSAGE_LENGTH`.

Also, removes an extraneous comment about removing null bytes. If
there are null bytes in the message content, then `normalize_body`
will raise an error.

Note that the previous check had intentionally reduced any message over
the 10000 character limit to 3900 characters, with the code in
question dating to 2012's 100df7e349.

The 3900 character truncating rule was implemented for incoming emails
with the email gateway, and predated other features to help with
overly long messages (better stripping of email footers via Talon,
introduced in f1f48f305e, and
condensing, introduced in c92d664b44).
While we could preserve that logic if desired, it likely is no longer
a necessary or useful variation from our usual truncation rules.
2023-02-20 12:55:15 -08:00
Alex Vandiver 8998aa00cd Revert "create_user: Use transaction.atomic decorator for do_create_user."
This reverts commit 851d68e0fc.

That commit widened how long the transaction is open, which made it
much more likely that after the user was created in the transaction,
and the memcached caches were flushed, some other request will fill
the `get_realm_user_dicts` cache with data which did not include the
new user (because it had not been committed yet).

If a user creation request lost this race, the user would, upon first
request to `/`, get a blank page and a Javascript error:

    Unknown user_id in get_by_user_id: 12345

...where 12345 was their own user-id.  This error would persist until
the cache expired (in 7 days) or something else expunged it.

Reverting this does not prevent the race, as the post_save hook's call
to flush_user_profile is still in a transaction (and has been since
168f241ff0), and thus leaves the potential race window open.
However, it much shortens the potential window of opportunity, and is
a reasonable short-term stopgap.
2023-02-17 21:20:52 -05:00
Abhijeet Prasad Bodas 9bc49ea4be message_send: Extract out stream notification user-set calculation logic.
This will allow us to re-use this logic later, when we add support for
re-checking notification settings just before sending email/push
notifications to the user.

Also, since this is essentially part of the notifiability logic,
this better belongs to `notification_data.py` and this change will
hopefully reduce the reading complexity of the message-send codepath.
2023-02-10 22:47:30 -08:00
Abhijeet Prasad Bodas 167ea63c1f message_send: Add comment on wildcard_mention_user_ids optimization.
The earlier commit explained the code, and not why we have this
optimization in the first place.
2023-02-10 22:47:30 -08:00
Anders Kaseorg 41aadc8c63 message_send: Convert RecipientInfoResult from TypedDict to dataclass.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-02-10 15:51:22 -08:00
Abhijeet Prasad Bodas 9d1c131dc6 muted users: Make file naming consistent.
This makes the names of the relevant files consistant with the database
model name as well as the frontend JS files.
2023-02-10 15:39:57 -08:00
Anders Kaseorg df001db1a9 black: Reformat with Black 23.
Black 23 enforces some slightly more specific rules about empty line
counts and redundant parenthesis removal, but the result is still
compatible with Black 22.

(This does not actually upgrade our Python environment to Black 23
yet.)

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-02-02 10:40:13 -08:00
Sahil Batra 851d68e0fc create_user: Use transaction.atomic decorator for do_create_user.
We change the do_create_user function to use transaction.atomic
decorator instead of using with block. Due to this change, all
send_event calls are made inside transaction.on_commit.

Some other changes -
- Remove transaction.atomic decorator from send_inital_realm_messages
since it is now called inside a transaction.
- Made changes in tests which tests message events and notifications
to make sure on_commit callbacks are executed.
2023-01-26 10:49:19 -08:00
Anders Kaseorg 4eda29bd86 ruff: Fix RUF005 Consider spread instead of concatenation.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-26 10:16:30 -08:00
Prakhar Pratyush 1a400b21e7 notifications: Fix missed message email notifications of welcome bot.
A missed message email notification, where the message is the welcome
message sent by the welcome bot on account creation, get sent when
the user somehow not focuses the browser tab during account creation.

No missed message email or push notifications should be sent for the
messages generated by the welcome bot.

'internal_send_private_message' accepts a parameter
'disable_external_notifications' and is set to 'True' when the sender
is 'welcome bot'.

A check is introduced in `trivially_should_not_notify`, not to notify
if `disable_external_notifications` is true.

TestCases are updated to include the `disable_external_notifications`
check in the early (False) return patterns of `is_push_notifiable` and
`is_email_notifiable`.

One query reduced for both `test_create_user_with_multiple_streams`
and `test_register`.
Reason: When welcome bot sends message after user creation
`do_send_messages` calls `get_active_presence_idle_user_ids`,
`user_ids` in `get_active_presence_idle_user_ids` remains empty if
`disable_external_notifications` is true because `is_notifiable` returns
false.
`get_active_presence_idle_user_ids` calls `filter_presence_idle_user_ids`
and since the `user_ids` is empty, the query inside the function doesn't
get executed.

MissedMessageHookTest updated.

Fixes: #22884
2023-01-24 11:16:21 -08:00
Prakhar Pratyush b40bbd6ca8 message_send: Refactor internal_send_*, internal_prep_* & _internal_prep_*.
This commit makes all the parameters after 'content' in
'internal_send_*', 'internal_prep_*' and '_internal_prep_*'
a mandatory keyword argument to increase code readability.
2023-01-24 11:16:21 -08:00
Anders Kaseorg b0e569f07c ruff: Fix SIM102 nested `if` statements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-23 11:18:36 -08:00
Abhijeet Prasad Bodas 04e6b04213 message send: Ignore alert words for presence idle users.
For alert words, we currently don't send email/push notifications --
only desktop notifications. Thus, we don't need to consider alert words
here, since desktop notifications do not utilize the presence status
calculated at this stage.

Tested manually that alert word desktop notifications work as expected.

When we implement email/push notifications for alert words (issues #5137
and #13127), we can add new fields like
`notifications_data.alert_word_email_notify`, similar to the existing
`notifications_data.wildcard_mention_email_notify`, which will allow us
to keep the alert word notifiability check inside the dataclass, similar
to how the mentions checks are done currently. So, even when that
feature is implemented, the code which this commit removes would be
unnecessary.
2023-01-06 17:08:20 -08:00
Anders Kaseorg 73c4da7974 ruff: Fix N818 exception name should be named with an Error suffix.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-17 16:52:00 -08:00
Mateusz Mandera 78a325ac58 internal_send_private_message: Fix edge case of message to system bot.
I don't think this is used anywhere outside of tests, but we should have
this logic correct. If this function is used to send a message from a
user to a cross-realm bot, the message.realm should be the realm of the
user.

In the normal case, where a user send a message to a cross-realm bot
through the API is already handled correctly, this bug is unrelated.
2022-10-31 10:23:57 -07:00
Mateusz Mandera 00b3546c9f models: Add denormalized .realm column to Message.
This commit adds the OPTIONAL .realm attribute to Message
(and ArchivedMessage), with the server changes for making new Messages
have this set. Old Messages still have to be migrated to backfill this,
before it can be non-nullable.

Appropriate test changes to correctly set .realm for Messages the tests
manually create are included here as well.
2022-10-07 10:09:38 -07:00
Zixuan James Li 7fd8d77ce0 typing: Import ValuesQuerySet alias from django_stubs_ext.
This saves us from using a conditional import.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-10-05 16:15:56 -07:00
Kartik Srivastava 409ef040bb stream_topic: Refactor user_ids_muting_topic.
This refactors and renames user_ids_muting_topic to accept a parameter
'visibility_policy' and fetch user IDs that have a specific
visibility_policy(provided as the parameter) set for a topic.
2022-09-27 17:18:48 -07:00
Mateusz Mandera 01d659b672 internal_prep_private_message: Remove redundant realm arg.
This is redundant and is taken based on recipient_user anyway.
2022-09-06 15:03:17 -07:00
Zixuan James Li 9f2dfd848f message_send: Add service_queue_events to SendMessageRequest.
We should not monkey-patch message when unnecessary. Adding
`service_queue_events` to `SendMessageRequests` suits our need to type
safety here.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-30 18:03:56 -07:00
Anders Kaseorg b945aa3443 python: Use a real parser for email addresses.
Now that we can assume Python 3.6+, we can use the
email.headerregistry module to replace hacky manual email address
parsing.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-07-29 15:47:33 -07:00
Anders Kaseorg 7d8be670a5 message_send: Remove TypeGuard.
The type safety of a TypeGuard is unchecked by mypy.  While this
particular TypeGuard is safe given the current context, one could
imagine future changes that make it unsafe, so it’s preferable to
avoid unchecked constructs whenever possible.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-07-16 10:44:16 -07:00
Zixuan James Li b210c22329 message_send: Extract ActiveUserDict for accurate typing.
django-stubs types the return type of query.values(...) as a TypedDict.
This makes Dict[str, Any] that we have been using incompatible with it.

We use TypeGuard to ensure that `service_bot_tuples` is correctly
inferred to be `Tuple[int, int]` instead of `Tuple[int, Optional[int]]`.
Given that `bot_type` is optional for `ActiveUserDict`, we need to
narrow `row` to `ActiveBotUserDict` to make sure that `bot_type` is
non-optional. An advantage of this approach is that no assertions or
type casts are needed.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-15 14:02:40 -07:00
Zixuan James Li ab1bbdda65 typing: Broaden type annotations for QuerySet compatibility.
To explain the rationale of this change, for example, there is
`get_user_activity_summary` which accepts either a `Collection[UserActivity]`,
where `QuerySet[T]` is not strictly `Sequence[T]` because its slicing behavior
is different from the `Protocol`, making `Collection` necessary.

Similarily, we should have `Iterable[T]` instead of `List[T]` so that
`QuerySet[T]` will also be an acceptable subtype, or `Sequence[T]` when we
also expect it to be indexed.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-07 11:27:42 -07:00
Anders Kaseorg df69e1d979 mypy: Enable truthy-bool errors.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-06-23 19:22:12 -07:00
Alex Vandiver cd9c69cd12 message_send: Remove unnecessary user_ids argument.
cfcbf58cd1 rightly removed the use of `user_ids` in
`render_markdown`, which in turn makes it unnecessary in
`render_incoming_message`.

Remove the unnecessary parameter from `render_incoming_message`.
2022-05-04 14:45:18 -07:00
Anders Kaseorg a2825e5984 python: Use Python 3.8 typing.{Protocol,TypedDict}.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-27 12:57:49 -07:00
Alex Vandiver 351bdfaf78 preview: Use cache only as a non-durable cache, not an IPC.
The `get_link_embed_data` / `link_embed_data_from_cache` pair as
introduced in c93f1d4eda uses the cache
as a temporary store inside of the `embed_links` worker; this means
that it must be durable storage, or the worker will stall and re-fetch
the same links to preview them.

Switch to plumbing through the fetched URL embed data as an parameter
to the Markdown evaluation which uses them, rather than using the
cache as an intermediary.  This frees up the cache to be merely a
non-durable cache.

As a side-effect, this removes get_cache_with_key, and
link_embed_data_from_cache which was its only callsite.
2022-04-15 14:48:12 -07:00
Anders Kaseorg 975066e3f0 actions: Split out zerver.actions.message_send.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:34 -07:00