Commit Graph

146 Commits

Author SHA1 Message Date
Prakhar Pratyush c798d192dc message_send: Update do_send_messages codepath to send event on commit.
Earlier, we were using 'send_event' & 'queue_json_publish' in
'do_send_messages' which can lead to a situation where we enqueue
events but the transaction fails at a later stage.

Events should not be sent until we know we're not rolling back.
2024-05-19 23:18:43 -07:00
Sahil Batra e78d0aacaf tests: Use NamedUserGroup for queries. 2024-04-26 17:03:09 -07:00
Lauryn Menard 0b5f325e48 exceptions: Update translated errors for stream to channel rename.
In zerver/lib/exceptions.py, updates translated error strings
to use channel instead of stream.

Note that the STREAM_WILDCARD_MENTION_NOT_ALLOWED and the
STREAM_DOES_NOT_EXIST error codes, and any additional fields
included with those errors, are not changed in these updates.

Part of stream to channel rename project.
2024-04-24 14:35:05 -07:00
Lauryn Menard 91ffb548cc streams: Update translated errors for stream to channel rename.
Updates translated JsonableError strings that relate to streams
to use channel instead of stream. Separated from other error string
updates as this is a dense area of changes for this rename.

Part of stream to channel rename project.
2024-04-24 14:35:05 -07:00
Lauryn Menard c1f1dc4b25 message-send: Update translated errors for stream to channel rename.
In zerver/actions/message_send.py, updates translated error strings
to use channel instead of stream.

Also, updates the messages sent to bot owners when a stream doesn't
exist or has no subscribers.

Part of stream to channel rename project.
2024-04-24 14:35:05 -07:00
Lauryn Menard 01b59c5aa2 message-type: Add support for "channel" as value for type parameter.
For endpoints with a type parameter to indicate whether a message is
a direct or stream message, adds support for passing "channel" as a
value for stream messages.

Part of stream to channel rename project.
2024-04-11 09:40:25 -07:00
Anders Kaseorg 570f3dd447 python: Reformat with Ruff formatter.
https://docs.astral.sh/ruff/formatter/

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-02-29 17:07:16 -08:00
sayyedarib 5c993f0479 direct_message: Allow DMs to bots and self regardless of policy.
Previously, in DM disabled org messaging to bot was not working when
starting new conversation and adding bot as recipient because of not
updating on recipient change. And secondly, self messaging was not
allowed.
This commit ensures that the DM to bot and self are allowed irrespective
of dm restrictions.

tests: Verify DMs adhere to DM restriction policy.

Fixes #28412

Signed-off-by: sayyedarib <sayyedaribhussain4321@gmail.com>
2024-02-15 16:13:36 -08:00
Alex Vandiver 23baabba86 message: Merge unnecessary cache_transformer step.
Having a non-identity `cache_transformer` is no different from running
it on every row of the query_function.  Simplify understanding of the
codepath used in caching by merging the pieces of code.
2024-02-14 12:27:03 -08:00
Alex Vandiver 737a751f5c message: Split MessageDict and friends into its own file. 2024-02-14 12:27:03 -08:00
Prakhar Pratyush 676c07be5b tests: Rename *topic local variables to *topic_name.
This is preparatory work towards adding a Topic model.
We plan to use the local variable name as 'topic' for
the Topic model objects.

Currently, we use *topic as the local variable name for
topic names.

We rename local variables of the form *topic to *topic_name
so that we don't need to think about type collisions in
individual code paths where we might want to talk about both
Topic objects and strings for the topic name.
2024-01-15 09:40:43 -08:00
Prakhar Pratyush 1eef052bd1 actions: Rename *topic local variables to *topic_name.
This is preparatory work towards adding a Topic model.
We plan to use the local variable name as 'topic' for
the Topic model objects.

Currently, we use *topic as the local variable name for
topic names.

We rename local variables of the form *topic to *topic_name
so that we don't need to think about type collisions in
individual code paths where we might want to talk about both
Topic objects and strings for the topic name.
2024-01-15 09:40:43 -08:00
Sahil Batra c0c9623ae4 message: Allow system bots to mention group if everyone else can.
We now allow system bots to mention a group if can_mention_group
setting is set to "role:everyone" group and not when it is set
to some other group.
2024-01-10 14:57:21 -08:00
Anders Kaseorg 4aa2d76bea models: Extract zerver.models.streams.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-12-16 22:08:44 -08:00
Anders Kaseorg 51f1dc257d models: Extract zerver.models.recipients.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-12-16 22:08:44 -08:00
Anders Kaseorg cd96193768 models: Extract zerver.models.realms.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-12-16 22:08:44 -08:00
Anders Kaseorg 7001a0dfc0 models: Extract zerver.models.groups.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-12-16 22:08:44 -08:00
Anders Kaseorg 45bb8d2580 models: Extract zerver.models.users.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-12-16 22:08:44 -08:00
Anders Kaseorg 37a9c4501f models: Extract zerver.models.constants.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-12-16 22:08:44 -08:00
Vector73 2e71ec78e3 settings: Add "Automatically follow topics where I'm mentioned" setting.
Fixes: #26795
2023-12-11 23:26:11 -08:00
Sahil Batra 198568522a message: Do not include details of inaccessible users in message data.
This commit adds code to not include original details of senders like
name, email and avatar url in the message objects sent through events
and in the response of endpoint used to fetch messages.

This is the last major commit for the project to add support for
limiting guest access to an entire organization.

Fixes #10970.
2023-12-09 17:23:16 -08:00
Anders Kaseorg 8a7916f21a python: Consistently use from…import for datetime.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-12-05 12:01:18 -08:00
Prakhar Pratyush 49388d5d3d topic_mentions: Fix restriction rule for @-topic mentions.
Now, the topic wildcard mention follows the following
rules:
* If the topic has less than 15 participants , anyone
can use @ topic mentions.
* For more than 15, the org setting 'wildcard_mention_policy'
determines who can use @ topic mentions.

Earlier, topic wildcard mentions followed the same restriction
as stream wildcard mentions, which was incorrect.

Fixes part of #27700.
2023-11-23 12:52:25 -08:00
Sahil Batra e4a97dd3ac message: Restrict sending DMs to inaccessible users.
This commit adds code to not allow guest users to send DMs
to users they cannot access.
2023-11-21 23:58:45 -08:00
Prakhar Pratyush b0ef76bf27 topic_mentions: Set 'topic_wildcard_mentioned' flag on @topic mention.
Earlier, the 'wildcard_mentioned' flag was set for both the
stream and topic wildcard mentions.

Now, the 'topic_wildcard_mentioned' flag is set for topic
wildcard mentions, and the 'wildcard_mentioned' flag is set for
stream wildcard mentions.

We will rename the 'wildcard_mentioned' flag to
'stream_wildcard_mentioned' in a later commit.
2023-11-02 09:25:51 -07:00
Sahil Batra e458b73a01 user_groups: Move constants for system group names to a new class.
This commit moves constants for system group names to a new
"SystemGroups" class so that we can use these group names
in multiple classes in models.py without worrying about the
order of defining them.
2023-11-01 10:42:56 -07:00
Prakhar Pratyush 17a0304309 send_message: Add an optional parameter in the success response.
Add an optional `automatic_new_visibility_policy` enum field
in the success response to indicate the new visibility policy
value due to the `automatically_follow_topics_policy` and
`automatically_unmute_topics_in_muted_streams_policy` user settings
during the send message action.

Only present if there is a change in the visibility policy.
2023-10-17 15:38:16 -07:00
Prakhar Pratyush 58568a60d6 settings: Add automatically follow and unmute topics policy settings.
This commit adds two user settings, named
* `automatically_follow_topics_policy`
* `automatically_unmute_topics_in_muted_streams_policy`

The settings control the user's preference on which topics they
will automatically 'follow' or 'unmute in muted streams'.

The policies offer four options:
1. Topics I participate in
2. Topics I send a message to
3. Topics I start
4. Never (default)

There is no support for configuring the settings through the UI yet.
2023-10-04 13:04:29 -07:00
Prakhar Pratyush 379a08eb1e message_send: Fix wildcard_mentioned flag unset for few participants.
For topic wildcard mentions, the 'wildcard_mentioned' flag is set
for those user messages having 'user_profile_id' in
'topic_participant_user_ids', i.e. all topic participants.

Earlier, the flag was set if the 'user_profile_id' exists in
'all_topic_wildcard_mention_user_ids'.
'all_topic_wildcard_mention_user_ids' contains the ids of those
users who are topic participants and have enabled notifications
for '@topic' mentions.

The earlier approach was incorrect, as it would set the
'wildcard_mentioned' flag only for those topic participants
who have enabled the notifications for '@topic' mention instead
of setting the flag for all the topic participants.

The bug was introduced in 4c9d26c.
2023-08-16 11:31:56 -07:00
Steve Howell 51db22c86c per-request caches: Add per_request_cache library.
We have historically cached two types of values
on a per-request basis inside of memory:

    * linkifiers
    * display recipients

Both of these caches were hand-written, and they
both actually cache values that are also in memcached,
so the per-request cache essentially only saves us
from a few memcached hits.

I think the linkifier per-request cache is a necessary
evil. It's an important part of message rendering, and
it's not super easy to structure the code to just get
a single value up front and pass it down the stack.

I'm not so sure we even need the display recipient
per-request cache any more, as we are generally pretty
smart now about hydrating recipient data in terms of
how the code is organized. But I haven't done thorough
research on that hypotheseis.

Fortunately, it's not rocket science to just write
a glorified memoize decorator and tie it into key
places in the code:

    * middleware
    * tests (e.g. asserting db counts)
    * queue processors

That's what I did in this commit.

This commit definitely reduces the amount of code
to maintain. I think it also gets us closer to
possibly phasing out this whole technique, but that
effort is beyond the scope of this PR. We could
add some instrumentation to the decorator to see
how often we get a non-trivial number of saved
round trips to memcached.

Note that when we flush linkifiers, we just use
a big hammer and flush the entire per-request
cache for linkifiers, since there is only ever
one realm in the cache.
2023-08-11 11:09:34 -07:00
Sahil Batra 91a58d026b models: Remove get_huddle_recipient and use get_or_create_huddle.
This commit removes get_huddle_recipient function and we now use
get_or_create_huddle in get_recipient_from_user_profiles.

As a result of this change, we do not fetch the recipient from
Huddle object but instead get it using the "id" and "recipient_id"
fields available from Huddle object like we do for a personal
message. This change allows us to not fetch recipient object
using select_related when querying the Huddle object.
2023-08-10 17:35:43 -07:00
Anders Kaseorg 562a79ab76 ruff: Fix PERF401 Use a list comprehension to create a transformed list.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-08-07 17:23:55 -07:00
Anders Kaseorg e932e2ce52 ruff: Fix UP032 Use f-string instead of `format` call.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-08-02 15:58:55 -07:00
Anders Kaseorg 3b09197fdf ruff: Fix RUF015 Prefer `next(...)` over single element slice.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-07-23 15:20:53 -07:00
Prakhar Pratyush 4c9d26ce17 mention: Send notifications for @topic wildcard mentions.
This commit completes the notifications part of the @topic
wildcard mention feature.

Notifications are sent to the topic participants for the
@topic wildcard mention.
2023-07-17 09:39:24 -07:00
Steve Howell 89381a8072 cache: Eliminate get-stream-by-name cache.
We remove the cache functionality for the
get_realm_stream function, and we also change it to
return a thin Stream object (instead of calling
select_related with no arguments).

The main goal here is to remove code complexity, as we
have been prone to at least one caching validation bug
related to how Realm and UserGroup interact. That
particular bug was more theoretical than practical in
terms of its impact, to be clear.

Even if we were to be perfectly disciplined about only
caching thin stream objects and always making sure to
delete cache entries when stream data changed, we would
still be prone to ugly situations like having
transactions get rolled back before we delete the cache
entry. The do_deactivate_stream is a perfect example of
where we have to consider the best time to unset the
cache. If you unset it too early, then you are prone to
races where somebody else churns the cache right before
you update the database. If you set it too late, then
you can have an invalid entry after a rollback or
deadlock situation. If you just eliminate the cache as
a moving part, that whole debate is moot.

As the lack of test changes here indicates, we rarely
fetch streams by name any more in critical sections of
our code.

The one place where we fetch by name is in loading the
home page, but that is **only** when you specify a
stream name. And, of course, that only causes about an
extra millisecond of time.
2023-07-11 13:45:40 -07:00
Prakhar Pratyush 179d5cb37d mention: Replace 'wildcards' with 'stream_wildcards'.
This prep commit replaces the 'wildcard' keyword in the codebase
with 'stream_wildcard' at some places for better readability, as
we plan to introduce 'topic_wildcards' as a part of the
'@topic mention' project.

Currently, 'wildcards = ["all", "everyone", "stream"]' which is an
alias to mention everyone in the stream, hence better renamed as
'stream_wildcards'.

Eventually, we will have:
'stream_wildcard' as an alias to mention everyone in the stream.
'topic_wildcard' as an alias to mention everyone in the topic.
'wildcard' refers to 'stream_wildcard' and 'topic_wildcard' as a whole.
2023-07-03 22:03:17 -07:00
Sahil Batra 1fdffaec73 message_send: Check group mention permission when sending message.
This commit adds backend code to check whether user has permission
to mention a group while sending message as per the can_mention_group
setting of the group.

Fixes a part of #25927.
2023-06-30 17:28:33 -07:00
Lauryn Menard d53b854a7c backend-tests: Update "private message" or "PM" to "direct message".
Updates comments and test strings/names with "private message" or
"PM" to use "direct message" instead.
2023-06-23 11:24:13 -07:00
Aman Agrawal cddf25656f test_message_send: Move out scheduled message tests. 2023-04-28 17:25:00 -07:00
Lauryn Menard 2c043c6242 message-type: Add support for "direct" as value for type parameter.
For endpoints with a `type` parameter to indicate whether the message
is a stream or direct message, `POST /typing` and `POST /messages`,
adds support for passing "direct" as the preferred value for direct
messages, group and 1-on-1.

Maintains support for "private" as a deprecated value to indicate
direct messages.

Fixes #24960.
2023-04-18 12:29:33 -07:00
Lauryn Menard 42d9560413 message: Use `recipient_type_name` for API message type references.
Refactors instances of `message_type_name` and `message_type`
that are referring to API message type value ("stream" or
"private") to use `recipient_type_name` instead.

Prep commit for adding "direct" as a value for endpoints with a
`type` parameter to indicate whether the message is a stream or
direct  message.
2023-04-18 12:29:33 -07:00
Aman Agrawal a06f3d26d0 scheduled_messages: Add endpoints to fetch and delete them. 2023-04-14 17:38:37 -07:00
Aman Agrawal c0ef1c360a message_send: Edit scheduled message if its ID is present.
If the ID of the scheduled message is passed by the client, we
edit the existing scheduled message instead of creating a new one.

However, this will soon be moved into its own API endpoint.
2023-04-14 17:38:37 -07:00
Aman Agrawal b63f440fb1 models: Allow scheduled msgs to store rendered content.
This is required by the client to display a list of currently
scheduled messages.
2023-04-14 17:38:37 -07:00
Prakhar Pratyush d96048b0af test_classes: Rename and refactor 'tornado_redirected_to_list'.
This commit renames the 'tornado_redirected_to_list' context
manager to 'capture_send_event_calls' to improve readability.

It also refactors the function to yield a list of events
instead of passing in a list data structure as a parameter
and appending events to it.
2023-04-07 09:45:26 -07:00
Anders Kaseorg 2d9b2a2a05 models: Remove type prefixes from __str__ values.
The Django convention is for __repr__ to include the type and __str__
to omit it.  In fact its default __repr__ implementation for models
automatically adds a type prefix to __str__, which has resulted in the
type being duplicated:

    >>> UserProfile.objects.first()
    <UserProfile: <UserProfile: emailgateway@zulip.com <Realm: zulipinternal 1>>>

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-03-08 22:56:55 -08:00
Lauryn Menard 084dd216f0 tests: Remove ignored `realm_str` parameter from message send test.
In commit 8181ec4b56, we removed the `realm_str` as a parameter
for `send_message_backed`. This removes a missed test that included
this as a parameter for that endpoint/function.
2023-03-06 09:41:53 -08:00
Sahil Batra 3950a8e19d test_helpers: Rename reset_emails_in_zulip_realm.
This commit renames reset_emails_in_zulip_realm function to
reset_email_visibility_to_everyone_in_zulip_realm which makes
it more clear to understand what the function actually does.

This commit also adds a comment explaining what this function
does.
2023-03-01 12:17:11 -08:00
Mateusz Mandera f0f02d05ab send_message_backend: Remove the realm_str API param.
This already became useless in 6e11754642,
as detailed in the API changelog entry here. At this point, we should
eliminate this param and the weird code around it.

This commit also deletes the associated tests added in
6e11754642, since with realm_str removed,
they make no sense anymore (and actually fail with an OpenAPI error due
to using params not used in the API). Hypothetically they could be
translated to use the subdomain= kwarg, but that also doesn't make
sense, since at that point they'd be just testing the case of a user
making an API request on a different subdomain than their current one
and that's just redundant and already tested generally in
test_decorators.
2023-02-28 12:12:15 -08:00