Commit Graph

729 Commits

Author SHA1 Message Date
Sahil Batra 86f73fcb3d user_groups: Create NamedUserGroup objects when creating new groups. 2024-04-26 17:03:09 -07:00
Lauryn Menard 9be4d07442 i18n: Update translated errors for stream to channel rename.
Updates various areas of the backend code that generate
JsonableErrors with translated strings to use channel
instead of stream.

Part of stream to channel rename project.
2024-04-24 14:35:05 -07:00
Lauryn Menard 51b3ef8ee3 streams-events: Update notifications for stream to channel rename.
Updates notification messages that are sent to "stream events"
topic when a permission or policy setting is changed to use channel
instead of stream. Also, updates some strings that were not marked
for translation in the message that was sent when the retention
policy was changed.

Updates notification messages that are sent when a stream/channel
is created.

Updates notification messages that are sent when a user is
subscribed to stream/channel(s).

Part of stream to channel rename project.
2024-04-24 14:35:05 -07:00
Lauryn Menard 91ffb548cc streams: Update translated errors for stream to channel rename.
Updates translated JsonableError strings that relate to streams
to use channel instead of stream. Separated from other error string
updates as this is a dense area of changes for this rename.

Part of stream to channel rename project.
2024-04-24 14:35:05 -07:00
Lauryn Menard c1f1dc4b25 message-send: Update translated errors for stream to channel rename.
In zerver/actions/message_send.py, updates translated error strings
to use channel instead of stream.

Also, updates the messages sent to bot owners when a stream doesn't
exist or has no subscribers.

Part of stream to channel rename project.
2024-04-24 14:35:05 -07:00
Lauryn Menard abc6f5b8a2 i18n: Update translated string variables for stream to channel rename.
Generally updates variables that appear in translated strings that use
"stream" to instead use "channel".

Two exceptions are ErrorCode.STREAM_DOES_NOT_EXIST JsonableErrors as
changing the variable would also change the fields returned by these
errors to clients.

Changes to context variables in emails and variables in onboarding
welcome bot messages are addressed in separate commits.

Part of stream to channel rename project.
2024-04-24 14:35:05 -07:00
tnmkr 5128d8f9af streams: Add creator field.
Adds nullable creator field, containing a reference to the user who
created the stream. When creating a stream, acting user is set as
the creator of the stream. Since API calls to create streams always
have an acting user, this field should always be set when streams
are created using the API.

Because streams can be created with no acting user, this field is
nullable. We try to backfill existing streams using RealmAuditLog table,
but not all streams are guaranteed to have a recorded create log. Thus
this new field is left null when it cannot be backfilled. We also set
this field to null when the creator user is deleted.
2024-04-20 19:03:43 -07:00
Ujjawal Modi 706c380971 audit_log: Update audit log entries created while creating a group.
Earlier a extra audit log entry of type
USER_GROUP_GROUP_BASED_SETTING_CHANGED was made when a new user
group is created. This commit updates the code to not create
that audit log entry.

There is no need to create these entry as we would still
have the required data from the "OLD_VALUE" field in the
audit log entry created when changing the setting and this
also makes it consistent with the entries created for
other operations like stream creation.
2024-04-19 10:18:45 -07:00
Anders Kaseorg 95a1481f99 message_flags: Remove inappropriate use of zerver.lib.timeout.
zerver.lib.timeout abuses asynchronous exceptions, so it’s only safe
to use on CPU computations with no side effects.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-04-18 11:50:38 -07:00
Anders Kaseorg f31579a220 python: Avoid relying on Collection supertype of QuerySet.
QuerySet doesn’t implement __contains__, so it can’t be a subtype of
Container or Collection (https://code.djangoproject.com/ticket/35154).
This incorrect subtyping annotation was removed in
https://github.com/typeddjango/django-stubs/pull/1925, so we need to
stop relying on it before upgrading to django-stubs 5.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-04-16 23:02:16 -07:00
roanster007 68b4298d8e settings: Add option to disable seeing typing notifications.
This commit adds an option to the advanced section of
Preferences settings, that would allow users to choose
whether to receive typing notifications from other
users.

Fixes #29642
2024-04-16 11:38:12 -07:00
Shubham Padia 1abd356a91 custom_profile_fields: Make required field optional during update. 2024-04-15 16:52:11 -07:00
Shubham Padia 8c30b61201 custom_profile_fields: Display_in_profile_summary optional for update. 2024-04-15 16:35:42 -07:00
Shubham Padia 137776b092 custom_profile_fields: Make field_data optional during update. 2024-04-15 16:35:42 -07:00
Shubham Padia c6c392bcc7 custom_profile_fields: Make name & hint optional during update.
Fixes #23022.
Added a new validation function that validates updates to custom
profile fields.
2024-04-15 16:35:39 -07:00
Mateusz Mandera 8038e2322c realm: Change implementation approach for upload_quota_gb.
Most importantly, fixes a bug where a realm with a custom
.upload_quota_gb value (set by changing it in the database via e.g.
manage.py shell) would end up having it lowered while upgrading their
plan via the do_change_realm_plan_type function, which used to just set
it to the value implied by the new plan without caring about whether
that isn't lower than the original limit.

The new approach is cleaner since we don't do db queries by
upload_quota_gb so it's nicer to just generate these dynamically, making
changes to our limit-per-plan rules much easier - skipping the need for
migrations.
2024-04-15 15:08:56 -07:00
Alex Vandiver a49111c89a invites: Use query.count() not len(query). 2024-04-12 17:37:11 -07:00
Alex Vandiver 436dab0e01 messages: Remove use of @overload in access_message.
f92d43c690 added uses of `@overload` to probide multiple type
signatures for `access_message`, based on the `get_user_message`
parameter.  Unfortunately, mypy does not check the function body
against overload signatures, so it allows type errors to go
undetected.

Replace the overloads with two functions, for one of which also
returns the usermessage.  The third form, of only returning if the
usermessage exists, is not in a high-enough performance endpoint that
a third form is worth maintaining; it uses the usermessage form.
2024-04-12 11:41:27 -07:00
Mateusz Mandera c9ca4e68e5 scim: Add config option to disable initial streams for guests.
When an organization (without open ability for anyone to join) invites a
guest user, the invitation prompts allows them to choose whether the
guest should be added to default streams or not. This is useful, because
since we don't have per-role default streams configs, they may want
default streams to be for full Members.

SCIM provisioning doesn't have this control, since a newly provisioned
user gets created via a direct do_create_user call, thus adding them to
the organization's default streams, with no workaround possible aside of
just getting rid of default streams in the organization.

To make provisioning guests in such an organization usable, we add a
simple config option to create them with no streams. It's configured by
adding
```
"create_guests_without_streams": True
```

to the config dict in settings.SCIM_CONFIG.
2024-04-11 12:28:26 -07:00
Sahil Singh 6c2535fe46 bots: Avoid multiple active bots with the same name.
Creating a bot with a name that is already in use
will raise an error. However, by deactivating
the existing bot, creating a new bot with the
same name, and then reactivating the original bot,
it is possible to have multiple bots with the same name.

To fix this, we check if the bot name is already
in use in the active bots list. If it is,
an error will be raised, prompting either the
name of the existing bot to be changed or
the bot to be deactivated.

Co-authored-by: Sujal Shah <sujalshah28092004@gmail.com>
2024-04-10 13:14:25 -07:00
Prakhar Pratyush 2abc25ac4c create_user: Send a group DM to admins for low licenses warning.
Earlier, low licenses warning message was sent to the
"New user announcements" stream.

We plan to disable the stream by default as a part of improving
onboarding experience.

Now, we send a group DM to admins for low licenses warning
to make it independent of the setting. These warning messages
are important and shouldn't be missed.
2024-04-03 12:28:05 -07:00
Prakhar Pratyush db214c8377 message: Add recipient_users parameter to internal_send_huddle_message.
This is a prep commit to add a 'recipient_users' parameter to
the 'internal_send_huddle_message' function.

'emails' is no longer a required parameter. We can use either
of the 'emails' or 'recipient_users' parameter. 'emails' is
eventually used to fetch 'recipient_users', so if the
'recipient_users' is already available we should use that to
skip database query.
2024-04-03 12:28:05 -07:00
roanster007 c7a08f3b77 settings: Add permission to enforce unique names in realm.
Previously, users were allowed to signup or change their names to
those which already existed in the realm.

This commit adds an Organization Permission, that shall enforce
users to use unique names while signing up or changing their
names. If a same or normalized full name is found in realm,
then a validation error is thrown.

Fixes #7830.
2024-04-02 14:55:59 -07:00
Aman Agrawal d21f5c9b75 registration: Ask user how they found Zulip. 2024-04-01 12:44:12 -07:00
Alex Vandiver 7988aad159 user_message: Use INSERT ... ON CONFLICT for historical UM creation.
Rather than use a bulk insert via Django, use the faster
`bulk_insert_all_ums` that we already have.  This also adds a `ON
CONFLICT` clause, to make the insert resilient to race conditions.

There are currently two callsites, with different desired `ON
CONFLICT` behaviours:
 - For `notify_reaction_update`, if the `UserMessage` had already been
   created, we would have done nothing to change it.
 - For `do_update_message_flags`, we would have ensured a specific bit
   was (un)set.

Extend `create_historical_user_messages` and `bulk_insert_all_ums` to
support `ON CONFLICT (...) UPDATE SET flags = ...`.
2024-03-26 10:52:37 -07:00
Alex Vandiver 52e3c8e1b2 user_message: Move create_historical_user_messages to zerver.lib.user_message.
It is not even used in zerver.actions.create_user.
2024-03-26 10:52:37 -07:00
Alex Vandiver f92d43c690 messages: Use overloads to only return a user_message if needed. 2024-03-22 09:30:17 -07:00
John Lu a5cf0ec526
refactor: Replace HUDDLE with DIRECT_MESSAGE_GROUP.
Replaced HUDDLE attribute with DIRECT_MESSAGE_GROUP using VS Code search,
part of a general renaming of the object class.

Fixes part of #28640.

Co-authored-by: JohnLu2004 <JohnLu10212004@gmail.com>
2024-03-21 16:39:33 -07:00
Vector73 f758ca596b custom_profile_fields: Add "required" parameter to the profile fields.
Fixes #28512.
2024-03-21 10:48:54 -07:00
Anders Kaseorg a1a341f0ae ruff: Fix UP032 Use f-string instead of `format` call.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-03-21 08:59:17 -07:00
Prakhar Pratyush 118a7e8d9d zulip_updates: Send zulip updates based on zulip_update_*_level.
This commit adds a management command that will run regularly
as a cron job to send zulip updates to realms based on their
current and latest zulip_update_announcements_level.

For realms with:
* level = None: Send a group DM to admins notifying them about
this new feature & suggestion to set the stream accordingly.

* level = 0:
  * If stream is still not configured, wait for a week
    before setting their level to latest level. They will
    miss updates until their configure the stream.
  * If stream is configured, send updates.

* level > 0: Send one message/update per level & increase
  the level by 1 till the latest level.

Fixes #28604.
2024-03-20 11:48:06 -07:00
Prakhar Pratyush 336d46001c message_send: Add 'internal_prep_huddle_message' function.
This is a prep commit to extract out the logic to
create message from 'internal_send_huddle_message'
into a separate function 'internal_prep_huddle_message'.

We will use this new function to get the huddle message
without sending it immediately.
2024-03-20 11:48:06 -07:00
Prakhar Pratyush 23fc04577b message_send: Don't create savepoint in do_send_messages.
In general, we never want to use savepoints.

This prep commit adds savepoint=False in do_send_messages
as we don't want to just rollback to this savepoint and
proceed if we encounter any error while sending zulip updates
via cron.
2024-03-20 11:48:06 -07:00
Alex Vandiver 0f0631813f message_edit: Do set differences in QuerySets. 2024-03-19 18:22:24 -04:00
Alex Vandiver c71b7afe9f message_edit: Handle user, not subscription, collections.
Nothing about the subscription is relevant -- we instead use
collections of User objects for readability.
2024-03-19 18:22:24 -04:00
Alex Vandiver f15e006873 message_edit: Perform cross join in SQL, not Python.
The database can do this faster than we can, and it reduces the amount
of information transmitted over the wire.
2024-03-19 18:22:24 -04:00
Alex Vandiver a8795c5d0b message_edit: Pull user_ids_gaining_usermessages higher, rename. 2024-03-19 18:22:24 -04:00
Alex Vandiver 7ce6095003 message_edit: Handle previous subscribers and private-history streams. 2024-03-19 18:22:24 -04:00
Alex Vandiver 7b1feac06a CVE-2024-27286: Remove UserMessage rows for non-active Subscriptions.
A user who was no longer subscribed to a private stream kept their
UserMessage row for a message sent while they were in it; this is
expected.  However, they _also_ kept that row even if the message was
moved to a different private stream that they were also not subscribed
to.  This violates the invariant that users without subscriptions
never have UserMessage rows.
2024-03-19 18:22:24 -04:00
Alex Vandiver e964536139 CVE-2024-27286: Run usermessage modifications even for change_one.
This `if new_stream is not None` block was improperly indented,
causing it to only run if the propagation mode was not `change_one`.
Since the block controlled creation and deletion of UserMessage rows,
this led to messages being improperly still visible to members of the
old stream if they were being moved from public to private streams.
Clients also failed to receive `delete_message` events, so the
messages remained visible in their feeds until they reloaded the
application.
2024-03-19 18:22:24 -04:00
Mateusz Mandera 1ede8da46c events: Change format of authentication_methods realm_update_dict event.
Makes the event in line with state_data.realm_authentication_methods.
2024-03-05 11:48:58 -08:00
Mateusz Mandera da9e4e6e54 backends: Implementation of restricting certain backends by plan.
Only affects zulipchat, by being based on the BILLING_ENABLED setting.

The restricted backends in this commit are
- AzureAD - restricted to Standard plan
- SAML - restricted to Plus plan, although it was already practically
  restricted due to requiring server-side configuration to be done by us

This restriction is placed upon **enabling** a backend - so
organizations that already have a backend enabled, will continue to be
able to use it. This allows us to make exceptions and enable a backend
for an org manually via the shell, and to grandfather organizations into
keeping the backend they have been relying on.
2024-03-05 11:48:58 -08:00
Alex Vandiver 352dbf9387 caches: Only take the realm_id, not the Realm, as a cache key function.
This saves a hit to the database to fetch the Realm of a UserProfile
that we are trying to flush.
2024-03-04 16:35:57 -08:00
Anders Kaseorg 570f3dd447 python: Reformat with Ruff formatter.
https://docs.astral.sh/ruff/formatter/

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-02-29 17:07:16 -08:00
Prakhar Pratyush fe1a20ebb3 settings: Add realm-level setting 'zulip_update_announcements_stream'.
This commit adds a realm-level setting named
'zulip_update_announcements_stream' that configures the
stream to which zulip updates should be posted.

Fixes part of #28604.
2024-02-25 09:33:00 -08:00
Prakhar Pratyush ee612dafac settings: Rename signup_notifications_stream realm setting.
This commit renames the realm-level setting
'signup_notifications_stream' to 'signup_announcements_stream'.

The new name reflects better what the setting does.
2024-02-21 09:04:23 -08:00
Prakhar Pratyush ab453fbe20 settings: Rename notifications_stream to new_stream_announce..._stream.
This commit renames the realm-level setting 'notifications_stream'
to 'new_stream_announcements_stream'.

The new name reflects better what the setting does.
2024-02-21 09:04:23 -08:00
sayyedarib 5c993f0479 direct_message: Allow DMs to bots and self regardless of policy.
Previously, in DM disabled org messaging to bot was not working when
starting new conversation and adding bot as recipient because of not
updating on recipient change. And secondly, self messaging was not
allowed.
This commit ensures that the DM to bot and self are allowed irrespective
of dm restrictions.

tests: Verify DMs adhere to DM restriction policy.

Fixes #28412

Signed-off-by: sayyedarib <sayyedaribhussain4321@gmail.com>
2024-02-15 16:13:36 -08:00
Alex Vandiver b3e8878252 message_cache: Rename update_to_dict_cache to update_message_cache.
This better describes what it does.
2024-02-14 12:27:03 -08:00
Alex Vandiver 737a751f5c message: Split MessageDict and friends into its own file. 2024-02-14 12:27:03 -08:00
Alex Vandiver 0f9b7f112b message: Move render_markdown into zerver.lib.markdown. 2024-02-14 12:27:03 -08:00
Alex Vandiver 22837fc1b4 message_edit: Carry the QuerySet through as much as possible.
Rather than pass around a list of message objects in-memory, we
instead keep the same constructed QuerySet which includes the later
propagated messages (if any), and use that same query to pick out
affected Attachment objects, rather than limiting to the set of ids.
This is not necessarily a win -- the list of message-ids *may* be very
long, and thus the query may be more concise, easier to send to
PostgreSQL, and faster for PostgreSQL to parse.  However, the list of
ids is almost certainly better-indexed.

After processing the move, the QuerySet must be re-defined as a search
of ids (and possibly a very long list of such), since there is no
other way which is guaranteed to correctly single out the moved
messages.  At this point, it is mostly equivalent to the list of
Message objects, and certainly takes no less memory.
2024-02-14 12:27:03 -08:00
Alex Vandiver 822131fef4 message: Add a bulk_access_stream_messages_query method.
This applies access restrictions in SQL, so that individual messages
do not need to be walked one-by-one.  It only functions for stream
messages.

Use of this method significantly speeds up checks if we moved "all
visible messages" in a topic, since we no longer need to walk every
remaining message in the old topic to determine that at least one was
visible to the user.  Similarly, it significantly speeds up merging
into existing topics, since it no longer must walk every message in
the new topic to determine if the user could see at least one.

Finally, it unlocks the ability to bulk-update only messages the user
has access to, in a single query (see subsequent commit).
2024-02-14 12:27:03 -08:00
Alex Vandiver e04891779f message_edit: len(changed_messages) is always > 0.
It is only ever added to, and starts as a one-element list.
2024-02-14 12:27:03 -08:00
Tim Abbott 79ddfa1e6f message_edit: Ensure gaining_usermessage_user_ids is set.
This makes this code path a bit easier to follow.
2024-02-14 12:27:03 -08:00
Tim Abbott 4a049299ee message_edit: Clarify unmodified_user_messages logic. 2024-02-14 12:27:03 -08:00
roanster007 6ab63874f2 message_edit: Refactor check_update_message method.
This is a preparatory commit that refactors the check_update_message
method to extract the checks containing whether a user can edit the
message or not into a separate method -validate_message_content_edit,
so that it can be re used later.
2024-02-12 10:51:12 -08:00
Tim Abbott 9efce0b16c create_user: Fix initial unreads ignoring private streams.
This logic was apparently missed when we implemented private streams
with shared history; the correct check is to look at whether the user
can access message history in the stream, which used to be equivalent
to whether it's a private stream.
2024-02-09 12:49:30 -08:00
shashank-23002 3bf04ffab5 messages: Update new user signup notification.
Fixes: #28834.
2024-02-06 18:41:05 -08:00
Prakhar Pratyush c54fa91f1a do_send_messages: Remove the stale 'email_gateway' parameter.
This commit removes the stale 'email_gateway' parameter
from 'do_send_messages' function.

This should have been removed in 6c473ed75f,
when the call to 'build_message_send_dict' was removed
from 'do_send_messages'.
2024-02-05 23:19:13 -08:00
Mateusz Mandera 6dd6fc045f realm_settings: Improve authentication_methods param validation.
The endpoint was lacking validation that the authentication_methods dict
submitted by the user made sense. So e.g. it allowed submitting a
nonsense key like NoSuchBackend or modifying the realm's configured
authentication methods for a backend that's not enabled on the server,
which should not be allowed.

Both were ultimately harmless, because:
1. Submitting NoSuchBackend would luckily just trigger a KeyError inside
   the transaction.atomic() block in do_set_realm_authentication_methods
   so it would actually roll back the database changes it was trying to
   make. So this couldn't actually create some weird
   RealmAuthenticationMethod entries.
2. Silently enabling or disabling e.g. GitHub for a realm when GitHub
   isn't enabled on the server doesn't really change anything. And this
   action is only available to the realm's admins to begin with, so
   there's no attack vector here.

test_supported_backends_only_updated wasn't actually testing anything,
because the state it was asserting:
```
        self.assertFalse(github_auth_enabled(realm))
        self.assertTrue(dev_auth_enabled(realm))
        self.assertFalse(password_auth_enabled(realm))
```

matched the desired state submitted to the API...
```
        result = self.client_patch(
            "/json/realm",
            {
                "authentication_methods": orjson.dumps(
                    {"Email": False, "Dev": True, "GitHub": False}
                ).decode()
            },
        )
```

so we just replace it with a new test that tests the param validation.
2024-02-02 17:26:32 -08:00
Anders Kaseorg 93198a19ed requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-01-29 10:41:54 -08:00
Alex Vandiver 37fa181e5f queue_processors: Process user_activity in one query.
This leads to significant speedups.  In a test, with 100 random unique
event classes, the old code processed a batch of 100 rows (on average
66-ish unique in the batch) in 0.45 seconds.  Doing this in a single
query processes the same batch in 0.0076 seconds.
2024-01-22 16:25:13 -08:00
Prakhar Pratyush 1eef052bd1 actions: Rename *topic local variables to *topic_name.
This is preparatory work towards adding a Topic model.
We plan to use the local variable name as 'topic' for
the Topic model objects.

Currently, we use *topic as the local variable name for
topic names.

We rename local variables of the form *topic to *topic_name
so that we don't need to think about type collisions in
individual code paths where we might want to talk about both
Topic objects and strings for the topic name.
2024-01-15 09:40:43 -08:00
Tim Abbott bafa476fd3 presence: Avoid having incomplete missing test coverage.
Rename and restructure these comparison variables such that we don't
have a possibly impossible case for presence.last_connected_time being
None.

Fixes #25498.
2024-01-14 18:32:01 -08:00
Anders Kaseorg c343d7c30e models: Move query_for_ids to zerver.lib.query_helpers.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-12-16 22:08:44 -08:00
Anders Kaseorg 1f1b2f9a68 models: Extract zerver.models.bots.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-12-16 22:08:44 -08:00
Anders Kaseorg 27c0b507af models: Extract zerver.models.custom_profile_fields.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-12-16 22:08:44 -08:00
Anders Kaseorg c9c819e1d7 models: Extract zerver.models.scheduled_jobs.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-12-16 22:08:44 -08:00
Anders Kaseorg cff0b78771 models: Move some functions to zerver.lib.attachments.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-12-16 22:08:44 -08:00
Anders Kaseorg bac027962f models: Extract zerver.models.clients.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-12-16 22:08:44 -08:00
Anders Kaseorg 4aa2d76bea models: Extract zerver.models.streams.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-12-16 22:08:44 -08:00
Anders Kaseorg 927d7a9a60 models: Extract zerver.models.prereg_users.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-12-16 22:08:44 -08:00
Anders Kaseorg 51f1dc257d models: Extract zerver.models.recipients.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-12-16 22:08:44 -08:00
Anders Kaseorg 3c11fd9466 models: Move some functions to zerver.lib.display_recipient.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-12-16 22:08:44 -08:00
Anders Kaseorg 33610c8782 models: Extract zerver.models.realm_playgrounds.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-12-16 22:08:44 -08:00
Anders Kaseorg 21ab3858a7 models: Extract zerver.models.linkifiers.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-12-16 22:08:44 -08:00
Anders Kaseorg 67fb485797 models: Extract zerver.models.realm_emoji.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-12-16 22:08:44 -08:00
Anders Kaseorg cd96193768 models: Extract zerver.models.realms.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-12-16 22:08:44 -08:00
Anders Kaseorg 7001a0dfc0 models: Extract zerver.models.groups.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-12-16 22:08:44 -08:00
Anders Kaseorg 45bb8d2580 models: Extract zerver.models.users.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-12-16 22:08:44 -08:00
Anders Kaseorg c7245d5f3b models: Make ScheduledMessage.read_by_sender non-nullable.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-12-14 08:16:31 -08:00
Anders Kaseorg 77a6f44455 message_send: Add read_by_sender API parameter.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-12-14 08:16:31 -08:00
Anders Kaseorg d7d5b6c73e models: Move Message.sent_by_human to Client.default_read_by_sender.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-12-14 08:16:31 -08:00
Sahil Batra 92c1dfc248 message_send: Optimize checking access to DM recipients.
We previously used get_accessible_user_ids to check whether the
sender can access all DM recipients, which was not efficient as
it queries the Message table. This commit updates the code to
make sure we use get_inaccessible_user_ids which is much more
efficient as it limits the queries to only DM recipients and
also queries the Message table only if needed.

This can still be optimized further as mentioned in #27835 but
this commit is a nice first step.
2023-12-14 08:14:09 -08:00
Tim Abbott f60312b2da realm_settings: Clear sessions outside transaction.
This fixes a bug introduced in
6f93ab72c0 where deactivating a realm
would fail with an exception that sessions cannot be cleared inside
database transactions.
2023-12-12 09:20:47 -08:00
Mateusz Mandera b09f3a2da1 do_set_realm_property: Noop if value isn't really changing.
It makes no sense to do operations if the value isn't changing. In
particular, this creates RealmAuditLog entries and sends useless events.
2023-12-12 08:06:12 -08:00
Vector73 2e71ec78e3 settings: Add "Automatically follow topics where I'm mentioned" setting.
Fixes: #26795
2023-12-11 23:26:11 -08:00
Prakhar Pratyush 41ceae9c31 remote_server: Immediately send analytics on user count change.
Actions that change the number of user counts adds a deferred_work
queue processor job immediately update the billing service about your
change.

This helps to avoid having users see stale state for how many
users they have when trying to pay.
2023-12-11 14:07:39 -08:00
Tim Abbott 4fe02be825 remote_server: Rename maybe_enqueue_audit_log_upload.
This is a rename of the previous
enqueue_register_realm_with_push_bouncer_if_needed but is clearer
about the fact that this will also upload audit logs if available.
2023-12-11 14:07:39 -08:00
Tim Abbott 6f93ab72c0 realm: Deactivate realms inside a transaction.
This is the only operating editing audit logs not already using a
transaction, and having it do so will simplify an upcoming interface
to be able to assume it is always inside a transaction.
2023-12-11 14:07:39 -08:00
Tim Abbott 629492e63b realm: Use delete_realm_user_sessions bulk query.
The previous logic was accidentally quadratic in a horrible way
involving querying all sessions on the server in a loop.
2023-12-11 14:07:39 -08:00
Sahil Batra 198568522a message: Do not include details of inaccessible users in message data.
This commit adds code to not include original details of senders like
name, email and avatar url in the message objects sent through events
and in the response of endpoint used to fetch messages.

This is the last major commit for the project to add support for
limiting guest access to an entire organization.

Fixes #10970.
2023-12-09 17:23:16 -08:00
Sahil Batra 72aa4b256d message: Do not allow guest to mention inaccessible users. 2023-12-09 16:59:38 -08:00
Prakhar Pratyush 83bd9955e3 events: Add 'onboarding_steps' event deprecating 'hotspots'.
Earlier, the event sent when an onboarding step (hotspot till now)
is marked as read generated an event with type='hotspots' and
'hotspots' named array in it.

This commit renames the type to 'onboarding_steps' and the array
to 'onboarding_steps' to reflect the fact that it'll also contain
data for elements other than hotspots.
2023-12-06 18:19:20 -08:00
Prakhar Pratyush dde3d72100 onboarding_steps: Add 'OneTimeNotice' dataclass.
This commit adds a 'OneTimeNotice' dataclass to
support one time banner and similar UI elements.
2023-12-06 18:19:20 -08:00
Prakhar Pratyush ac8af3d6de urls: Add a new endpoint for hotspot and deprecate the old one.
This commit adds a new endpoint 'users/me/onboarding_steps'
deprecating the older 'users/me/hotspots' to mark hotspot as read.

We also renamed the view `mark_hotspot_as_read` to
`mark_onboarding_step_as_read`.

Reason: Our plan is to make this endpoint flexible to support
other types of UI elements not just restricted to hotspots.
2023-12-06 18:19:20 -08:00
Prakhar Pratyush 62bfc20ebc models: Rename 'UserHotspot' model to 'OnboardingStep'.
This commit renames the 'UserHotspot' model to 'OnboardingStep'.

Also, it renames the 'hotspot' field in that model
to 'onboarding_step'.
2023-12-06 18:19:20 -08:00
Lauryn Menard 906e667a70 corporate: Implement support_url for all BillingSession child classes.
Adds a helper since there are only a few different parameters for
all BillingSession child clases, `build_support_url`.

Also, renames `get_support_url` to more explicitly note that it
is for realms: `get_realm_support_url`.
2023-12-06 11:04:54 -08:00
Sahil Batra 965869d3f8 register: Add client capability to not receive unknown users data.
This commit adds a new client capability to decide whether the
client needs unknown users data or not.
2023-12-06 00:09:53 -08:00
Sahil Batra 3697df1971 realm: Allow enabling restricted user access for guests only on plus plans.
This commit adds code to not allow Zulip Cloud organizations that are not
on the Plus plan to change the "can_access_all_users_group" setting.

Fixes #27877.
2023-12-06 00:09:53 -08:00
Prakhar Pratyush e5d71fe5ac stripe: Move `update_license_ledger_if_needed` to BillingSession.
This commit moves the 'update_license_ledger_if_needed' and its
helper function 'update_license_ledger_for_automanaged_plan'
to the 'BillingSession' abstract class.

This refactoring will help in minimizing duplicate code while
supporting both realm and remote_server customers.
2023-12-05 12:51:41 -08:00
Anders Kaseorg 8a7916f21a python: Consistently use from…import for datetime.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-12-05 12:01:18 -08:00
Tim Abbott f6c7eaf1e5 models: Add push_notifications_enabled & corresponding end_timestamp.
Add two fields to Realm model:
*push_notifications_enabled
*push_notifications_enabled_end_timestamp

Co-authored-by: Prakhar Pratyush <prakhar@zulip.com>
2023-12-04 07:08:39 -08:00
Mateusz Mandera a67dd6dc1f realms: Call send_realms_only_to_push_bouncer at realm creation/import. 2023-12-03 08:49:58 -08:00
Alex Vandiver 569c364392 users: Fetch and lock the user row before updating its role.
We want to both (a) take a lock on the UserProfile row, and (b)
modify the passed-in UserProfile object, so that callers see the
changes in the object they hold.  Unfortunately,
`select_for_update` cannot be combined with `refresh_from_db`
(https://code.djangoproject.com/ticket/28344).  Call
`select_for_update` and throw away the result, so that we know we have
the lock on the row, then re-fill the `user_profile` object with the
values now that the lock exists.
2023-11-30 16:15:23 -08:00
Sahil Batra a6fa875c23 models: Refactor get_fake_email_domain to take realm.host as arg.
This commit updates get_fake_email_domain to accept realm.host as
argument instead of the Realm object since we only use realm.host
to get the fake email domain.

This is a preparatory commit for the limited guest feature as we
would be sending the fake email of the message sender in message
event object to a guest user who cannot access the sender and
there we would need to compute the fake email.
2023-11-29 12:01:37 -08:00
Alex Vandiver 037eaa07e2 user_settings: Revoke previous email changes on new one. 2023-11-28 22:11:41 -08:00
Alex Vandiver 6ecfbd2ae8 create_realm: Fix copy/paste error in assertion message. 2023-11-28 18:15:30 -08:00
Prakhar Pratyush 49388d5d3d topic_mentions: Fix restriction rule for @-topic mentions.
Now, the topic wildcard mention follows the following
rules:
* If the topic has less than 15 participants , anyone
can use @ topic mentions.
* For more than 15, the org setting 'wildcard_mention_policy'
determines who can use @ topic mentions.

Earlier, topic wildcard mentions followed the same restriction
as stream wildcard mentions, which was incorrect.

Fixes part of #27700.
2023-11-23 12:52:25 -08:00
Prakhar Pratyush 51b39cb682 stripe: Add 'do_update_plan' method to the 'BillingSession' class.
This commit moves a major portion of the 'update_plan`
view to a new shared 'BillingSession.do_update_plan' method.

This refactoring will help in minimizing duplicate code
while supporting both realm and remote_server customers.
2023-11-23 09:01:45 -08:00
Mateusz Mandera 1ec0d5bd9d requests: Add SELF_HOSTING_MANAGEMENT_SUBDOMAIN. 2023-11-22 14:22:26 -08:00
Sahil Batra f75b4f65c1 streams: Send user remove events when deactivating streams. 2023-11-21 23:58:45 -08:00
Sahil Batra 45e1b32447 users: Send user remove events on user deactivation.
Guests might lose access to deactivated users if the user
is not involved in any DM with guest. This commit adds
code to send "realm_user/remove" events for such cases.
2023-11-21 23:58:45 -08:00
Sahil Batra 32c15d67b5 users: Send user creation events when sending DMs.
We now send user creation events to recipient users
when sending DMs if recipients gain access to either
sender or other pariticpating users in the DM.
2023-11-21 23:58:45 -08:00
Sahil Batra e4a97dd3ac message: Restrict sending DMs to inaccessible users.
This commit adds code to not allow guest users to send DMs
to users they cannot access.
2023-11-21 23:58:45 -08:00
Sahil Batra 39a31170ee streams: Send event when guest loses access to a user.
This commit adds code to send "realm_user/remove" event
when a guest user loses access to a user due to the user
being unsubscribed from one or more streams.
2023-11-21 23:58:45 -08:00
Sahil Batra d394cfc4db streams: Send user creation events on adding subscribers.
This commit adds code to send user creation events to
guests who gain access to new subscribers and to the
new guest subscribers who gain access to existing
stream subscribers.
2023-11-21 23:58:45 -08:00
Sahil Batra dbcc9ea826 users: Update presence and user status code to support restricted users.
The presence and user status update events are only sent to accessible
users, i.e. guests do not receive presence and user status updates for
users they cannot access.
2023-11-21 23:58:45 -08:00
Sahil Batra 650e55fef8 users: Send events only to users who can access the modified user.
This commit adds code to make sure that update events for changing
a user's role, email, etc. are not sent to guests who cannot access
the modified user.
2023-11-21 23:58:45 -08:00
Sahil Batra 6f14d105a7 create_user: Update data in user creation events for guests.
We do not send the original user data in user creation events
to guests if user access is restricted in realm, as they would
receive the information about user if user is subscribed to some
common streams after account creation.
2023-11-21 23:58:45 -08:00
Sahil Batra dc98136346 streams: Send stream deletion events on unsubscribing users.
This commit adds code to send stream deletion events when
unsubscribing non-admin users from private streams and
when unsubscribing guests from public streams since
non-admins cannot access unsubscribed private streams
and guests cannot access unsubscribed public streams.
2023-11-16 13:25:33 -05:00
Sahil Batra d4fb244d2d CVE-2023-47642: Invalid metadata access for formerly subscribed streams.
It was discovered by the Zulip development team that active users who
had previously been subscribed to a stream incorrectly continued being
able to use the Zulip API to access metadata for that stream. As a
result, users who had been removed from a stream, but still had an
account in the organization, could still view metadata for that
stream (including the stream name, description, settings, and an email
address used to send emails into the stream via the incoming email
integration). This potentially allowed users to see changes to a
stream’s metadata after they had lost access to the stream.

This bug was present in all Zulip releases prior to today's Zulip
Server 7.5.
2023-11-16 13:25:33 -05:00
Sahil Batra 432001656e streams: Remove "email_address" field from Subscription objects.
This commit removes "email_address" field from Subscription objects
and we would instead a new endpoint in next commit to get email
address for stream with proper access check.

This change also fixes the bug where we would include email address
for the unsubscribed private stream as well when user did not have
permission to send message to the stream, and having email allowed
the unsubscribed user to send message to the stream.

Note that the unsubscribed user can still send message to the stream
if the user had noted down the email before being unsubscribed
and the stream token is not changed after unsubscribing the user.
2023-11-16 13:25:33 -05:00
Mateusz Mandera 1819b85b85 management: Allow changing is_billing_admin using change_user_role. 2023-11-15 18:36:07 -08:00
Tim Abbott 2e2997bd7d typing: Limit typing notifications in large streams. 2023-11-15 09:42:25 -08:00
Sahil Batra 532293c4cf streams: Unset is_web_public and is_realm_public fields on attachments.
This commit adds code to unset is_web_public and is_realm_public fields
on attachments when deactivating a stream as we do not want to allow
spectators to access them after the stream is deactivated.

This commit also adds a comment explaining why we don't use
do_change_stream_permission to set the privacy fields on deactivating
a stream.

Fixes #27634.
2023-11-13 10:26:14 -08:00
Sahil Batra 30e2c14f4a streams: Unset is_realm_public on attachment when unarchiving streams.
We did not unset is_realm_public field on attachements when unarchiving
streams, but we do unset is_web_public field. This commit adds code to
unset the is_realm_public field as well as we make the stream private
while unarchiving it.
2023-11-13 10:26:14 -08:00
Sahil Batra df01d0725c message_send: Remove select_related call for stream.
We do not access any related field from the stream object,
so there is no need for select_related.
2023-11-13 08:04:45 -08:00
Prakhar Pratyush c597de6a1d topic_mentions: Rename wildcard_mentioned to stream_wildcard_mentioned.
Rename the existing 'wildcard_mentioned' flag to
'stream_wildcard_mentioned'.

The 'wildcard_mentioned' flag is deprecated and exists for
backwards compatibility.

We have two separate flags for stream and topic wildcard mentions,
i.e., 'stream_wildcard_mentioned' and 'topic_wildcard_mentioned',
respectively.

* stream wildcard mentions: `@all`, `@everyone`, and `@stream`
* topic wildcard mentions: `@topic`

The `wildcard_mentioned` flag is included in the events and
API response if either `stream_wildcard_mentioned` or
`topic_wildcard_mentioned` is set.
2023-11-10 11:06:26 -08:00
Prakhar Pratyush e6e156709a typing_notifications: Don't notify long_term_idle subscribers.
The event for stream typing notifications is no longer sent
to the long_term_idle subscribers of the stream.

This helps to reduce the tornado's work of parsing super-long
JSON-encoded lists of user IDs in large streams. Now the lists
are shorter.
2023-11-07 09:30:27 -08:00
Prakhar Pratyush b0ef76bf27 topic_mentions: Set 'topic_wildcard_mentioned' flag on @topic mention.
Earlier, the 'wildcard_mentioned' flag was set for both the
stream and topic wildcard mentions.

Now, the 'topic_wildcard_mentioned' flag is set for topic
wildcard mentions, and the 'wildcard_mentioned' flag is set for
stream wildcard mentions.

We will rename the 'wildcard_mentioned' flag to
'stream_wildcard_mentioned' in a later commit.
2023-11-02 09:25:51 -07:00
Sahil Batra bb15b2d708 users: Send "update" events when deactivating or reactivating users.
We now send "realm_user/update" (and "realm_bot/update" for bots)
events with "is_active" field when deactivating and reactivating
users, including bots.

We would want to use "remove" event for a user losing access
to another user for #10970, so it is better to use "update"
event for deactivation as we only update "is_active" field
in the user objects and the clients still have the data for
deactivated users.

Previously, we used to send "add" event for reactivation along
with complete user objects, but clients should have the data
for deactivated users as well, so an "update" event is enough
like we do when deactivating users.
2023-11-02 08:31:12 -07:00
Sahil Batra e458b73a01 user_groups: Move constants for system group names to a new class.
This commit moves constants for system group names to a new
"SystemGroups" class so that we can use these group names
in multiple classes in models.py without worrying about the
order of defining them.
2023-11-01 10:42:56 -07:00
Sahil Batra 08d41f9eb3 settings: Rename permissions_configuration variable.
This commit renames permissions_configuration variable to
permission_configuration since the object contains config for
a single permission setting and thus permission_configuration
seems like a better name.
2023-11-01 10:42:56 -07:00
Sahil Batra 9a6cf82adc streams: Fix sending stream-related events to guests.
Previous behavior-
- Guest did not receive stream creation events for new
web-public streams.
- Guest did not receive peer_add and peer_remove events
for web-public and subscribed public streams.

This commit fixes the behavior to be -
- Guests now receive stream creation events for new
web-public streams.
- Guest now receive peer_add and peer_remove events for
web-public and subscribed public streams.
2023-10-31 10:54:21 -07:00
Sahil Batra 71b8f49614 streams: Return early if there is no change in subscriptions.
This commit updates code in bulk_remove_subscriptions and
bulk_add_subscriptions to return early if there are no
subscribers to remove or add to the streams.

This change helps us in avoiding unnecessary queries like the
one used to get subscribers list of streams, which is then used
to send events but we would not send any events if no subscribers
are added or removed and some more similar queries.
2023-10-31 10:54:21 -07:00
Hemant Umre ac1f711fef registration: Set the organization language at creation time.
In this commit, we add a new dropdown 'Organization language' on
the `/new` and `/realm/register` pages. This dropdown allows setting
the language of the organization during its creation. This allows
messages from Welcome Bot and introductory messages in streams to be
internationalized.

Fixes a part of #25729.
2023-10-26 16:27:35 -07:00
Sayyed Arib Hussain 7e38d06130
create_user: Use silent mentions in user join notices.
This provides more information (email, etc.) when clicked on
while being better adapted to email_address_visibility existing.

Fixes #27243.
2023-10-24 09:58:55 -07:00
Prakhar Pratyush 17a0304309 send_message: Add an optional parameter in the success response.
Add an optional `automatic_new_visibility_policy` enum field
in the success response to indicate the new visibility policy
value due to the `automatically_follow_topics_policy` and
`automatically_unmute_topics_in_muted_streams_policy` user settings
during the send message action.

Only present if there is a change in the visibility policy.
2023-10-17 15:38:16 -07:00
Prakhar Pratyush 8b12cc606a request: Extract out methods from 'scheduled_messages' to reuse.
This is a prep commit that extracts the following two methods
from '/actions/scheduled_messages' to reuse in the next commit.
* extract_stream_id
* extract_direct_message_recipient_ids

The 'to' parameter for 'POST /typing' will follow the same pattern
in the next commit as we currently have for the 'to' parameter in
'POST /scheduled_messages', so we can reuse these functions.
2023-10-10 17:15:28 -07:00
Prakhar Pratyush a22f637769 typing_indicator: Replace private with direct in 'message_type' field.
This commit replaces the value `private` with `direct` in the
`message_type` field for the `typing` events sent when a user
starts or stops typing a message.
2023-10-10 17:15:28 -07:00
Anders Kaseorg 835ee69c80 docs: Fix grammar errors found by mwic.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-10-09 13:24:09 -07:00
Prakhar Pratyush 02f3f73b0c recipient_info: Include 'sender_id' in 'topic_participant_user_ids'.
This commit includes the message's sender id in the
'topic_participant_user_ids' set.

The 'participants_for_topic' function doesn't include the sender_id,
if the user is sending their first message in the topic, because
'participants_for_topic' queries the 'Message' table, but the message
is actually sent at a later stage in the codepath, resulting in
missing the sender_id in this case.

This is needed to set the 'wildcard_mentioned' flag for the sender's
user message in the case of topic wildcard mentions.

This doesn't lead to sending email and push notifications to the
sender because we have a check to skip notifications if the user
to receive notifications is the sender itself.

This should have been included in c0c30bc.
2023-10-06 11:38:37 -07:00
Prakhar Pratyush 58568a60d6 settings: Add automatically follow and unmute topics policy settings.
This commit adds two user settings, named
* `automatically_follow_topics_policy`
* `automatically_unmute_topics_in_muted_streams_policy`

The settings control the user's preference on which topics they
will automatically 'follow' or 'unmute in muted streams'.

The policies offer four options:
1. Topics I participate in
2. Topics I send a message to
3. Topics I start
4. Never (default)

There is no support for configuring the settings through the UI yet.
2023-10-04 13:04:29 -07:00
Tim Abbott c6fe799369 i18n: Fix default language for users created via API/LDAP.
This fixes a regression introduced in
9954db4b59, where the realm's default
language would be ignored for users created via API/LDAP/SAML,
resulting in all such users having English as their default language.

The API/LDAP/SAML account creation code paths don't have a request,
and thus cannot pull default language from the user's browser.

We have the `realm.default_language` field intended for this use case,
but it was not being passed through the system.

Rather than pass `realm.default_language` through from each caller, we
make the low-level user creation code set this field, as that seems
more robust to the creation of future callers.
2023-10-01 21:10:13 +02:00
Alex Vandiver c5373e9934 users: Prevent mirror dummy users from being "reactivated."
These do not represent users who were ever active, so they should not
be able to be reactivated.

See also #26703.
2023-10-01 09:29:23 -07:00
Tim Abbott ef5fee4778 user_groups: Improve bulk membership change logic.
Adds support for bulk-adjusting a single user's membership in multiple
user groups in a single transaction in the low-level actions
functions, for future use by work on #9957.
2023-09-29 15:08:02 +02:00
Mateusz Mandera 41d76969cb build_message_send_dict: Remove redundant realm arg.
Since the function asserts that the realm arg matches message.realm
anyway, having the argument is redundant here.
2023-09-20 09:55:25 -07:00
David Rosa cb123d4f15 management: Rename command reactivate_stream -> unarchive_stream. 2023-09-19 15:03:33 -07:00
Lauryn Menard 7c618b57cf emails: Update subject for confirm_new_email to have realm host.
Updates the email subject for confirming an email change to include
the realm host.
2023-09-17 15:12:21 -07:00
Mateusz Mandera 3e15ea3f3f scim: Add supporting for syncing the user role.
This adds support for syncing user role via the newly added "role"
attribute, which can be set to either of
['owner', 'administrator', 'moderator', 'member', 'guest'].

Removes durable=True from the atomic decorator of do_change_user_role,
as django-scim2 runs PATCH operations in an atomic block.
2023-09-15 13:15:45 -07:00
Anders Kaseorg 28597365da python: Delete superfluous parens.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-09-13 13:40:19 -07:00
Anders Kaseorg 2665a3ce2b python: Elide unnecessary list wrappers.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-09-13 12:41:23 -07:00
Alex Vandiver b94402152d models: Always search Messages with a realm_id or id limit.
Unless there is a limit on `id`, always provide a `realm_id` limit as
well.  We also notate which index is expected to be used in each
query.
2023-09-11 15:00:37 -07:00
Alex Vandiver d6745209f2 django: Use .exists() instead of .count() when possible. 2023-09-11 15:00:37 -07:00
Alex Vandiver 9d3d57e786 message_send: Inline single use of filter_by_exact_message_topic.
Matching the topic exactly, as opposed to case-insensitively, is not a
common operation, and one that we want to make difficult to do
accidentally.  Inline the single use case of it.
2023-09-11 15:00:37 -07:00
Anders Kaseorg 1905df2342 requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-09-09 12:53:39 -07:00
Anders Kaseorg 0ce6dcb905 mypy: Upgrade mypy from 1.4.1 to 1.5.1.
_default_manager is the same as objects on most of our models. But
when a model class is stored in a variable, the type system doesn’t
know which model the variable is referring to, so it can’t know that
objects even exists (Django doesn’t add it if the user added a custom
manager of a different name). django-stubs used to incorrectly assume
it exists unconditionally, but it no longer does.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-09-07 17:51:42 -07:00
Ujjawal Modi 88ec312b21 events: Send invites changes event to non-admin users also.
Earlier whenever a new invitation is created a event was sent
to only admin users. So, if invites by a non-admins user are changed
the invite panel does not live update.

This commit makes changes to also send event to non-admin
user if invites by them are changed.
2023-09-07 14:21:01 -07:00
Ujjawal Modi 5e31a6b1c0 invites: Make it possible for non-admins to revoke multiuse invites.
This commit makes changes to allow non-admins to revoke multiuse
invitations created by them.
2023-09-07 14:21:01 -07:00
Ujjawal Modi f67cef8885 invite: Add new setting for "Who can create multiuse invite links".
This commit does the backend changes required for adding a realm
setting based on groups permission model and does the API changes
required for the new setting `Who can create multiuse invite link`.
2023-09-07 14:21:01 -07:00
Lauryn Menard 91b40a45fe demo-orgs: Add email and password process for demo organization owners.
Creates process for demo organization owners to add an email address
and password to their account.

Uses the same flow as changing an email (via user settings) at the
beginning, but then sends a different email template to the user
for the email confirmation process.

We also encourage users to set their full name field in the modal for
adding an email in a demo organization. We disable the submit button
on the form if either input is empty, email or full name.

When the user clicks the 'confirm and set password' button in the
email sent to confirm the email address sent via the form, their
email is updated via confirm_email_change, but the user is redirected
to the reset password page for their account (instead of the page for
confirming an email change has happened).

Once the user successfully sets a password, then they will be
prompted to log in with their newly configured email and password.
2023-08-31 15:02:16 -07:00
Eeshan Garg 5e33ae8adf demo-orgs: Create dev environment demo organization without email.
To make creation of demo organizations feel lightweight for users,
we do not want to require an email address at sign-up. Instead an
empty string will used for the new realm owner's email. Currently
implements that for new demo organizations in the development
environment.

Because the user's email address does not exist, we don't enqueue
any of the welcome emails upon account/realm creation, and we
don't create/send new login emails.

This is a part of #19523.

Co-authored by: Tim Abbott <tabbott@zulip.com>
Co-authored by: Lauryn Menard <lauryn@zulip.com>
2023-08-31 15:02:16 -07:00
Sahil Batra ada2991f1c users: Send stream creation/deletion events on role change.
We now send stream creation and stream deletion events on
changing a user's role because a user can gain or lose
access to some streams on changing their role.
2023-08-25 12:56:36 -07:00
Mateusz Mandera c908b518ef CVE-2023-32678: Prevent unauthorized editing/deletion in priv streams.
Users who used to be subscribed to a private stream and have been
removed from it since retain the ability to edit messages/topics, and
delete messages that they used to have access to, if other relevant
organization permissions allow these actions. For example, a user may be
able to edit or delete their old messages they posted in such a private
stream. An administrator will be able to delete old messages (that they
had access to) from the private stream.

We fix this by fixing the logic in has_message_access (which lies at the
core of our message access checks - access_message() and
bulk_access_messages())
to not rely on only a UserMessage row for checking access but also
verify stream type and subscription status.
2023-08-25 14:10:27 -04:00
Zixuan James Li a081428ad2 user_groups: Make locks required for updating user group memberships.
**Background**

User groups are expected to comply with the DAG constraint for the
many-to-many inter-group membership. The check for this constraint has
to be performed recursively so that we can find all direct and indirect
subgroups of the user group to be added.

This kind of check is vulnerable to phantom reads which is possible at
the default read committed isolation level because we cannot guarantee
that the check is still valid when we are adding the subgroups to the
user group.

**Solution**

To avoid having another transaction concurrently update one of the
to-be-subgroup after the recursive check is done, and before the subgroup
is added, we use SELECT FOR UPDATE to lock the user group rows.

The lock needs to be acquired before a group membership change is about
to occur before any check has been conducted.

Suppose that we are adding subgroup B to supergroup A, the locking protocol
is specified as follows:

1. Acquire a lock for B and all its direct and indirect subgroups.
2. Acquire a lock for A.

For the removal of user groups, we acquire a lock for the user group to
be removed with all its direct and indirect subgroups. This is the special
case A=B, which is still complaint with the protocol.

**Error handling**

We currently rely on Postgres' deadlock detection to abort transactions
and show an error for the users. In the future, we might need some
recovery mechanism or at least better error handling.

**Notes**

An important note is that we need to reuse the recursive CTE query that
finds the direct and indirect subgroups when applying the lock on the
rows. And the lock needs to be acquired the same way for the addition and
removal of direct subgroups.

User membership change (as opposed to user group membership) is not
affected. Read-only queries aren't either. The locks only protect
critical regions where the user group dependency graph might violate
the DAG constraint, where users are not participating.

**Testing**

We implement a transaction test case targeting some typical scenarios
when an internal server error is expected to happen (this means that the
user group view makes the correct decision to abort the transaction when
something goes wrong with locks).

To achieve this, we add a development view intended only for unit tests.
It has a global BARRIER that can be shared across threads, so that we
can synchronize them to consistently reproduce certain potential race
conditions prevented by the database locks.

The transaction test case lanuches pairs of threads initiating possibly
conflicting requests at the same time. The tests are set up such that exactly N
of them are expected to succeed with a certain error message (while we don't
know each one).

**Security notes**

get_recursive_subgroups_for_groups will no longer fetch user groups from
other realms. As a result, trying to add/remove a subgroup from another
realm results in a UserGroup not found error response.

We also implement subgroup-specific checks in has_user_group_access to
keep permission managing in a single place. Do note that the API
currently don't have a way to violate that check because we are only
checking the realm ID now.
2023-08-24 17:21:08 -07:00
Zixuan James Li 006b2acd5d user_groups: Make acting user requried for deletion.
When doing a checked deletion, the acting user is always required.
It is effectively the user_profile parameter we had been previously
requiring.
2023-08-24 17:21:08 -07:00
Zixuan James Li a3f4341934 user_groups: Make for_read required.
We want to make the callers be more explicit about the use of the
user group being accessed, so that the later implemented database lock
can be benefited from the visibility.
2023-08-24 17:21:08 -07:00
Sahil Batra 58aecbe443 message: Pass realm as argument to wildcard_mention_allowed.
We do not want to access realm from "sender" field so that
we do not need to pass "sender__realm" argument to
select_related call when querying messages. We can instead
pass realm as argument to wildcard_mention_allowed.
2023-08-23 11:38:32 -07:00
Sahil Batra df2407f97a message: Access realm from SendMessageRequest object directly.
We store realm object in SendMessageRequest object, so we can
access it directly instead of getting it from "sender" field.
2023-08-23 11:38:32 -07:00
Sahil Batra 7295028194 message: Access realm object directly from message.
We can directly get the realm object from Message object now
and there is no need to get the realm object from "sender"
field of Message object.

After this change, we would not need to fetch "sender__realm"
field using "select_related" and instead only passing "realm"
to select_related when querying Message objects would be enough.

This commit also updates a couple of cases to directly access
realm ID from message object and not message.sender. Although
we have fetched sender object already, so accessing realm_id
from message directly or from message.sender should not matter,
but we can be consistent to directly get realm from Message
object whenever possible.
2023-08-23 11:38:32 -07:00
Sahil Batra 7137eba222 streams: Don't compute traffic data for sub objects in zephyr realm.
We set stream_weekly_traffic field to "null" for Subscription
objects in zephyr mirror realm as we do not need stream traffic
data in zephyr mirror realm. This makes the subscription data
consistent with steams data.

This commit also udpates test to check never_subscribed data for
zephyr mirror realm.
2023-08-21 15:21:58 -07:00
Sahil Batra 6776e380b2 stream_traffic: Update get_streams_traffic to return None for zephyr realm.
Instead of having a "realm.is_zephyr_mirror_realm" check for every
get_streams_traffic call, this commit udpates get_streams_traffic to
accept realm as parameter and return "None" for zephyr mirror realm.
2023-08-21 15:21:58 -07:00
Zixuan James Li 30495cec58 migration: Rename extra_data_json to extra_data in audit log models.
This migration applies under the assumption that extra_data_json has
been populated for all existing and coming audit log entries.

- This removes the manual conversions back and forth for extra_data
throughout the codebase including the orjson.loads(), orjson.dumps(),
and str() calls.

- The custom handler used for converting Decimal is removed since
DjangoJSONEncoder handles that for extra_data.

- We remove None-checks for extra_data because it is now no longer
nullable.

- Meanwhile, we want the bouncer to support processing RealmAuditLog entries for
remote servers before and after the JSONField migration on extra_data.

- Since now extra_data should always be a dict for the newer remote
server, which is now migrated, the test cases are updated to create
RealmAuditLog objects by passing a dict for extra_data before
sending over the analytics data. Note that while JSONField allows for
non-dict values, a proper remote server always passes a dict for
extra_data.

- We still test out the legacy extra_data format because not all
remote servers have migrated to use JSONField extra_data.
This verifies that support for extra_data being a string or None has not
been dropped.

Co-authored-by: Siddharth Asthana <siddharthasthana31@gmail.com>
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2023-08-16 17:18:14 -07:00
Sahil Batra 98b0fa8ae2 bots: Remove subscription from inaccessible streams on reactivating bot.
This commit removes the private stream suscriptions of the bot if the
original owner is deactivated and we change the owner to the user who
is reactivating the bot. We unsusbcribe the bot from private streams
that the new owner is not subscribed to.

Fixes part of #21700.
2023-08-16 15:37:37 -07:00
Sahil Batra 35d5609996 bots: Remove private stream subscriptions on changing bot owner.
We remove bot's subscriptions for private streams to which the
new owner is not subscribed and keep the ones to which the new
owner is subscribed on changing owner.

This commit also changes the code for sending subscription
remove events to use transaction.on_commit since we call
the function inside a transactopn in do_change_bot_owner and
this also requires some changes in tests in test_events.
2023-08-16 15:37:37 -07:00
Sahil Batra ba00907946 bots: Re-parent bot while reactivating if original owner is deactivated.
Since the original owner is deactivated, we change the owner to the
admin who is reactivating the bot.
2023-08-16 15:37:37 -07:00
Sahil Batra 133bd1e4cb bots: Extract events sent on chaning bot owner in a separate function.
This is a prep commit to re-parent the bots on reactivating them if
original owner is deactivated such that we can avoid duplicating code.
2023-08-16 15:37:37 -07:00
Prakhar Pratyush 665f491f22 message_edit: Fix wildcard_mentioned flag set for all user-messages.
Earlier, for topic wildcard mentions, the 'wildcard_mentioned'
flag was set for all the user-messages. (similar to stream wildcard
mention).

The flag should be set for the topic participants only.

The bug was introduced in 4c9d26c.
2023-08-16 11:31:56 -07:00
Prakhar Pratyush 379a08eb1e message_send: Fix wildcard_mentioned flag unset for few participants.
For topic wildcard mentions, the 'wildcard_mentioned' flag is set
for those user messages having 'user_profile_id' in
'topic_participant_user_ids', i.e. all topic participants.

Earlier, the flag was set if the 'user_profile_id' exists in
'all_topic_wildcard_mention_user_ids'.
'all_topic_wildcard_mention_user_ids' contains the ids of those
users who are topic participants and have enabled notifications
for '@topic' mentions.

The earlier approach was incorrect, as it would set the
'wildcard_mentioned' flag only for those topic participants
who have enabled the notifications for '@topic' mention instead
of setting the flag for all the topic participants.

The bug was introduced in 4c9d26c.
2023-08-16 11:31:56 -07:00
bjorn3 762c9667a7 uploads: Rewrite comment in do_claim_attachments to improve clarity.
Co-Authored-By: Tim Abbott <tabbott@zulip.com>
2023-08-15 10:08:11 -07:00
bjorn3 8279ad603d uploads: Fix double negation in do_claim_attachments comment. 2023-08-15 10:08:11 -07:00
Zixuan James Li 37660dd0e7 linkifier: Support reordering linkifiers.
This adds API support to reorder linkifiers and makes sure that the
returned lists of linkifiers from `GET /events`, `POST /register`, and
`GET /realm/linkifiers` are always sorted with the order that they
should processed when rendering linkifiers.

We set the new `order` field to the ID with the migration. This
preserves the order of the existing linkifiers.

New linkifiers added will always be ordered the last. When reordering,
the `order` field of all linkifiers in the same realm is updated, in
a manner similar to how we implement ordering for
`custom_profile_fields`.
2023-08-14 15:21:48 -07:00
Prakhar Pratyush 860eee94fd notifications: Rename 'pm' to 'dm' in 'RecipientInfoResult' dataclass.
This commit renames the keyword 'pm' to 'dm' in the
'pm_mention_email_disabled_user_ids' and
'pm_mention_push_disabled_user_ids' attributes of the
'RecipientInfoResult' dataclass.

'pm' and 'dm' are the acronyms for 'private message' and
'direct message' respectively.

It includes 'TODO/compatibility' code to support the old format
fields in the tornado queues during the Zulip server upgrades.
2023-08-10 17:41:49 -07:00
Prakhar Pratyush c4e4737cc6 notification_trigger: Rename `private_message` to `direct_message`.
This commit renames the 'PRIVATE_MESSAGE' attribute of the
'NotificationTriggers' class to 'DIRECT_MESSAGE'.

Custom migration to update the existing value in the database.

It includes 'TODO/compatibility' code to support the old
notification trigger value 'private_message' in the
push notification queue during the Zulip server upgrades.

Earlier 'private_message' was one of the possible values for the
'trigger' property of the '[`POST /zulip-outgoing-webhook`]' response;
Update the docs to reflect the change in the above-mentioned trigger
value.
2023-08-10 17:41:49 -07:00
Anders Kaseorg 562a79ab76 ruff: Fix PERF401 Use a list comprehension to create a transformed list.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-08-07 17:23:55 -07:00
Anders Kaseorg c2c96eb0cf python: Annotate type aliases with TypeAlias.
This is not strictly necessary but it’s clearer and improves mypy’s
error messages.

https://docs.python.org/3/library/typing.html#typing.TypeAlias
https://mypy.readthedocs.io/en/stable/kinds_of_types.html#type-aliases

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-08-07 10:02:49 -07:00
Sahil Batra ae72151ec1 streams: Pass stream_weekly_traffic field in stream objects.
This commit adds code to pass stream traffic data using
the "stream_weekly_traffic" field in stream objects.

We already include the traffic data in Subscription objects,
but the traffic data does not depend on the user to stream
relationship and is stream-only information, so it's better
to include it in Stream objects. We may remove the traffic
data and other stream information fields for Subscription
objects in future.

This will help clients to correctly display the stream
traffic data in case where client receives a stream
creation event and no subscription event, for an already
existing stream which the user did not have access to before.
2023-08-06 18:06:42 -07:00
Sahil Batra 8c9ad30411 streams: Add stream_to_dict method.
This commit adds stream_to_dict method which is same as
Stream.to_dict method as of now. This is a prep commit
to include stream traffic data in stream objects.
2023-08-06 18:02:47 -07:00
Alex Vandiver b67108c8c6 retention: Prevent deletion of partially-archived messages.
Previously, this code:
```python3
old_archived_attachments = ArchivedAttachment.objects.annotate(
    has_other_messages=Exists(
        Attachment.objects.filter(id=OuterRef("id"))
        .exclude(messages=None)
        .exclude(scheduled_messages=None)
    )
).filter(messages=None, create_time__lt=delta_weeks_ago, has_other_messages=False)
```

...protected from removal any ArchivedAttachment objects where there
was an Attachment which had _both_ a message _and_ a scheduled
message, instead of _either_ a message _or_ a scheduled message.
Since files are removed from disk when the ArchivedAttachment rows are
deleted, this meant that if an upload was referenced in two messages,
and one was deleted, the file was permanently deleted when the
ArchivedMessage and ArchivedAttachment were cleaned up, despite being
still referenced in live Messages and Attachments.

Switch from `.exclude(messages=None).exclude(scheduled_messages=None)`
to `.exclude(messages=None, scheduled_messages=None)` which "OR"s
those conditions appropriately.

Pull the relevant test into its own file, and expand it significantly
to cover this, and other, corner cases.
2023-08-06 13:40:02 -07:00
Anders Kaseorg 733083c65d ruff: Collapse short multi-line import statements.
isort did this by default, though it’s unclear whether that was
intended; see https://github.com/astral-sh/ruff/issues/4153.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-08-02 17:41:41 -07:00
Steve Howell 1c607f8467 user fetch: Pass realm_id to format_user_row. 2023-07-25 23:08:52 -07:00
Ujjawal Modi fbcc3b5c84 user_groups: Rename `can_mention_group_id` parameter.
Earlier the API endpoints related to user_group accepts and returns a
field `can_mention_group_id` which represents the ID
of user_group whose members can mention the group.

This commit renames this field to `can_mention_group`.
2023-07-25 18:33:04 -07:00
Ujjawal Modi c8bcb422f5 streams: Rename `can_remove_subscribers_group_id` parameter.
Earlier the API endpoints related to streams accepts and returns a
field `can_remove_subscribers_group_id` which represents the ID
of user_group whose members can remove subscribers from stream.

This commit renames this field to `can_remove_subscribers_group`.
2023-07-25 18:33:04 -07:00
Zixuan James Li fe1a2f6f02 realm_playgrounds: Refactor error handling for validation on creation.
Previously, the view function was responsible for doing a first pass of
the validations done for RealmPlayground. It is no longer true now. This
refactors do_add_realm_playground to check_add_realm_playground and make
it responsible for validating the playground fields and doing error
handling for the ValidationError raised.
2023-07-24 17:40:59 -07:00
Zixuan James Li 000761ac0c realm_playgrounds: Replace url_prefix with url_template.
Dropping support for url_prefix for RealmPlayground, the server now uses
url_template instead only for playground creation, retrieval and audit
logging upon removal.

This does the necessary handling so that url_template is expanded with
the extracted code.

Fixes #25723.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2023-07-24 17:40:59 -07:00
Prakhar Pratyush af648833f2 notifications: Remove the stray notification trigger strings.
This commit removes the stray strings used to refer to
various types of notification triggers.

We use the attributes of the 'NotificationTriggers' class instead.
2023-07-24 11:02:14 -07:00
Zixuan James Li 641f60305d realm_playgrounds: Add url_template field.
As an intermediate step before we fully support url_template for realm
playgrounds, we populate url_template in the backend ensuring that all
the new entries will be validated. With a later backfilling migration,

we prepare the database such that all the records will have a valid URL
template.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2023-07-24 10:29:40 -07:00
Zixuan James Li 131729a06c realm_playgrounds: Remove unnecessary Any for kwargs.
Having a more precise type annotation helps with ensuring the migration
to use URL templates gets type checked.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2023-07-24 10:29:40 -07:00
Anders Kaseorg 2ae285af7c ruff: Fix PLR1714 Consider merging multiple comparisons.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-07-23 15:21:33 -07:00
Sahil Batra c11cf8eb54 users: Directly access id of foreign keys instead of full object.
We used to access the complete objects for UserProfile foreign
keys like "bot_owner" and "default_sending_stream", where we only
needed ID of them.

This commit fixes some of such instances and now we directly get
the id using "bot_owner_id" and "default_sending_stream_id" so
that we can avoid the unnecessary complexity of accessing the
complete object.
2023-07-20 10:44:39 -07:00
Sahil Batra bb3945a32f models: Remove select_related call in get_active_users.
We do not use any related fields for the UserProfile objects
fetched by get_active_users, so we can simply remove the
select_related call.

The user object from get_active_users was used to get realm
but since get_active_users called from a realm object we can
directly use that realm object. This change also leads to
some changes in the cache code where we now pass the realm
to the function instead of selecting it from UserProfile object.
2023-07-20 10:44:39 -07:00
Alex Vandiver 871a668dd2 reactions: Add error code for duplicate addition/removal. 2023-07-19 16:18:31 -07:00
Anders Kaseorg 0efc662eab ruff: Fix RUF015 Prefer `next(iter(…))` over `list(…)[0]`.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-07-19 16:14:59 -07:00
Anders Kaseorg 143baa4243 python: Convert translated positional {} fields to {named} fields.
Translators benefit from the extra information in the field names, and
need the reordering freedom that isn’t available with multiple
positional fields.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-07-18 15:19:07 -07:00
Alex Vandiver b188e6fa04 management: Add a reactivate-stream command.
Fixes #601.
2023-07-17 17:42:54 -07:00
Alex Vandiver 003fa7adda message_edit: Lock the Message row in check_update_message.
Fundamentally, we should take a write lock on the message, check its
validity for a change, and then make and commit that change.
Previously, `check_update_message` did not operate in a transaction,
but `do_update_message` did -- which led to the ordering:

 - `check_update_message` reads Message, not in a transaction
 - `check_update_message` verifies properties of the Message
 - `do_update_message` starts a transaction
 - `do_update_message` takes a read lock on UserMessage
 - `do_update_message` writes on UserMessage
 - `do_update_message` writes Message
 - `do_update_message` commits

This leads to race conditions, where the `check_update_message` may
have verified based on stale data, and `do_update_message` may
improperly overwrite it; as well as deadlocks, where
other (properly-written) codepaths take a write lock on Message
_before_ updating UserMessage, and thus deadlock with
`do_update_message`.

Change `check_update_message` to open a transaction, and take the
write lock when first accessing the Message row.  We update the
comment above `do_update_message` to clarify this expectation.

The new ordering is thus:

 - `check_update_message` starts a transaction
 - `check_update_message` takes a write lock on Message
 - `check_update_message` verifies properties of the Message
 - `do_update_message` writes on UserMessage
 - `do_update_message` writes Message
 - `check_update_message` commits
2023-07-17 10:53:38 -07:00
Prakhar Pratyush 4c9d26ce17 mention: Send notifications for @topic wildcard mentions.
This commit completes the notifications part of the @topic
wildcard mention feature.

Notifications are sent to the topic participants for the
@topic wildcard mention.
2023-07-17 09:39:24 -07:00
Steve Howell 67cdf1a7b4 emojis: Use get_emoji_data.
The previous function was poorly named, asked for a
Realm object when realm_id sufficed, and returned a
tuple of strings that had different semantics.

I also avoid calling it duplicate times in a couple
places, although it was probably rarely the case that
both invocations actually happened if upstream
validations were working.

Note that there is a TypedDict called EmojiInfo, so I
chose EmojiData here.  Perhaps a better name would be
TinyEmojiData or something.

I also simplify the reaction tests with a verify
helper.
2023-07-17 09:35:53 -07:00
Steve Howell b742f1241f realm emoji: Use a single cache for all lookups.
The active realm emoji are just a subset of all your
realm emoji, so just use a single cache entry per
realm.

Cache misses should be very infrequent per realm.

If a realm has lots of deactivated realm emoji, then
there's a minor expense to deserialize them, but that
is gonna be dwarfed by all the other more expensive
operations in message-send.

I also renamed the two related functions.  I erred on
the side of using somewhat verbose names, as we don't
want folks to confuse the two use cases. Fortunately
there are somewhat natural affordances to use one or
the other, and mypy helps too.

Finally, I use realm_id instead of realm in places
where we don't need the full Realm object.
2023-07-17 09:35:53 -07:00
Sahil Batra 75b61a8261 streams: Send stream creation events when subscribing guests.
We did not send the stream creation events when subscribing
guests to public streams while we do send them when subscribing
non-admin users to private streams.

This commit adds code to send the stream creation events when
subscribing guests to public streams, so the clients can know
that the stream exists and fixes the bug where client tries
to process a subscription add event for a stream which it does
not know about.
2023-07-13 14:04:51 -07:00
Steve Howell 4533ff3671 onboarding: Rename variable to cutoff_date.
This is just as clear in terms of intent, and it's robust to
us tweaking the number of weeks.
2023-07-13 11:46:34 -07:00
Prakhar Pratyush c0c30bc5f7 topic_mentions: Fetch users to be notified of @topic mentions.
This commit adds the 'topic_wildcard_mention_user_ids' and
'topic_wildcard_mention_in_followed_topic_user_ids'
attributes to the 'RecipientInfoResult' dataclass.

Only topic participants are notified of @topic mentions.

Topic participants are anyone who sent a message to a topic
or reacted to a message on the topic.

'topic_wildcard_mention_in_followed_topic_user_ids' stores the
ids of the topic participants who follow the topic and have
enabled the wildcard mention notifications for followed topics.

'topic_wildcard_mention_user_ids' stores the ids of the topic
participants for whom 'user_allows_notifications_in_StreamTopic'
with setting 'wildcard_mentions_notify' returns True.
2023-07-13 11:34:48 -07:00
Steve Howell 418057048a onboarding: Backfill unread messages up to 12 weeks old.
Previously this limit was 1 week, which was fine for busy
organizations, but for organizations that send a few messages a week,
or have occasional bursts of activity but the last one was a few weeks
ago, this should give a significantly better new user experience.

There are still caps like 1000 messages total and 20
unread, but we're a bit more flexible about time.
2023-07-13 10:40:12 -07:00
Lauryn Menard 3dfdbbc775 welcome-emails: Separate followup_day1 email from other welcome emails.
The initial followup_day1 email confirms that the new user account
has been successfully created and should be sent to the user
independently of an organization's setting for send_welcome_emails.

Here we separate out the followup_day1 email into a separate function
from enqueue_welcome_emails and create a helper function for setting
the shared welcome email sender information.

The followup_day1 email is still a scheduled email so that the initial
account creation and log-in process for the user remains unchanged.

Fixes #25268.
2023-07-11 14:15:52 -07:00
Steve Howell 89381a8072 cache: Eliminate get-stream-by-name cache.
We remove the cache functionality for the
get_realm_stream function, and we also change it to
return a thin Stream object (instead of calling
select_related with no arguments).

The main goal here is to remove code complexity, as we
have been prone to at least one caching validation bug
related to how Realm and UserGroup interact. That
particular bug was more theoretical than practical in
terms of its impact, to be clear.

Even if we were to be perfectly disciplined about only
caching thin stream objects and always making sure to
delete cache entries when stream data changed, we would
still be prone to ugly situations like having
transactions get rolled back before we delete the cache
entry. The do_deactivate_stream is a perfect example of
where we have to consider the best time to unset the
cache. If you unset it too early, then you are prone to
races where somebody else churns the cache right before
you update the database. If you set it too late, then
you can have an invalid entry after a rollback or
deadlock situation. If you just eliminate the cache as
a moving part, that whole debate is moot.

As the lack of test changes here indicates, we rarely
fetch streams by name any more in critical sections of
our code.

The one place where we fetch by name is in loading the
home page, but that is **only** when you specify a
stream name. And, of course, that only causes about an
extra millisecond of time.
2023-07-11 13:45:40 -07:00
Steve Howell adb548c7a2 stream creation: Avoid stream.realm references.
We want to avoid Django going back to the database to
get a realm object that the caller already has.

It's actually currently the case that we often
pre-fetch realm objects when we get stream objects
using get_stream (using a call to select_related() with
no arguments), but that is an expensive operation that
we want to avoid going forward.

This commit prepares us to just fetch slim objects.
2023-07-11 13:45:40 -07:00
Zixuan James Li 3349ac9f86 user_groups: Audit UserGroup group based setting changes.
This add audit log entries when any group based setting of a user group
is updated. We store both the old and new values in extra_data, along
with the name of that setting. Entries populated during user group creation
are hardcoded to track "can_mention_group".

Potentially we can adjust "set_defaults_for_group_settings" so that it
populates realm audit logs with it, but that is out of scope for this change.

We use an atomic transaction so that the audit logs are committed
together with the updates.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2023-07-11 08:56:55 -07:00
Zixuan James Li 4d0b7fe682 user_groups: Audit UserGroup properties changes.
This add audit log entries when the name or description of a user group
is updated. We store both the old and new values in extra_data. We wrap
the functions inside an atomic transaction so that the audit logs and
the updates are committed together.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2023-07-11 08:56:55 -07:00
Zixuan James Li 3035854dca user_groups: Audit UserGroup supergroup memberships changes.
This is mostly the same as tracking subgroup changes, except that now
modified_user_group is the subgroup.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2023-07-11 08:56:55 -07:00
Zixuan James Li ad698d597a user_groups: Audit UserGroup subgroup memberships changes.
It's worth noting that instead of adding another field to the
RealmAuditLog model, we store the modified subgroup ids in extra_data as
a JSON encoded dict with the key "subgroup_ids". We don't create audit
log entries for supergroup changes at this point.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2023-07-11 08:56:55 -07:00
Zixuan James Li 44781ddfa9 user_groups: Audit UserGroup memberships changes.
This also add audit log entries during user creation and role change,
because we modify system group memberships there.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2023-07-11 08:56:55 -07:00
Zixuan James Li 63f5936207 user_groups: Audit UserGroup creation.
We also create RealmAuditLog entries for the initial memberships that
get added along with the creation of a UserGroup. System user groups are
not created with members so no audit logs are populated for that.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2023-07-11 08:56:55 -07:00
Steve Howell b31bbc6148 signup: Clean up add_new_user_history.
Various cleanups:

    * clean up comments
    * improve names for constants and variables
    * express first ORM query as a single statement
    * use set differences to simplify logic
    * avoid all the reversing churn
    * avoid early-exit idiom since this function is so small

Note that it's plausible that we should just combine the two
queries and let the database exclude the already-used ids,
but that felt a little risky for now.  As I mentioned on
Zulip, I think the one-week window has dubious value, but
I am biased by having wasted time chasing down a test
flake related to the time window.
2023-07-10 13:41:28 -07:00
Steve Howell 225e826fb2 deactivate streams: Remove unused "log" parameter. 2023-07-10 13:41:28 -07:00
Steve Howell bc3afe9127 default stream groups: Make deleting streams efficient.
This pulls one query out the loop, and then it makes
another query a bulk query, and then it finally eliminates
an unnecessary query at the end.
2023-07-10 13:41:28 -07:00
Steve Howell 1156a50109 signup: Avoid bloated Stream objects for default streams.
Basically, I eliminate the use of select_all() in a query
that still makes a single round trip.  We have good test
enforcement that Django never needs to lazily fetch
objects off the Stream object. (It used to be common
to fetch stream.realm a while back, but we upgraded
bulk_add_subscription, in particular, a while back.)
2023-07-10 13:41:28 -07:00
Steve Howell 8894ff89ac signup: Extract set_up_streams_for_new_human_user.
We extract code from process_new_human_user with
no modifications.

This has all the best outcomes of extracting a function:

    * better profile info
    * easier to test for query counts (signup gets real noisy)
    * simplifies a long, messy function

It has no real drawbacks, since the helper function doesn't need
to pass back any intermediate state to the parent for the rest
of what the parent does.

When you profile test_signup and test_invite, with a decent
sample size, the set_up_streams_for_new_human_user function
does about 20% of the work for process_new_human_user, which
is a lot considering that most tests don't create a ton of
pre-registered or default streams.
2023-07-10 13:41:28 -07:00
Steve Howell 763b5e0741 default streams: Extract library functions.
I created zerver/lib/default_streams.py, so that various
views and events.py don't have to awkwardly reach into
an "actions" file.

I copied over two functions verbatim from actions/default_streams.py:

    get_default_streams_for_realm
    streams_to_dicts_sorted

The latter only remains as an internal detail in the new library.

I also created two new helpers:

    get_default_stream_ids_for_realm:

        This is both faster and easier to use in all the places
        where we only need to get a set of default stream ids.

    get_default_streams_for_realm_as_dicts:

        This just wraps the prior calls to
        streams_to_dicts_sorted(get_default_streams_for_realm(...)),
        and it doesn't yet address the slowness of the underlying
        code.

        All the "real" code should be functionally the same.

        In a few tests I now use this wrapper instead of
        calling get_default_streams_for_realm, just to get
        slightly deeper coverage.
2023-07-10 13:41:28 -07:00
Prakhar Pratyush 179d5cb37d mention: Replace 'wildcards' with 'stream_wildcards'.
This prep commit replaces the 'wildcard' keyword in the codebase
with 'stream_wildcard' at some places for better readability, as
we plan to introduce 'topic_wildcards' as a part of the
'@topic mention' project.

Currently, 'wildcards = ["all", "everyone", "stream"]' which is an
alias to mention everyone in the stream, hence better renamed as
'stream_wildcards'.

Eventually, we will have:
'stream_wildcard' as an alias to mention everyone in the stream.
'topic_wildcard' as an alias to mention everyone in the topic.
'wildcard' refers to 'stream_wildcard' and 'topic_wildcard' as a whole.
2023-07-03 22:03:17 -07:00
Sahil Batra 6b2ca03174 user_groups: Add support to update can_mention_group setting.
This commit adds API support to update can_mention_group setting
of a user group.

Fixes a part of #25927.
2023-06-30 17:28:33 -07:00
Sahil Batra 4bea6ffaa8 user_groups: Add support to set can_mention_group during creation.
This commit adds API support to set can_mention_group while
creating a user group.

Fixes a part of #25927.
2023-06-30 17:28:33 -07:00
Sahil Batra e6accb0ad9 user_groups: Add can_mention_group_id field to UserGroup objects.
This commit adds code to include can_mention_group_id field to
UserGroup objects passed with response of various endpoints
including "/register" endpoint and also in the group object
send with user group creation event.

Fixes a part of #25927.
2023-06-30 17:28:33 -07:00
Sahil Batra 7aaf34fd7e message_edit: Check group mention permission when editing message.
This commit adds backend code to check whether a user is allowed
to mention a user group while editing a message as per
can_mention_group setting of that group.

Fixes a part of #25927.
2023-06-30 17:28:33 -07:00
Sahil Batra 1fdffaec73 message_send: Check group mention permission when sending message.
This commit adds backend code to check whether user has permission
to mention a group while sending message as per the can_mention_group
setting of the group.

Fixes a part of #25927.
2023-06-30 17:28:33 -07:00
Sahil Batra 2763f9b575 user_groups: Add can_mention_group setting.
This commit adds a new can_mention_group setting which will be
used to determine who can mention a particular group.

Fixes a part of #25927.
2023-06-30 17:28:33 -07:00
Lauryn Menard d3f7cfccbc zerver: Update comments with "private message" or "PM".
Updates comments/doc-strings that use "private message" or "PM" in
files in the `/zerver` directory to instead use "direct message".
2023-06-23 11:24:13 -07:00
Anders Kaseorg 7657cb4a0f django_api: Extract send_event_on_commit helper.
django-stubs 4.2.1 gives transaction.on_commit a more accurate type
annotation, but this exposed that mypy can’t handle the lambda default
parameters that we use to recapture loop variables such as

    for stream_id in public_stream_ids:
        peer_user_ids = …
        event = …

        transaction.on_commit(
            lambda event=event, peer_user_ids=peer_user_ids: send_event(
                realm, event, peer_user_ids
            )
        )

https://github.com/python/mypy/issues/15459

A workaround that mypy accepts is

        transaction.on_commit(
            (
                lambda event, peer_user_ids: lambda: send_event(
                    realm, event, peer_user_ids
                )
            )(event, peer_user_ids)
        )

But that’s kind of ugly and potentially error-prone, so let’s make a
helper function for this very common pattern.

        send_event_on_commit(realm, event, peer_user_ids)

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-06-19 13:42:40 -07:00
Prakhar Pratyush e71d3ada87 settings: Add wildcard mention notifications for the followed topics.
This commit makes it possible for users to control the wildcard
mention notifications for messages sent to followed topics
via a global notification setting.

There is no support for configuring this setting
through the UI yet.
2023-06-13 18:01:41 -07:00
Prakhar Pratyush d73c715dc2 settings: Add push notifications for the followed topics.
This commit makes it possible for users to control
the push notifications for messages sent to followed topics
via a global notification setting.

There is no support for configuring this setting
through the UI yet.
2023-06-13 18:01:41 -07:00
Prakhar Pratyush 5e5538886f settings: Add email notifications for the followed topics.
This commit makes it possible for users to control
the email notifications for messages sent to followed topics
via a global notification setting.
Although there is no support for configuring this setting
through the UI yet.

Add five new fields to the UserBaseSettings class for
the "followed topic notifications" feature, similar to
stream notifications. But this commit consists only of
the implementation of email notifications.
2023-06-13 18:01:41 -07:00
Zixuan Li e39e04c3ce
migration: Add `extra_data_json` for audit log models.
Note that we use the DjangoJSONEncoder so that we have builtin support
for parsing Decimal and datetime.

During this intermediate state, the migration that creates
extra_data_json field has been run. We prepare for running the backfilling
migration that populates extra_data_json from extra_data.

This change implements double-write, which is important to keep the
state of extra data consistent. For most extra_data usage, this is
handled by the overriden `save` method on `AbstractRealmAuditLog`, where
we either generates extra_data_json using orjson.loads or
ast.literal_eval.

While backfilling ensures that old realm audit log entries have
extra_data_json populated, double-write ensures that any new entries
generated will also have extra_data_json set. So that we can then safely
rename extra_data_json to extra_data while ensuring the non-nullable
invariant.

For completeness, we additionally set RealmAuditLog.NEW_VALUE for
the USER_FULL_NAME_CHANGED event. This cannot be handled with the
overridden `save`.

This addresses: https://github.com/zulip/zulip/pull/23116#discussion_r1040277795

Note that extra_data_json at this point is not used yet. So the test
cases do not need to switch to testing extra_data_json. This is later
done after we rename extra_data_json to extra_data.

Double-write for the remote server audit logs is special, because we only
get the dumped bytes from an external source. Luckily, none of the
payload carries extra_data that is not generated using orjson.dumps for
audit logs of event types in SYNC_BILLING_EVENTS. This can be verified
by looking at:

`git grep -A 6 -E "event_type=.*(USER_CREATED|USER_ACTIVATED|USER_DEACTIVATED|USER_REACTIVATED|USER_ROLE_CHANGED|REALM_DEACTIVATED|REALM_REACTIVATED)"`

Therefore, we just need to populate extra_data_json doing an
orjson.loads call after a None-check.

Co-authored-by: Zixuan James Li <p359101898@gmail.com>
2023-06-07 12:14:43 -07:00
Sahil Batra a143d6b62c message_edit: Add period to error message raised due to time limit. 2023-05-29 14:49:32 -07:00
Mateusz Mandera db7a7d589c actions: Handle cross-realm messages in do_scrub_realm.
This is necessary to properly scrub all data from the realm.
2023-05-28 15:24:31 -07:00
Lauryn Menard 957382253a scheduled-messages: Split out edit scheduled message endpoint.
Part of splitting creating and editing scheduled messages.
Should be merged with final commit in series. Breaks tests.

Splits out editing an existing scheduled message into a new
view function and updated `edit_scheduled_message` function.
2023-05-26 18:05:55 -07:00
Lauryn Menard 154af5bb6b scheduled-messages: Remove ID from create scheduled message.
Part of splitting creating and editing scheduled messages.
Should be merged with final commit in series. Breaks tests.

Removes `scheduled_message_id` parameter from the create scheduled
message path.
2023-05-26 18:05:55 -07:00
Lauryn Menard a2a50e1408 scheduled-messages: Simplify message_to type in check_schedule_message.
As of commit 38f6807af1, we accept only stream and user IDs for
the recipient information for scheduled messages, which means we
can simplify the type for `message_to` in `check_schedule_message`.
2023-05-18 08:45:51 -07:00
Sahil Batra 007a51f277 accounts: Allow user to change email visibility during first login.
We now allow users to change email address visibility setting
on the "Terms of service" page during first login. This page is
not shown for users creating account using normal registration
process, but is useful for imported users and users created
through API, LDAP, SCIM and management commands.
2023-05-16 13:52:56 -07:00
Aman Agrawal c2145a8993 scheduled_message: Send `update` event on failed delivery. 2023-05-14 16:46:20 -07:00