Commit Graph

366 Commits

Author SHA1 Message Date
Sahil Batra 8f79cec51a users: Pass bogus data for inaccessible users.
We now pass bogus data for inaccessible users when sending
the users data in "realm_users" field of "register" response
or when using endpoints like "GET /users" to get data of
all the users in realm.

We would add a client capability field in future commits
such that new clients would receive data only for accessible
users and they can form the bogus data by themselves.
2023-11-13 08:04:45 -08:00
Sahil Batra e458b73a01 user_groups: Move constants for system group names to a new class.
This commit moves constants for system group names to a new
"SystemGroups" class so that we can use these group names
in multiple classes in models.py without worrying about the
order of defining them.
2023-11-01 10:42:56 -07:00
Sahil Batra c9fccf476a settings: Rename default_view and escape_navigates_to_default_view.
This commit renames default_view and escape_navigates_to_default_view
settings to web_home_view and web_escape_navigates_to_home_view in
database and API to match with our recent renaming of user facing
strings related to this.

We also rename the variables, functions, comments in code and class
names and IDs for elements related to this.
2023-10-26 11:00:12 -07:00
Sayyed Arib Hussain 7e38d06130
create_user: Use silent mentions in user join notices.
This provides more information (email, etc.) when clicked on
while being better adapted to email_address_visibility existing.

Fixes #27243.
2023-10-24 09:58:55 -07:00
Prakhar Pratyush 02f3f73b0c recipient_info: Include 'sender_id' in 'topic_participant_user_ids'.
This commit includes the message's sender id in the
'topic_participant_user_ids' set.

The 'participants_for_topic' function doesn't include the sender_id,
if the user is sending their first message in the topic, because
'participants_for_topic' queries the 'Message' table, but the message
is actually sent at a later stage in the codepath, resulting in
missing the sender_id in this case.

This is needed to set the 'wildcard_mentioned' flag for the sender's
user message in the case of topic wildcard mentions.

This doesn't lead to sending email and push notifications to the
sender because we have a check to skip notifications if the user
to receive notifications is the sender itself.

This should have been included in c0c30bc.
2023-10-06 11:38:37 -07:00
Prakhar Pratyush 58568a60d6 settings: Add automatically follow and unmute topics policy settings.
This commit adds two user settings, named
* `automatically_follow_topics_policy`
* `automatically_unmute_topics_in_muted_streams_policy`

The settings control the user's preference on which topics they
will automatically 'follow' or 'unmute in muted streams'.

The policies offer four options:
1. Topics I participate in
2. Topics I send a message to
3. Topics I start
4. Never (default)

There is no support for configuring the settings through the UI yet.
2023-10-04 13:04:29 -07:00
Tim Abbott c6fe799369 i18n: Fix default language for users created via API/LDAP.
This fixes a regression introduced in
9954db4b59, where the realm's default
language would be ignored for users created via API/LDAP/SAML,
resulting in all such users having English as their default language.

The API/LDAP/SAML account creation code paths don't have a request,
and thus cannot pull default language from the user's browser.

We have the `realm.default_language` field intended for this use case,
but it was not being passed through the system.

Rather than pass `realm.default_language` through from each caller, we
make the low-level user creation code set this field, as that seems
more robust to the creation of future callers.
2023-10-01 21:10:13 +02:00
Alex Vandiver c5373e9934 users: Prevent mirror dummy users from being "reactivated."
These do not represent users who were ever active, so they should not
be able to be reactivated.

See also #26703.
2023-10-01 09:29:23 -07:00
Alex Vandiver b94402152d models: Always search Messages with a realm_id or id limit.
Unless there is a limit on `id`, always provide a `realm_id` limit as
well.  We also notate which index is expected to be used in each
query.
2023-09-11 15:00:37 -07:00
Sahil Batra ada2991f1c users: Send stream creation/deletion events on role change.
We now send stream creation and stream deletion events on
changing a user's role because a user can gain or lose
access to some streams on changing their role.
2023-08-25 12:56:36 -07:00
Sahil Batra 7295028194 message: Access realm object directly from message.
We can directly get the realm object from Message object now
and there is no need to get the realm object from "sender"
field of Message object.

After this change, we would not need to fetch "sender__realm"
field using "select_related" and instead only passing "realm"
to select_related when querying Message objects would be enough.

This commit also updates a couple of cases to directly access
realm ID from message object and not message.sender. Although
we have fetched sender object already, so accessing realm_id
from message directly or from message.sender should not matter,
but we can be consistent to directly get realm from Message
object whenever possible.
2023-08-23 11:38:32 -07:00
Lauryn Menard 5e29e025c5 email-templates: Add zulip_onboarding_topics email templates.
The "followup_day2" email template name is not clear or descriptive
about the purpose of the email. Creates a duplicate of those email
template files with the template name "zulip_onboarding_topics".

Because any existing scheduled emails that use the "followup_day2"
templates will need to be updated before the current templates can
be removed, we don't do a simple file rename here.
2023-08-18 16:25:48 -07:00
Prakhar Pratyush 379a08eb1e message_send: Fix wildcard_mentioned flag unset for few participants.
For topic wildcard mentions, the 'wildcard_mentioned' flag is set
for those user messages having 'user_profile_id' in
'topic_participant_user_ids', i.e. all topic participants.

Earlier, the flag was set if the 'user_profile_id' exists in
'all_topic_wildcard_mention_user_ids'.
'all_topic_wildcard_mention_user_ids' contains the ids of those
users who are topic participants and have enabled notifications
for '@topic' mentions.

The earlier approach was incorrect, as it would set the
'wildcard_mentioned' flag only for those topic participants
who have enabled the notifications for '@topic' mention instead
of setting the flag for all the topic participants.

The bug was introduced in 4c9d26c.
2023-08-16 11:31:56 -07:00
Steve Howell 549891266d tests: Add assert_memcached_count.
We use a specific name to distinguish from other caches
like per-request caches.
2023-08-11 11:09:34 -07:00
Prakhar Pratyush 860eee94fd notifications: Rename 'pm' to 'dm' in 'RecipientInfoResult' dataclass.
This commit renames the keyword 'pm' to 'dm' in the
'pm_mention_email_disabled_user_ids' and
'pm_mention_push_disabled_user_ids' attributes of the
'RecipientInfoResult' dataclass.

'pm' and 'dm' are the acronyms for 'private message' and
'direct message' respectively.

It includes 'TODO/compatibility' code to support the old format
fields in the tornado queues during the Zulip server upgrades.
2023-08-10 17:41:49 -07:00
Prakhar Pratyush f55711dae3 test_users: Use 'do_change_user_setting' instead of '.save()'.
Use the 'do_change_user_setting' function instead of directly
using '.save()' to change the user settings values.
2023-08-07 10:01:52 -07:00
Prakhar Pratyush ef64d8df97 test_users: Update a test to use the default values for 'hamlet'.
In 'test_copy_default_settings_from_another_user', we verify that
'cordelia' and 'iago' have the same values for their user settings,
but 'hamlet' has the defaults.

Earlier, we explicitly set the 'color_scheme' setting for 'hamlet' as
'UserProfile.COLOR_SCHEME_NIGHT', which is not needed.

As we verify, 'hamlet' should have the defaults.
So just verifying if the 'color_scheme' setting for 'hamlet' is
'UserProfile.COLOR_SCHEME_AUTOMATIC' (default) fulfils our purpose.

The extra line of code was introduced in b10f156.
2023-08-07 10:01:52 -07:00
Anders Kaseorg c419c6369e users: Fix TypedDict name for get_accounts_for_email return.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-08-02 17:37:30 -07:00
Steve Howell 61a9f701bd cache: Use a single cache entry for cross-realm bots.
The cross-realm bots rarely change, and there are only
a few of them, so we just query them all at once and
put them in the cache.

Also, we put the dictionaries in the cache, instead of
the user objects, since there is nothing time-sensitive
about the dictionaries, and they are small. This saves
us a little time computing the avatar url and things
like that, not to mention marshalling costs.

This commit also fixes a theoretical bug where we would
have stale cache entries if somebody somehow modified
the cross-realm bots without bumping KEY_PREFIX.

Internally we no longer pre-fetch the realm objects for
the bots, but we don't get overly precise about picking
individual fields from UserProfile, since we rarely hit
the database and since we don't store raw ORM objects
in the cache.

The test diffs make it look like we are hitting the
cache an extra time, but the tests weren't counting
bulk fetches.  Now we only use a single key for all
bots rather a key per bot.
2023-07-25 23:08:52 -07:00
Steve Howell 0c92879f2a cross realm bots: Eliminate bulk_get_users confusion.
The bulk_get_users() function was only being used to
get cross-realm bots.

It appears that it was introduced in
f02e5b90f6 for that
specific use case.

Now we make the function more specific and test it more
accurately.

We also eliminate a lot of janky code and comments,
including some code that never had test coverage.

Incidentally, it appears that we did not have any code
to invalidate the cache keys here, and that is still
the case. In practice I assume people rarely
re-configure their cross-realm bots unless they are
upgrading the server, and then KEY_PREFIX comes into
play. 25fd4c5508 seems
to have caused that hopefully harmless regression.

A further step will be to make this cache more coarse,
since there are only a few cross-realm bots. The next
commit will hopefully simplify the code and address the
validation pitfall.
2023-07-25 23:08:52 -07:00
Anders Kaseorg 3b09197fdf ruff: Fix RUF015 Prefer `next(...)` over single element slice.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-07-23 15:20:53 -07:00
Steve Howell d19c1f7438 message fetching: Avoid duplicate cache layers.
This code removes a lot of complexity with very likely
positive overall impact on system performance and
negligible downside.

We already cache display recipients on a per-user
level, so there's no need for another cache layer on
top of that that keys them with recipient ids.

We avoid strange things where Alice/Bob and Bob/Charlie
get put into the top layer cache and then we still have
a cache miss on Alice/Charlie despite the lower level
cache being able to support per-user lookups.

This change does introduce an extra database round trip
if any of our messages have a huddle, but the query is
extremely cheap, and we can always try to cache that
function more directly or try to re-use some of our
other huddle-based caches.

As part of this, we clean up the names for the
lower-level per-user cache of display recipients, and
we simplify the cache keys.

We also stop passing in a full Recipient object to the
`bulk_get_huddle_user_ids` functions.

The local impact of this change should be easy to
measure (at least approximately), since we use this
function every time a user gets messages via the
/messages endpoint.
2023-07-19 11:07:33 -07:00
Zixuan James Li e9e18454d2 user_groups: Populate membership audit logs during realm creation.
This tracks user group membership changes when the realm is first set
up, either through an import or not. This happens when we add users to
the system user groups by their roles.

For an imported realm, we do extra handling when the data doesn't include
user groups. This gets audited as well.
2023-07-13 11:55:38 -07:00
Prakhar Pratyush c0c30bc5f7 topic_mentions: Fetch users to be notified of @topic mentions.
This commit adds the 'topic_wildcard_mention_user_ids' and
'topic_wildcard_mention_in_followed_topic_user_ids'
attributes to the 'RecipientInfoResult' dataclass.

Only topic participants are notified of @topic mentions.

Topic participants are anyone who sent a message to a topic
or reacted to a message on the topic.

'topic_wildcard_mention_in_followed_topic_user_ids' stores the
ids of the topic participants who follow the topic and have
enabled the wildcard mention notifications for followed topics.

'topic_wildcard_mention_user_ids' stores the ids of the topic
participants for whom 'user_allows_notifications_in_StreamTopic'
with setting 'wildcard_mentions_notify' returns True.
2023-07-13 11:34:48 -07:00
Lauryn Menard 0e1acd595b welcome-emails: Use followup_day2 for scheduled email tests.
The followup_day2 email is scheduled with a delay as a welcome email
and is therefore more likely to exist as a scheduled email in these
deactivation cases.
2023-07-11 14:15:52 -07:00
Zixuan James Li 44781ddfa9 user_groups: Audit UserGroup memberships changes.
This also add audit log entries during user creation and role change,
because we modify system group memberships there.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2023-07-11 08:56:55 -07:00
Prakhar Pratyush 179d5cb37d mention: Replace 'wildcards' with 'stream_wildcards'.
This prep commit replaces the 'wildcard' keyword in the codebase
with 'stream_wildcard' at some places for better readability, as
we plan to introduce 'topic_wildcards' as a part of the
'@topic mention' project.

Currently, 'wildcards = ["all", "everyone", "stream"]' which is an
alias to mention everyone in the stream, hence better renamed as
'stream_wildcards'.

Eventually, we will have:
'stream_wildcard' as an alias to mention everyone in the stream.
'topic_wildcard' as an alias to mention everyone in the topic.
'wildcard' refers to 'stream_wildcard' and 'topic_wildcard' as a whole.
2023-07-03 22:03:17 -07:00
Alex Vandiver 77c146b8b0 send_email: Delete ScheduledEmail objects with no recipients.
9d97af6ebb addressed the one major source of inconsistent data which
would be solved by simply re-attempting the ScheduledEmail row.  Every
other instance that we have seen since then has been a corrupt or
modified database in some way, which does not self-resolve.  This
results in an endless stream of emails to the administrator, and no
forward progress.

Drop this to a warning, and make it remove the offending row.  This
ensures we make forward progress.
2023-06-19 13:40:50 -07:00
Prakhar Pratyush e71d3ada87 settings: Add wildcard mention notifications for the followed topics.
This commit makes it possible for users to control the wildcard
mention notifications for messages sent to followed topics
via a global notification setting.

There is no support for configuring this setting
through the UI yet.
2023-06-13 18:01:41 -07:00
Prakhar Pratyush d73c715dc2 settings: Add push notifications for the followed topics.
This commit makes it possible for users to control
the push notifications for messages sent to followed topics
via a global notification setting.

There is no support for configuring this setting
through the UI yet.
2023-06-13 18:01:41 -07:00
Prakhar Pratyush 5e5538886f settings: Add email notifications for the followed topics.
This commit makes it possible for users to control
the email notifications for messages sent to followed topics
via a global notification setting.
Although there is no support for configuring this setting
through the UI yet.

Add five new fields to the UserBaseSettings class for
the "followed topic notifications" feature, similar to
stream notifications. But this commit consists only of
the implementation of email notifications.
2023-06-13 18:01:41 -07:00
Zixuan James Li a5cc3c5d45 users: Use a less generic response for unauthorized user creation.
This reduces confusion when an admin user tries to create users.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2023-06-05 17:33:26 -07:00
Sahil Batra 7f01b3fb63 users: Set tos_version to -1 for users who have not logged-in yet.
We now set tos_version to "-1" for imported users and the ones
created using API or using other methods like LDAP, SCIM and
management commands. This value will help us to allow users to
change email address visibility setting during first login.
2023-05-16 13:52:56 -07:00
Mateusz Mandera ffa3aa8487 auth: Rewrite data model for tracking enabled auth backends.
So far, we've used the BitField .authentication_methods on Realm
for tracking which backends are enabled for an organization. This
however made it a pain to add new backends (requiring altering the
column and a migration - particularly troublesome if someone wanted to
create their own custom auth backend for their server).

Instead this will be tracked through the existence of the appropriate
rows in the RealmAuthenticationMethods table.
2023-04-18 09:22:56 -07:00
Lauryn Menard f391bfeec6 emails: Add new onboarding email with guide for organization type.
Adds a new welcome email, `onboarding_zulip_guide`, to be sent four
days after a new user registers with a Zulip organization if the
organization has specified a particular organization type that has
a guide in the corporate `/for/.../` pages. If there is no guide,
then no email is scheduled or sent.

The current `for/communities/` page is not very useful for users
who are not organization administrators, so these onboarding guide
emails are further restricted for those organization types to
only go to new users who are invited/registered as admins for the
organzation.

Adds two database queries for new user registrations: one to get
the organization's type and one to create the scheduled email.

Adds two email logs because the email is sent both to a new user
who registers with an existing organization and to the organization
owner when they register a new organization.

Co-authored by: Alya Abbott <alya@zulip.com>
2023-04-10 08:38:09 -07:00
Prakhar Pratyush d96048b0af test_classes: Rename and refactor 'tornado_redirected_to_list'.
This commit renames the 'tornado_redirected_to_list' context
manager to 'capture_send_event_calls' to improve readability.

It also refactors the function to yield a list of events
instead of passing in a list data structure as a parameter
and appending events to it.
2023-04-07 09:45:26 -07:00
Mateusz Mandera f3db618cb5 actions: Implement do_delete_user_preserving_messages function.
With an extensive docstring by tabbott documenting its limitations.

Co-authored-by: Tim Abbott <tabbott@zulip.com>
2023-03-29 12:51:21 -07:00
Abhijeet Prasad Bodas ba0dd70904 user_topics: Introduce visibility policy enum subclass in models.py.
This commit gives more readable code than using the `VISIBILITY_POLICY`
prefix.
This commit does not alter the database schema.
2023-03-14 17:50:45 -07:00
Lauryn Menard 213d0f4990 create-user: Remove notifications sent to admin realm.
Removes the notification message that was sent if a stream named
"signups" exists in the `settings.SYSTEM_BOT_REALM`. This was a
undocumented feature that would send a notification message when
a new user registered with a Zulip organization that was hosted
by an admin realm like Zulip Cloud.

This removes two database queries when a new user is created: one
to get the system bot realm and the other to get the notification
bot in said realm.

Note that there are still notification messages sent when a new
organization is registered with the admin realm if the "signups"
stream exists.
2023-03-13 12:28:26 -07:00
Kartik Srivastava ce5d13f9b2 message_send: Handle notifications for UNMUTED topic in a muted stream.
This commit adds 'visibility_policy' as a
parameter to user_allows_notifications_in_StreamTopic
function.

This adds logic inside the user_allows_notifications_in_StreamTopic
function, to not return False when a stream is muted
but the topic is UNMUTED.

Adds a method `user_id_to_visibility_policy_dict`
to 'StreamTopicTarget' class to fetch
(user_id => visibility_policy) in single db query.

Co-authored-by: Kartik Srivastava <kaushiksri0908@gmail.com>
Co-authored-by: Prakhar Pratyush <prakhar841301@gmail.com>
2023-03-06 19:15:45 -08:00
Prakhar Pratyush 2df2ef9f0f user_topics: Refactor 'do_mute_topic'.
This commit is a step in the direction of having a common
function to handle visibility_policy changes and event
generation instead of separate functions for each
visibility policy.

In order to support different types of topic visibility policies,
this renames 'do_topic_mute' to 'do_set_user_topic_visibility_policy'
and refactors it to accept a parameter 'visibility_policy'.
2023-03-06 19:15:45 -08:00
Prakhar Pratyush 9ecc610681 mute_topic: Update tests to not use lib functions directly.
The "add_topic_mute" and "remove_topic_mute" library functions
shouldn't be called directly from tests.
They should instead call "do_mute_topic" and "do_unmute_topic"

The reason being:
Library functions are meant to be internal interfaces
for just changing the database, and shouldn't generally be
called elsewhere.
2023-03-06 19:15:45 -08:00
Lauryn Menard e9bfdd1bf2 response: Implement ignored parameters with MutableJsonResponse class.
Creates `MutableJsonResponse` as a subclass of Django's `HttpResponse`
that we can modify for ignored parameters in the response content.

Updates responses to include `ignored_parameters_unsupported` in
the response data through `has_request_variables`. Creates unit
test for this implementation in `test_decorators.py`.

The `method` parameter processed in `rest_dispatch` is not in the
`REQ` framework, so for any tests that pass that parameter, assert
for the ignored parameter with a comment.

Updates OpenAPI documentation for `ignored_parameters_unsupported`
being returned in the JSON success response for all endpoints.
Adds detailed documentation in the error handling article, and
links to that page in relevant locations throughout the API docs.

For the majority of endpoints, the documentation does not include
the array in any examples of return values, and instead links to
the error handling page. The exceptions are the three endpoints
that had previously supported this return value. The changes note
and example for these endpoints is also used in the error
handling page.
2023-03-06 10:33:13 -08:00
Sahil Batra 3950a8e19d test_helpers: Rename reset_emails_in_zulip_realm.
This commit renames reset_emails_in_zulip_realm function to
reset_email_visibility_to_everyone_in_zulip_realm which makes
it more clear to understand what the function actually does.

This commit also adds a comment explaining what this function
does.
2023-03-01 12:17:11 -08:00
Sahil Batra 0ed5f76063 settings: Add backend code for using user email_address_visibility setting.
This commits update the code to use user-level email_address_visibility
setting instead of realm-level to set or update the value of UserProfile.email
field and to send the emails to clients.

Major changes are -

- UserProfile.email field is set while creating the user according to
RealmUserDefault.email_address_visbility.

- UserProfile.email field is updated according to change in the setting.

- 'email_address_visibility' is added to person objects in user add event
and in avatar change event.

- client_gravatar can be different for different users when computing
avatar_url for messages and user objects since email available to clients
is dependent on user-level setting.

- For bots, email_address_visibility is set to EVERYONE while creating
them irrespective of realm-default value.

- Test changes are basically setting user-level setting instead of realm
setting and modifying the checks accordingly.
2023-02-10 17:35:49 -08:00
Sahil Batra 2396e6858f users: Always pass "delivery_email" field in user objects.
Previously, user objects contained delivery_email field
only when user had access to real email. Also, delivery_email
was not present if visibility setting is set to "everyone"
as email field was itself set to real email.

This commit changes the code to pass "delivery_email" field
always in the user objects with its value being "None" if
user does not have access to real email and real email otherwise.
The "delivery_email" field value is None for logged-out users.
For bots, the "delivery_email" is always set to real email
irrespective of email_address_visibility setting.

Also, since user has access to real email if visibility is set
to "everyone", "delivery_email" field is passed in that case
too.

There is no change in email field and it is same as before.

This commit also adds code to send event to update delivery_email
field when email_address_visibility setting changes to all the
users whose access to emails changes and also changes the code to
send event on changing delivery_email to users who have access
to email.
2023-02-10 17:35:49 -08:00
Anders Kaseorg 41aadc8c63 message_send: Convert RecipientInfoResult from TypedDict to dataclass.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-02-10 15:51:22 -08:00
Anders Kaseorg df001db1a9 black: Reformat with Black 23.
Black 23 enforces some slightly more specific rules about empty line
counts and redundant parenthesis removal, but the result is still
compatible with Black 22.

(This does not actually upgrade our Python environment to Black 23
yet.)

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-02-02 10:40:13 -08:00
Prakhar Pratyush 1a400b21e7 notifications: Fix missed message email notifications of welcome bot.
A missed message email notification, where the message is the welcome
message sent by the welcome bot on account creation, get sent when
the user somehow not focuses the browser tab during account creation.

No missed message email or push notifications should be sent for the
messages generated by the welcome bot.

'internal_send_private_message' accepts a parameter
'disable_external_notifications' and is set to 'True' when the sender
is 'welcome bot'.

A check is introduced in `trivially_should_not_notify`, not to notify
if `disable_external_notifications` is true.

TestCases are updated to include the `disable_external_notifications`
check in the early (False) return patterns of `is_push_notifiable` and
`is_email_notifiable`.

One query reduced for both `test_create_user_with_multiple_streams`
and `test_register`.
Reason: When welcome bot sends message after user creation
`do_send_messages` calls `get_active_presence_idle_user_ids`,
`user_ids` in `get_active_presence_idle_user_ids` remains empty if
`disable_external_notifications` is true because `is_notifiable` returns
false.
`get_active_presence_idle_user_ids` calls `filter_presence_idle_user_ids`
and since the `user_ids` is empty, the query inside the function doesn't
get executed.

MissedMessageHookTest updated.

Fixes: #22884
2023-01-24 11:16:21 -08:00
Anders Kaseorg 73c4da7974 ruff: Fix N818 exception name should be named with an Error suffix.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-17 16:52:00 -08:00
Zixuan James Li 46329a2710 test_classes: Create a dedicate helper for query count check.
This adds a helper based on testing patterns of using the "queries_captured"
context manager with "assert_length" to check the number of queries
executed for preventing performance regression.

It explains the rationale of checking the query count through an
"AssertionError" and prints the queries captured as assert_length does,
but with a format optimized for displaying the queries in a more
readable manner.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-10-17 11:32:52 -07:00
Mateusz Mandera a94b2572be do_deactivate_user: Revoke sessions in transaction.on_commit().
Fixes #21709.
2022-10-11 10:49:07 -07:00
Sahil Batra 2bf70fe4db custom_profile_field: Add "Pronouns" custom field type.
This commit adds "Pronouns" custom profile field type. We also
add "Pronouns" type field in the development environment
2022-10-06 17:56:26 -07:00
Yogesh Sirsat c355934179 custom_profile_fields: Change data of default external account type.
change the names of "github" and "twitter" external account fields to
"GitHub username" and "Twitter username" respectively and remove the
hints of them.
2022-09-12 14:29:11 -07:00
Mateusz Mandera d48075de58 logout_all_users: Add --rotate-api-keys option to the command.
Fixes #19397.
2022-08-15 17:58:05 -07:00
Julia Bichler 0a278c39d2 settings: Send email after deactivating user.
This adds a feature where an admin can choose to send an email
with custom content to an user after they deactivated them.

Fixes #18943.
2022-08-06 21:41:53 -07:00
Sahil Batra 690420ffa0 populate_db: Fix data for "Favorite editor" custom field.
The field_data sent from client while creating a select
type field is a dict with a number as key.

In development database the field data for "Favorite editor"
field was of different form where the option label was used
as key in the dict.

This commit fixes it to be of the same as it is when creating
a field from web-app. As a result, we also need to update
the tests and this commit also update field_data for other
select-type fields.
2022-07-15 16:51:24 -07:00
Zixuan James Li 58e95cbfae typing: Add none-check for test_users.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-06-23 19:25:48 -07:00
Zixuan James Li a142fbff85 tests: Refactor away result.json() calls with helpers.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-06-06 23:06:00 -07:00
Sahil Batra 708204290e streams: Capitalize "ID" in invalid stream errors in API.
This commit changes the error message from "Invalid stream id"
to "Invalid stream ID" for cases where invalid stream IDs are
passed to API endpoints to make it consistent with other similar
error messages.
2022-05-27 17:06:03 -07:00
Tim Abbott 53518e8a24 events: Pass an explicit realm to do_events_register. 2022-05-05 15:17:07 -07:00
Sahil Batra 9a6886f630 users: Always pass delivery_email in user's own object.
This commit changes the code to always pass delivery_email
field in the user's own object in 'realm_users'.

This commit also fixes the events sent by notify_created_user.
In the "realm_user/add" event sent when creating the user,
the delivery_email field was set according to the access
for the created user itself as the created user was passed as
acting_user to format_user_row. But now since we have changed
the code to always allow the user themselves to have access
to the email, this bug was caught in tests and we fix the person
object in the event to have delivery_email field based on whether
the user receiving the event has access to email or not.
2022-05-04 12:52:43 -07:00
Sahil Batra 61365fbe21 invites: Use expiration time in minutes instead of days.
This commit changes the invite API to accept invitation
expiration time in minutes since we are going to add a
custom option in further commits which would allow a user
to set expiration time in minutes, hours and weeks as well.
2022-04-20 13:31:37 -07:00
Tim Abbott f446560607 do_delete_user: Clean up acting_user logic.
This now uses our standard acting_user convention for functions called
only from management commands.
2022-04-15 15:55:21 -07:00
Anders Kaseorg 59f6b090c7 actions: Split out zerver.actions.realm_settings.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:37 -07:00
Anders Kaseorg 5d1a5a3877 actions: Split out zerver.actions.muted_users.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:36 -07:00
Anders Kaseorg cbad5739ab actions: Split out zerver.actions.create_user.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:35 -07:00
Anders Kaseorg 975066e3f0 actions: Split out zerver.actions.message_send.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:34 -07:00
Anders Kaseorg d7981dad62 actions: Split out zerver.actions.users.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:34 -07:00
Anders Kaseorg f6a06ba6e3 actions: Move part into zerver.lib.bulk_create.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:33 -07:00
Anders Kaseorg ca8d374e21 actions: Split out zerver.actions.invites.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:31 -07:00
Alex Vandiver 62ba8e455d CVE-2022-24751: Clear sessions outside of the transaction.
Clearing the sessions inside the transaction makes Zulip vulnerable to
a narrow window where the deleted session has not yet been committed,
but has been removed from the memcached cache.  During this window, a
request with the session-id which has just been deleted can
successfully re-fill the memcached cache, as the in-database delete is
not yet committed, and thus not yet visible.  After the delete
transaction commits, the cache will be left with a cached session,
which allows further site access until it expires (after
SESSION_COOKIE_AGE seconds), is ejected from the cache due to memory
pressure, or the server is upgraded.

Move the session deletion outside of the transaction.

Because the testsuite runs inside of a transaction, it is impossible
to test this is CI; the testsuite uses the non-caching
`django.contrib.sessions.backends.db` backend, regardless.  The test
added in this commit thus does not fail before this commit; it is
merely a base expression that the session should be deleted somehow,
and does not exercise the assert added in the previous commit.
2022-03-15 13:52:15 -07:00
Sahil Batra 9345c344d7 user_groups: Add members to the System groups created.
This commit adds users to the appropriate system user group
based on their role. We also change the user groups when
changing role of the user.

We also add migration to add existing users to the appropriate
user groups.

This commit adds update_users_in_full_members_system_group which
is currently used to update the full members group on changing
role of a user. This function will be modified in next commit such
that it can be used to update full members group on changing
waiting_period_threshold setting of realm.
2022-03-14 18:53:46 -07:00
Kartik Srivastava eefaa9120f user_topic: Rename topic_mutes.py to user_topics.py. 2022-03-11 14:26:55 -08:00
Sahil Batra 392b17da5f invite: Add backend support for "Never expires" option.
The database value for expiry_date is None for the invite
that will never expire and the clients send -1 as value
in the API similar to the message retention setting.

Also, when passing invite_expire_in_days as an argument
in various functions, invite_expire_in_days is passed as
-1 for "Never expires" option since invite_expire_in_days
is an optional argument in some functions and thus we cannot
pass "None" value.
2022-02-24 16:32:19 -08:00
Mateusz Mandera 0dd97eeaab do_set_realm_property: Use transaction.atomic. 2022-01-28 13:03:39 -08:00
Mateusz Mandera 93e8740218 do_deactivate_user: Revoke invitations generated by the user.
Leaving old invitations valid, potentially for a very long time, is
clearly unexpected and undesired behavior under normal circumstances. A
user shouldn't be able to e.g. generate a multiuse invite link, get
banned from the organization by being deactivated and then just re-join
using the link they've created for themselves.
2022-01-12 13:53:34 -08:00
Mateusz Mandera 30ccb76e19 do_delete_user: Preserve date_joined value of the user. 2022-01-04 15:42:03 -08:00
Mateusz Mandera 208c0c3034 do_delete_user: Use get_fake_email_domain for dummy user email domain.
Otherwise the dummy user can be created with an invalid email domain -
e.g. in development environment with the domain
"@http://localhost:9991". get_fake_email_domain exists exactly for
handling these kinds of scenarios.
2022-01-04 15:42:03 -08:00
Mateusz Mandera dffdeb48e7 do_delete_user: Make the replacement dummy user inactive.
Otherwise, the dummy user will show up in the user list in the right
sidebar.
2022-01-04 15:42:03 -08:00
Abhijeet Prasad Bodas 15e8717847 notifications: Don't enqueue notifications for bots.
This replaces the temporary (and testless) fix in
24b1439e93 with a more permanent
fix.

Instead of checking if the user is a bot just before
sending the notifications, we now just don't enqueue
notifications for bots. This is done by sending a list
of bot IDs to the event_queue code, just like other
lists which are used for creating NotificationData objects.

Credit @andersk for the test code in `test_notification_data.py`.
2022-01-03 09:55:06 -08:00
Steve Howell 1e4593b2ae performance: Avoid Recipient lookup.
We don't have to go to the database to get the Recipient
fields for `user_profile.recipient`.

See also 85ed6f332a from a little
over a year ago--it's very similar.
2021-12-28 12:15:02 -08:00
Steve Howell f638fd6f72 performance: Get used stream colors in separate trip.
This commit sets us up for the next commit, which will
save us a very expensive query.

If you are adding 15k users to a stream, and each user
has about 20 existing streams, then we need to retrieve
300k rows from the database to figure out which stream
colors they already have.  We don't need all the extra
fields from Subscription, so now we get just the two
values we need for making a color map.

In the next commit we'll eliminate the other use case
for the big query, and I will explain in greater
depth how splitting out the color-picking code can
be a huge win. It is possible that some product decisions
could make this codepath easier. We could also do some
engineering specific to stream colors, such as caching
which colors users have already used.

This does cost us an extra round trip to the database.
2021-12-28 12:15:02 -08:00
Nikhil Maske 091772b534 hotspots: Remove intro_reply hotspot.
Zulip shows two guides on How to reply, first one by
the welcome bot and second one is intro_reply hotspot.
To simply and avoid redundancy, intro_reply hotspot is
removed.

Fixes #20482.
2021-12-07 21:55:59 -08:00
Sahil Batra 9a03a78e3d test_users: Use delivery_email instead of email to generate gravatar url.
We always use delivery_email to generate gravatar_url, but in
test_admin_api_hide_emails we were passing email to get_gravatar_url
and matched with the avatar_url field of the fetched user object.

The tests were passing because the email_address_is_realm_public
was using old realm object and thus email field was incorrectly
set to delivery_email even when email_address_visibility was set
to EMAIL_ADDRESS_VISIBILITY_ADMINS.

This commit fixes the test to pass delivery_email to get_gravatar_url.
2021-10-29 16:19:55 -07:00
Gaurav Pandey 1c1a1e2cad api: Remove encoding of string in update_user endpoint.
* Remove unnecessary json_validator for full_name parameter.
* Update frontend to pass the right parameter.
* Update documentation and note the change.

Fixes #18409.
2021-10-18 19:03:43 -07:00
Sahil Batra 6be20a8c6d users: Allow admins to get subscription status in private stream.
We pass allow_realm_admin as True to access_stream_by_id for
`GET users/{user_id}/subscriptions{stream_id}` endpoint
because we want to allow non-subscribed admins to get
subscription status in private streams.

Fixes #19077.
2021-10-15 10:58:56 -07:00
Mateusz Mandera c4edda016f models: Remove property decorator from UserProfile.profile_data.
Having this as a property is misleading, given that it does db queries
to fetch the CustomProfileField data.
2021-09-22 16:22:46 -07:00
shanukun 8c1ea78d7d invite: Extend invite api for handling expiration duration.
This extends the invite api endpoints to handle an extra
argument, expiration duration, which states the number of
days before the invitation link expires.

For prereg users, expiration info is attached to event
object to pass it to invite queue processor in order to
create and send confirmation link.
In case of multiuse invites, confirmation links are
created directly inside do_create_multiuse_invite_link(),

For filtering valid user invites, expiration info stored in
Confirmation object is used, which is accessed by a prereg
user using reverse generic relations.

Fixes #16359.
2021-09-10 16:53:03 -07:00
Tim Abbott f1c0309971 test_users: Fix comparison of differently sorted sets.
This fixes a nondeterministic test failure in main.
2021-09-09 11:30:43 -07:00
sahil839 8bab712cb4 create_user: Initialize settings according to realm-level defaults.
This commit modifies the copy_user_settings code such that instead
of source user profile, we can have two types of sources - a user
profile and RealmUserDefault table of realm and then set the
settings from RealmUserDefault only is there is no user profile
as a source.

We also rename copy_user_settings to copy_default_settings for
clarity.
2021-09-09 10:55:19 -07:00
Sahil Batra 03a0cfda64 settings: Set enable_marketing_emails after copying user settings.
We set the enable_marketing_emails setting after copying user
settings to override the value selected in registration form.
This change is also necessary because enable_marketing_emails
field is present in RealmUserDefault to avoid copying code
but we do not use this value actually and instead we want
the setting to be set according to the value in registration
form.

We set this setting only for non-bot users since we generally
do not set any settings for bots.
2021-09-09 10:15:07 -07:00
Wesley Aptekar-Cassels f743dda008 emoji: Change default emojiset from Google classic to Google modern.
This is a step towards updating our emoji to support newer Unicode
emojis (#19371).

See CZO for discussion:
https://chat.zulip.org/#narrow/stream/2-general/topic/emoji.20version.3F/near/1239756
2021-09-09 02:00:36 +05:30
Shelly 6d92534d51 models: Add setters for is_realm_owner and is_moderator.
This fixes a regression where one could end up deactivating all owners
of a realm when trying to synchronize LDAP with the `is_realm_admin`
flag configured in `AUTH_LDAP_USER_FLAGS_BY_GROUP`.

With tweaks by tabbott to add is_moderator as well.

Fixes #18677.
2021-09-07 17:15:39 -07:00
PIG208 aa9d73c9f6 typing: Improve typing with assertions.
This fixes some mypy errors discovered with django-stubs.
2021-08-20 05:54:19 -07:00
Anders Kaseorg 79f7f9b8d9 actions: Remove unused get_emails_from_user_ids function.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-19 01:51:37 -07:00
Alex Vandiver 56344e4765 scheduled_email: Only call .save() if necessary.
Calling `email.save()` is only needed if we altered `email.address`;
it is unnecessary if we called `email.users.add(...)` which will have
done its own INSERT.
2021-08-19 00:44:33 -07:00
Alex Vandiver ebaafb32f3 send_email: Change clear_scheduled_emails to only take one user.
No codepath except tests passes in more than one user_profile -- and
doing so is what makes the deduplication necessary.

Simplify the API by making it only take one user_profile id.
2021-08-19 00:44:33 -07:00
Anders Kaseorg 3665deb93a python: Remove unnecessary intermediate lists.
Generated automatically by pyupgrade.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-02 15:53:52 -07:00
Abhijeet Prasad Bodas de04f0ad67 notifications: Calculate PMs/mentions settings like other settings.
Previously, we checked for the `enable_offline_email_notifications` and
`enable_offline_push_notifications` settings (which determine whether the
user will receive notifications for PMs and mentions) just before sending
notifications. This has a few problem:

1. We do not have access to all the user settings in the notification
handlers (`handle_missedmessage_emails` and `handle_push_notifications`),
and therefore, we cannot correctly determine whether the notification should
be sent. Checks like the following which existed previously, will, for
example, incorrectly not send notifications even when stream email
notifications are enabled-
```
if not receives_offline_email_notifications(user_profile):
    return
```
With this commit, we simply do not enqueue notifications if the "offline"
settings are disabled, which fixes that bug.

Additionally, this also fixes a bug with the "online push notifications"
feature, which was, if someone were to:
* turn off notifications for PMs and mentions (`enable_offline_push_notifications`)
* turn on stream push notifications (`enable_stream_push_notifications`)
* turn on "online push" (`enable_online_push_notifications`)

then, they would still receive notifications for PMs when online.
This isn't how the "online push enabled" feature is supposed to work;
it should only act as a wrapper around the other notification settings.

The buggy code was this in `handle_push_notifications`:
```
if not (
    receives_offline_push_notifications(user_profile)
    or receives_online_push_notifications(user_profile)
):
    return

    // send notifications
```

This commit removes that code, and extends our `notification_data.py` logic
to cover this case, along with tests.

2. The name for these settings is slightly misleading. They essentially
talk about "what to send notifications for" (PMs and mentions), and not
"when to send notifications" (offline). This commit improves this condition
by restricting the use of this term only to the database field, and using
clearer names everywhere else. This distinction will be important to have
non-confusing code when we implement multiple options for notifications
in the future as dropdown (never/when offline/when offline or online, etc).

3. We should ideally re-check all notification settings just before the
notifications are sent. This is especially important for email notifications,
which may be sent after a long time after the message was sent. We will
in the future add code to thoroughly re-check settings before sending
notifications in a clean manner, but temporarily not re-checking isn't
a terrible scenario either.
2021-07-28 13:55:25 -07:00