Commit Graph

52432 Commits

Author SHA1 Message Date
Sahil Batra d0fe378075 cache_helpers: Pass "realm" and "bot_owner" args to select_related.
This commit updates code to pass "realm" and "bot_owner" args to
select_related call in get_users. We pass "realm" and "bot_owner"
args to get_users because the caches which this function is used
to populate are used for get_user and get_user_profile_by_api_key
functions and they also select both these fields when querying for
UserProfile objects.
2023-07-20 10:44:39 -07:00
Sahil Batra 3e09a21929 models: Pass realm and bot_owner as args to select_related.
This commit updates the select_related calls in queries to get
UserProfile objects in get_user, get_user_by_delivery_email,
get_user_profile_by_id, get_user_profile_by_id_in_realm and
get_user_profile_by_api_key functions to pass "realm" and
"bot_owner" as arguments to select_related call.

These functions are used in different parts of code to get
the UserProfile object and realm is accessed using the user
object at many places.

"bot_owner" field is also used in some places like to check
whether a bot can access a stream, to check whether a user
can change modify another user, in webhooks code to send the
message to the bot owner, and in tests as well. There can be
some places where the bot owner is not required and in most
such cases the code would only be accessed for human users,
which means the bot_owner will be null for these cases and
would avoid complexity and performance issues.

Note that previously, no arguments were passed to select_related
and thus only realm field was fetched during the query.
2023-07-20 10:44:39 -07:00
Sahil Batra 71c66cd75c models: Pass realm as arg to select_related in get_system_bot.
This commit updates the select_related calls in queries to
get UserProfile object in get_syste_bot function pass "realm"
as argument to select_related call.

The "get_system_bot" call function is mostly used to get cross
realm bot which are used as senders to send messages.
The fields like default_events_register_stream and recipient
are not required for these cases. The bot_owner field is used
to check access to a stream to send message but the cross-realm
bots are handled differently and the bot_owner check is not
required.

Also, note that "realm" is the only non-null foreign key field
in UserProfile object, so select_related() was only fetching
realm object previously as well. But we should still pass
"realm" as argument in select_related call so that we can make
sure that only required fields are selected in case we add
more foreign keys to UserProfile in future.
2023-07-20 10:44:39 -07:00
Sahil Batra ce89cab667 management: Pass realm as arg to select_related in get_user.
This commit updates the select_related calls in queries to
get UserProfile objects in get_user function called in
management commands to pass "realm" as argument to
select_related call.

There are some management commands like deactivate_user,
change_full_name, etc. which might need fields like
"default_sending_stream" when changing full name of a bot
or something similar, but we don't think that would happen
often and we can afford to have a DB round trip to get
these fields if needed.

Also, note that "realm" is the only non-null foreign key
field in UserProfile object, so select_related() was only
fetching realm object previously as well. But we should
still pass "realm" as argument in select_related call so
that we can make sure that only required fields are
selected in case we add more foreign keys to UserProfile
in future.
2023-07-20 10:44:39 -07:00
Sahil Batra 63cc18a94b sync_ldap_user_data: Pass realm as arg to select_related.
This commit updates the select_related calls in queries
to get UserProfile objects in sync_ldap_user_data code
to pass "realm" as argument to select_related call.

Also, note that "realm" is the only non-null foreign key
field in UserProfile object, so select_related() was only
fetching realm object previously as well. But we should
still pass "realm" as argument in select_related call so
that we can make sure that only required fields are
selected in case we add more foreign keys to UserProfile
in future.
2023-07-20 10:44:39 -07:00
Sahil Batra 584026b21f models: Pass realm as arg to select_related in get_user_profile_by_email.
This commit updates select_related call in get_user_profile_by_email
to pass "realm" as argument.

This function is intended to be used for manual manage.py shell
work so we just keep the behavior same as before as "realm" is
the only non-null related field in UserProfile.
2023-07-20 10:44:39 -07:00
Sahil Batra 577555e529 send_custom_email: Pass realm as arg to select_related.
This commit updates the select_related calls in queries
to get UserProfile objects in send_custom_email code to
pass "realm" as argument to select_related call.

Also, note that "realm" is the only non-null foreign key
field in UserProfile object, so select_related() was only
fetching realm object previously as well. But we should
still pass "realm" as argument in select_related call so
that we can make sure that only required fields are selected
in case we add more foreign keys to UserProfile in future.
2023-07-20 10:44:39 -07:00
Sahil Batra 3ae0b4f913 dev_login: Pass realm as arg to select_related calls.
This commit updates the select_related calls in queries to
get UserProfile objects in dev_login code to pass "realm"
as argument to select_related call.

Also, note that "realm" is the only non-null foreign key field
in UserProfile object, so select_related() was only fetching
realm object previously as well. But we should still pass "realm"
as argument in select_related call so that we can make sure that
only required fields are selected in case we add more foreign
keys to UserProfile in future.
2023-07-20 10:44:39 -07:00
Sahil Batra 62f01edee3 populate_db: Pass realm as arg to select_related calls.
This commit updates the select_related calls in queries to
get UserProfile objects in populate_db code to pass "realm"
as argument to select_related call.

Also, note that "realm" is the only non-null foreign key field
in UserProfile object, so select_related() was only fetching
realm object previously as well. But we should still pass "realm"
as argument in select_related call so that we can make sure that
only required fields are selected in case we add more foreign
keys to UserProfile in future.
2023-07-20 10:44:39 -07:00
Sahil Batra c0029319f9 users: Pass realm as arg to select_related in fetch_users_by_id.
This commit updates select_related call to pass "realm" as
argument in select_related call in fetch_users_by_id function
as we only require realm for the UserProfile objects fetched
using fetch_users_by_id.

Also, note that "realm" is the only non-null foreign key field
in UserProfile object, so select_related() was only fetching
realm object previously as well. But we should still pass "realm"
as argument in select_related call so that we can make sure that
only required fields are selected in case we add more foreign
keys to UserProfile in future.
2023-07-20 10:44:39 -07:00
Sahil Batra bb3945a32f models: Remove select_related call in get_active_users.
We do not use any related fields for the UserProfile objects
fetched by get_active_users, so we can simply remove the
select_related call.

The user object from get_active_users was used to get realm
but since get_active_users called from a realm object we can
directly use that realm object. This change also leads to
some changes in the cache code where we now pass the realm
to the function instead of selecting it from UserProfile object.
2023-07-20 10:44:39 -07:00
Sahil Batra eda3879733 soft_deactivation: Remove select_related call.
This commit removes select_related call from
get_soft_deactivated_users_for_catch_up as
we do not use any related fields for the
UserProfile objects fetched using this call.
2023-07-20 10:44:39 -07:00
Sahil Batra 290973585c cache_helpers: Remove select_related call for Client.
There are no foreign keys for Client and so there are
no related objects to select using select_related.
2023-07-20 10:44:39 -07:00
Lalit a871d7d90e poll_data: Revert to using "canned" instead of -1.
We revert to using "canned" instead of -1 when populating
`key_to_option` map so that it does not break for the old polls.
2023-07-20 09:55:09 -07:00
Lauryn Menard 072a62d5b5 api-changelog: Add entry for feature level 193 updates.
Adds an entry to the API changelog for feature level 193. The
changes documented here are from commit 3802503a8a.
2023-07-20 09:52:28 -07:00
Alex Vandiver 8743602648 puppet: Allow access to smokescreen metrics on CZO. 2023-07-19 16:20:39 -07:00
Alex Vandiver d957559371 uploads: Allow uploads to set storage class.
Uploads are well-positioned to use S3's "intelligent tiering" storage
class.  Add a setting to let uploaded files to declare their desired
storage class at upload time, and document how to move existing files
to the same storage class.
2023-07-19 16:19:34 -07:00
Alex Vandiver ba7492a314 web: Do not assume error responses are valid, or our, JSON.
Partially fixes #24815.
2023-07-19 16:18:31 -07:00
Alex Vandiver 7efe989a72 web: Switch from JSON.parse(xhr.responseText) to xhr.responseJSON. 2023-07-19 16:18:31 -07:00
Alex Vandiver d05a1e9efa web: Remove nonsense round-trip through JSON.parse() + JSON.stringify(). 2023-07-19 16:18:31 -07:00
Alex Vandiver d05a9d4000 reactions: Do not send server error reports for duplicate requests. 2023-07-19 16:18:31 -07:00
Alex Vandiver 871a668dd2 reactions: Add error code for duplicate addition/removal. 2023-07-19 16:18:31 -07:00
Anders Kaseorg 29bdaaf5b5 requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-07-19 16:14:59 -07:00
Anders Kaseorg 32a8151ce8 profile_request: Support only synchronous responses for now.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-07-19 16:14:59 -07:00
Anders Kaseorg ff6cf54716 pyproject: Remove mypy exemption for coverage.
coverage added type annotations in 7.2.0.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-07-19 16:14:59 -07:00
Anders Kaseorg 195efb3802 name_restrictions: Update disposable_email_domains usage.
‘blocklist’ was added in 0.0.35 (with backwards compatibility for the
old name), and type annotations were added in 0.0.91 (with only the
new name).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-07-19 16:14:59 -07:00
Anders Kaseorg d87eea1a67 ruff: Fix B034 `re.split`, `re.sub` should pass keyword arguments.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-07-19 16:14:59 -07:00
Anders Kaseorg 50e6cba1af ruff: Fix UP032 Use f-string instead of `format` call.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-07-19 16:14:59 -07:00
Anders Kaseorg 0efc662eab ruff: Fix RUF015 Prefer `next(iter(…))` over `list(…)[0]`.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-07-19 16:14:59 -07:00
Anders Kaseorg 9bb3d15a79 openapi: Switch to new openapi_core validation API.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-07-19 16:14:59 -07:00
Alex Vandiver 60ce5e1955 wal-g: Use "start_time" field, not "time" which is S3 modified-at.
The `time` field is based on the file metadata in S3, which means that
touching the file contents in S3 can move backups around in the list.

Switch to using `start_time` as the sort key, which is based on the
contents of the JSON file stored as part of the backup, so is not
affected by changes in S3 metadata.
2023-07-19 14:57:51 -07:00
Alex Vandiver 1accc6929e upgrade: Check PostgreSQL versions after venv setup.
The claim in the comment from c8ec3dfcf6, that we can and should use
the current deploy's venv, misses one key case -- when upgrading the
operating system, the current deploy's venv is unworkable, since it
was configured for a previous version of Python.  As such, any attempt
to load Django to verify the version of PostgreSQL it is talking to
must happen after the venv is configured.

Move the database version check into
`scripts/lib/check-database-compatibility`, which also moves it after
the new venv is configured.

Because we no longer reliably know, at `apt-get upgrade` time, what
version of PostgreSQL is installed, we hold all versions of the
pgroonga packages.
2023-07-19 14:19:28 -07:00
Aman Agrawal b7fcce305e recent_topics: Fix clear search button misaligned.
Looks like some changes actually improved aligned of this icon so
that we don't need this hacky margin.
2023-07-19 11:09:24 -07:00
Steve Howell 3599b1662e cache: Eliminate transformed_bulk_cached_fetch.
Its two callers now just directly call
generic_bulk_cached_fetch with the explicit `lambda
obj: obj` helpers.
2023-07-19 11:07:33 -07:00
Steve Howell d19c1f7438 message fetching: Avoid duplicate cache layers.
This code removes a lot of complexity with very likely
positive overall impact on system performance and
negligible downside.

We already cache display recipients on a per-user
level, so there's no need for another cache layer on
top of that that keys them with recipient ids.

We avoid strange things where Alice/Bob and Bob/Charlie
get put into the top layer cache and then we still have
a cache miss on Alice/Charlie despite the lower level
cache being able to support per-user lookups.

This change does introduce an extra database round trip
if any of our messages have a huddle, but the query is
extremely cheap, and we can always try to cache that
function more directly or try to re-use some of our
other huddle-based caches.

As part of this, we clean up the names for the
lower-level per-user cache of display recipients, and
we simplify the cache keys.

We also stop passing in a full Recipient object to the
`bulk_get_huddle_user_ids` functions.

The local impact of this change should be easy to
measure (at least approximately), since we use this
function every time a user gets messages via the
/messages endpoint.
2023-07-19 11:07:33 -07:00
Steve Howell b85d3dd65b recipient caches: Split up bulk-fetching.
The only overlap between how we fetched streams and
users was to share some really complicated data
structures.

We can also short-circuit some logic if a message
batch is either all-stream or all-DM.
2023-07-19 11:07:33 -07:00
Steve Howell 03557a5568 huddles: Find huddle user ids more efficiently.
We restrict the columns, avoid quadratic looping,
and don't bother with order_by.

We also return the user ids (per recipient) as
sets, since that's how the only caller uses the
info (albeit implicitly via set.union accepting
a list).
2023-07-19 11:07:33 -07:00
Alex Vandiver 5a26237b54 wal-g: Support alternate S3 storage classes. 2023-07-19 10:55:18 -07:00
Alex Vandiver 52eacd30c5 wal-g: Set WALG_S3_PREFIX, instead of WALE_S3_PREFIX.
The `WALE_` prefix was only used for backwards compatibility.  Switch
to the canonical variable name.
2023-07-19 10:55:18 -07:00
Anders Kaseorg 052984bc14 utils: Remove make_safe_digest wrapper.
It’s unclear what was supposed to be “safe” about this wrapper.  The
hashlib API is fine without it, and we don’t want to encourage further
use of SHA-1.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-07-19 10:54:05 -07:00
Anders Kaseorg 143baa4243 python: Convert translated positional {} fields to {named} fields.
Translators benefit from the extra information in the field names, and
need the reordering freedom that isn’t available with multiple
positional fields.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-07-18 15:19:07 -07:00
Anders Kaseorg db6323b2c4 python: Remove unused arguments for translated format strings.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-07-18 10:38:46 -07:00
Alex Vandiver b188e6fa04 management: Add a reactivate-stream command.
Fixes #601.
2023-07-17 17:42:54 -07:00
Prakhar Pratyush 1a2831059d notifications: Enable audible notifications for the FOLLOWED topic.
This commit adds support for triggering audible desktop
notifications when:

* A message is sent to a followed topic with the global user setting
'enable_followed_topic_audible_notifications' enabled.

* A message with wildcard mentions is sent to a followed topic with
the global user setting
'enable_followed_topic_wildcard_mentions_notify' enabled.
2023-07-17 17:39:17 -07:00
Prakhar Pratyush 67d4334bf1 notifications: Enable desktop notifications for the FOLLOWED topic.
This commit adds support for triggering visual desktop
notifications when:

* A message is sent to a followed topic with the global user setting
'enable_followed_topic_desktop_notifications' enabled.

* A message with wildcard mentions is sent to a followed topic with
the global user setting
'enable_followed_topic_wildcard_mentions_notify' enabled.
2023-07-17 17:39:17 -07:00
Prakhar Pratyush 237f9f4799 topic_popover: Add support to follow a topic from the topic_popover.
This commit adds support for following a topic from the topic
three-dot menu in the left sidebar.

Three options, i.e., 'Mute topic', 'Unmute topic', and 'Follow topic',
are shown at the top of the menu (in the mentioned order), regardless
of whether the stream is muted or unmuted.

We can no longer set the topic's visibility_policy to INHERIT
(the default value) from this menu.

The changes are visible in the development environment only.

Fixes #25917.

Co-authored-by: Prakhar Pratyush <prakhar841301@gmail.com>
Co-authored-by: Hardik Dharmani <Ddharmani99@gmail.com>
2023-07-17 17:39:17 -07:00
Prakhar Pratyush 93a18b999a settings: Update SETTINGS/TOPICS panel to include followed topics.
This commit updates the existing 'Topics' settings UI
to add support for followed topics.

Changes:
- Add "Followed" as the first option in the topic status dropdowns.
- Modify the intro text.
- Replace the "Learn more." link with a question mark linking to
'/help/mute-a-topic' on the "Topic settings" heading.

The changes are visible in the development environment only.

Fixes #25918.
2023-07-17 17:39:17 -07:00
Prakhar Pratyush 352c48f7c6 user_topics: Fix live-update bug in the 'SETTINGS / TOPICS' panel.
The change in the visibility_policy or status of a topic from
the topic popover, message header bar, or recent conversations
is not live-reflected in the 'SETTINGS / TOPICS' panel.

This commit fixes the above-mentioned bug by updating
the 'handle_topic_updates' function to update the settings panel.
2023-07-17 17:39:17 -07:00
Prakhar Pratyush 08c952a2a3 user_topics: Add the missing node tests.
This commit adds the missing node tests for the case
when the visibility policy is UNMUTED.

This should have been included in c526543.
2023-07-17 17:39:17 -07:00
Prakhar Pratyush 32b201d5e0 user_topics: Rename 'muted_topics_ui' to 'user_topics_ui'.
This commit renames the file 'muted_topics_ui.js' to
'user_topics_ui.js', as we also support visibility policies
other than MUTED.
2023-07-17 17:39:17 -07:00