Commit Graph

46954 Commits

Author SHA1 Message Date
Mateusz Mandera 6a6fd38d03 models: Reserve a RealmAuditLog.USER_DELETED_PRESERVING_MESSAGES number.
The relevant function is waiting to be merged in #21299 - but we have
already used it on Zulip Cloud, creating RealmAuditLog entries with the
number 107 and thus should reserve it before another PR takes
it for another purpose, creating confusion in the logs.
2022-04-17 16:17:58 -07:00
Tim Abbott b63757621a realm_audit_log: Set acting_user=None explicitly, with comments.
Based on an audit, this closes out the last core instances in which
acting_user was not being passed explicitly when creating
RealmAuditLog instances.

There are some outstanding issues in the billing system, which we plan
to extract as a separate issue.

Fixes #14808.
2022-04-15 15:55:21 -07:00
Tim Abbott f446560607 do_delete_user: Clean up acting_user logic.
This now uses our standard acting_user convention for functions called
only from management commands.
2022-04-15 15:55:21 -07:00
Alex Vandiver 7cc9b93b91 settings: Remove misleading and irrelevant comment.
This comment was _originally_ for the `default` memcached cache, back
when it was added all of the way back in 0a84d7ac62.  9e64750083
made it a lie, and edc718951c made it even more confusing when it
removed the `default` cache configuration block, leaving the wrong
comment next to the wrong cache configuration block.

Banish the comment.
2022-04-15 14:48:12 -07:00
Alex Vandiver 9030d53acb test_link_embed: Remove unnecessary TEST_CACHES.
The only purpose of this seems to be to not have to reset the cache;
fae59502ab added it without any explanation for why it is necessary.

Remove it, and explicitly flush the cache in the one place where it is
necessary.
2022-04-15 14:48:12 -07:00
Alex Vandiver 56058f3316 caches: Remove unnecessary "in-memory" cache.
This cache was added in da33b72848 to serve as a replacement for the
durable database cache, in development; the previous commit has
switched that to be the non-durable memcached backend.

The special-case for "in-memory" in development is mostly-unnecessary
in contrast to memcached -- `./tools/run-dev.py` flushes memcached on
every startup.  This differs in behaviour slightly, in that if the
codepath is changed and `run-dev` restarts Django, the cache is not
cleared.  This seems an unlikely occurrence, however, and the code
cleanup from its removal is worth it.
2022-04-15 14:48:12 -07:00
Alex Vandiver 04ca2e92f7 caches: Cache link preview data in memcached, not in PostgreSQL.
The choice to cache these in the database dates back to c93f1d4eda,
with the comment added in da33b72848 while working around the
durability of the "database" cache in local development.

The values were stored in a durable cache, as they needed to be
ensured to persist between when they were inserted in
`get_link_embed_data` and when they were used in
`render_incoming_message` via `link_embed_data_from_cache`.

However, database accesses are not fast compared to memcached, and we
wish to avoid the overhead of the database connection from the
`embed_links` worker.  Specifically, making the connection may not be
thread-safe -- and in low-memory (and Docker) configurations, all
workers run as separate threads in a single process.  This can lead to
stalled database connections in `embed_links` workers, and failed
previews.

Since the previous commit made the durability of the cache no longer
necessary, this will have minimal effect; at worst, posting the same
URL twice, on either side of an upgrade, will result in two preview
fetches of it.
2022-04-15 14:48:12 -07:00
Alex Vandiver 351bdfaf78 preview: Use cache only as a non-durable cache, not an IPC.
The `get_link_embed_data` / `link_embed_data_from_cache` pair as
introduced in c93f1d4eda uses the cache
as a temporary store inside of the `embed_links` worker; this means
that it must be durable storage, or the worker will stall and re-fetch
the same links to preview them.

Switch to plumbing through the fetched URL embed data as an parameter
to the Markdown evaluation which uses them, rather than using the
cache as an intermediary.  This frees up the cache to be merely a
non-durable cache.

As a side-effect, this removes get_cache_with_key, and
link_embed_data_from_cache which was its only callsite.
2022-04-15 14:48:12 -07:00
Alex Vandiver 327ff9ea0f preview: Use a dataclass for the embed data.
This is significantly cleaner than passing around `Dict[str, Any]` all
of the time.
2022-04-15 14:48:12 -07:00
Alex Vandiver ede4a88b49 populate_db: Remove unnecessary pre-population of URL cache.
76deb30312 changed this to not just be the URL, but rather a
prefixed hash of the URL, but failed to update this location which
wrote to it.  This meant that this pre-population step was writing to
the wrong keys in the durable cache, and thus ineffective.

Then, da33b72848 switched the cache to be in-memory, making this
write to the wrong keys in an in-process memory store.  There is no
way to pre-fill this sort of cache, except at server start-up.

Finally, and most fundamentally, 8c0c9ca7a4 then disabled
`inline_url_embed_preview` by default, making the code entirely moot.

Remove the triply-unnecessary code.
2022-04-15 14:48:12 -07:00
Alex Vandiver aaa58a49db cache: Make the cache_name=None behaviour clearer.
`django.core.cache.cache` is equal to
`django.core.cache.caches["default"]`; the latter is more
understandable in context.
2022-04-15 14:48:12 -07:00
Alex Vandiver 661c333377 markdown: Use named parameters to add_a helper.
This has enough parameters that it benefits from making which is which
explicit.
2022-04-15 14:48:12 -07:00
Alex Vandiver 452a30305d markdown: Clarify url parameter of "add_a" helper. 2022-04-15 14:48:12 -07:00
Mateusz Mandera fcf82bf047 digest: Don't send emails to deactivated users, even if queued. 2022-04-15 14:32:55 -07:00
Mateusz Mandera 7a13836d26 test_digest: Fix typo in a comment. 2022-04-15 14:32:55 -07:00
Anders Kaseorg cc30ed8ec7 actions: Delete zerver.lib.actions.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:38 -07:00
Anders Kaseorg 729019acdd actions: Move part into zerver.lib.test_classes.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:37 -07:00
Anders Kaseorg e01faebd7e actions: Split out zerver.actions.create_realm.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:37 -07:00
Anders Kaseorg 53f4a395bc actions: Split out zerver.actions.realm_domains.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:37 -07:00
Anders Kaseorg 59f6b090c7 actions: Split out zerver.actions.realm_settings.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:37 -07:00
Anders Kaseorg 12de8d797e actions: Move part into zerver.forms.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:36 -07:00
Anders Kaseorg eda000899b actions: Split out zerver.actions.message_edit.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:36 -07:00
Anders Kaseorg 5d1a5a3877 actions: Split out zerver.actions.muted_users.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:36 -07:00
Anders Kaseorg ec174dfb47 actions: Split out zerver.actions.bots.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:36 -07:00
Anders Kaseorg eb4e9fe1e7 actions: Split out zerver.actions.message_flags.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:36 -07:00
Anders Kaseorg e5500a2226 actions: Split out zerver.actions.reactions.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:35 -07:00
Anders Kaseorg cbad5739ab actions: Split out zerver.actions.create_user.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:35 -07:00
Anders Kaseorg 5fcbc412cf actions: Split out zerver.actions.streams.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:35 -07:00
Anders Kaseorg 975066e3f0 actions: Split out zerver.actions.message_send.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:34 -07:00
Anders Kaseorg ec6355389a actions: Split out zerver.actions.user_settings.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:34 -07:00
Anders Kaseorg d7981dad62 actions: Split out zerver.actions.users.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:34 -07:00
Anders Kaseorg bbce879c81 actions: Split out zerver.actions.custom_profile_fields.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:33 -07:00
Anders Kaseorg f6a06ba6e3 actions: Move part into zerver.lib.bulk_create.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:33 -07:00
Anders Kaseorg c041b68578 actions: Move part into zerver.lib.message.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:33 -07:00
Anders Kaseorg 9dd7e34ab3 actions: Move part into zerver.lib.subscription_info.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:33 -07:00
Anders Kaseorg b7adfb02f6 actions: Split out zerver.actions.presence.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:32 -07:00
Anders Kaseorg ab04068294 actions: Move part into zerver.lib.users.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:32 -07:00
Anders Kaseorg e230ea2598 actions: Split out zerver.actions.uploads.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:32 -07:00
Anders Kaseorg a29f1b39da actions: Move part into zerver.lib.streams.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:32 -07:00
Anders Kaseorg 6168c0110a actions: Split out zerver.actions.user_activity.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:32 -07:00
Anders Kaseorg df4849bb15 actions: Split out zerver.actions.user_topics.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:32 -07:00
Anders Kaseorg 385616f27f actions: Split out zerver.actions.realm_emoji.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:31 -07:00
Anders Kaseorg 8fc5922ebd actions: Split out zerver.actions.realm_export.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:31 -07:00
Anders Kaseorg 3d7aa98c45 actions: Split out zerver.actions.realm_icon.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:31 -07:00
Anders Kaseorg 7f088f3403 actions: Split out zerver.actions.realm_logo.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:31 -07:00
Anders Kaseorg ca8d374e21 actions: Split out zerver.actions.invites.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:31 -07:00
Anders Kaseorg 241463e215 actions: Split out zerver.actions.alert_words.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:31 -07:00
Anders Kaseorg 1ac7496855 actions: Split out zerver.actions.default_streams.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:31 -07:00
Anders Kaseorg 12130da339 actions: Split out zerver.actions.hotspots.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:31 -07:00
Anders Kaseorg 975f5a3c2d actions: Split out zerver.actions.realm_linkifiers.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:31 -07:00