zulip/docs/subsystems/caching.md

# Caching in Zulip

Like any product with good performance characteristics, Zulip makes
extensive use of caching. This article talks about our caching
strategy, focusing on how we use `memcached` (since it's the thing
people generally think about when they ask about how a server does
caching).

## Backend caching with memcached

On the backend, Zulip uses `memcached`, a popular key-value store, for
caching. Our `memcached` caching helps let us optimize Zulip's
performance and scalability, since most requests don't need to talk to
the database (which, even for a trivial query with everything on the
same machine, usually takes 3-10x as long as a memcached fetch).

We use Django's built-in caching integration to manage talking to
memcached, and then a small application-layer library
(`zerver/lib/cache.py`).

It's common for projects using a caching system like `memcached` to
either have the codebase littered with explicit requests to interact
with the cache (or flush data from a cache), or (worse) be littered
with weird bugs that disappear after you flush memcached.

Caching bugs are a pain to track down, because they generally require
an extra and difficult-to-guess step to reproduce (namely, putting the
wrong data into the cache).

So we've designed our backend to ensure that if we write a small
amount of Zulip's core caching code correctly, then the code most developers
naturally write will both benefit from caching and not create any cache
consistency problems.

The overall result of this design is that in the vast majority of
Zulip's Django codebase, all one needs to do is call the standard
accessor functions for data (like `get_user` or `get_stream` to fetch
user and stream objects, or for view code, functions like
`access_stream_by_id`, which checks permissions), and everything will
work great. The data fetches automatically benefit from `memcached`
caching, since those accessor methods have already been written to
transparently use Zulip's memcached caching system, and the developer
doesn't need to worry about whether the data returned is up-to-date:
it is. In the following sections, we'll talk about how we make this
work.

As a side note, the policy of using these accessor functions wherever
possible is a good idea, regardless of caching, because the functions
also generally take care of details you might not think about
(e.g. case-insensitive matching of stream names or email addresses).
It's amazing how slightly tricky logic that's duplicated in several
places invariably ends up buggy in some of those places, and in
aggregate we call these accessor functions hundreds of times in
Zulip. But the caching is certainly a nice bonus.

### The core implementation

The `get_user` function is a pretty typical piece of code using this
framework; as you can see, it's very little code on top of our
`cache_with_key` decorator:

```python
def user_profile_cache_key_id(email: str, realm_id: int) -> str:
    return f"user_profile:{hashlib.sha1(email.strip().encode()).hexdigest()}:{realm_id}"

def user_profile_cache_key(email: str, realm: "Realm") -> str:
    return user_profile_cache_key_id(email, realm.id)

@cache_with_key(user_profile_cache_key, timeout=3600 * 24 * 7)
def get_user(email: str, realm: Realm) -> UserProfile:
    return UserProfile.objects.select_related().get(
        email__iexact=email.strip(), realm=realm)
```

This decorator implements a pretty classic caching paradigm:

- The `user_profile_cache_key` function defines a unique map from a
  canonical form of its arguments to a string. These strings are
  namespaced (the `user_profile:` part) so that they won't overlap
  with other caches, and encode the arguments so that two uses of this
  cache won't overlap. In this case, a hash of the email address and
  realm ID are those canonicalized arguments. (The `make_safe_digest`
  is important to ensure we don't send special characters to
  memcached). And we have two versions, depending whether the caller
  has access to a `Realm` or just a `realm_id`.
- When `get_user` is called, `cache_with_key` will compute the key,
  and do a Django `cache_get` query for the key (which goes to
  memcached). If the key is in the cache, it just returns the value.
  Otherwise, it fetches the value from the database (using the actual
  code in the body of `get_user`), and then stores the value back to
  that memcached key before returning the result to the caller.
- Cache entries expire after the timeout; in this case, a week.
  Though in frequently deployed environments like chat.zulip.org,
  often cache entries will stop being used long before that, because
  `KEY_PREFIX` is rotated every time we deploy to production; see
  below for details.

We use this decorator in more than 30 places in Zulip, and it saves a
huge amount of otherwise very self-similar caching code.

### Cautions

The one thing to be really careful with in using `cache_with_key` is
that if an item is in the cache, the body of `get_user` (above) is
never called. This means some things that might seem like clever code
reuse are actually a really bad idea. For example:

- Don't add a `get_active_user` function that uses the same cache key
  function as `get_user` (but with a different query that filters our
  deactivated users). If one called `get_active_user` to access a
  deactivated user, the right thing would happen, but if you called
  `get_user` to access that user first, then the `get_active_user`
  function would happily return the user from the cache, without ever
  doing your more restrictive query.

So remember: Use separate cache key functions for different data sets,
even if they feature the same objects.

### Cache invalidation after writes

The caching strategy described above works pretty well for anything
where the state it's storing is immutable (i.e. never changes). With
mutable state, one needs to do something to ensure that the Python
processes don't end up fetching stale data from the cache after a
write to the database.

We handle this using Django's longstanding
[post_save signals][post-save-signals] feature. Django signals let
you configure some code to run every time Django does something (for
`post_save`, right after any write to the database using Django's
`.save()`).

There's a handful of lines in `zerver/models.py` like these that
configure this:

```python
post_save.connect(flush_realm, sender=Realm)
post_save.connect(flush_user_profile, sender=UserProfile)
```

Once this `post_save` hook is registered, whenever one calls
`user_profile.save(...)` with a UserProfile object in our Django
project, Django will call the `flush_user_profile` function. Zulip is
systematic about using the standard Django `.save()` function for
modifying `user_profile` objects (and passing the `update_fields`
argument to `.save()` consistently, which encodes which fields on an
object changed). This means that all we have to do is write those
cache-flushing functions correctly, and people writing Zulip code
won't need to think about (or even know about!) the caching.

Each of those flush functions basically just computes the list of
cache keys that might contain data that was modified by the
`.save(...)` call (based on the object changed and the `update_fields`
data), and then sends a bulk delete request to `memcached` to remove
those keys from the cache (if present).

Maintaining these flush functions requires some care (every time we
add a new cache, we need to look through them), but overall it's a
pretty simple algorithm: If the changed data appears in any form in a
given cache key, that cache key needs to be cleared. E.g. the
`active_user_ids_cache_key` cache for a realm needs to be flushed
whenever a new user is created in that realm, or user is
deactivated/reactivated, even though it's just a list of IDs and thus
doesn't explicitly contain the `is_active` flag.

Once you understand how that works, it's pretty easy to reason about
when a particular flush function should clear a particular cache; so
the main thing that requires care is making sure we remember to reason
about that when changing cache semantics.

But the overall benefit of this cache system is that almost all the
code in Zulip just needs to modify Django model objects and call
`.save()`, and the caching system will do the right thing.

### Production deployments and database migrations

When upgrading a Zulip server, it's important to avoid having one
version of the code interact with cached objects from another version
that has a different data layout. In Zulip, we avoid this through
some clever caching strategies. Each "deployment directory" for Zulip
in production has inside it a `var/remote_cache_prefix` file,
containing a cache prefix (`KEY_PREFIX` in the code) that is
automatically appended to the start of any cache keys accessed by that
deployment directory (this is all handled internally by
`zerver/lib/cache.py`).

This completely solves the problem of potentially having contamination
from inconsistent versions of the source code / data formats in the cache.

### Automated testing and memcached

For Zulip's `test-backend` unit tests, we use the same strategy. In
particular, we just edit `KEY_PREFIX` before each unit test; this
means each of the thousands of test cases in Zulip has its own
independent memcached key namespace on each run of the unit tests. As
a result, we never have to worry about memcached caching causing
problems across multiple tests.

This is a really important detail. It makes it possible for us to do
assertions in our tests on the number of database queries or memcached
queries that are done as part of a particular function/route, and have
those checks consistently get the same result (those tests are great
for catching bugs where we accidentally do database queries in a
loop). And it means one can debug failures in the test suite without
having to consider the possibility that memcached is somehow confusing
the situation.

Further, this `KEY_PREFIX` model means that running the backend tests
won't potentially conflict with whatever you're doing in a Zulip
development environment on the same machine, which also saves a ton of
time when debugging, since developers don't need to think about things
like whether some test changed Hamlet's email address and that's why
login is broken.

More full-stack test suites like `test-js-with-puppeteer` or `test-api`
use a similar strategy (set a random `KEY_PREFIX` at the start of the
test run).

### Manual testing and memcached

Zulip's development environment will automatically flush (delete all
keys in) `memcached` when provisioning and when starting `run-dev`.
You can run the server with that behavior disabled using
`tools/run-dev --no-clear-memcached`.

### Performance

One thing be careful about with memcached queries is to avoid doing
them in loops (the same applies for database queries!). Instead, one
should use a bulk query. We have a fancy function,
`generate_bulk_cached_fetch`, which is super magical and handles this
for us, with support for a bunch of fancy features like marshalling
data before/after going into the cache (e.g. to compress `message`
objects to minimize data transfer between Django and memcached).

## In-process caching in Django

We generally try to avoid in-process backend caching in Zulip's Django
codebase, because every Zulip production installation involves
multiple servers. We do have a few, however:

- `per_request_display_recipient_cache`: A cache flushed at the start
  of every request; this simplifies correctly implementing our goal of
  not repeatedly fetching the "display recipient" (e.g. stream name)
  for each message in the `GET /messages` codebase.
- Caches of various data, like the `SourceMap` object, that are
  expensive to construct, not needed for most requests, and don't
  change once a Zulip server has been deployed in production.

## Browser caching of state

Zulip makes extensive use of caching of data in the browser and mobile
apps; details like which users exist, with metadata like names and
avatars, similar details for streams, recent message history, etc.

This data is fetched in the `/register` endpoint (or `page_params`
for the web app), and kept correct over time. The key to keeping these
state up to date is Zulip's
[real-time events system](events-system.md), which
allows the server to notify clients whenever state that might be
cached by clients is changed. Clients are responsible for handling
the events, updating their state, and rerendering any UI components
that might display the modified state.

[post-save-signals]: https://docs.djangoproject.com/en/3.2/ref/signals/#post-save