mirror of https://github.com/zulip/zulip.git
243 lines
8.7 KiB
Markdown
243 lines
8.7 KiB
Markdown
# Exporting data
|
|
|
|
## Overview
|
|
|
|
Occasionally Zulip administrators will need to move data from one
|
|
server to another.
|
|
|
|
There are many major operational aspects to doing a conversion. I will
|
|
list them here, noting that several are not within the scope of this
|
|
document:
|
|
|
|
- Get new servers running.
|
|
- Export data from the old DB.
|
|
- Export files from S3.
|
|
- Import files into new storage.
|
|
- Import data into new DB.
|
|
- Restart new servers.
|
|
- Decommission old server.
|
|
|
|
This document focuses almost entirely on the **export** piece. Issues
|
|
with getting Zulip itself running are totally out of scope here. For the
|
|
import side of things, I only touch on it implicity. (My reasoning is
|
|
that we *have* to get the export piece right in a timely fashion, even
|
|
if it means we have to sort out some straggling issues on the import side
|
|
later.)
|
|
|
|
## Export
|
|
|
|
We have tools that essentially export Zulip data to the file system.
|
|
|
|
A good overview of the process is here:
|
|
[management/export.py](https://github.com/zulip/zulip/blob/master/zerver/management/commands/export.py)
|
|
|
|
This document supplements that explanation, but here we focus more
|
|
on the logistics of a big conversion. For some historical perspective,
|
|
this document was originally drafted as part of a big Zulip cut-over.
|
|
|
|
The main exporting tools in place as of summer 2016 are below:
|
|
|
|
- We can export single realms (but not yet limit users within the realm).
|
|
- We can export single users (but then we get no realm-wide data in the process).
|
|
- We can run exports simultaneously (but have to navigate a bunch of /tmp directories).
|
|
|
|
Things that we still may need:
|
|
- We may want to export multiple realms simultaneously.
|
|
- We may want to export multiple single users simultaneously.
|
|
- We may want to limit users within realm exports.
|
|
- We may want more operational robustness/convenience while doing several exports simultaenously.
|
|
- We may want to merge multiple export files to remove duplicates.
|
|
|
|
We have a few major classes of data. They are listed below in the order
|
|
that we process them in `do_export_realm()`:
|
|
|
|
#### Public Realm Data
|
|
|
|
Realm/RealmAlias/RealmEmoji/RealmFilter/DefaultStream.
|
|
|
|
#### Cross Realm Data
|
|
|
|
Client/zerver_userprofile_cross_realm
|
|
|
|
This includes Client and three bots.
|
|
|
|
Client is unique in being a fairly core table that is
|
|
not tied to UserProfile or Realm (unless you somewhat painfully tie
|
|
it back to users in a bottom-up fashion though other tables).
|
|
|
|
#### Disjoint User Data
|
|
|
|
UserProfile/UserActivity/UserActivityInterval/UserPresence.
|
|
|
|
#### Recipient Data
|
|
|
|
Recipient/Stream/Subscription/Huddle.
|
|
|
|
These tables are tied back to users, but they introduce complications
|
|
when you try to deal with multi-user subsets.
|
|
|
|
#### File-related Data
|
|
|
|
Attachment
|
|
|
|
This includes Attachment, and it referencs the avatar_source field of
|
|
UserProfile. Most importantly, of course, it requires us to grab files
|
|
from S3. Finally, Attachment's m2m relationship ties to Message.
|
|
|
|
#### Message Data
|
|
|
|
Message/UserMessage
|
|
|
|
### Summary
|
|
|
|
Here are the same classes of data, listed in roughly
|
|
decreasing order of riskiness:
|
|
|
|
- Message Data (sheer volume/lack of time/security)
|
|
- File-Related Data (S3/security/lots of moving parts)
|
|
- Recipient Data (complexity/security/cross-realm considerations)
|
|
- Cross Realm Data (duplicate ids)
|
|
- Disjoint User Data
|
|
- Public Realm Data
|
|
|
|
(Note the above list is essentially in reverse order of how we
|
|
process the data, which isn't surprising for a top-down approach.)
|
|
|
|
The next section of the document talks about risk factors.
|
|
|
|
# Risk Mitigation
|
|
|
|
## Generic considerations
|
|
|
|
We have two major mechanisms for getting data:
|
|
|
|
##### Top Down
|
|
|
|
Get realm data, then all users in realm, then all recipients, then all messages, etc.
|
|
|
|
The problem with the top down approach will be **filtering**. Also, if
|
|
errors arise during top-down passes, it may be time consuming to re-run
|
|
the processes.
|
|
|
|
##### Bottom Up
|
|
|
|
Start with users, get their recipient data, etc.
|
|
|
|
The problems with the bottom up approach will be **merging**. Also, if
|
|
we run multiple bottom-up passes, there is the danger of duplicating some
|
|
work, particularly on the message side of things.
|
|
|
|
### Approved Transfers
|
|
|
|
We have not yet integrated the approved-transfer model, which tells us
|
|
which users can be moved.
|
|
|
|
## Risk factors broken out by data categories
|
|
|
|
### Message Data
|
|
|
|
- models: Message/UserMessage.
|
|
- assets: messages-*.json, subprocesses, partial files
|
|
|
|
Rows in the Message model depend on Recipient/UserProfile.
|
|
|
|
Rows in the UserMessage model depend on UserProfile/Message.
|
|
|
|
The biggest concern here is the **sheer volume** of data, with
|
|
security being a close second. (They are interrelated, as without
|
|
security concerns, we could just bulk-export everything one time.)
|
|
|
|
We currently have these measures in place for top-down processing:
|
|
- chunking
|
|
- multi-processing
|
|
- messages are filtered by both sender and recipient
|
|
|
|
|
|
### File Related Data
|
|
|
|
- models: Attachment
|
|
- assets: S3, attachment.json, uploads-temp/, image files in avatars/, assorted files in uploads/, avatars/records.json, uploads/records.json, zerver_attachment_messages
|
|
|
|
When it comes to exporting attachment data, we have some minor volume issues, but the
|
|
main concern is just that there are **lots of moving parts**:
|
|
|
|
- S3 needs to be up, and we get some metadata from it as well as files.
|
|
- We have security concerns about copying over only files that belong to users who approved the transfer.
|
|
- This piece is just different in how we store data from all the other DB-centric pieces.
|
|
- At import time we have to populate the m2m table (but fortunately, this is pretty low
|
|
risk in terms of breaking anything.)
|
|
|
|
### Recipient Data
|
|
- models: Recipient/Stream/Subscription/Huddle
|
|
- assets: realm.json, (user,stream,huddle)_(recipient,subscription)
|
|
|
|
This data is fortunately low to medium in volume. The risk here will come
|
|
from **model complexity** and **cross-realm concerns**.
|
|
|
|
From the top down, here are the dependencies:
|
|
|
|
- Recipient depends on UserProfile
|
|
- Subscription depends on Recipient
|
|
- Stream currently depends on Realm (but maybe it should be tied to Subscription)
|
|
- Huddle depends on Subscription and UserProfile
|
|
|
|
The biggest risk factor here is probably just the possibility that we could introduce
|
|
some bug in our code as we try to segment Recipient into user, stream, and huddle components,
|
|
especially if we try to handle multiple users or realms.
|
|
I think this can be largely mitigated by the new Config approach.
|
|
|
|
And then we also have some complicated Huddle logic that will be customized
|
|
regardless. The fiddliest part
|
|
of the Huddle logic is creating the set of unsafe_huddle_recipient_ids.
|
|
|
|
Last but not least, if we go with some hybrid of bottom-up and top-down, these tables
|
|
are neither close to the bottom nor close to the top, so they may have the most
|
|
fiddly edge cases when it comes to filtering and merging.
|
|
|
|
Recommendation: We probably want to get a backup of all this data that is very simply
|
|
bulk-exported from the entire DB, and we should obviously put it in a secure place.
|
|
|
|
### Cross Realm Data
|
|
- models: Client
|
|
- assets: realm.json, three bots (notification/email/welcome), id_maps
|
|
|
|
The good news here is that Client is a small table, and there are
|
|
only three special bots.
|
|
|
|
The bad news is that cross-realm data **complicates everything else**,
|
|
and we have to avoid **database id conflicts**.
|
|
|
|
If we use bottom-up approaches to load small user populations at a time, we may
|
|
have **merging** issues here. We will need to consolidate ids either by merging
|
|
exports in /tmp or handle it import time.
|
|
|
|
For the three bots, they live in zerver_userprofile_crossrealm, and we re-map
|
|
their ids on the new server.
|
|
|
|
Recommendation: Do not sweat the exports too much. Deal with all the messiness at
|
|
import time, and rely on the tables being really small. We already have logic
|
|
to catch Client.DoesNotExist exceptions, for example. As for possibly missing
|
|
messages that the welcome bot and friends have sent in the past, I am not sure
|
|
what our risk profile is there, but I imagine it is relatively low.
|
|
|
|
### Disjoint User Data
|
|
- models: UserProfile/UserActivity/UserActivityInterval/UserPresence
|
|
- assets: realm.json, password, api_key, avatar salt, id_maps
|
|
|
|
On the DB side this data should be fairly easy to deal with. All of these
|
|
tables are basically disjoint by user profile id. Our biggest
|
|
risk is **remapped user ids** at import time, but this is mostly covered
|
|
in the section above.
|
|
|
|
We have code in place to exclude password and api_key from UserProfile
|
|
rows. The import process calls set_unusable_password().
|
|
|
|
### Public Realm Data
|
|
|
|
- models: Realm/RealmAlias/RealmEmoji/RealmFilter/DefaultStream
|
|
- asserts: realm.json
|
|
|
|
All of these tables are public (per-realm), and they are keyed by
|
|
realm id. There is not a ton to worry about here, except possibly
|
|
**merging** if we run multiple bottom-up jobs for a single realm.
|