docs: Better document backup/export tool tradeoffs.

- More consistent export/import vs backup bullets at the top.

- Remove misleading documentation regarding the `zulip_org_id` reuse
  problems. This documentation was written for Zulip 2.1.0 in
  c6fe6cf0a4 and largely made obsolete
  in d800ac33a0 (Zulip 5.0).

- Light editing for readability/crispness.

Fixes #28925.
This commit is contained in:
Tim Abbott 2024-02-27 13:51:44 -08:00
parent 3d30fcd10a
commit c1269f1038
1 changed files with 88 additions and 73 deletions

View File

@ -7,42 +7,51 @@ service (or back):
- The [Backup](#backups) tool is designed for exact restoration of a - The [Backup](#backups) tool is designed for exact restoration of a
Zulip server's state, for disaster recovery, testing with production Zulip server's state, for disaster recovery, testing with production
data, or hardware migration. This tool has a few limitations: data, and hardware migrations.
We highly recommend this tool in situations where it is applicable,
because it is fast, robust, and minimizes disruption for your
users. This tool has a few limitations:
- Backups must be restored on a server running the same Zulip - Backups must be restored on a server running the same Zulip
version (most precisely, one where `manage.py showmigrations` has version (most precisely, one where `manage.py showmigrations` has
the same output). identical output).
- Backups must be restored on a server running the same PostgreSQL - Backups must be restored on a server running the same PostgreSQL
version. To install Zulip with the same version of PostgreSQL that version. To install Zulip with the same version of PostgreSQL that
the backup was taken on, pass the desired version with [the the backup was taken on, pass the desired version with [the
`--postgresql-version` argument][installer-options] when `--postgresql-version` argument][installer-options] when
installing. installing. Note that PostgreSQL is easy to [upgrade
independently][postgres-upgrade] from the rest of your Zulip
installation.
- Backups aren't useful for migrating organizations between - Backups aren't useful for migrating organizations between
self-hosting and Zulip Cloud (which may require renumbering all self-hosting and Zulip Cloud (which may require renumbering all
the users/messages/etc.). the users/messages/etc.).
We highly recommend this tool in situations where it is applicable, We also document [backup details](#backup-details) for users
because it is highly optimized and highly stable, since the hard managing backups manually.
work is done by the built-in backup feature of PostgreSQL. We also
document [backup details](#backup-details) for users managing
backups manually.
- The logical [Data export](#data-export) tool is designed for - The [logical data export](#data-export) tool is designed for
migrating data between Zulip Cloud and other Zulip servers, as well migrating data between Zulip Cloud and other Zulip servers, as well
as various auditing purposes. The logical export tool produces a as various auditing purposes.
`.tar.gz` archive with most of the Zulip database data encoded in
JSON filesa format shared by our [data
import](#import-into-a-new-zulip-server) tools for third-party
services like
[Slack](https://zulip.com/help/import-from-slack).
Like the backup tool, logical data exports must be imported on a We recommend this tool in cases where the backup tool isn't
Zulip server running the same version. However, logical data applicable, including situations where an easily machine-parsable
exports can be imported on Zulip servers running a different export format is desired. This tool has a few limitations and
PostgreSQL version or hosting a different set of Zulip caveats:
organizations. We recommend this tool in cases where the backup
tool isn't applicable, including situations where an easily - Like the backup tool, logical data exports must be imported on a
machine-parsable export format is desired. Zulip server running the same Zulip version. However, logical data
exports can be imported on Zulip servers running a different
PostgreSQL version or hosting a different set of Zulip
organizations.
- Transferring an organization via the data export tool results in
significant user-facing disruption, such as logging all users out of
their accounts and requiring them to reset their passwords.
The logical export tool produces a `.tar.gz` archive with most of
the Zulip database data encoded in JSON filesa format shared by our
[data import](#import-into-a-new-zulip-server) tools for third-party
services like [Slack](https://zulip.com/help/import-from-slack).
- [Compliance exports](#compliance-exports) allow a server - [Compliance exports](#compliance-exports) allow a server
administrator to export messages matching a search query. administrator to export messages matching a search query.
@ -60,6 +69,7 @@ service (or back):
as part of a high availability environment. as part of a high availability environment.
[installer-options]: deployment.md#advanced-installer-options [installer-options]: deployment.md#advanced-installer-options
[postgres-upgrade]: upgrade.md#upgrading-postgresql
## Backups ## Backups
@ -235,9 +245,9 @@ To restore from a manual backup, the process is basically the reverse of the abo
This restoration process can also be used to migrate a Zulip This restoration process can also be used to migrate a Zulip
installation from one server to another. installation from one server to another.
We recommend running a disaster recovery after setting up your backups to We recommend running a disaster recovery test after setting up your
confirm that your backups are working. You may also want to monitor backups to confirm that your backups are working. You may also want to
that they are up to date using the Nagios plugin at: monitor that they are up to date using the Nagios plugin at:
`puppet/zulip/files/nagios_plugins/zulip_postgresql_backups/check_postgresql_backup`. `puppet/zulip/files/nagios_plugins/zulip_postgresql_backups/check_postgresql_backup`.
## Data export ## Data export
@ -245,24 +255,37 @@ that they are up to date using the Nagios plugin at:
Zulip's powerful data export tool is designed to handle migration of a Zulip's powerful data export tool is designed to handle migration of a
Zulip organization between different Zulip installations; as a result, Zulip organization between different Zulip installations; as a result,
these exports contain all non-transient data for a Zulip organization, these exports contain all non-transient data for a Zulip organization,
with the exception of passwords and API keys. with the exception of secrets, like passwords and API keys.
We recommend using the [backup tool](#backups) if your primary goal is We recommend instead using the [backup tool](#backups) in all
backups. scenarios where it is applicable, because this data export process has
a few downsides in comparison:
### Upgrade if exporting for import into Zulip Cloud - All users will have their passwords randomized and be logged out of
their accounts, both on web and mobile clients.
- All bots and integrations will need to be updated with new API keys.
- Users, streams, and messages are usually renumbered, which will
break most links from external programs referencing these objects.
If you are exporting data from a self-hosted version of Zulip for purposes of ### Consider upgrading
importing into Zulip Cloud, you should first [upgrade your server to the
We recommend [upgrading your Zulip server](../production/upgrade.md)
to the latest release [maintenance
release](../overview/release-lifecycle.md), or at least the latest
maintenance release for your major Zulip version.
**For Zulip Cloud imports**: If you are exporting data from a
self-hosted version of Zulip for purposes of importing into Zulip
Cloud, you should first [upgrade your server to the
`zulip-cloud-current` branch][upgrade-zulip-from-git]: `zulip-cloud-current` branch][upgrade-zulip-from-git]:
```bash ```bash
/home/zulip/deployments/current/scripts/upgrade-zulip-from-git zulip-cloud-current /home/zulip/deployments/current/scripts/upgrade-zulip-from-git zulip-cloud-current
``` ```
It is not sufficient to be on the latest stable release, as zulip.com runs It is not sufficient to be on the latest stable release, because Zulip
pre-release versions of Zulip that are often several months of development ahead Cloud runs pre-release versions of Zulip that are often several months
of the latest release. of development ahead of the latest release.
### Preventing changes during the export ### Preventing changes during the export
@ -303,10 +326,10 @@ cd /home/zulip/deployments/current
(The `-r` option lets you specify the organization to export; `''` is (The `-r` option lets you specify the organization to export; `''` is
the default organization hosted at the Zulip server's root domain.) the default organization hosted at the Zulip server's root domain.)
This will generate a tarred archive with a name like This will generate a compressed archive with a name like
`/tmp/zulip-export-zcmpxfm6.tar.gz`. The archive contains several `/tmp/zulip-export-zcmpxfm6.tar.gz`. The archive contains several JSON
JSON files (containing the Zulip organization's data) as well as an files (containing the Zulip organization's data) as well as an archive
archive of all the organization's uploaded files. of all the organization's uploaded files.
## Import into a new Zulip server ## Import into a new Zulip server
@ -336,44 +359,34 @@ archive of all the organization's uploaded files.
budget extra RAM for running the data import tool. budget extra RAM for running the data import tool.
2. If your new Zulip server is meant to fully replace a previous Zulip 2. If your new Zulip server is meant to fully replace a previous Zulip
server, you may want to copy some settings from `/etc/zulip` to your server, copying `/etc/zulip/settings.py` and
new server to reuse the server-level configuration and secret keys `/etc/zulip/zulip.conf` is safe and recommended, to avoid
from your old server. There are a few important details to understand unnecessarily repeating configuration work.
about doing so:
- Copying `/etc/zulip/settings.py` and `/etc/zulip/zulip.conf` is Copying `/etc/zulip/zulip-secrets.conf` is also safe and
safe and recommended. Care is required when copying secrets from recommended, with the following important exceptions and notes:
`/etc/zulip/zulip-secrets.conf` (details below).
- If you copy `zulip_org_id` and `zulip_org_key` (the credentials - Copying `avatar_salt` is not recommended. Zulip will
for the [mobile push notifications
service](mobile-push-notifications.md)), you should
be very careful to make sure the no users had their IDs
renumbered during the import process (this can be checked using
`manage.py shell` with some care). The push notifications
service has a mapping of which `user_id` values are associated
with which devices for a given Zulip server (represented by the
`zulip_org_id` registration). This means that if any `user_id`
values were renumbered during the import and you don't register a
new `zulip_org_id`, push notifications meant for the user who now
has ID 15 may be sent to devices registered by the user who had
user ID 15 before the data export (yikes!). The solution is
simply to not copy these settings and re-register your server for
mobile push notifications if any users had their IDs renumbered
during the logical export/import process.
- If you copy the `rabbitmq_password` secret from
`zulip-secrets.conf`, you'll need to run
`scripts/setup/configure-rabbitmq` as root to update your local RabbitMQ
installation to use the password in your Zulip secrets file.
- You will likely want to copy `camo_key` (required to avoid
breaking certain links) and any settings you added related to
authentication and email delivery so that those work on your new
server.
- Copying `avatar_salt` is not recommended, due to similar issues
to the mobile push notifications service. Zulip will
automatically rewrite avatars at URLs appropriate for the new automatically rewrite avatars at URLs appropriate for the new
user IDs, and using the same avatar salt (and same server URL) user IDs, and using the same avatar salt (and same server URL)
post import could result in issues with browsers caching the post import could result in issues with browsers caching and
avatar image improperly for users whose ID was renumbered. displaying avatar images improperly for users whose ID was
renumbered.
- Copying `zulip_org_id` and `zulip_org_key` is recommended to
avoid disconnecting your Zulip server from its registration with
the [Mobile Push Notifications Service][mobile-push].
- If you copy the `rabbitmq_password` secret from
`zulip-secrets.conf`, you'll need to run
`scripts/setup/configure-rabbitmq` as root to update your local
RabbitMQ installation to use the password in your Zulip secrets
file.
- Copying `camo_key` is required to avoid breaking links from Zulip
messages to externally hosted images.
- If your Zulip server is on an old Zulip Server release that
predates Zulip 5.0, and you use the [Mobile Push Notifications
Service][mobile-push], you should upgrade before you do the
export/import process if at all possible, and [ask for support][contact-support] if
it is not.
3. Log in to a shell on your Zulip server as the `zulip` user. Run the 3. Log in to a shell on your Zulip server as the `zulip` user. Run the
following commands, replacing the filename with the path to your data following commands, replacing the filename with the path to your data
@ -390,6 +403,7 @@ cd /home/zulip/deployments/current
This could take several minutes to run depending on how much data you're This could take several minutes to run depending on how much data you're
importing. importing.
[contact-support]: https://zulip.com/help/contact-support
[upgrade-zulip-from-git]: upgrade.md#upgrading-from-a-git-repository [upgrade-zulip-from-git]: upgrade.md#upgrading-from-a-git-repository
#### Import options #### Import options
@ -538,3 +552,4 @@ the Nagios plugin installed into
[wal]: https://www.postgresql.org/docs/current/wal-intro.html [wal]: https://www.postgresql.org/docs/current/wal-intro.html
[archive-timeout]: https://www.postgresql.org/docs/current/runtime-config-wal.html#GUC-ARCHIVE-TIMEOUT [archive-timeout]: https://www.postgresql.org/docs/current/runtime-config-wal.html#GUC-ARCHIVE-TIMEOUT
[mobile-push]: ../production/mobile-push-notifications.md