mirror of https://github.com/zulip/zulip.git
docs: Better document backup/export tool tradeoffs.
- More consistent export/import vs backup bullets at the top. - Remove misleading documentation regarding the `zulip_org_id` reuse problems. This documentation was written for Zulip 2.1.0 inc6fe6cf0a4
and largely made obsolete ind800ac33a0
(Zulip 5.0). - Light editing for readability/crispness. Fixes #28925.
This commit is contained in:
parent
3d30fcd10a
commit
c1269f1038
|
@ -7,42 +7,51 @@ service (or back):
|
||||||
|
|
||||||
- The [Backup](#backups) tool is designed for exact restoration of a
|
- The [Backup](#backups) tool is designed for exact restoration of a
|
||||||
Zulip server's state, for disaster recovery, testing with production
|
Zulip server's state, for disaster recovery, testing with production
|
||||||
data, or hardware migration. This tool has a few limitations:
|
data, and hardware migrations.
|
||||||
|
|
||||||
|
We highly recommend this tool in situations where it is applicable,
|
||||||
|
because it is fast, robust, and minimizes disruption for your
|
||||||
|
users. This tool has a few limitations:
|
||||||
|
|
||||||
- Backups must be restored on a server running the same Zulip
|
- Backups must be restored on a server running the same Zulip
|
||||||
version (most precisely, one where `manage.py showmigrations` has
|
version (most precisely, one where `manage.py showmigrations` has
|
||||||
the same output).
|
identical output).
|
||||||
- Backups must be restored on a server running the same PostgreSQL
|
- Backups must be restored on a server running the same PostgreSQL
|
||||||
version. To install Zulip with the same version of PostgreSQL that
|
version. To install Zulip with the same version of PostgreSQL that
|
||||||
the backup was taken on, pass the desired version with [the
|
the backup was taken on, pass the desired version with [the
|
||||||
`--postgresql-version` argument][installer-options] when
|
`--postgresql-version` argument][installer-options] when
|
||||||
installing.
|
installing. Note that PostgreSQL is easy to [upgrade
|
||||||
|
independently][postgres-upgrade] from the rest of your Zulip
|
||||||
|
installation.
|
||||||
- Backups aren't useful for migrating organizations between
|
- Backups aren't useful for migrating organizations between
|
||||||
self-hosting and Zulip Cloud (which may require renumbering all
|
self-hosting and Zulip Cloud (which may require renumbering all
|
||||||
the users/messages/etc.).
|
the users/messages/etc.).
|
||||||
|
|
||||||
We highly recommend this tool in situations where it is applicable,
|
We also document [backup details](#backup-details) for users
|
||||||
because it is highly optimized and highly stable, since the hard
|
managing backups manually.
|
||||||
work is done by the built-in backup feature of PostgreSQL. We also
|
|
||||||
document [backup details](#backup-details) for users managing
|
|
||||||
backups manually.
|
|
||||||
|
|
||||||
- The logical [Data export](#data-export) tool is designed for
|
- The [logical data export](#data-export) tool is designed for
|
||||||
migrating data between Zulip Cloud and other Zulip servers, as well
|
migrating data between Zulip Cloud and other Zulip servers, as well
|
||||||
as various auditing purposes. The logical export tool produces a
|
as various auditing purposes.
|
||||||
`.tar.gz` archive with most of the Zulip database data encoded in
|
|
||||||
JSON files–a format shared by our [data
|
|
||||||
import](#import-into-a-new-zulip-server) tools for third-party
|
|
||||||
services like
|
|
||||||
[Slack](https://zulip.com/help/import-from-slack).
|
|
||||||
|
|
||||||
Like the backup tool, logical data exports must be imported on a
|
We recommend this tool in cases where the backup tool isn't
|
||||||
Zulip server running the same version. However, logical data
|
applicable, including situations where an easily machine-parsable
|
||||||
exports can be imported on Zulip servers running a different
|
export format is desired. This tool has a few limitations and
|
||||||
PostgreSQL version or hosting a different set of Zulip
|
caveats:
|
||||||
organizations. We recommend this tool in cases where the backup
|
|
||||||
tool isn't applicable, including situations where an easily
|
- Like the backup tool, logical data exports must be imported on a
|
||||||
machine-parsable export format is desired.
|
Zulip server running the same Zulip version. However, logical data
|
||||||
|
exports can be imported on Zulip servers running a different
|
||||||
|
PostgreSQL version or hosting a different set of Zulip
|
||||||
|
organizations.
|
||||||
|
- Transferring an organization via the data export tool results in
|
||||||
|
significant user-facing disruption, such as logging all users out of
|
||||||
|
their accounts and requiring them to reset their passwords.
|
||||||
|
|
||||||
|
The logical export tool produces a `.tar.gz` archive with most of
|
||||||
|
the Zulip database data encoded in JSON files–a format shared by our
|
||||||
|
[data import](#import-into-a-new-zulip-server) tools for third-party
|
||||||
|
services like [Slack](https://zulip.com/help/import-from-slack).
|
||||||
|
|
||||||
- [Compliance exports](#compliance-exports) allow a server
|
- [Compliance exports](#compliance-exports) allow a server
|
||||||
administrator to export messages matching a search query.
|
administrator to export messages matching a search query.
|
||||||
|
@ -60,6 +69,7 @@ service (or back):
|
||||||
as part of a high availability environment.
|
as part of a high availability environment.
|
||||||
|
|
||||||
[installer-options]: deployment.md#advanced-installer-options
|
[installer-options]: deployment.md#advanced-installer-options
|
||||||
|
[postgres-upgrade]: upgrade.md#upgrading-postgresql
|
||||||
|
|
||||||
## Backups
|
## Backups
|
||||||
|
|
||||||
|
@ -235,9 +245,9 @@ To restore from a manual backup, the process is basically the reverse of the abo
|
||||||
This restoration process can also be used to migrate a Zulip
|
This restoration process can also be used to migrate a Zulip
|
||||||
installation from one server to another.
|
installation from one server to another.
|
||||||
|
|
||||||
We recommend running a disaster recovery after setting up your backups to
|
We recommend running a disaster recovery test after setting up your
|
||||||
confirm that your backups are working. You may also want to monitor
|
backups to confirm that your backups are working. You may also want to
|
||||||
that they are up to date using the Nagios plugin at:
|
monitor that they are up to date using the Nagios plugin at:
|
||||||
`puppet/zulip/files/nagios_plugins/zulip_postgresql_backups/check_postgresql_backup`.
|
`puppet/zulip/files/nagios_plugins/zulip_postgresql_backups/check_postgresql_backup`.
|
||||||
|
|
||||||
## Data export
|
## Data export
|
||||||
|
@ -245,24 +255,37 @@ that they are up to date using the Nagios plugin at:
|
||||||
Zulip's powerful data export tool is designed to handle migration of a
|
Zulip's powerful data export tool is designed to handle migration of a
|
||||||
Zulip organization between different Zulip installations; as a result,
|
Zulip organization between different Zulip installations; as a result,
|
||||||
these exports contain all non-transient data for a Zulip organization,
|
these exports contain all non-transient data for a Zulip organization,
|
||||||
with the exception of passwords and API keys.
|
with the exception of secrets, like passwords and API keys.
|
||||||
|
|
||||||
We recommend using the [backup tool](#backups) if your primary goal is
|
We recommend instead using the [backup tool](#backups) in all
|
||||||
backups.
|
scenarios where it is applicable, because this data export process has
|
||||||
|
a few downsides in comparison:
|
||||||
|
|
||||||
### Upgrade if exporting for import into Zulip Cloud
|
- All users will have their passwords randomized and be logged out of
|
||||||
|
their accounts, both on web and mobile clients.
|
||||||
|
- All bots and integrations will need to be updated with new API keys.
|
||||||
|
- Users, streams, and messages are usually renumbered, which will
|
||||||
|
break most links from external programs referencing these objects.
|
||||||
|
|
||||||
If you are exporting data from a self-hosted version of Zulip for purposes of
|
### Consider upgrading
|
||||||
importing into Zulip Cloud, you should first [upgrade your server to the
|
|
||||||
|
We recommend [upgrading your Zulip server](../production/upgrade.md)
|
||||||
|
to the latest release [maintenance
|
||||||
|
release](../overview/release-lifecycle.md), or at least the latest
|
||||||
|
maintenance release for your major Zulip version.
|
||||||
|
|
||||||
|
**For Zulip Cloud imports**: If you are exporting data from a
|
||||||
|
self-hosted version of Zulip for purposes of importing into Zulip
|
||||||
|
Cloud, you should first [upgrade your server to the
|
||||||
`zulip-cloud-current` branch][upgrade-zulip-from-git]:
|
`zulip-cloud-current` branch][upgrade-zulip-from-git]:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
/home/zulip/deployments/current/scripts/upgrade-zulip-from-git zulip-cloud-current
|
/home/zulip/deployments/current/scripts/upgrade-zulip-from-git zulip-cloud-current
|
||||||
```
|
```
|
||||||
|
|
||||||
It is not sufficient to be on the latest stable release, as zulip.com runs
|
It is not sufficient to be on the latest stable release, because Zulip
|
||||||
pre-release versions of Zulip that are often several months of development ahead
|
Cloud runs pre-release versions of Zulip that are often several months
|
||||||
of the latest release.
|
of development ahead of the latest release.
|
||||||
|
|
||||||
### Preventing changes during the export
|
### Preventing changes during the export
|
||||||
|
|
||||||
|
@ -303,10 +326,10 @@ cd /home/zulip/deployments/current
|
||||||
(The `-r` option lets you specify the organization to export; `''` is
|
(The `-r` option lets you specify the organization to export; `''` is
|
||||||
the default organization hosted at the Zulip server's root domain.)
|
the default organization hosted at the Zulip server's root domain.)
|
||||||
|
|
||||||
This will generate a tarred archive with a name like
|
This will generate a compressed archive with a name like
|
||||||
`/tmp/zulip-export-zcmpxfm6.tar.gz`. The archive contains several
|
`/tmp/zulip-export-zcmpxfm6.tar.gz`. The archive contains several JSON
|
||||||
JSON files (containing the Zulip organization's data) as well as an
|
files (containing the Zulip organization's data) as well as an archive
|
||||||
archive of all the organization's uploaded files.
|
of all the organization's uploaded files.
|
||||||
|
|
||||||
## Import into a new Zulip server
|
## Import into a new Zulip server
|
||||||
|
|
||||||
|
@ -336,44 +359,34 @@ archive of all the organization's uploaded files.
|
||||||
budget extra RAM for running the data import tool.
|
budget extra RAM for running the data import tool.
|
||||||
|
|
||||||
2. If your new Zulip server is meant to fully replace a previous Zulip
|
2. If your new Zulip server is meant to fully replace a previous Zulip
|
||||||
server, you may want to copy some settings from `/etc/zulip` to your
|
server, copying `/etc/zulip/settings.py` and
|
||||||
new server to reuse the server-level configuration and secret keys
|
`/etc/zulip/zulip.conf` is safe and recommended, to avoid
|
||||||
from your old server. There are a few important details to understand
|
unnecessarily repeating configuration work.
|
||||||
about doing so:
|
|
||||||
|
|
||||||
- Copying `/etc/zulip/settings.py` and `/etc/zulip/zulip.conf` is
|
Copying `/etc/zulip/zulip-secrets.conf` is also safe and
|
||||||
safe and recommended. Care is required when copying secrets from
|
recommended, with the following important exceptions and notes:
|
||||||
`/etc/zulip/zulip-secrets.conf` (details below).
|
|
||||||
- If you copy `zulip_org_id` and `zulip_org_key` (the credentials
|
- Copying `avatar_salt` is not recommended. Zulip will
|
||||||
for the [mobile push notifications
|
|
||||||
service](mobile-push-notifications.md)), you should
|
|
||||||
be very careful to make sure the no users had their IDs
|
|
||||||
renumbered during the import process (this can be checked using
|
|
||||||
`manage.py shell` with some care). The push notifications
|
|
||||||
service has a mapping of which `user_id` values are associated
|
|
||||||
with which devices for a given Zulip server (represented by the
|
|
||||||
`zulip_org_id` registration). This means that if any `user_id`
|
|
||||||
values were renumbered during the import and you don't register a
|
|
||||||
new `zulip_org_id`, push notifications meant for the user who now
|
|
||||||
has ID 15 may be sent to devices registered by the user who had
|
|
||||||
user ID 15 before the data export (yikes!). The solution is
|
|
||||||
simply to not copy these settings and re-register your server for
|
|
||||||
mobile push notifications if any users had their IDs renumbered
|
|
||||||
during the logical export/import process.
|
|
||||||
- If you copy the `rabbitmq_password` secret from
|
|
||||||
`zulip-secrets.conf`, you'll need to run
|
|
||||||
`scripts/setup/configure-rabbitmq` as root to update your local RabbitMQ
|
|
||||||
installation to use the password in your Zulip secrets file.
|
|
||||||
- You will likely want to copy `camo_key` (required to avoid
|
|
||||||
breaking certain links) and any settings you added related to
|
|
||||||
authentication and email delivery so that those work on your new
|
|
||||||
server.
|
|
||||||
- Copying `avatar_salt` is not recommended, due to similar issues
|
|
||||||
to the mobile push notifications service. Zulip will
|
|
||||||
automatically rewrite avatars at URLs appropriate for the new
|
automatically rewrite avatars at URLs appropriate for the new
|
||||||
user IDs, and using the same avatar salt (and same server URL)
|
user IDs, and using the same avatar salt (and same server URL)
|
||||||
post import could result in issues with browsers caching the
|
post import could result in issues with browsers caching and
|
||||||
avatar image improperly for users whose ID was renumbered.
|
displaying avatar images improperly for users whose ID was
|
||||||
|
renumbered.
|
||||||
|
- Copying `zulip_org_id` and `zulip_org_key` is recommended to
|
||||||
|
avoid disconnecting your Zulip server from its registration with
|
||||||
|
the [Mobile Push Notifications Service][mobile-push].
|
||||||
|
- If you copy the `rabbitmq_password` secret from
|
||||||
|
`zulip-secrets.conf`, you'll need to run
|
||||||
|
`scripts/setup/configure-rabbitmq` as root to update your local
|
||||||
|
RabbitMQ installation to use the password in your Zulip secrets
|
||||||
|
file.
|
||||||
|
- Copying `camo_key` is required to avoid breaking links from Zulip
|
||||||
|
messages to externally hosted images.
|
||||||
|
- If your Zulip server is on an old Zulip Server release that
|
||||||
|
predates Zulip 5.0, and you use the [Mobile Push Notifications
|
||||||
|
Service][mobile-push], you should upgrade before you do the
|
||||||
|
export/import process if at all possible, and [ask for support][contact-support] if
|
||||||
|
it is not.
|
||||||
|
|
||||||
3. Log in to a shell on your Zulip server as the `zulip` user. Run the
|
3. Log in to a shell on your Zulip server as the `zulip` user. Run the
|
||||||
following commands, replacing the filename with the path to your data
|
following commands, replacing the filename with the path to your data
|
||||||
|
@ -390,6 +403,7 @@ cd /home/zulip/deployments/current
|
||||||
This could take several minutes to run depending on how much data you're
|
This could take several minutes to run depending on how much data you're
|
||||||
importing.
|
importing.
|
||||||
|
|
||||||
|
[contact-support]: https://zulip.com/help/contact-support
|
||||||
[upgrade-zulip-from-git]: upgrade.md#upgrading-from-a-git-repository
|
[upgrade-zulip-from-git]: upgrade.md#upgrading-from-a-git-repository
|
||||||
|
|
||||||
#### Import options
|
#### Import options
|
||||||
|
@ -538,3 +552,4 @@ the Nagios plugin installed into
|
||||||
|
|
||||||
[wal]: https://www.postgresql.org/docs/current/wal-intro.html
|
[wal]: https://www.postgresql.org/docs/current/wal-intro.html
|
||||||
[archive-timeout]: https://www.postgresql.org/docs/current/runtime-config-wal.html#GUC-ARCHIVE-TIMEOUT
|
[archive-timeout]: https://www.postgresql.org/docs/current/runtime-config-wal.html#GUC-ARCHIVE-TIMEOUT
|
||||||
|
[mobile-push]: ../production/mobile-push-notifications.md
|
||||||
|
|
Loading…
Reference in New Issue