script: Add ready-to-run tooling for doing backups.

Based on an initial version by Tim Abbott (#11204).

Fixes #552.

docs/production/install.md

@@ -82,13 +82,14 @@ If the script gives an error, consult [Troubleshooting](#troubleshooting) below.
 
 ## Step 3: Create a Zulip organization, and log in
 
-On success, the install script prints a link.  If you're importing
-your data from e.g. [HipChat][hipchat-import] or
-[Slack][slack-import], you should stop here and follow the import
-instructions.
+On success, the install script prints a link.  If you're [restoring a
+backup][zulip-backups] or importing your data from [HipChat][hipchat-import],
+[Slack][slack-import], or another Zulip server, you should stop here
+and return to the the import instructions.
 
 [hipchat-import]: https://zulipchat.com/help/import-from-hipchat
 [slack-import]: https://zulipchat.com/help/import-from-slack
+[zulip-backups]: ../production/maintain-secure-upgrade.html#backups
 
 Otherwise, open the link in a browser.  Follow the prompts to set up
 your organization, and your own user account as an administrator.

docs/production/maintain-secure-upgrade.md

@@ -239,7 +239,79 @@ dependencies are installed by default.
 
 ## Backups
 
-There are several pieces of data that you might want to back up:
+Starting with Zulip 2.0, Zulip has a built-in backup tool:
+
+```
+# As the zulip user
+/home/zulip/deployments/current/manage.py backup
+# Or as root
+su zulip -c '/home/zulip/deployments/current/manage.py backup'
+```
+
+This will generate a `.tar.gz` archive containing all the data stored
+on your Zulip server that would be needed to restore your Zulip
+server's state on another machine perfectly.
+
+### Restoring backups
+
+Backups generated using the Zulip 2.0 backup tool can be restored as
+follows.
+
+First, [install a new Zulip server through Step 3][install-server]
+with the version of both the base OS and Zulip from your previous
+installation.  Then, run as root:
+
+```
+/home/zulip/deployments/current/scripts/setup/restore-backup /path/to/backup
+```
+
+[install-server]: ../production/install.html
+
+### What is included
+
+Zulip's backup tools includes everything you need to fully restore
+your Zulip server from a user perspective.
+
+The following data present on a Zulip server is not included in these
+backup archives, and you may want to backup separately:
+
+* Transient data present in Zulip's RabbitMQ queues.  For example, a
+  record that a missed-message email for a given Zulip message is
+  scheduled to be sent to a given user in 2 minutes if the recipient
+  user doesn't interact with Zulip during that time window.  You can
+  check their status using `rabbitmq list_queues` as root.
+
+* Certain highly transient state that Zulip doesn't store in a
+  database, such as typing status, API rate-limiting counters,
+  etc. that would have no value 1 minute after the backup is
+  completed.
+
+* The server access/error logs from `/var/log/zulip`, because a Zulip
+  server only appends to those log files (i.e. they aren't necessarily
+  to precisely restore your Zulip data), and they can be very large
+  compared to the rest of the data for a Zulip server.
+
+* Files uploaded with the Zulip
+  [S3 file upload backend](../production/upload-backends.html).  We
+  don't include these for two reasons. First, the uploaded file data
+  in S3 can easily be many times larger than the rest of the backup,
+  and downloading it all to a server doing a backup could easily
+  exceed its disk capacity.  Additionally, S3 is a reliable persistent
+  storage system with its own high-quality tools for doing backups.
+  Contributions of (documentation on) ready-to-use scripting for S3
+  backups are welcome.
+
+* SSL certificates.  Since these are security-sensitive and either
+  trivially replaced (if generated via Certbot) or provided by the
+  system administrator, we do not include them in these backups.
+
+### Backup details
+
+This section is primarily for users managing backups themselves
+(E.g. if they're using a remote postgres database with an existing
+backup strategy), and also serves as documentation for what is
+included in the backups generated by Zulip's standard tools.  That
+data includes:
 
 * The postgres database.  That you can back up like any postgres
 database; we have some example tooling for doing that incrementally
@@ -264,24 +336,23 @@ user-uploaded avatars will need to be re-uploaded (since avatar
 filenames are computed using a hash of `avatar_salt` and user's
 email), etc.
 
-* The logs under `/var/log/zulip` can be handy to have backed up, but
-they do get large on a busy server, and it's definitely
-lower-priority.
-
-Zulip also has a [data export and import tool][export-import], which
-is useful for migrating data between Zulip Cloud and other Zulip
+Zulip also has a logical [data export and import tool][export-import],
+which is useful for migrating data between Zulip Cloud and other Zulip
 servers, as well as various auditing purposes.  The big advantage of
-the `postgres` layer backups over the export/import process is that
+the `manage.py backup` system over the export/import process is that
 it's structurally very unlikely for the `postgres` process to ever
-develop bugs.  The export tool's advantage is that the export is more
-human-readable and easier to parse, and doesn't have the requirement
-that the same set of Zulip organizations exist on the two servers.
+develop bugs, whereas the import/export tool requires some work for
+every new feature we add to Zulip, and thus may occasionally have bugs
+aroun corner cases.  The export tool's advantage is that the export is
+more human-readable and easier to parse, and doesn't have the
+requirement that the same set of Zulip organizations exist on the two
+servers (which is critical for migrations to and from Zulip Cloud).
 
 [export-import]: ../production/export-and-import.html
 
-### Restore from backups
+### Restore from manual backups
 
-To restore from backups, the process is basically the reverse of the above:
+To restore from a manual backup, the process is basically the reverse of the above:
 
 * Install new server as normal by downloading a Zulip release tarball
   and then using `scripts/setup/install`, you don't need

scripts/lib/zulip_tools.py

@@ -113,8 +113,8 @@ def subprocess_text_output(args):
 def get_zulip_uid() -> int:
     return os.stat(get_deploy_root()).st_uid
 
-def su_to_zulip():
-    # type: () -> None
+def su_to_zulip(save_suid=False):
+    # type: (bool) -> None
     """Warning: su_to_zulip assumes that the zulip checkout is owned by
     the zulip user (or whatever normal user is running the Zulip
     installation).  It should never be run from the installer or other
@@ -122,7 +122,10 @@ def su_to_zulip():
     created."""
     pwent = pwd.getpwuid(get_zulip_uid())
     os.setgid(pwent.pw_gid)
-    os.setuid(pwent.pw_uid)
+    if save_suid:
+        os.setresuid(pwent.pw_uid, pwent.pw_uid, os.getuid())
+    else:
+        os.setuid(pwent.pw_uid)
     os.environ['HOME'] = pwent.pw_dir
 
 def make_deploy_path():

scripts/setup/restore-backup

@@ -0,0 +1,104 @@
+#!/usr/bin/env python3
+
+import argparse
+import os
+import subprocess
+import sys
+import tempfile
+
+BASE_DIR = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+sys.path.append(BASE_DIR)
+from scripts.lib.zulip_tools import su_to_zulip, run
+
+POSTGRES_USER = "postgres"
+
+parser = argparse.ArgumentParser()
+parser.add_argument("tarball", help="Filename of input tarball")
+
+if __name__ == "__main__":
+    args = parser.parse_args()
+    su_to_zulip(save_suid=True)
+
+    import scripts.lib.setup_path_on_import
+
+    # First, we unpack the /etc/zulip configuration, so we know how
+    # this server is supposed to be configured (and can import
+    # /etc/zulip/settings.py via `from zproject import settings`,
+    # next).  Ignore errors if zulip-backup/settings is not present
+    # (E.g. because this is a development backup).
+    subprocess.call(
+        [
+            "tar",
+            "-C",
+            "/etc/zulip",
+            "--strip-components=2",
+            "-xzf",
+            args.tarball,
+            "zulip-backup/settings",
+        ]
+    )
+
+    from zproject import settings
+
+    paths = [
+        ("settings", "/etc/zulip"),
+        # zproject will only be present for development environment backups.
+        ("zproject", os.path.join(settings.DEPLOY_ROOT, "zproject")),
+        ("uploads", os.path.join(settings.DEPLOY_ROOT, settings.LOCAL_UPLOADS_DIR)),
+    ]
+
+    with tempfile.TemporaryDirectory(prefix="zulip-restore-backup-") as tmp:
+        uid = os.getuid()
+        gid = os.getgid()
+        os.setresuid(0, 0, 0)
+        for name, path in paths:
+            os.makedirs(path, exist_ok=True)
+            os.chown(path, uid, gid)
+        os.setresuid(uid, uid, 0)
+
+        # We create symlinks so that we can do a single `tar -x`
+        # command to unpack the uploads, settings, etc. to their
+        # appropriate places.
+        os.mkdir(os.path.join(tmp, "zulip-backup"))
+        for name, path in paths:
+            os.symlink(path, os.path.join(tmp, "zulip-backup", name))
+        run(["tar", "-C", tmp, "--keep-directory-symlink", "-xzf", args.tarball])
+
+        # Now, restore the the database backup using pg_restore.
+        db_name = settings.DATABASES["default"]["NAME"]
+        assert isinstance(db_name, str)
+        db_dir = os.path.join(tmp, "zulip-backup", "database")
+        os.setresuid(0, 0, 0)
+        run(["chown", "-R", POSTGRES_USER, "--", tmp])
+        run(
+            [
+                os.path.join(
+                    settings.DEPLOY_ROOT, "scripts", "setup", "terminate-psql-sessions"
+                ),
+                "zulip",
+                "zulip",
+                "zulip_base",
+            ]
+        )
+        as_postgres = ["su", "-s", "/usr/bin/env", "--", POSTGRES_USER]
+        run(as_postgres + ["dropdb", "--if-exists", "--", db_name])
+        run(as_postgres + ["createdb", "-T", "template0", "--", db_name])
+        run(as_postgres + ["pg_restore", "-d", db_name, "--", db_dir])
+        run(["chown", "-R", str(uid), "--", tmp])
+        os.setresuid(uid, uid, 0)
+
+        # In production, we also need to do a `zulip-puppet-apply` in
+        # order to adjust any configuration from /etc/zulip/zulip.conf
+        # to this system.
+        if settings.PRODUCTION:
+            os.setresuid(0, 0, 0)
+            run(
+                [
+                    os.path.join(settings.DEPLOY_ROOT, "scripts", "zulip-puppet-apply"),
+                    "-f",
+                ]
+            )
+            os.setresuid(uid, uid, 0)
+            run(["supervisorctl", "restart", "all"])
+
+        run([os.path.join(settings.DEPLOY_ROOT, "scripts", "setup", "flush-memcached")])

zerver/management/commands/backup.py

@@ -0,0 +1,73 @@
+import os
+import tempfile
+from argparse import ArgumentParser, RawTextHelpFormatter
+from typing import Any
+
+from django.conf import settings
+from django.utils.timezone import now as timezone_now
+
+from zerver.lib.management import ZulipBaseCommand
+from scripts.lib.zulip_tools import run, TIMESTAMP_FORMAT
+
+
+class Command(ZulipBaseCommand):
+    # Fix support for multi-line usage strings
+    def create_parser(self, *args: Any, **kwargs: Any) -> ArgumentParser:
+        parser = super().create_parser(*args, **kwargs)
+        parser.formatter_class = RawTextHelpFormatter
+        return parser
+
+    def add_arguments(self, parser: ArgumentParser) -> None:
+        parser.add_argument(
+            "output", default=None, nargs="?", help="Filename of output tarball"
+        )
+
+    def handle(self, *args: Any, **options: Any) -> None:
+        timestamp = timezone_now().strftime(TIMESTAMP_FORMAT)
+
+        with tempfile.TemporaryDirectory(
+            prefix="zulip-backup-%s-" % (timestamp,)
+        ) as tmp:
+            os.mkdir(os.path.join(tmp, "zulip-backup"))
+            members = []
+
+            if settings.DEVELOPMENT:
+                os.symlink(
+                    os.path.join(settings.DEPLOY_ROOT, "zproject"),
+                    os.path.join(tmp, "zulip-backup", "zproject"),
+                )
+                members.append("zulip-backup/zproject/dev-secrets.conf")
+            else:
+                os.symlink("/etc/zulip", os.path.join(tmp, "zulip-backup", "settings"))
+                members.append("zulip-backup/settings")
+
+            db_name = settings.DATABASES["default"]["NAME"]
+            db_dir = os.path.join(tmp, "zulip-backup", "database")
+            run(["pg_dump", "--format=directory", db_name, "--file", db_dir])
+            members.append("zulip-backup/database")
+
+            if settings.LOCAL_UPLOADS_DIR is not None and os.path.exists(
+                os.path.join(settings.DEPLOY_ROOT, settings.LOCAL_UPLOADS_DIR)
+            ):
+                os.symlink(
+                    os.path.join(settings.DEPLOY_ROOT, settings.LOCAL_UPLOADS_DIR),
+                    os.path.join(tmp, "zulip-backup", "uploads"),
+                )
+                members.append("zulip-backup/uploads")
+
+            try:
+                if options["output"] is None:
+                    tarball_path = tempfile.NamedTemporaryFile(
+                        prefix="zulip-backup-%s-" % (timestamp,),
+                        suffix=".tar.gz",
+                        delete=False,
+                    ).name
+                else:
+                    tarball_path = options["output"]
+
+                run(["tar", "-C", tmp, "-chzf", tarball_path, "--"] + members)
+                print("Backup tarball written to %s" % (tarball_path,))
+            except BaseException:
+                if options["output"] is None:
+                    os.unlink(tarball_path)
+                raise