zulip/docs/production/upgrade-or-modify.md

Upgrade or modify Zulip

This page explains how to upgrade, patch, or modify Zulip, including:

• Upgrading to a release
• Upgrading from a git repository
• Troubleshooting and rollback
• Preserving local changes to configuration files
• Upgrading the operating system
• Modifying Zulip
• Applying changes from master

Upgrading to a release

Note that there are additional instructions if you're using docker-zulip, have patched Zulip, or have modified Zulip-managed configuration files. To upgrade to a new Zulip release:

1. Download the appropriate release tarball from https://www.zulip.org/dist/releases/ You can download the latest release with:

   wget https://www.zulip.org/dist/releases/zulip-server-latest.tar.gz
   

   You also have the option of upgrading Zulip to a version in a Git repository directly or creating your own release tarballs from a copy of the zulip.git repository using tools/build-release-tarball.

2. Login to your Zulip and run as root:

   /home/zulip/deployments/current/scripts/upgrade-zulip zulip-server-VERSION.tar.gz
   

   The upgrade process will:

   • Run apt-get upgrade
   • Install new versions of Zulip's dependencies (mainly Python packages).
   • (upgrade-zulip-from-git only) Build Zulip's frontend assets using webpack.
   • Shut down the Zulip service
   • Run a puppet apply
   • Run any database migrations
   • Bring the Zulip service back up on the new version.

Upgrading will result in brief downtime for the service, which should be under 30 seconds unless there is an expensive database migration involved (these will be documented in the release notes, and usually can be avoided with some care). If downtime is problematic for your organization, consider testing the upgrade on a backup in advance, doing the final upgrade at off hours, or buying a support contract.

See the troubleshooting guide if you run into any issues or need to roll back the upgrade.

Upgrading from a git repository

Zulip supports upgrading a production installation to any commit in a Git repository, which is great for running pre-release changes from master or maintaining a fork. The process is simple:

# Upgrade to an official release
/home/zulip/deployments/current/scripts/upgrade-zulip-from-git 1.8.1
# Upgrade to a branch (or other Git ref)
/home/zulip/deployments/current/scripts/upgrade-zulip-from-git 2.1.x
/home/zulip/deployments/current/scripts/upgrade-zulip-from-git master

Zulip will automatically fetch the relevant Git commit and upgrade to that version of Zulip.

Branches with names like 2.1.x are stable release branches, containing the changes planned for the next minor release (E.g. 2.1.5); we support these stable release branches as though they were a published release.

The master branch contains changes planned for the next major release (E.g. 2.2.0); see our documentation on running master before upgrading to it.

By default, this uses the main upstream Zulip server repository, but you can configure any other Git repository by adding a section like this to /etc/zulip/zulip.conf:

[deployment]
git_repo_url = https://github.com/zulip/zulip.git

See also our documentation on upgrading docker-zulip.

Troubleshooting and rollback

See also the general Zulip server troubleshooting guide.

The upgrade scripts are idempotent, so there's no harm in trying again after resolving an issue. The most common causes of errors are:

• Networking issues (e.g. your Zulip server doesn't have reliable Internet access or needs a proxy setup). Fix the networking issue and try again.
• Especially when using upgrade-zulip-from-git, systems with the minimal RAM for running Zulip can run into out-of-memory issues during the upgrade process (generally tools/webpack is the step that fails). You can get past this by shutting down the Zulip server with supervisorctl stop all to free up RAM before running the upgrade process.

Useful logs are available in a few places:

• The Zulip upgrade scripts log all output to /var/log/zulip/upgrade.log.
• The Zulip server logs all Internal Server Errors to /var/log/zulip/errors.log.

If you need help and don't have a support contract, you can visit #production help in the Zulip development community server for best-effort help. Please include the relevant error output from the above logs in a markdown code block in any reports.

Rolling back to a prior version

This rollback process is intended for minor releases (e.g. 2.0.3 to 2.0.6); a more complicated process is required to roll back database migrations before downgrading to an older major release.

The Zulip upgrade process works by creating a new deployment under /home/zulip/deployments/ containing a complete copy of the Zulip server code, and then moving the symlinks at /home/zulip/deployments/{current,last,next} as part of the upgrade process.

This means that if the new version isn't working, you can quickly downgrade to the old version by running /home/zulip/deployments/last/scripts/restart-server, or to an earlier previous version by running /home/zulip/deployments/DATE/scripts/restart-server. The restart-server script stops any running Zulip server, and starts the version corresponding to the restart-server path you call.

Preserving local changes to configuration files

.. warning::
    If you have modified configuration files installed by
    Zulip (e.g. the nginx configuration), the Zulip upgrade process will
    overwrite your configuration when it does the ``puppet apply``.

You can test whether this will happen assuming no upstream changes to the configuration using scripts/zulip-puppet-apply (without the -f option), which will do a test puppet run and output and changes it would make. Using this list, you can save a copy of any files that you've modified, do the upgrade, and then restore your configuration.

That said, Zulip's configuration files are designed to be flexible enough for a wide range of installations, from a small self-hosted system to Zulip Cloud. Before making local changes to a configuration file, first check whether there's an option supported by /etc/zulip/zulip.conf for the customization you need. And if you need to make local modifications, please report the issue so that we can make the Zulip puppet configuration flexible enough to handle your setup.

nginx configuration changes

If you need to modify Zulip's nginx configuration, we recommend first attempting to add configuration to /etc/nginx/conf.d or /etc/nginx/zulip-include/app.d; those directories are designed for custom configuration.

Upgrading the operating system

When you upgrade the operating system on which Zulip is installed (E.g. Ubuntu 18.04 Bionic to Ubuntu 20.04 Focal), you need to take some additional steps to update your Zulip installation, documented below.

The steps are largely the same for the various OS upgrades aside from the versions of postgres, so you should be able to adapt these instructions for other supported platforms.

Upgrading from Ubuntu 18.04 Bionic to 20.04 Focal

1. Upgrade your server to the latest Zulip release (at least 2.2.0, which adds support for Ubuntu Focal).

2. As the Zulip user, stop the Zulip server and run the following to back up the system:

   supervisorctl stop all
   /home/zulip/deployments/current/manage.py backup --output=/home/zulip/release-upgrade.backup.tar.gz
   

3. Switch to the root user and upgrade the operating system using the OS's standard tooling. E.g. for Ubuntu, this means running do-release-upgrade and following the prompts until it completes successfully:

   sudo -i # Or otherwise get a root shell
   do-release-upgrade
   

   When do-release-upgrade asks you how to upgrade configuration files for services that Zulip manages like redis, postgres, nginx, and memcached, the best choice is N to keep the currently installed version. But it's not important; the next step will re-install Zulip's configuration in any case.

4. As root, upgrade the database installation and OS configuration to match the new OS version:

   touch /usr/share/postgresql/12/pgroonga_setup.sql.applied
   /home/zulip/deployments/current/scripts/zulip-puppet-apply -f
   pg_dropcluster 12 main --stop
   systemctl stop postgresql
   pg_upgradecluster 10 main
   pg_dropcluster 10 main
   apt remove postgresql-10
   systemctl start postgresql
   systemctl restart memcached
   

5. At this point, you are now running the version of postgres that comes with the new Ubuntu version. Finally, we need to reinstall the current version of Zulip, which among other things will recompile Zulip's Python module dependencies for your new version of Python:

   rm -rf /srv/zulip-venv-cache/*
   /home/zulip/deployments/current/scripts/lib/upgrade-zulip-stage-2 \
       /home/zulip/deployments/current/ --ignore-static-assets
   

That last command will finish by restarting your Zulip server; you should now be able to navigate to its URL and confirm everything is working correctly.

Upgrading from Ubuntu 16.04 Xenial to 18.04 Bionic

1. Upgrade your server to the latest Zulip 2.1.x release, since newer releases don't support Ubuntu 16.04 Xenial.

2. Same as for Bionic to Focal.

3. Same as for Bionic to Focal.

4. As root, upgrade the database installation and OS configuration to match the new OS version:

   touch /usr/share/postgresql/10/pgroonga_setup.sql.applied
   /home/zulip/deployments/current/scripts/zulip-puppet-apply -f
   pg_dropcluster 10 main --stop
   systemctl stop postgresql
   pg_upgradecluster 9.5 main
   pg_dropcluster 9.5 main
   apt remove postgresql-9.5
   systemctl start postgresql
   systemctl restart memcached
   

5. Same as for Bionic to Focal.

That last command will finish by restarting your Zulip server; you should now be able to navigate to its URL and confirm everything is working correctly.

Upgrading from Ubuntu 14.04 Trusty to 16.04 Xenial

1. Upgrade your server to the latest Zulip 2.0.x release, since newer releases don't support Ubuntu 14.04 Trusty.

2. Same as for Bionic to Focal.

3. Same as for Bionic to Focal.

4. As root, upgrade the database installation and OS configuration to match the new OS version:

   apt remove upstart -y
   /home/zulip/deployments/current/scripts/zulip-puppet-apply -f
   pg_dropcluster 9.5 main --stop
   systemctl stop postgresql
   pg_upgradecluster -m upgrade 9.3 main
   pg_dropcluster 9.3 main
   apt remove postgresql-9.3
   systemctl start postgresql
   service memcached restart
   

5. Same as for Bionic to Focal.

Upgrading from Debian Stretch to Debian Buster

1. Upgrade your server to the latest Zulip 2.1.x release, since newer releases don't support Debian Stretch.

2. Same as for Xenial to Bionic, above.

3. Follow Debian's instructions to upgrade the OS.

   When prompted for you how to upgrade configuration files for services that Zulip manages like redis, postgres, nginx, and memcached, the best choice is N to keep the currently installed version. But it's not important; the next step will re-install Zulip's configuration in any case.

4. As root, upgrade the database installation and OS configuration to match the new OS version:

   apt remove upstart -y
   /home/zulip/deployments/current/scripts/zulip-puppet-apply -f
   pg_dropcluster 9.5 main --stop
   systemctl stop postgresql
   pg_upgradecluster -m upgrade 9.3 main
   pg_dropcluster 9.3 main
   apt remove postgresql-9.3
   systemctl start postgresql
   service memcached restart
   

5. Same as for Xenial to Bionic.

Modifying Zulip

Zulip is 100% free and open source software, and you're welcome to modify it! This section explains how to make and maintain modifications in a safe and convenient fashion.

If you do modify Zulip and then report an issue you see in your modified version of Zulip, please be responsible about communicating that fact:

• Ideally, you'd reproduce the issue in an unmodified version (e.g. on chat.zulip.org or zulip.com).
• Where that is difficult or you think it's very unlikely your changes are related to the issue, just mention your changes in the issue report.

If you're looking to modify Zulip by applying changes developed by the Zulip core team and merged into master, skip to this section.

Making changes

One way to modify Zulip is to just edit files under /home/zulip/deployments/current and then restart the server. This can work OK for testing small changes to Python code or shell scripts. But we don't recommend this approach for maintaining changes because:

• You cannot modify JavaScript, CSS, or other frontend files this way, because we don't include them in editable form in our production release tarballs (doing so would make our release tarballs much larger without any runtime benefit).
• You will need to redo your changes after you next upgrade your Zulip server (or they will be lost).
• You need to remember to restart the server or your changes won't have effect.
• Your changes aren't tracked, so mistakes can be hard to debug.

Instead, we recommend the following GitHub-based workflow (see our Git guide if you need a primer):

• Decide where you're going to edit Zulip's code. We recommend using the Zulip development environment on a desktop or laptop as it will make it extremely convenient for you to test your changes without deploying them in production. But if your changes are small or you're OK with risking downtime, you don't strictly need it; you just need an environment with Git installed.
• Important. Determine what Zulip version you're running on your server. You can check by inspecting ZULIP_VERSION in /home/zulip/deployments/current/version.py (we'll use 2.0.4 below). If you apply your changes to the wrong version of Zulip, it's likely to fail and potentially cause downtime.
• Fork and clone the zulip/zulip repository on GitHub.
• Create a branch (named acme-branch below) containing your changes:

cd zulip
git checkout -b acme-branch 2.0.4

• Use your favorite code editor to modify Zulip.
• Commit your changes and push them to GitHub:

git commit -a

# Use `git diff` to verify your changes are what you expect
git diff 2.0.4 acme-branch

# Push the changes to your GitHub fork
git push origin +acme-branch

• Login to your Zulip server and configure and use upgrade-zulip-from-git to install the changes; remember to configure git_repo_url to point to your fork on GitHub and run it as upgrade-zulip-from-git acme-branch.

This workflow solves all of the problems described above: your change will be compiled and installed correctly (restarting the server), and your changes will be tracked so that it's convenient to maintain them across future Zulip releases.

Upgrading to future releases

Eventually, you'll want to upgrade to a new Zulip release. If your changes were integrated into that Zulip release or are otherwise no longer needed, you can just upgrade as usual. Otherwise, you'll need to update your branch by rebasing your changes (starting from a clone of the zulip/zulip repository). The example below assumes you have a branch off of 2.0.4 and want to upgrade to 2.1.0.

cd zulip
git fetch --tags upstream
git checkout acme-branch
git rebase --onto 2.1.0 2.0.4
# Fix any errors or merge conflicts; see Zulip's Git Guide for advice

# Use `git diff` to verify your changes are what you expect
git diff 2.1.0 acme-branch

git push origin +acme-branch

And then use upgrade-zulip-from-git to install your updated branch, as before.

Making changes with docker-zulip

If you are using docker-zulip, there are two things that are different from the above:

• Because of how container images work, editing files directly is even more precarious, because Docker is designed for working with container images and may lose your changes.
• Instead of running upgrade-zulip-from-git, you will need to use the docker upgrade workflow to build a container image based on your modified version of Zulip.

Applying changes from master

If you are experiencing an issue that has already been fixed by the Zulip development community, and you'd like to get the fix now, you have a few options. There are two possible ways you might get those fixes on your local Zulip server without waiting for an official release.

Applying a small change

Many bugs have small/simple fixes. In this case, you can use the Git workflow described above, using:

git fetch upstream
git cherry-pick abcd1234

instead of "making changes locally" (where abcd1234 is the commit ID of the change you'd like).

In general, we can't provide unpaid support for issues caused by cherry-picking arbitrary commits if the issues don't also affect master or an official release.

The exception to this rule is when we ask or encourage a user to apply a change to their production system to help verify the fix resolves the issue for them. You can expect the Zulip community to be responsive in debugging any problems caused by a patch we asked you to apply.

Also, consider asking whether a small fix that is important to you can be added to the current stable release branch (E.g. 2.1.x). In addition to scheduling that change for Zulip's next bug fix release, we support changes in stable release branches as though they were released.

Upgrading to master

It's unsafe to backport arbitrary patches from master to an older version. Common issues include:

• Changes containing database migrations (new files under */migrations/), which includes most new features. We don't support applying database migrations out of order.
• Changes that are stacked on top of other changes to the same system.
• Essentially any patch with hundreds of lines of changes.

While it's possible to backport these sorts of changes, you're unlikely to succeed without help from the core team via a support contract.

If you need an unreleased feature, the best path is usually to upgrade to Zulip master using upgrade-zulip-from-git. Before upgrading to master, make sure you understand:

• The master branch is under very active development; dozens of new changes are integrated into it on most days. Master can have thousands of changes not present in the latest release (all of which will be included in our next release). There are probably some bugs.
• We deploy master to chat.zulip.org and zulip.com on a regular basis (often daily), so it's very important to the project that it be stable. Most regressions will be minor UX issues or be fixed quickly, because we need them to be fixed.
• The development community is very interested in helping debug issues that arise when upgrading from the latest release to master, since they provide us an opportunity to fix that category of issue before our next major release. (Much more so than we are in helping folks debug other custom changes). That said, we cannot make any guarantees about how quickly we'll resolve an issue to folks without a formal support contract.
• We do not support downgrading from master to earlier versions, so if downtime for your Zulip server is unacceptable, make sure you have a current backup in case the upgrade fails.
• Our changelog contains draft release notes available listing major changes since the last release. The Upgrade notes section will always be current, even if some new features aren't documented.

Contributing patches

Zulip contains thousands of changes submitted by volunteer contributors like you. If your changes are likely to be of useful to other organizations, consider contributing them.