zulip/docs/production/postgresql.md

PostgreSQL database details

Starting with Zulip 3.0, Zulip supports a range of PostgreSQL versions:

We recommend that installations upgrade to the latest PostgreSQL supported by their version of Zulip.

Separate PostgreSQL database

It is possible to run Zulip against a PostgreSQL database which is not on the primary application server. There are two possible flavors of this -- using a managed PostgreSQL instance from a cloud provider, or separating the PostgreSQL server onto a separate (but still Zulip-managed) server for scaling purposes.

Cloud-provider-managed PostgreSQL (e.g. Amazon RDS)

You can use a database-as-a-service like Amazon RDS for the Zulip database. The experience is slightly degraded, in that most providers don't include useful dictionary files in their installations, and don't provide a way to provide them yourself, resulting in a degraded full-text search experience around issues dictionary files are relevant (e.g. stemming).

Step 1: Set up Zulip

Follow the standard install instructions, with modified install arguments:

./zulip-server-*/scripts/setup/install --certbot \
    --email=YOUR_EMAIL --hostname=YOUR_HOSTNAME \
    --puppet-classes=zulip::profile::standalone_nodb \
    --postgresql-missing-dictionaries

Step 2: Create the PostgreSQL database

Access an administrative psql shell on your PostgreSQL database, and run the commands in scripts/setup/create-db.sql to:

• Create a database called zulip with C.UTF-8 collation.
• Create a user called zulip with full rights on that database.
• Log in with the zulip user to create a schema called zulip in the zulip database. You might have to grant create privileges first for the zulip user to do this.

If you cannot run that SQL directly, you should perform the equivalent actions in the service's web UI.

Depending on how authentication works for your PostgreSQL installation, you may also need to set a password for the Zulip user, generate a client certificate, or similar; consult the documentation for your database provider for the available options.

Step 3: Configure Zulip to use the PostgreSQL database

In /etc/zulip/settings.py on your Zulip server, configure the following settings with details for how to connect to your PostgreSQL server. Your database provider should provide these details.

• REMOTE_POSTGRES_HOST: Name or IP address of the PostgreSQL server.
• REMOTE_POSTGRES_PORT: Port on the PostgreSQL server.
• REMOTE_POSTGRES_SSLMODE: SSL Mode used to connect to the server.

If you're using password authentication, you should specify the password of the zulip user in /etc/zulip/zulip-secrets.conf as follows:

postgres_password = abcd1234

Set the remote server's PostgreSQL version in /etc/zulip/zulip.conf:

[postgresql]
# Set this to match the version running on your remote PostgreSQL server
version = 16

Now complete the installation by running the following commands.

# Ask Zulip installer to initialize the PostgreSQL database.
su zulip -c '/home/zulip/deployments/current/scripts/setup/initialize-database'

# And then generate a realm creation link:
su zulip -c '/home/zulip/deployments/current/manage.py generate_realm_creation_link'

Remote PostgreSQL database

This assumes two servers; one hosting the PostgreSQL database, and one hosting the remainder of the Zulip services.

Step 1: Set up Zulip

Follow the standard install instructions, with modified install arguments:

./zulip-server-*/scripts/setup/install --certbot \
    --email=YOUR_EMAIL --hostname=YOUR_HOSTNAME \
    --puppet-classes=zulip::profile::standalone_nodb

Step 2: Create the PostgreSQL database server

On the host that will run PostgreSQL, download the Zulip tarball and install just the PostgreSQL server part:

./zulip-server-*/scripts/setup/install \
    --puppet-classes=zulip::profile::postgresql

./zulip-server-*/scripts/setup/create-database

You will need to configure /etc/postgresql/*/main/pg_hba.conf to allow connections to the zulip database as the zulip user from the application frontend host. How you configure this is up to you (i.e. password authentication, certificates, etc), and is outside the scope of this document.

Step 3: Configure Zulip to use the PostgreSQL database

In /etc/zulip/settings.py on your Zulip server, configure the following settings with details for how to connect to your PostgreSQL server.

• REMOTE_POSTGRES_HOST: Name or IP address of the PostgreSQL server.
• REMOTE_POSTGRES_PORT: Port on the PostgreSQL server; this is likely 5432
• REMOTE_POSTGRES_SSLMODE: SSL Mode used to connect to the server.

If you're using password authentication, you should specify the password of the zulip user in /etc/zulip/zulip-secrets.conf as follows:

postgres_password = abcd1234

Set the remote server's PostgreSQL version in /etc/zulip/zulip.conf:

[postgresql]
# Set this to match the version running on your remote PostgreSQL server
version = 16

Now complete the installation by running the following commands.

# Ask Zulip installer to initialize the PostgreSQL database.
su zulip -c '/home/zulip/deployments/current/scripts/setup/initialize-database'

# And then generate a realm creation link:
su zulip -c '/home/zulip/deployments/current/manage.py generate_realm_creation_link'

PostgreSQL warm standby

Zulip's configuration allows for warm standby database replicas as a disaster recovery solution; see the linked PostgreSQL documentation for details on this type of deployment. Zulip's configuration can, but is not required to, build on top of wal-g, our streaming database backup solution, for ease of establishing base images without incurring load on the primary.

Warm standby replicas should configure the hostname of their primary replica, and username to use for replication, in /etc/zulip/zulip.conf:

[postgresql]
replication_user = replicator
replication_primary = hostname-of-primary.example.com

The postgres user on the replica will need to be able to authenticate as the replication_user user, which may require further configuration of pg_hba.conf and client certificates on the replica. If you are using password authentication, you can set a postgresql_replication_password secret in /etc/zulip/zulip-secrets.conf.

Use zulip-puppet-apply to rebuild the PostgreSQL configuration with those values. If wal-g is used, use env-wal-g backup-fetch to fetch the latest copy of the base backup; otherwise, use pg_basebackup. The PostgreSQL replica should then begin live-replicating from the primary.

In the event of a primary failure, use pg_ctl promote on the warm standby to promote it to primary. As with all database promotions, care should be taken to ensure that the old primary cannot come back online, and leave the cluster with two diverging timelines.

To make fail-over to the warm-standby faster, without requiring a restart of Zulip services, you can configure Zulip with a comma-separated list of remote PostgreSQL servers to connect to; it will choose the first which accepts writes (i.e. is not a read-only replica). In the event that the primary fails, it will repeatedly retry the list, in order, until the replica is promoted and becomes writable. To configure this, in /etc/zulip/settings.py, set:

REMOTE_POSTGRES_HOST = 'primary-database-host,warm-standby-host'

PostgreSQL vacuuming alerts

The autovac_freeze PostgreSQL alert from check_postgres is particularly important. This alert indicates that the age (in terms of number of transactions) of the oldest transaction id (XID) is getting close to the autovacuum_freeze_max_age setting. When the oldest XID hits that age, PostgreSQL will force a VACUUM operation, which can often lead to sudden downtime until the operation finishes. If it did not do this and the age of the oldest XID reached 2 billion, transaction id wraparound would occur and there would be data loss. To clear the nagios alert, perform a VACUUM in each indicated database as a database superuser (i.e. postgres).

See the PostgreSQL documentation for more details on PostgreSQL vacuuming.