Mirror
/
zulip
mirror of https://github.com/zulip/zulip.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity

docs: Improve PostgreSQL documentation. 

This removes a bunch of out-of-date documentation which is better
addressed outside of Zulip, and provides more details on running a
PostgreSQL server on a separate host.
This commit is contained in:
Alex Vandiver 2024-04-04 21:27:01 +00:00 committed by Tim Abbott
parent 7187146422
commit 44c4b93f6e
2 changed files with 166 additions and 215 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

105
docs/production/deployment.md
View File

@ -81,12 +81,11 @@ as well as those mentioned in the
default database user will be `zulip`. This setting can only be set
on the first install.
- `--postgresql-missing-dictionaries`: Set
`postgresql.missing_dictionaries` ([docs][missing-dicts]) in the
Zulip settings, which omits some configuration needed for full-text
indexing. This should be used with [cloud managed databases like
RDS](#using-zulip-with-amazon-rds-as-the-database). This option
conflicts with `--no-overwrite-settings`.
- `--postgresql-missing-dictionaries`: Set `postgresql.missing_dictionaries`
([docs][missing-dicts]) in the Zulip settings, which omits some configuration
needed for full-text indexing. This should be used with [cloud managed
databases like RDS][postgresql]. This option conflicts with
`--no-overwrite-settings`.
- `--no-init-db`: This option instructs the installer to not do any
database initialization. This should be used when you already have a
@ -190,18 +189,6 @@ Smokescreen in `/etc/zulip/settings.py`; just search for the service name in
that file and you'll find inline documentation in comments for how to configure
it.
Since some of these services require some configuration on the node itself
(e.g. installing our PostgreSQL extensions), we have designed the Puppet
configuration that Zulip uses for installing and upgrading configuration to be
completely modular.
For example, to install a Zulip Redis server on a machine, you can run
the following after unpacking a Zulip production release tarball:
```bash
./scripts/setup/install --puppet-classes zulip::profile::redis
```
All puppet modules under `zulip::profile` are allowed to be configured
stand-alone on a host. You can see most likely manifests you might
want to choose in the list of includes in [the main manifest for the
@ -211,78 +198,20 @@ directory if you want to customize. A good example of doing this is
in the [kandra Puppet configuration][zulipchat-puppet] that we use
as part of managing chat.zulip.org and zulip.com.
For example, to install a Zulip Redis server on a machine, you can run
the following after unpacking a Zulip production release tarball:
```bash
./scripts/setup/install --puppet-classes zulip::profile::redis
```
To run the database on a separate server, including a cloud provider's managed
PostgreSQL instance (e.g. AWS RDS), see out [dedicated PostgreSQL
documentation][postgresql].
[standalone.pp]: https://github.com/zulip/zulip/blob/main/puppet/zulip/manifests/profile/standalone.pp
[zulipchat-puppet]: https://github.com/zulip/zulip/tree/main/puppet/kandra/manifests
### Using Zulip with Amazon RDS as the database
You can use DBaaS services like Amazon RDS for the Zulip database.
The experience is slightly degraded, in that most DBaaS provides don't
include useful dictionary files in their installations and don't
provide a way to provide them yourself, resulting in a degraded
[full-text search](../subsystems/full-text-search.md) experience
around issues dictionary files are relevant (e.g. stemming).
You also need to pass some extra options to the Zulip installer in
order to avoid it throwing an error when Zulip attempts to configure
the database's dictionary files for full-text search; the details are
below.
#### Step 1: Set up Zulip
Follow the [standard instructions](install.md), with modified `install`
arguments:
```bash
./zulip-server-*/scripts/setup/install --certbot \
--email=YOUR_EMAIL --hostname=YOUR_HOSTNAME \
--puppet-classes=zulip::profile::standalone_nodb \
--postgresql-missing-dictionaries
```
#### Step 2: Create the PostgreSQL database
Access an administrative `psql` shell on your PostgreSQL database, and
run the commands in `scripts/setup/create-db.sql` to:
- Create a database called `zulip`.
- Create a user called `zulip`.
- Now log in with the `zulip` user to create a schema called
`zulip` in the `zulip` database. You might have to grant `create`
privileges first for the `zulip` user to do this.
Depending on how authentication works for your PostgreSQL installation,
you may also need to set a password for the Zulip user, generate a
client certificate, or similar; consult the documentation for your
database provider for the available options.
#### Step 3: Configure Zulip to use the PostgreSQL database
In `/etc/zulip/settings.py` on your Zulip server, configure the
following settings with details for how to connect to your PostgreSQL
server. Your database provider should provide these details.
- `REMOTE_POSTGRES_HOST`: Name or IP address of the PostgreSQL server.
- `REMOTE_POSTGRES_PORT`: Port on the PostgreSQL server.
- `REMOTE_POSTGRES_SSLMODE`: SSL Mode used to connect to the server.
If you're using password authentication, you should specify the
password of the `zulip` user in /etc/zulip/zulip-secrets.conf as
follows:
```ini
postgres_password = abcd1234
```
Now complete the installation by running the following commands.
```bash
# Ask Zulip installer to initialize the PostgreSQL database.
su zulip -c '/home/zulip/deployments/current/scripts/setup/initialize-database'
# And then generate a realm creation link:
su zulip -c '/home/zulip/deployments/current/manage.py generate_realm_creation_link'
```
[postgresql]: postgresql.md
## Using an alternate port

276
docs/production/postgresql.md
View File

@ -12,41 +12,73 @@ PostgreSQL][upgrade-postgresql] supported by their version of Zulip.
[upgrade-postgresql]: upgrade.md#upgrading-postgresql
#### Remote PostgreSQL database
## Separate PostgreSQL database
This is a bit annoying to set up, but you can configure Zulip to use a
dedicated PostgreSQL server by setting the `REMOTE_POSTGRES_HOST`
variable in /etc/zulip/settings.py, and configuring PostgreSQL
certificate authentication (see
http://www.postgresql.org/docs/9.1/static/ssl-tcp.html and
http://www.postgresql.org/docs/9.1/static/libpq-ssl.html for
documentation on how to set this up and deploy the certificates) to
make the DATABASES configuration in `zproject/computed_settings.py`
work (or override that configuration).
It is possible to run Zulip against a PostgreSQL database which is not on the
primary application server. There are two possible flavors of this -- using a
managed PostgreSQL instance from a cloud provider, or separating the PostgreSQL
server onto a separate (but still Zulip-managed) server for scaling purposes.
If you want to use a remote PostgreSQL database, you should configure
the information about the connection with the server. You need a user
called "zulip" in your database server. You can configure these
options in `/etc/zulip/settings.py` (the below descriptions are from the
PostgreSQL documentation):
### Cloud-provider-managed PostgreSQL (e.g. Amazon RDS)
- `REMOTE_POSTGRES_HOST`: Name or IP address of the remote host
- `REMOTE_POSTGRES_SSLMODE`: SSL Mode used to connect to the server,
different options you can use are:
- disable: I don't care about security, and I don't want to pay the
overhead of encryption.
- allow: I don't care about security, but I will pay the overhead of
encryption if the server insists on it.
- prefer: I don't care about encryption, but I wish to pay the
overhead of encryption if the server supports it.
- require: I want my data to be encrypted, and I accept the
overhead. I trust that the network will make sure I always connect
to the server I want.
- verify-ca: I want my data encrypted, and I accept the overhead. I
want to be sure that I connect to a server that I trust.
- verify-full: I want my data encrypted, and I accept the
overhead. I want to be sure that I connect to a server I trust,
and that it's the one I specify.
You can use a database-as-a-service like Amazon RDS for the Zulip database. The
experience is slightly degraded, in that most providers don't include useful
dictionary files in their installations, and don't provide a way to provide them
yourself, resulting in a degraded [full-text search][fts] experience around
issues dictionary files are relevant (e.g. stemming).
[fts]: ../subsystems/full-text-search.md
#### Step 1: Set up Zulip
Follow the [standard install instructions](install.md), with modified `install`
arguments:
```bash
./zulip-server-*/scripts/setup/install --certbot \
--email=YOUR_EMAIL --hostname=YOUR_HOSTNAME \
--puppet-classes=zulip::profile::standalone_nodb \
--postgresql-missing-dictionaries
```
#### Step 2: Create the PostgreSQL database
Access an administrative `psql` shell on your PostgreSQL database, and
run the commands in `scripts/setup/create-db.sql` to:
- Create a database called `zulip` with `C.UTF-8` collation.
- Create a user called `zulip` with full rights on that database.
- Log in with the `zulip` user to create a schema called `zulip` in the `zulip`
database. You might have to grant `create` privileges first for the `zulip`
user to do this.
If you cannot run that SQL directly, you should perform the equivalent actions
in the service's web UI.
Depending on how authentication works for your PostgreSQL installation, you may
also need to set a password for the Zulip user, generate a client certificate,
or similar; consult the documentation for your database provider for the
available options.
#### Step 3: Configure Zulip to use the PostgreSQL database
In `/etc/zulip/settings.py` on your Zulip server, configure the
following settings with details for how to connect to your PostgreSQL
server. Your database provider should provide these details.
- `REMOTE_POSTGRES_HOST`: Name or IP address of the PostgreSQL server.
- `REMOTE_POSTGRES_PORT`: Port on the PostgreSQL server.
- `REMOTE_POSTGRES_SSLMODE`: [SSL Mode][ssl-mode] used to connect to the server.
[ssl-mode]: https://www.postgresql.org/docs/current/libpq-ssl.html#LIBPQ-SSL-PROTECTION
If you're using password authentication, you should specify the
password of the `zulip` user in /etc/zulip/zulip-secrets.conf as
follows:
```ini
postgres_password = abcd1234
```
Set the remote server's PostgreSQL version in `/etc/zulip/zulip.conf`:
@ -56,109 +88,99 @@ Set the remote server's PostgreSQL version in `/etc/zulip/zulip.conf`:
version = 16
```
Then you should specify the password of the user zulip for the
database in /etc/zulip/zulip-secrets.conf:
Now complete the installation by running the following commands.
```bash
# Ask Zulip installer to initialize the PostgreSQL database.
su zulip -c '/home/zulip/deployments/current/scripts/setup/initialize-database'
# And then generate a realm creation link:
su zulip -c '/home/zulip/deployments/current/manage.py generate_realm_creation_link'
```
### Remote PostgreSQL database
This assumes two servers; one hosting the PostgreSQL database, and one hosting
the remainder of the Zulip services.
#### Step 1: Set up Zulip
Follow the [standard install instructions](install.md), with modified `install`
arguments:
```bash
./zulip-server-*/scripts/setup/install --certbot \
--email=YOUR_EMAIL --hostname=YOUR_HOSTNAME \
--puppet-classes=zulip::profile::standalone_nodb
```
#### Step 2: Create the PostgreSQL database server
On the host that will run PostgreSQL, download the Zulip tarball and install
just the PostgreSQL server part:
```bash
./zulip-server-*/scripts/setup/install \
--puppet-classes=zulip::profile::postgresql
./zulip-server-*/scripts/setup/create-database
```
You will need to [configure `/etc/postgresql/*/main/pg_hba.conf`][pg-hba] to
allow connections to the `zulip` database as the `zulip` user from the
application frontend host. How you configure this is up to you (i.e. password
authentication, certificates, etc), and is outside the scope of this document.
[pg-hba]: https://www.postgresql.org/docs/current/auth-pg-hba-conf.html
#### Step 3: Configure Zulip to use the PostgreSQL database
In `/etc/zulip/settings.py` on your Zulip server, configure the following
settings with details for how to connect to your PostgreSQL server.
- `REMOTE_POSTGRES_HOST`: Name or IP address of the PostgreSQL server.
- `REMOTE_POSTGRES_PORT`: Port on the PostgreSQL server; this is likely `5432`
- `REMOTE_POSTGRES_SSLMODE`: [SSL Mode][ssl-mode] used to connect to the server.
If you're using password authentication, you should specify the
password of the `zulip` user in /etc/zulip/zulip-secrets.conf as
follows:
```ini
postgres_password = xxxx
postgres_password = abcd1234
```
Finally, you can stop your database on the Zulip server via:
Set the remote server's PostgreSQL version in `/etc/zulip/zulip.conf`:
```ini
[postgresql]
# Set this to match the version running on your remote PostgreSQL server
version = 16
```
Now complete the installation by running the following commands.
```bash
sudo service postgresql stop
sudo update-rc.d postgresql disable
# Ask Zulip installer to initialize the PostgreSQL database.
su zulip -c '/home/zulip/deployments/current/scripts/setup/initialize-database'
# And then generate a realm creation link:
su zulip -c '/home/zulip/deployments/current/manage.py generate_realm_creation_link'
```
In future versions of this feature, we'd like to implement and
document how to the remote PostgreSQL database server itself
automatically by using the Zulip install script with a different set
of Puppet manifests than the all-in-one feature; if you're interested
in working on this, post to the Zulip development mailing list and we
can give you some tips.
## PostgreSQL vacuuming alerts
#### Debugging PostgreSQL database issues
The `autovac_freeze` PostgreSQL alert from `check_postgres` is particularly
important. This alert indicates that the age (in terms of number of
transactions) of the oldest transaction id (XID) is getting close to the
`autovacuum_freeze_max_age` setting. When the oldest XID hits that age,
PostgreSQL will force a VACUUM operation, which can often lead to sudden
downtime until the operation finishes. If it did not do this and the age of the
oldest XID reached 2 billion, transaction id wraparound would occur and there
would be data loss. To clear the nagios alert, perform a `VACUUM` in each
indicated database as a database superuser (i.e. `postgres`).
When debugging PostgreSQL issues, in addition to the standard `pg_top`
tool, often it can be useful to use this query:
See [the PostgreSQL documentation][vacuum] for more details on PostgreSQL
vacuuming.
```postgresql
SELECT procpid,waiting,query_start,current_query FROM pg_stat_activity ORDER BY procpid;
```
which shows the currently running backends and their activity. This is
similar to the pg_top output, with the added advantage of showing the
complete query, which can be valuable in debugging.
To stop a runaway query, you can run
`SELECT pg_cancel_backend(pid int)` or
`SELECT pg_terminate_backend(pid int)` as the 'postgres' user. The
former cancels the backend's current query and the latter terminates
the backend process. They are implemented by sending SIGINT and
SIGTERM to the processes, respectively. We recommend against sending
a PostgreSQL process SIGKILL. Doing so will cause the database to kill
all current connections, roll back any pending transactions, and enter
recovery mode.
#### Stopping the Zulip PostgreSQL database
To start or stop PostgreSQL manually, use the pg_ctlcluster command:
```bash
pg_ctlcluster 9.1 [--force] main {start|stop|restart|reload}
```
By default, using stop uses "smart" mode, which waits for all clients
to disconnect before shutting down the database. This can take
prohibitively long. If you use the --force option with stop,
pg_ctlcluster will try to use the "fast" mode for shutting
down. "Fast" mode is described by the manpage thusly:
> With the --force option the "fast" mode is used which rolls back all
> active transactions, disconnects clients immediately and thus shuts
> down cleanly. If that does not work, shutdown is attempted again in
> "immediate" mode, which can leave the cluster in an inconsistent state
> and thus will lead to a recovery run at the next start. If this still
> does not help, the postmaster process is killed. Exits with 0 on
> success, with 2 if the server is not running, and with 1 on other
> failure conditions. This mode should only be used when the machine is
> about to be shut down.
Many database parameters can be adjusted while the database is
running. Just modify /etc/postgresql/9.1/main/postgresql.conf and
issue a reload. The logs will note the change.
#### Debugging issues starting PostgreSQL
pg_ctlcluster often doesn't give you any information on why the
database failed to start. It may tell you to check the logs, but you
won't find any information there. pg_ctlcluster runs the following
command underneath when it actually goes to start PostgreSQL:
```bash
/usr/lib/postgresql/9.1/bin/pg_ctl start -D /var/lib/postgresql/9.1/main -s -o \
'-c config_file="/etc/postgresql/9.1/main/postgresql.conf"'
```
Since pg_ctl doesn't redirect stdout or stderr, running the above can
give you better diagnostic information. However, you might want to
stop PostgreSQL and restart it using pg_ctlcluster after you've debugged
with this approach, since it does bypass some of the work that
pg_ctlcluster does.
#### PostgreSQL vacuuming alerts
The `autovac_freeze` PostgreSQL alert from `check_postgres` is
particularly important. This alert indicates that the age (in terms
of number of transactions) of the oldest transaction id (XID) is
getting close to the `autovacuum_freeze_max_age` setting. When the
oldest XID hits that age, PostgreSQL will force a VACUUM operation,
which can often lead to sudden downtime until the operation finishes.
If it did not do this and the age of the oldest XID reached 2 billion,
transaction id wraparound would occur and there would be data loss.
To clear the nagios alert, perform a `VACUUM` in each indicated
database as a database superuser (`postgres`).
See
http://www.postgresql.org/docs/9.1/static/routine-vacuuming.html#VACUUM-FOR-WRAPAROUND
for more details on PostgreSQL vacuuming.
[vacuum]: http://www.postgresql.org/docs/current/static/routine-vacuuming.html#VACUUM-FOR-WRAPAROUND