diff --git a/docs/_templates/layout.html b/docs/_templates/layout.html
index eceb87fae4..4217f0bf1b 100644
--- a/docs/_templates/layout.html
+++ b/docs/_templates/layout.html
@@ -6,7 +6,7 @@
# version e.g. to say that something is likely to have changed.
# For more info see: https://www.sphinx-doc.org/en/master/templating.html
#}
- {% if pagename in [] and release.endswith('+git') %}
+ {% if pagename in ["production/system-configuration", "production/reverse-proxies"] and release.endswith('+git') %}
{#
# This page doesn't exist in the stable documentation yet.
# This temporary workaround prevents test failures and should be removed after the next release.
diff --git a/docs/overview/changelog.md b/docs/overview/changelog.md
index 2c147437a6..adc748e652 100644
--- a/docs/overview/changelog.md
+++ b/docs/overview/changelog.md
@@ -345,7 +345,7 @@ _Released 2023-08-25_
- Fixed a bug, introduced in Zulip Server 7.2, when the
[email gateway](../production/email-gateway.md)
was used in conjunction with a
- [reverse proxy](../production/deployment.md#putting-the-zulip-application-behind-a-reverse-proxy).
+ [reverse proxy](../production/reverse-proxies.md).
- Improved the performance of
[resolving](https://zulip.com/help/resolve-a-topic) or
[moving](https://zulip.com/help/move-content-to-another-topic) long topics.
@@ -404,7 +404,7 @@ _Released 2023-07-05_
`X-Forwarded-Proto` is also necessary.
- Removed [reverse proxy][proxies] nginx configuration files when the
- [`loadbalancer.ips`](../production/deployment.md#ips)
+ [`loadbalancer.ips`](../production/system-configuration.md#ips)
setting has been unset.
- Improved error-handling of scheduled emails, so they cannot attempt infinite
deliveries of a message with no recipients.
@@ -422,10 +422,10 @@ _Released 2023-07-05_
[import](https://zulip.com/help/import-from-slack#export-your-slack-data),
such as a token having too few permissions.
- Added support for IPv6
- [nameservers in the nginx configuration](../production/deployment.md#nameserver).
+ [nameservers in the nginx configuration](../production/system-configuration.md#nameserver).
- Updated translations.
-[proxies]: ../production/deployment.md#configuring-zulip-to-trust-proxies
+[proxies]: ../production/reverse-proxies.md#configuring-zulip-to-trust-proxies
### Zulip Server 7.1
@@ -593,7 +593,7 @@ _Released 2023-05-31_
- High volume log files like `server.log` are now by default retained
for 14 days, configured via the `access_log_retention_days`
[deployment
- option](../production/deployment.md#system-and-deployment-configuration). This
+ option](../production/system-configuration.md). This
replaces a harder to understand size-based algorithm that was not
easily configurable.
- The URL patterns for
@@ -613,8 +613,8 @@ _Released 2023-05-31_
- Zulip's Twitter preview integration has been disabled due to Twitter
desupporting the API that it relied on.
-[reverse-proxy-docs]: ../production/deployment.md#putting-the-zulip-application-behind-a-reverse-proxy
-[loadbalancer-ips]: ../production/deployment.md#configuring-zulip-to-trust-proxies
+[reverse-proxy-docs]: ../production/reverse-proxies.md
+[loadbalancer-ips]: ../production/reverse-proxies.md#configuring-zulip-to-trust-proxies
## Zulip Server 6.x series
@@ -1772,7 +1772,7 @@ _Released 2021-05-13_
codebase with Prettier.
- Migrated testing from CircleCI to GitHub Actions.
-[zulip-conf-settings]: ../production/deployment.md#system-and-deployment-configuration
+[zulip-conf-settings]: ../production/system-configuration.md
## Zulip Server 3.x series
diff --git a/docs/production/deployment.md b/docs/production/deployment.md
index e8a36ba220..c3a603144f 100644
--- a/docs/production/deployment.md
+++ b/docs/production/deployment.md
@@ -79,7 +79,7 @@ as well as those mentioned in the
- `--no-overwrite-settings`: This option preserves existing
`/etc/zulip` configuration files.
-[missing-dicts]: #missing_dictionaries
+[missing-dicts]: system-configuration.md#missing_dictionaries
## Installing on an existing server
@@ -301,7 +301,7 @@ configure that as follows:
We also have documentation for a Zulip server [using HTTP][using-http] for use
behind reverse proxies.
-[using-http]: #configuring-zulip-to-allow-http
+[using-http]: reverse-proxies.md#configuring-zulip-to-allow-http
## Customizing the outgoing HTTP proxy
@@ -349,7 +349,7 @@ In Zulip 4.7 and older, to enable SSRF protection via Smokescreen, you
will need to explicitly add the `zulip::profile::smokescreen` Puppet
class, and configure the `[http_proxy]` block as above.
-[proxy.enable_for_camo]: #enable_for_camo
+[proxy.enable_for_camo]: system-configuration.md#enable_for_camo
[smokescreen]: https://github.com/stripe/smokescreen
[smokescreen-acls]: https://github.com/stripe/smokescreen#acls
[ssrf]: https://owasp.org/www-community/attacks/Server_Side_Request_Forgery
@@ -367,284 +367,6 @@ some other proxy, you can override this default by setting
[s3]: upload-backends.md#s3-backend-configuration
-## Putting the Zulip application behind a reverse proxy
-
-Zulip is designed to support being run behind a reverse proxy server.
-This section contains notes on the configuration required with
-variable reverse proxy implementations.
-
-### Installer options
-
-If your Zulip server will not be on the public Internet, we recommend,
-installing with the `--self-signed-cert` option (rather than the
-`--certbot` option), since Certbot requires the server to be on the
-public Internet.
-
-#### Configuring Zulip to allow HTTP
-
-Zulip requires clients to connect to Zulip servers over the secure
-HTTPS protocol; the insecure HTTP protocol is not supported. However,
-we do support using a reverse proxy that speaks HTTPS to clients and
-connects to the Zulip server over HTTP; this can be secure when the
-Zulip server is not directly exposed to the public Internet.
-
-After installing the Zulip server as [described
-above](#installer-options), you can configure Zulip to accept HTTP
-requests from a reverse proxy as follows:
-
-1. Add the following block to `/etc/zulip/zulip.conf`:
-
- ```ini
- [application_server]
- http_only = true
- ```
-
-1. As root, run
- `/home/zulip/deployments/current/scripts/zulip-puppet-apply`. This
- will convert Zulip's main `nginx` configuration file to allow HTTP
- instead of HTTPS.
-
-1. Finally, restart the Zulip server, using
- `/home/zulip/deployments/current/scripts/restart-server`.
-
-Note that Zulip must be able to accurately determine if its connection to the
-client was over HTTPS or not; if you enable `http_only`, it is very important
-that you correctly configure Zulip to trust the `X-Forwarded-Proto` header from
-its proxy (see the next section), or clients may see infinite redirects.
-
-#### Configuring Zulip to trust proxies
-
-Before placing Zulip behind a reverse proxy, it needs to be configured to trust
-the client IP addresses that the proxy reports via the `X-Forwarded-For` header,
-and the protocol reported by the `X-Forwarded-Proto` header. This is important
-to have accurate IP addresses in server logs, as well as in notification emails
-which are sent to end users. Zulip doesn't default to trusting all
-`X-Forwarded-*` headers, because doing so would allow clients to spoof any IP
-address, and claim connections were over a secure connection when they were not;
-we specify which IP addresses are the Zulip server's incoming proxies, so we
-know which `X-Forwarded-*` headers to trust.
-
-1. Determine the IP addresses of all reverse proxies you are setting up, as seen
- from the Zulip host. Depending on your network setup, these may not be the
- same as the public IP addresses of the reverse proxies. These can also be IP
- address ranges, as expressed in CIDR notation.
-
-1. Add the following block to `/etc/zulip/zulip.conf`.
-
- ```ini
- [loadbalancer]
- # Use the IP addresses you determined above, separated by commas.
- ips = 192.168.0.100
- ```
-
-1. Reconfigure Zulip with these settings. As root, run
- `/home/zulip/deployments/current/scripts/zulip-puppet-apply`. This will
- adjust Zulip's `nginx` configuration file to accept the `X-Forwarded-For`
- header when it is sent from one of the reverse proxy IPs.
-
-1. Finally, restart the Zulip server, using
- `/home/zulip/deployments/current/scripts/restart-server`.
-
-### nginx configuration
-
-Below is a working example of a full nginx configuration. It assumes
-that your Zulip server sits at `https://10.10.10.10:443`; see
-[above](#configuring-zulip-to-allow-http) to switch to HTTP.
-
-1. Follow the instructions to [configure Zulip to trust
- proxies](#configuring-zulip-to-trust-proxies).
-
-1. Configure the root `nginx.conf` file. We recommend using
- `/etc/nginx/nginx.conf` from your Zulip server for our recommended
- settings. E.g. if you don't set `client_max_body_size`, it won't be
- possible to upload large files to your Zulip server.
-
-1. Configure the `nginx` site-specific configuration (in
- `/etc/nginx/sites-available`) for the Zulip app. The following
- example is a good starting point:
-
- ```nginx
- server {
- listen 80;
- listen [::]:80;
- location / {
- return 301 https://$host$request_uri;
- }
- }
-
- server {
- listen 443 ssl http2;
- listen [::]:443 ssl http2;
- server_name zulip.example.com;
-
- ssl_certificate /etc/letsencrypt/live/zulip.example.com/fullchain.pem;
- ssl_certificate_key /etc/letsencrypt/live/zulip.example.com/privkey.pem;
-
- location / {
- proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
- proxy_set_header X-Forwarded-Proto $scheme;
- proxy_set_header Host $host;
- proxy_http_version 1.1;
- proxy_buffering off;
- proxy_read_timeout 20m;
- proxy_pass https://10.10.10.10:443;
- }
- }
- ```
-
- Don't forget to update `server_name`, `ssl_certificate`,
- `ssl_certificate_key` and `proxy_pass` with the appropriate values
- for your deployment.
-
-[nginx-proxy-longpolling-config]: https://github.com/zulip/zulip/blob/main/puppet/zulip/files/nginx/zulip-include-common/proxy_longpolling
-[standalone.pp]: https://github.com/zulip/zulip/blob/main/puppet/zulip/manifests/profile/standalone.pp
-[zulipchat-puppet]: https://github.com/zulip/zulip/tree/main/puppet/kandra/manifests
-
-### Apache2 configuration
-
-Below is a working example of a full Apache2 configuration. It assumes
-that your Zulip server sits at `https://internal.zulip.hostname:443`.
-Note that if you wish to use SSL to connect to the Zulip server,
-Apache requires you use the hostname, not the IP address; see
-[above](#configuring-zulip-to-allow-http) to switch to HTTP.
-
-1. Follow the instructions to [configure Zulip to trust
- proxies](#configuring-zulip-to-trust-proxies).
-
-1. Set `USE_X_FORWARDED_HOST = True` in `/etc/zulip/settings.py` and
- restart Zulip.
-
-1. Enable some required Apache modules:
-
- ```bash
- a2enmod ssl proxy proxy_http headers rewrite
- ```
-
-1. Create an Apache2 virtual host configuration file, similar to the
- following. Place it the appropriate path for your Apache2
- installation and enable it (E.g. if you use Debian or Ubuntu, then
- place it in `/etc/apache2/sites-available/zulip.example.com.conf`
- and then run
- `a2ensite zulip.example.com && systemctl reload apache2`):
-
- ```apache
-
- ServerName zulip.example.com
- RewriteEngine On
- RewriteRule ^ https://%{HTTP_HOST}%{REQUEST_URI} [R=301,L]
-
-
-
- ServerName zulip.example.com
-
- RequestHeader set "X-Forwarded-Proto" expr=%{REQUEST_SCHEME}
-
- RewriteEngine On
- RewriteRule /(.*) https://internal.zulip.hostname:443/$1 [P,L]
-
-
- Require all granted
- ProxyPass https://internal.zulip.hostname:443/ timeout=1200
-
-
- SSLEngine on
- SSLProxyEngine on
- SSLCertificateFile /etc/letsencrypt/live/zulip.example.com/fullchain.pem
- SSLCertificateKeyFile /etc/letsencrypt/live/zulip.example.com/privkey.pem
- # This file can be found in ~zulip/deployments/current/puppet/zulip/files/nginx/dhparam.pem
- SSLOpenSSLConfCmd DHParameters "/etc/nginx/dhparam.pem"
- SSLProtocol all -SSLv3 -TLSv1 -TLSv1.1
- SSLCipherSuite ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
- SSLHonorCipherOrder off
- SSLSessionTickets off
- Header set Strict-Transport-Security "max-age=31536000"
-
- ```
-
- Don't forget to update `ServerName`, `RewriteRule`, `ProxyPass`,
- `SSLCertificateFile`, and `SSLCertificateKeyFile` as are
- appropriate for your deployment.
-
-### HAProxy configuration
-
-Below is a working example of a HAProxy configuration. It assumes that
-your Zulip server sits at `https://10.10.10.10:443`; see
-[above](#configuring-zulip-to-allow-http) to switch to HTTP.
-
-1. Follow the instructions to [configure Zulip to trust
- proxies](#configuring-zulip-to-trust-proxies).
-
-1. Configure HAProxy. The below is a minimal `frontend` and `backend`
- configuration:
-
- ```text
- frontend zulip
- mode http
- bind *:80
- bind *:443 ssl crt /etc/ssl/private/zulip-combined.crt
- http-request redirect scheme https code 301 unless { ssl_fc }
- http-request set-header X-Forwarded-Proto http unless { ssl_fc }
- http-request set-header X-Forwarded-Proto https if { ssl_fc }
- default_backend zulip
-
- backend zulip
- mode http
- timeout server 20m
- server zulip 10.10.10.10:443 check ssl ca-file /etc/ssl/certs/ca-certificates.crt
- ```
-
- Don't forget to update `bind *:443 ssl crt` and `server` as is
- appropriate for your deployment.
-
-### Other proxies
-
-If you're using another reverse proxy implementation, there are few
-things you need to be careful about when configuring it:
-
-1. Configure your reverse proxy (or proxies) to correctly maintain the
- `X-Forwarded-For` HTTP header, which is supposed to contain the series
- of IP addresses the request was forwarded through. Additionally,
- [configure Zulip to respect the addresses sent by your reverse
- proxies](#configuring-zulip-to-trust-proxies). You can verify
- your work by looking at `/var/log/zulip/server.log` and checking it
- has the actual IP addresses of clients, not the IP address of the
- proxy server.
-
-1. Configure your reverse proxy (or proxies) to correctly maintain the
- `X-Forwarded-Proto` HTTP header, which is supposed to contain either `https`
- or `http` depending on the connection between your browser and your
- proxy. This will be used by Django to perform CSRF checks regardless of your
- connection mechanism from your proxy to Zulip. Note that the proxies _must_
- set the header, overriding any existing values, not add a new header.
-
-1. Configure your proxy to pass along the `Host:` header as was sent
- from the client, not the internal hostname as seen by the proxy.
- If this is not possible, you can set `USE_X_FORWARDED_HOST = True`
- in `/etc/zulip/settings.py`, and pass the client's `Host` header to
- Zulip in an `X-Forwarded-Host` header.
-
-1. Ensure your proxy doesn't interfere with Zulip's use of
- long-polling for real-time push from the server to your users'
- browsers. This [nginx code snippet][nginx-proxy-longpolling-config]
- does this.
-
- The key configuration options are, for the `/json/events` and
- `/api/1/events` endpoints:
-
- - `proxy_read_timeout 1200;`. It's critical that this be
- significantly above 60s, but the precise value isn't important.
- - `proxy_buffering off`. If you don't do this, your `nginx` proxy may
- return occasional 502 errors to clients using Zulip's events API.
-
-1. The other tricky failure mode we've seen with `nginx` reverse
- proxies is that they can load-balance between the IPv4 and IPv6
- addresses for a given hostname. This can result in mysterious errors
- that can be quite difficult to debug. Be sure to declare your
- `upstreams` equivalent in a way that won't do load-balancing
- unexpectedly (e.g. pointing to a DNS name that you haven't configured
- with multiple IPs for your Zulip machine; sometimes this happens with
- IPv6 configuration).
-
## PostgreSQL warm standby
Zulip's configuration allows for [warm standby database
@@ -673,341 +395,3 @@ If you are using password authentication, you can set a
[warm-standby]: https://www.postgresql.org/docs/current/warm-standby.html
[wal-g]: export-and-import.md#database-only-backup-tools
-
-## System and deployment configuration
-
-The file `/etc/zulip/zulip.conf` is used to configure properties of
-the system and deployment; `/etc/zulip/settings.py` is used to
-[configure the application itself](settings.md). The `zulip.conf`
-sections and settings are described below.
-
-When a setting refers to "set to true" or "set to false", the values
-`true` and `false` are canonical, but any of the following values will
-be considered "true", case-insensitively:
-
-- 1
-- y
-- t
-- yes
-- true
-- enable
-- enabled
-
-Any other value (including the empty string) is considered false.
-
-### `[machine]`
-
-#### `puppet_classes`
-
-A comma-separated list of the Puppet classes to install on the server.
-The most common is **`zulip::profile::standalone`**, used for a
-stand-alone single-host deployment.
-[Components](../overview/architecture-overview.md#components) of
-that include:
-
-- **`zulip::profile::app_frontend`**
-- **`zulip::profile::memcached`**
-- **`zulip::profile::postgresql`**
-- **`zulip::profile::redis`**
-- **`zulip::profile::rabbitmq`**
-
-If you are using a [Apache as a single-sign-on
-authenticator](authentication-methods.md#apache-based-sso-with-remote_user),
-you will need to add **`zulip::apache_sso`** to the list.
-
-#### `pgroonga`
-
-Set to true if enabling the [multi-language PGroonga search
-extension](../subsystems/full-text-search.md#multi-language-full-text-search).
-
-#### `timesync`
-
-What time synchronization daemon to use; defaults to `chrony`, but also supports
-`ntpd` and `none`. Installations should not adjust this unless they are aligning
-with a fleet-wide standard of `ntpd`. `none` is only reasonable in containers
-like LXC which do not allow adjustment of the clock; a Zulip server will not
-function correctly without an accurate clock.
-
-### `[deployment]`
-
-#### `deploy_options`
-
-Options passed by `upgrade-zulip` and `upgrade-zulip-from-git` into
-`upgrade-zulip-stage-2`. These might be any of:
-
-- **`--skip-puppet`** skips doing Puppet/apt upgrades. The user will need
- to run `zulip-puppet-apply` manually after the upgrade.
-- **`--skip-migrations`** skips running database migrations. The
- user will need to run `./manage.py migrate` manually after the upgrade.
-- **`--skip-purge-old-deployments`** skips purging old deployments;
- without it, only deployments with the last two weeks are kept.
-
-Generally installations will not want to set any of these options; the
-`--skip-*` options are primarily useful for reducing upgrade downtime
-for servers that are upgraded frequently by core Zulip developers.
-
-#### `git_repo_url`
-
-Default repository URL used when [upgrading from a Git
-repository](upgrade.md#upgrading-from-a-git-repository).
-
-### `[application_server]`
-
-#### `http_only`
-
-If set to true, [configures Zulip to allow HTTP access][using-http];
-use if Zulip is deployed behind a reverse proxy that is handling
-SSL/TLS termination.
-
-#### `nginx_listen_port`
-
-Set to the port number if you [prefer to listen on a port other than
-443](#using-an-alternate-port).
-
-#### `nginx_worker_connections`
-
-Adjust the [`worker_connections`][nginx_worker_connections] setting in
-the nginx server. This defaults to 10000; increasing it allows more
-concurrent connections per CPU core, at the cost of more memory
-consumed by NGINX. This number, times the number of CPU cores, should
-be more than twice the concurrent number of users.
-
-[nginx_worker_connections]: http://nginx.org/en/docs/ngx_core_module.html#worker_connections
-
-#### `queue_workers_multiprocess`
-
-By default, Zulip automatically detects whether the system has enough
-memory to run Zulip queue processors in the higher-throughput but more
-multiprocess mode (or to save 1.5GiB of RAM with the multithreaded
-mode). The calculation is based on whether the system has enough
-memory (currently 3.5GiB) to run a single-server Zulip installation in
-the multiprocess mode.
-
-Set explicitly to true or false to override the automatic
-calculation. This override is useful both Docker systems (where the
-above algorithm might see the host's memory, not the container's)
-and/or when using remote servers for postgres, memcached, redis, and
-RabbitMQ.
-
-#### `rolling_restart`
-
-If set to true, when using `./scripts/restart-server` to restart
-Zulip, restart the uwsgi processes one-at-a-time, instead of all at
-once. This decreases the number of 502's served to clients, at the
-cost of slightly increased memory usage, and the possibility that
-different requests will be served by different versions of the code.
-
-#### `service_file_descriptor_limit`
-
-The number of file descriptors which [Supervisor is configured to allow
-processes to use][supervisor-minfds]; defaults to 40000. If your Zulip deployment
-is very large (hundreds of thousands of concurrent users), your Django processes
-hit this limit and refuse connections to clients. Raising it above this default
-may require changing system-level limits, particularly if you are using a
-virtualized environment (e.g. Docker, or Proxmox LXC).
-
-[supervisor-minfds]: http://supervisord.org/configuration.html?highlight=minfds#supervisord-section-values
-
-#### `s3_memory_cache_size`
-
-Used only when the [S3 storage backend][s3-backend] is in use.
-Controls the in-memory size of the cache _index_; the default is 1MB,
-which is enough to store about 8 thousand entries.
-
-#### `s3_disk_cache_size`
-
-Used only when the [S3 storage backend][s3-backend] is in use.
-Controls the on-disk size of the cache _contents_; the default is
-200MB.
-
-#### `s3_cache_inactive_time`
-
-Used only when the [S3 storage backend][s3-backend] is in use.
-Controls the longest amount of time an entry will be cached since last
-use; the default is 30 days. Since the contents of the cache are
-immutable, this serves only as a potential additional limit on the
-size of the contents on disk; `s3_disk_cache_size` is expected to be
-the primary control for cache sizing.
-
-#### `nameserver`
-
-When the [S3 storage backend][s3-backend] is in use, downloads from S3 are
-proxied from nginx, whose configuration requires an explicit value of a DNS
-nameserver to resolve the S3 server's hostname. Zulip defaults to using the
-resolver found in `/etc/resolv.conf`; this setting overrides any value found
-there.
-
-[s3-backend]: upload-backends.md
-
-#### `uwsgi_listen_backlog_limit`
-
-Override the default uwsgi backlog of 128 connections.
-
-#### `uwsgi_processes`
-
-Override the default `uwsgi` (Django) process count of 6 on hosts with
-more than 3.5GiB of RAM, 4 on hosts with less.
-
-#### `access_log_retention_days`
-
-Number of days of access logs to keep, for both nginx and the application.
-Defaults to 14 days.
-
-### `[postfix]`
-
-#### `mailname`
-
-The hostname that [Postfix should be configured to receive mail
-at](email-gateway.md#local-delivery-setup), as well as identify itself as for
-outgoing email.
-
-### `[postgresql]`
-
-#### `effective_io_concurrency`
-
-Override PostgreSQL's [`effective_io_concurrency`
-setting](https://www.postgresql.org/docs/current/runtime-config-resource.html#GUC-EFFECTIVE-IO-CONCURRENCY).
-
-#### `listen_addresses`
-
-Override PostgreSQL's [`listen_addresses`
-setting](https://www.postgresql.org/docs/current/runtime-config-connection.html#GUC-LISTEN-ADDRESSES).
-
-#### `random_page_cost`
-
-Override PostgreSQL's [`random_page_cost`
-setting](https://www.postgresql.org/docs/current/runtime-config-query.html#GUC-RANDOM-PAGE-COST)
-
-#### `replication_primary`
-
-On the [warm standby replicas](#postgresql-warm-standby), set to the
-hostname of the primary PostgreSQL server that streaming replication
-should be done from.
-
-#### `replication_user`
-
-On the [warm standby replicas](#postgresql-warm-standby), set to the
-username that the host should authenticate to the primary PostgreSQL
-server as, for streaming replication. Authentication will be done
-based on the `pg_hba.conf` file; if you are using password
-authentication, you can set a `postgresql_replication_password` secret
-for authentication.
-
-#### `skip_backups`
-
-If set to as true value, inhibits the nightly [`wal-g` backups][wal-g] which
-would be taken on all non-replicated hosts and [all warm standby
-replicas](#postgresql-warm-standby). This is generally only set if you have
-multiple warm standby replicas, in order to avoid taking multiple backups, one
-per replica.
-
-#### `backups_disk_concurrency`
-
-Number of concurrent disk reads to use when taking backups. Defaults to 1; you
-may wish to increase this if you are taking backups on a replica, so can afford
-to affect other disk I/O, and have an SSD which is good at parallel random
-reads.
-
-#### `backups_storage_class`
-
-What [storage class](https://aws.amazon.com/s3/storage-classes/) to use when
-uploading database backups. Defaults to `STANDARD`, meaning "[S3
-standard][s3-standard]", but many deployments will have overall lower costs if
-"[S3 Standard - Infrequent Access][s3-ia]" is used, via the `STANDARD_IA`
-value. Also supported is "[S3 Reduced Redundancy][s3-rr]", by setting
-`REDUCED_REDUNDANCY`, but this is not suggested for production use.
-
-[s3-standard]: https://aws.amazon.com/s3/storage-classes/#General_purpose
-[s3-ia]: https://aws.amazon.com/s3/storage-classes/#Infrequent_access
-[s3-rr]: https://aws.amazon.com/s3/reduced-redundancy/
-
-#### `missing_dictionaries`
-
-If set to a true value during initial database creation, uses PostgreSQL's
-standard `pg_catalog.english` text search configuration, rather than Zulip's
-improved set of stopwords. Has no effect after initial database construction.
-
-#### `ssl_ca_file`
-
-Set to the path to the PEM-encoded certificate authority used to
-authenticate client connections.
-
-#### `ssl_cert_file`
-
-Set to the path to the PEM-encoded public certificate used to secure
-client connections.
-
-#### `ssl_key_file`
-
-Set to the path to the PEM-encoded private key used to secure client
-connections.
-
-#### `ssl_mode`
-
-The mode that should be used to verify the server certificate. The
-PostgreSQL default is `prefer`, which provides no security benefit; we
-strongly suggest setting this to `require` or better if you are using
-certificate authentication. See the [PostgreSQL
-documentation](https://www.postgresql.org/docs/current/libpq-ssl.html#LIBPQ-SSL-SSLMODE-STATEMENTS)
-for potential values.
-
-#### `version`
-
-The version of PostgreSQL that is in use. Do not set by hand; use the
-[PostgreSQL upgrade tool](upgrade.md#upgrading-postgresql).
-
-### `[memcached]`
-
-#### `memory`
-
-Override the number of megabytes of memory that memcached should be
-configured to consume; defaults to 1/8th of the total server memory.
-
-#### `max_item_size`
-
-Override the maximum size that an item in memcached can store. This defaults to
-1m; adjusting it should only be necessary if your Zulip server has organizations
-which have more than 20k users.
-
-### `[loadbalancer]`
-
-#### `ips`
-
-Comma-separated list of IP addresses or netmasks of external load balancers
-whose `X-Forwarded-For` and `X-Forwarded-Proto` should be respected. These can
-be individual IP addresses, or CIDR IP address ranges.
-
-### `[http_proxy]`
-
-#### `host`
-
-The hostname or IP address of an [outgoing HTTP `CONNECT`
-proxy](#customizing-the-outgoing-http-proxy). Defaults to `localhost`
-if unspecified.
-
-#### `port`
-
-The TCP port of the HTTP `CONNECT` proxy on the host specified above.
-Defaults to `4750` if unspecified.
-
-#### `listen_address`
-
-The IP address that Smokescreen should bind to and listen on.
-Defaults to `127.0.0.1`.
-
-#### `enable_for_camo`
-
-Because Camo includes logic to deny access to private subnets, routing
-its requests through Smokescreen is generally not necessary. Set to
-true or false to override the default, which uses the proxy only if
-it is not the default of Smokescreen on a local host.
-
-### `[sentry]`
-
-#### `organization`
-
-The Sentry organization used for the [Sentry deploy hook](#sentry-deploy-hook).
-
-#### `project`
-
-The Sentry project used for the [Sentry deploy hook](#sentry-deploy-hook).
diff --git a/docs/production/email-gateway.md b/docs/production/email-gateway.md
index 218d0a24db..7671c1eddf 100644
--- a/docs/production/email-gateway.md
+++ b/docs/production/email-gateway.md
@@ -101,7 +101,7 @@ using an [HTTP reverse proxy][reverse-proxy]).
Congratulations! The integration should be fully operational.
-[reverse-proxy]: deployment.md#putting-the-zulip-application-behind-a-reverse-proxy
+[reverse-proxy]: reverse-proxies.md
## Polling setup
diff --git a/docs/production/index.md b/docs/production/index.md
index 22bd713d91..dfe9dc9ea7 100644
--- a/docs/production/index.md
+++ b/docs/production/index.md
@@ -10,6 +10,7 @@ install
troubleshooting
management-commands
settings
+system-configuration
mobile-push-notifications
upgrade
modify
@@ -21,6 +22,7 @@ upload-backends
ssl-certificates
email
deployment
+reverse-proxies
multiple-organizations
email-gateway
video-calls
diff --git a/docs/production/requirements.md b/docs/production/requirements.md
index 5d36a36625..ff8ca5b7f1 100644
--- a/docs/production/requirements.md
+++ b/docs/production/requirements.md
@@ -119,7 +119,7 @@ access to incoming port 22 for SSH access for remote access.
[ssrf]: https://owasp.org/www-community/attacks/Server_Side_Request_Forgery
[smokescreen-proxy]: deployment.md#customizing-the-outgoing-http-proxy
-[reverse-proxy]: deployment.md#putting-the-zulip-application-behind-a-reverse-proxy
+[reverse-proxy]: reverse-proxies.md
## Credentials needed
diff --git a/docs/production/reverse-proxies.md b/docs/production/reverse-proxies.md
new file mode 100644
index 0000000000..6b7a8bea6b
--- /dev/null
+++ b/docs/production/reverse-proxies.md
@@ -0,0 +1,277 @@
+## Reverse proxies
+
+Zulip is designed to support being run behind a reverse proxy server.
+This section contains notes on the configuration required with
+variable reverse proxy implementations.
+
+### Installer options
+
+If your Zulip server will not be on the public Internet, we recommend,
+installing with the `--self-signed-cert` option (rather than the
+`--certbot` option), since Certbot requires the server to be on the
+public Internet.
+
+#### Configuring Zulip to allow HTTP
+
+Zulip requires clients to connect to Zulip servers over the secure
+HTTPS protocol; the insecure HTTP protocol is not supported. However,
+we do support using a reverse proxy that speaks HTTPS to clients and
+connects to the Zulip server over HTTP; this can be secure when the
+Zulip server is not directly exposed to the public Internet.
+
+After installing the Zulip server as [described
+above](#installer-options), you can configure Zulip to accept HTTP
+requests from a reverse proxy as follows:
+
+1. Add the following block to `/etc/zulip/zulip.conf`:
+
+ ```ini
+ [application_server]
+ http_only = true
+ ```
+
+1. As root, run
+ `/home/zulip/deployments/current/scripts/zulip-puppet-apply`. This
+ will convert Zulip's main `nginx` configuration file to allow HTTP
+ instead of HTTPS.
+
+1. Finally, restart the Zulip server, using
+ `/home/zulip/deployments/current/scripts/restart-server`.
+
+Note that Zulip must be able to accurately determine if its connection to the
+client was over HTTPS or not; if you enable `http_only`, it is very important
+that you correctly configure Zulip to trust the `X-Forwarded-Proto` header from
+its proxy (see the next section), or clients may see infinite redirects.
+
+#### Configuring Zulip to trust proxies
+
+Before placing Zulip behind a reverse proxy, it needs to be configured to trust
+the client IP addresses that the proxy reports via the `X-Forwarded-For` header,
+and the protocol reported by the `X-Forwarded-Proto` header. This is important
+to have accurate IP addresses in server logs, as well as in notification emails
+which are sent to end users. Zulip doesn't default to trusting all
+`X-Forwarded-*` headers, because doing so would allow clients to spoof any IP
+address, and claim connections were over a secure connection when they were not;
+we specify which IP addresses are the Zulip server's incoming proxies, so we
+know which `X-Forwarded-*` headers to trust.
+
+1. Determine the IP addresses of all reverse proxies you are setting up, as seen
+ from the Zulip host. Depending on your network setup, these may not be the
+ same as the public IP addresses of the reverse proxies. These can also be IP
+ address ranges, as expressed in CIDR notation.
+
+1. Add the following block to `/etc/zulip/zulip.conf`.
+
+ ```ini
+ [loadbalancer]
+ # Use the IP addresses you determined above, separated by commas.
+ ips = 192.168.0.100
+ ```
+
+1. Reconfigure Zulip with these settings. As root, run
+ `/home/zulip/deployments/current/scripts/zulip-puppet-apply`. This will
+ adjust Zulip's `nginx` configuration file to accept the `X-Forwarded-For`
+ header when it is sent from one of the reverse proxy IPs.
+
+1. Finally, restart the Zulip server, using
+ `/home/zulip/deployments/current/scripts/restart-server`.
+
+### nginx configuration
+
+Below is a working example of a full nginx configuration. It assumes
+that your Zulip server sits at `https://10.10.10.10:443`; see
+[above](#configuring-zulip-to-allow-http) to switch to HTTP.
+
+1. Follow the instructions to [configure Zulip to trust
+ proxies](#configuring-zulip-to-trust-proxies).
+
+1. Configure the root `nginx.conf` file. We recommend using
+ `/etc/nginx/nginx.conf` from your Zulip server for our recommended
+ settings. E.g. if you don't set `client_max_body_size`, it won't be
+ possible to upload large files to your Zulip server.
+
+1. Configure the `nginx` site-specific configuration (in
+ `/etc/nginx/sites-available`) for the Zulip app. The following
+ example is a good starting point:
+
+ ```nginx
+ server {
+ listen 80;
+ listen [::]:80;
+ location / {
+ return 301 https://$host$request_uri;
+ }
+ }
+
+ server {
+ listen 443 ssl http2;
+ listen [::]:443 ssl http2;
+ server_name zulip.example.com;
+
+ ssl_certificate /etc/letsencrypt/live/zulip.example.com/fullchain.pem;
+ ssl_certificate_key /etc/letsencrypt/live/zulip.example.com/privkey.pem;
+
+ location / {
+ proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+ proxy_set_header X-Forwarded-Proto $scheme;
+ proxy_set_header Host $host;
+ proxy_http_version 1.1;
+ proxy_buffering off;
+ proxy_read_timeout 20m;
+ proxy_pass https://10.10.10.10:443;
+ }
+ }
+ ```
+
+ Don't forget to update `server_name`, `ssl_certificate`,
+ `ssl_certificate_key` and `proxy_pass` with the appropriate values
+ for your deployment.
+
+[nginx-proxy-longpolling-config]: https://github.com/zulip/zulip/blob/main/puppet/zulip/files/nginx/zulip-include-common/proxy_longpolling
+[standalone.pp]: https://github.com/zulip/zulip/blob/main/puppet/zulip/manifests/profile/standalone.pp
+[zulipchat-puppet]: https://github.com/zulip/zulip/tree/main/puppet/kandra/manifests
+
+### Apache2 configuration
+
+Below is a working example of a full Apache2 configuration. It assumes
+that your Zulip server sits at `https://internal.zulip.hostname:443`.
+Note that if you wish to use SSL to connect to the Zulip server,
+Apache requires you use the hostname, not the IP address; see
+[above](#configuring-zulip-to-allow-http) to switch to HTTP.
+
+1. Follow the instructions to [configure Zulip to trust
+ proxies](#configuring-zulip-to-trust-proxies).
+
+1. Set `USE_X_FORWARDED_HOST = True` in `/etc/zulip/settings.py` and
+ restart Zulip.
+
+1. Enable some required Apache modules:
+
+ ```bash
+ a2enmod ssl proxy proxy_http headers rewrite
+ ```
+
+1. Create an Apache2 virtual host configuration file, similar to the
+ following. Place it the appropriate path for your Apache2
+ installation and enable it (E.g. if you use Debian or Ubuntu, then
+ place it in `/etc/apache2/sites-available/zulip.example.com.conf`
+ and then run
+ `a2ensite zulip.example.com && systemctl reload apache2`):
+
+ ```apache
+
+ ServerName zulip.example.com
+ RewriteEngine On
+ RewriteRule ^ https://%{HTTP_HOST}%{REQUEST_URI} [R=301,L]
+
+
+
+ ServerName zulip.example.com
+
+ RequestHeader set "X-Forwarded-Proto" expr=%{REQUEST_SCHEME}
+
+ RewriteEngine On
+ RewriteRule /(.*) https://internal.zulip.hostname:443/$1 [P,L]
+
+
+ Require all granted
+ ProxyPass https://internal.zulip.hostname:443/ timeout=1200
+
+
+ SSLEngine on
+ SSLProxyEngine on
+ SSLCertificateFile /etc/letsencrypt/live/zulip.example.com/fullchain.pem
+ SSLCertificateKeyFile /etc/letsencrypt/live/zulip.example.com/privkey.pem
+ # This file can be found in ~zulip/deployments/current/puppet/zulip/files/nginx/dhparam.pem
+ SSLOpenSSLConfCmd DHParameters "/etc/nginx/dhparam.pem"
+ SSLProtocol all -SSLv3 -TLSv1 -TLSv1.1
+ SSLCipherSuite ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
+ SSLHonorCipherOrder off
+ SSLSessionTickets off
+ Header set Strict-Transport-Security "max-age=31536000"
+
+ ```
+
+ Don't forget to update `ServerName`, `RewriteRule`, `ProxyPass`,
+ `SSLCertificateFile`, and `SSLCertificateKeyFile` as are
+ appropriate for your deployment.
+
+### HAProxy configuration
+
+Below is a working example of a HAProxy configuration. It assumes that
+your Zulip server sits at `https://10.10.10.10:443`; see
+[above](#configuring-zulip-to-allow-http) to switch to HTTP.
+
+1. Follow the instructions to [configure Zulip to trust
+ proxies](#configuring-zulip-to-trust-proxies).
+
+1. Configure HAProxy. The below is a minimal `frontend` and `backend`
+ configuration:
+
+ ```text
+ frontend zulip
+ mode http
+ bind *:80
+ bind *:443 ssl crt /etc/ssl/private/zulip-combined.crt
+ http-request redirect scheme https code 301 unless { ssl_fc }
+ http-request set-header X-Forwarded-Proto http unless { ssl_fc }
+ http-request set-header X-Forwarded-Proto https if { ssl_fc }
+ default_backend zulip
+
+ backend zulip
+ mode http
+ timeout server 20m
+ server zulip 10.10.10.10:443 check ssl ca-file /etc/ssl/certs/ca-certificates.crt
+ ```
+
+ Don't forget to update `bind *:443 ssl crt` and `server` as is
+ appropriate for your deployment.
+
+### Other proxies
+
+If you're using another reverse proxy implementation, there are few
+things you need to be careful about when configuring it:
+
+1. Configure your reverse proxy (or proxies) to correctly maintain the
+ `X-Forwarded-For` HTTP header, which is supposed to contain the series
+ of IP addresses the request was forwarded through. Additionally,
+ [configure Zulip to respect the addresses sent by your reverse
+ proxies](#configuring-zulip-to-trust-proxies). You can verify
+ your work by looking at `/var/log/zulip/server.log` and checking it
+ has the actual IP addresses of clients, not the IP address of the
+ proxy server.
+
+1. Configure your reverse proxy (or proxies) to correctly maintain the
+ `X-Forwarded-Proto` HTTP header, which is supposed to contain either `https`
+ or `http` depending on the connection between your browser and your
+ proxy. This will be used by Django to perform CSRF checks regardless of your
+ connection mechanism from your proxy to Zulip. Note that the proxies _must_
+ set the header, overriding any existing values, not add a new header.
+
+1. Configure your proxy to pass along the `Host:` header as was sent
+ from the client, not the internal hostname as seen by the proxy.
+ If this is not possible, you can set `USE_X_FORWARDED_HOST = True`
+ in `/etc/zulip/settings.py`, and pass the client's `Host` header to
+ Zulip in an `X-Forwarded-Host` header.
+
+1. Ensure your proxy doesn't interfere with Zulip's use of
+ long-polling for real-time push from the server to your users'
+ browsers. This [nginx code snippet][nginx-proxy-longpolling-config]
+ does this.
+
+ The key configuration options are, for the `/json/events` and
+ `/api/1/events` endpoints:
+
+ - `proxy_read_timeout 1200;`. It's critical that this be
+ significantly above 60s, but the precise value isn't important.
+ - `proxy_buffering off`. If you don't do this, your `nginx` proxy may
+ return occasional 502 errors to clients using Zulip's events API.
+
+1. The other tricky failure mode we've seen with `nginx` reverse
+ proxies is that they can load-balance between the IPv4 and IPv6
+ addresses for a given hostname. This can result in mysterious errors
+ that can be quite difficult to debug. Be sure to declare your
+ `upstreams` equivalent in a way that won't do load-balancing
+ unexpectedly (e.g. pointing to a DNS name that you haven't configured
+ with multiple IPs for your Zulip machine; sometimes this happens with
+ IPv6 configuration).
diff --git a/docs/production/security-model.md b/docs/production/security-model.md
index 06efd27756..f2bded5510 100644
--- a/docs/production/security-model.md
+++ b/docs/production/security-model.md
@@ -260,7 +260,7 @@ strength allowed is controlled by two settings in
[go-camo]: https://github.com/cactus/go-camo
[ssrf]: https://owasp.org/www-community/attacks/Server_Side_Request_Forgery
[smokescreen-setup]: deployment.md#customizing-the-outgoing-http-proxy
-[proxy.enable_for_camo]: deployment.md#enable_for_camo
+[proxy.enable_for_camo]: system-configuration.md#enable_for_camo
## Rate limiting
diff --git a/docs/production/settings.md b/docs/production/settings.md
index b6c8dcacb1..fafcee6c8f 100644
--- a/docs/production/settings.md
+++ b/docs/production/settings.md
@@ -33,7 +33,7 @@ to each new major release.
Since Zulip’s settings file is a Python script, there are a number of
other things that one can configure that are documented in
-[System and deployment configuration](deployment.md#system-and-deployment-configuration).
+[System and deployment configuration](system-configuration.md).
Otherwise, ask in [the Zulip development community](https://zulip.com/development-community/)
if there’s something you’d like to do but can’t figure out how to.
diff --git a/docs/production/system-configuration.md b/docs/production/system-configuration.md
new file mode 100644
index 0000000000..ad2898c51d
--- /dev/null
+++ b/docs/production/system-configuration.md
@@ -0,0 +1,337 @@
+## System configuration
+
+The file `/etc/zulip/zulip.conf` is used to configure properties of
+the system and deployment; `/etc/zulip/settings.py` is used to
+[configure the application itself](settings.md). The `zulip.conf`
+sections and settings are described below.
+
+When a setting refers to "set to true" or "set to false", the values
+`true` and `false` are canonical, but any of the following values will
+be considered "true", case-insensitively:
+
+- 1
+- y
+- t
+- yes
+- true
+- enable
+- enabled
+
+Any other value (including the empty string) is considered false.
+
+### `[machine]`
+
+#### `puppet_classes`
+
+A comma-separated list of the Puppet classes to install on the server.
+The most common is **`zulip::profile::standalone`**, used for a
+stand-alone single-host deployment.
+[Components](../overview/architecture-overview.md#components) of
+that include:
+
+- **`zulip::profile::app_frontend`**
+- **`zulip::profile::memcached`**
+- **`zulip::profile::postgresql`**
+- **`zulip::profile::redis`**
+- **`zulip::profile::rabbitmq`**
+
+If you are using a [Apache as a single-sign-on
+authenticator](authentication-methods.md#apache-based-sso-with-remote_user),
+you will need to add **`zulip::apache_sso`** to the list.
+
+#### `pgroonga`
+
+Set to true if enabling the [multi-language PGroonga search
+extension](../subsystems/full-text-search.md#multi-language-full-text-search).
+
+#### `timesync`
+
+What time synchronization daemon to use; defaults to `chrony`, but also supports
+`ntpd` and `none`. Installations should not adjust this unless they are aligning
+with a fleet-wide standard of `ntpd`. `none` is only reasonable in containers
+like LXC which do not allow adjustment of the clock; a Zulip server will not
+function correctly without an accurate clock.
+
+### `[deployment]`
+
+#### `deploy_options`
+
+Options passed by `upgrade-zulip` and `upgrade-zulip-from-git` into
+`upgrade-zulip-stage-2`. These might be any of:
+
+- **`--skip-puppet`** skips doing Puppet/apt upgrades. The user will need
+ to run `zulip-puppet-apply` manually after the upgrade.
+- **`--skip-migrations`** skips running database migrations. The
+ user will need to run `./manage.py migrate` manually after the upgrade.
+- **`--skip-purge-old-deployments`** skips purging old deployments;
+ without it, only deployments with the last two weeks are kept.
+
+Generally installations will not want to set any of these options; the
+`--skip-*` options are primarily useful for reducing upgrade downtime
+for servers that are upgraded frequently by core Zulip developers.
+
+#### `git_repo_url`
+
+Default repository URL used when [upgrading from a Git
+repository](upgrade.md#upgrading-from-a-git-repository).
+
+### `[application_server]`
+
+#### `http_only`
+
+If set to true, [configures Zulip to allow HTTP access][using-http];
+use if Zulip is deployed behind a reverse proxy that is handling
+SSL/TLS termination.
+
+#### `nginx_listen_port`
+
+Set to the port number if you [prefer to listen on a port other than
+443](deployment.md#using-an-alternate-port).
+
+#### `nginx_worker_connections`
+
+Adjust the [`worker_connections`][nginx_worker_connections] setting in
+the nginx server. This defaults to 10000; increasing it allows more
+concurrent connections per CPU core, at the cost of more memory
+consumed by NGINX. This number, times the number of CPU cores, should
+be more than twice the concurrent number of users.
+
+[nginx_worker_connections]: http://nginx.org/en/docs/ngx_core_module.html#worker_connections
+
+#### `queue_workers_multiprocess`
+
+By default, Zulip automatically detects whether the system has enough
+memory to run Zulip queue processors in the higher-throughput but more
+multiprocess mode (or to save 1.5GiB of RAM with the multithreaded
+mode). The calculation is based on whether the system has enough
+memory (currently 3.5GiB) to run a single-server Zulip installation in
+the multiprocess mode.
+
+Set explicitly to true or false to override the automatic
+calculation. This override is useful both Docker systems (where the
+above algorithm might see the host's memory, not the container's)
+and/or when using remote servers for postgres, memcached, redis, and
+RabbitMQ.
+
+#### `rolling_restart`
+
+If set to true, when using `./scripts/restart-server` to restart
+Zulip, restart the uwsgi processes one-at-a-time, instead of all at
+once. This decreases the number of 502's served to clients, at the
+cost of slightly increased memory usage, and the possibility that
+different requests will be served by different versions of the code.
+
+#### `service_file_descriptor_limit`
+
+The number of file descriptors which [Supervisor is configured to allow
+processes to use][supervisor-minfds]; defaults to 40000. If your Zulip deployment
+is very large (hundreds of thousands of concurrent users), your Django processes
+hit this limit and refuse connections to clients. Raising it above this default
+may require changing system-level limits, particularly if you are using a
+virtualized environment (e.g. Docker, or Proxmox LXC).
+
+[supervisor-minfds]: http://supervisord.org/configuration.html?highlight=minfds#supervisord-section-values
+
+#### `s3_memory_cache_size`
+
+Used only when the [S3 storage backend][s3-backend] is in use.
+Controls the in-memory size of the cache _index_; the default is 1MB,
+which is enough to store about 8 thousand entries.
+
+#### `s3_disk_cache_size`
+
+Used only when the [S3 storage backend][s3-backend] is in use.
+Controls the on-disk size of the cache _contents_; the default is
+200MB.
+
+#### `s3_cache_inactive_time`
+
+Used only when the [S3 storage backend][s3-backend] is in use.
+Controls the longest amount of time an entry will be cached since last
+use; the default is 30 days. Since the contents of the cache are
+immutable, this serves only as a potential additional limit on the
+size of the contents on disk; `s3_disk_cache_size` is expected to be
+the primary control for cache sizing.
+
+#### `nameserver`
+
+When the [S3 storage backend][s3-backend] is in use, downloads from S3 are
+proxied from nginx, whose configuration requires an explicit value of a DNS
+nameserver to resolve the S3 server's hostname. Zulip defaults to using the
+resolver found in `/etc/resolv.conf`; this setting overrides any value found
+there.
+
+[s3-backend]: upload-backends.md
+
+#### `uwsgi_listen_backlog_limit`
+
+Override the default uwsgi backlog of 128 connections.
+
+#### `uwsgi_processes`
+
+Override the default `uwsgi` (Django) process count of 6 on hosts with
+more than 3.5GiB of RAM, 4 on hosts with less.
+
+#### `access_log_retention_days`
+
+Number of days of access logs to keep, for both nginx and the application.
+Defaults to 14 days.
+
+### `[postfix]`
+
+#### `mailname`
+
+The hostname that [Postfix should be configured to receive mail
+at](email-gateway.md#local-delivery-setup), as well as identify itself as for
+outgoing email.
+
+### `[postgresql]`
+
+#### `effective_io_concurrency`
+
+Override PostgreSQL's [`effective_io_concurrency`
+setting](https://www.postgresql.org/docs/current/runtime-config-resource.html#GUC-EFFECTIVE-IO-CONCURRENCY).
+
+#### `listen_addresses`
+
+Override PostgreSQL's [`listen_addresses`
+setting](https://www.postgresql.org/docs/current/runtime-config-connection.html#GUC-LISTEN-ADDRESSES).
+
+#### `random_page_cost`
+
+Override PostgreSQL's [`random_page_cost`
+setting](https://www.postgresql.org/docs/current/runtime-config-query.html#GUC-RANDOM-PAGE-COST)
+
+#### `replication_primary`
+
+On the [warm standby replicas](deployment.md#postgresql-warm-standby), set to the
+hostname of the primary PostgreSQL server that streaming replication
+should be done from.
+
+#### `replication_user`
+
+On the [warm standby replicas](deployment.md#postgresql-warm-standby), set to the
+username that the host should authenticate to the primary PostgreSQL
+server as, for streaming replication. Authentication will be done
+based on the `pg_hba.conf` file; if you are using password
+authentication, you can set a `postgresql_replication_password` secret
+for authentication.
+
+#### `skip_backups`
+
+If set to as true value, inhibits the nightly [`wal-g` backups][wal-g] which
+would be taken on all non-replicated hosts and [all warm standby
+replicas](deployment.md#postgresql-warm-standby). This is generally only set if you have
+multiple warm standby replicas, in order to avoid taking multiple backups, one
+per replica.
+
+#### `backups_disk_concurrency`
+
+Number of concurrent disk reads to use when taking backups. Defaults to 1; you
+may wish to increase this if you are taking backups on a replica, so can afford
+to affect other disk I/O, and have an SSD which is good at parallel random
+reads.
+
+#### `backups_storage_class`
+
+What [storage class](https://aws.amazon.com/s3/storage-classes/) to use when
+uploading database backups. Defaults to `STANDARD`, meaning "[S3
+standard][s3-standard]", but many deployments will have overall lower costs if
+"[S3 Standard - Infrequent Access][s3-ia]" is used, via the `STANDARD_IA`
+value. Also supported is "[S3 Reduced Redundancy][s3-rr]", by setting
+`REDUCED_REDUNDANCY`, but this is not suggested for production use.
+
+[s3-standard]: https://aws.amazon.com/s3/storage-classes/#General_purpose
+[s3-ia]: https://aws.amazon.com/s3/storage-classes/#Infrequent_access
+[s3-rr]: https://aws.amazon.com/s3/reduced-redundancy/
+
+#### `missing_dictionaries`
+
+If set to a true value during initial database creation, uses PostgreSQL's
+standard `pg_catalog.english` text search configuration, rather than Zulip's
+improved set of stopwords. Has no effect after initial database construction.
+
+#### `ssl_ca_file`
+
+Set to the path to the PEM-encoded certificate authority used to
+authenticate client connections.
+
+#### `ssl_cert_file`
+
+Set to the path to the PEM-encoded public certificate used to secure
+client connections.
+
+#### `ssl_key_file`
+
+Set to the path to the PEM-encoded private key used to secure client
+connections.
+
+#### `ssl_mode`
+
+The mode that should be used to verify the server certificate. The
+PostgreSQL default is `prefer`, which provides no security benefit; we
+strongly suggest setting this to `require` or better if you are using
+certificate authentication. See the [PostgreSQL
+documentation](https://www.postgresql.org/docs/current/libpq-ssl.html#LIBPQ-SSL-SSLMODE-STATEMENTS)
+for potential values.
+
+#### `version`
+
+The version of PostgreSQL that is in use. Do not set by hand; use the
+[PostgreSQL upgrade tool](upgrade.md#upgrading-postgresql).
+
+### `[memcached]`
+
+#### `memory`
+
+Override the number of megabytes of memory that memcached should be
+configured to consume; defaults to 1/8th of the total server memory.
+
+#### `max_item_size`
+
+Override the maximum size that an item in memcached can store. This defaults to
+1m; adjusting it should only be necessary if your Zulip server has organizations
+which have more than 20k users.
+
+### `[loadbalancer]`
+
+#### `ips`
+
+Comma-separated list of IP addresses or netmasks of external load balancers
+whose `X-Forwarded-For` and `X-Forwarded-Proto` should be respected. These can
+be individual IP addresses, or CIDR IP address ranges.
+
+### `[http_proxy]`
+
+#### `host`
+
+The hostname or IP address of an [outgoing HTTP `CONNECT`
+proxy](deployment.md#customizing-the-outgoing-http-proxy). Defaults to
+`localhost` if unspecified.
+
+#### `port`
+
+The TCP port of the HTTP `CONNECT` proxy on the host specified above.
+Defaults to `4750` if unspecified.
+
+#### `listen_address`
+
+The IP address that Smokescreen should bind to and listen on.
+Defaults to `127.0.0.1`.
+
+#### `enable_for_camo`
+
+Because Camo includes logic to deny access to private subnets, routing
+its requests through Smokescreen is generally not necessary. Set to
+true or false to override the default, which uses the proxy only if
+it is not the default of Smokescreen on a local host.
+
+### `[sentry]`
+
+#### `organization`
+
+The Sentry organization used for the [Sentry deploy hook](deployment.md#sentry-deploy-hook).
+
+#### `project`
+
+The Sentry project used for the [Sentry deploy hook](deployment.md#sentry-deploy-hook).
diff --git a/docs/production/upload-backends.md b/docs/production/upload-backends.md
index b3609b7484..ae85fb4ae1 100644
--- a/docs/production/upload-backends.md
+++ b/docs/production/upload-backends.md
@@ -95,7 +95,7 @@ servers; Zulip defaults this value to the first nameserver found in
will need to run `/home/zulip/deployments/current/scripts/zulip-puppet-apply` to
update the nginx configuration for the new value.
-[s3-resolver]: deployment.md#nameserver
+[s3-resolver]: system-configuration.md#nameserver
## S3 bucket policy