Commit Graph

697 Commits

Author SHA1 Message Date
Lauryn Menard 3d997c4af2 docs: Add multiple-organizations.md to production section of TOC.
Adds the existing article about hosting multiple Zulip organizations
to the production section of the table of contents.
2023-08-11 11:11:44 -07:00
n0099 2b1d250154 nginx: Switch from `$host` to `$http_host` for HTTP/3 compatibility.
nginx sets the value of the `$http_host` variable to the empty string
when using http/3, as there is technically no `Host:` header sent:
https://github.com/nginx-quic/nginx-quic/issues/3

Users with a browser that support http/3 will send their first request
to nginx with http/2, and get an expected HTTP 200 -- but any
subsequent requests will fail with am HTTP 400, since the browser will
have upgraded to http/3, which has an empty `Host` header, which Zulip
rejects.

Switch to the `$host` variable, which works for all HTTP versions.

Co-authored-by: Alex Vandiver <alexmv@zulip.com>
2023-07-25 18:20:34 -04:00
Anders Kaseorg b285813beb error_notify: Remove custom email error reporting handler.
Restore the default django.utils.log.AdminEmailHandler when
ERROR_REPORTING is enabled.  Those with more sophisticated needs can
turn it off and use Sentry or a Sentry-compatible system.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-07-20 11:00:09 -07:00
Alex Vandiver d957559371 uploads: Allow uploads to set storage class.
Uploads are well-positioned to use S3's "intelligent tiering" storage
class.  Add a setting to let uploaded files to declare their desired
storage class at upload time, and document how to move existing files
to the same storage class.
2023-07-19 16:19:34 -07:00
Alex Vandiver 5a26237b54 wal-g: Support alternate S3 storage classes. 2023-07-19 10:55:18 -07:00
Alex Vandiver a7b9e67c06 docs: Mention the PostgreSQL upgrade in the release upgrade section.
Users can, quite understandably, assume that upgrading Zulip upgraded
the underlying PostgreSQL version.  Though it is mentioned at the top
of the page, mentioning it here clarifies that it is an additional
step.
2023-07-11 17:27:57 -07:00
Alex Vandiver e18d0e3331 docs: Reference PostgreSQL version in "Restoring backups" section.
This extends 8ede54fb1b, in the specific backups section, rather
than just in the overview.
2023-07-05 14:53:24 -07:00
Mateusz Mandera 8c530f30e3 docs: Update SCIM doc to not mention add_scim_client.
add_scim_client was removed in 55342efd33.
2023-07-05 14:36:22 -07:00
David Rosa 513eb510aa docs: Rename "private message" -> "direct message".
- Updates instances of "private message", "PM", and "private_message",
  excluding historical references in `overview/changelog.md`.
- Also excludes `/docs/translating` since we would need new
  translations for "direct messages" and "DMs".
2023-06-23 14:36:16 -07:00
Alex Vandiver d46279c41e docs: Clarify that trust of X-Fowarded-Proto is also necessary.
Previously, `X-Forwarded-Proto` did not need to be set, and failure to
set `loadbalancer.ips` would merely result in bad IP-address
rate-limiting and incorrect access logs; after 0935d388f0, however,
failure to do either of those, if Zulip is deployed with `http_only`,
will lead to infinite redirect loops after login.  These are
accompanied by a misleading error, from Tornado, of:

    Forbidden (Origin checking failed - https://zulip.example.com does not match any trusted origins.): /json/events

This is most common with Docker deployments, where deployments use
another docker container, such as nginx or Traefik, to do SSL
termination.  See zulip/docker-zulip#403.

Update the documentation to reinforce that `loadbalancer.ips` also
controls trust of `X-Forwarded-Proto`, and that failure to set it will
cause the application to not function correctly.
2023-06-23 10:47:14 -07:00
Alex Vandiver f4b20337a7 docs: Document supported versions of PostgreSQL.
Fixes: #25853.
2023-06-12 16:37:55 -07:00
Alex Vandiver bd217ad31b puppet: Read resolver from /etc/resolv.conf.
04cf68b45e make nginx responsible for downloading (and caching)
files from S3.  As noted in that commit, nginx implements its own
non-blocking DNS resolver, since the base syscall is blocking, so
requires an explicit nameserver configuration.  That commit used
127.0.0.53, which is provided by systemd-resolved, as the resolver.

However, that service may not always be enabled and running, and may
in fact not even be installed (e.g. on Docker).  Switch to parsing
`/etc/resolv.conf` and using the first-provided nameserver.  In many
deployments, this will still be `127.0.0.53`, but for others it will
provide a working DNS server which is external to the host.

In the event that a server is misconfigured and has no resolvers in
`/etc/resolv.conf`, it will error out:
```console
Error: Evaluation Error: Error while evaluating a Function Call, No nameservers found in /etc/resolv.conf!  Configure one by setting application_server.nameserver in /etc/zulip/zulip.conf (file: /home/zulip/deployments/current/puppet/zulip/manifests/app_frontend_base.pp, line: 76, column: 70) on node example.zulipdev.org
```
2023-06-12 20:18:28 +00:00
Tim Abbott 6ca5130cd8 docs: Improve troubleshooting overview intro. 2023-06-12 13:01:50 -07:00
Alya Abbott 582e88544c docs: Clarify instructions for getting help with self-hosting. 2023-06-12 13:01:50 -07:00
Alex Vandiver bad600e731 docs: Clarify access to port 25 is needed for local email delivery. 2023-06-07 08:56:01 -07:00
Alex Vandiver 9a6529f86a docs: Add language to code blocks. 2023-05-31 08:18:03 -07:00
Alex Vandiver adb30c4d18 docs: Remove unused link references. 2023-05-31 08:18:03 -07:00
Alex Vandiver 8212dccc91 docs: Fix missing and typo'd link references. 2023-05-31 08:18:03 -07:00
Alex Vandiver 679591ccce docs: Document postgresql.missing_dictionaries. 2023-05-31 08:18:03 -07:00
Alex Vandiver d058188fa8 docs: Update documentation for deploy hooks.
ecfb12404a updated how values were passed to hooks, but did not
update the documentation.
2023-05-30 14:52:01 -07:00
Alex Vandiver 9ca4574fae docs: Document zulip_notify deploy hook. 2023-05-30 14:52:01 -07:00
Alex Vandiver f45a6a6d99 docs: Add missing link in Sentry deploy docs. 2023-05-30 11:25:43 -07:00
Mateusz Mandera 8fb0fe96c6 saml: Save SessionIndex in session and use when making a LogoutRequest.
This is a useful improvement in general for making correct
LogoutRequests to Idps and a necessary one to make SP-initiated logout
fully work properly in the desktop application. During desktop auth
flow, the user goes through the browser, where they log in through their
IdP. This gives them a logged in  browser session at the IdP. However,
SAML SP-initiated logout is fully conducted within the desktop
application. This means that proper information needs to be given to the
the IdP in the LogoutRequest to let it associate the LogoutRequest with
that logged in session that was established in the browser. SessionIndex
is exactly the tool for that in the SAML spec.
2023-05-23 13:01:15 -07:00
Mateusz Mandera 5dd4dcdebb saml: Make SP-initiated SLO work in the desktop application. 2023-05-23 13:01:15 -07:00
Mateusz Mandera 3f55c10685 saml: Rework SP-initiated logout config to support IdP-level config.
This gives more flexibility on a server with multiple organizations and
SAML IdPs. Such a server can have some organizations handled by IdPs
with SLO set up, and some without it set up. In such a scenario, having
a generic True/False server-wide setting is insufficient and instead
being able to specify the IdPs/orgs for SLO is needed.
2023-05-23 13:01:15 -07:00
Mateusz Mandera 0bb0220ebb saml: Implement SP-initiated Logout.
Closes #20084

This is the flow that this implements:
1. A logged-in user clicks "Logout".
2. If they didn't auth via SAML, just do normal logout. Otherwise:
3. Form a LogoutRequest and redirect the user to
https://idp.example.com/slo-endpoint?SAMLRequest=<LogoutRequest here>
4. The IdP validates the LogoutRequest, terminates its own user session
and redirects the user to
https://thezuliporg.example.com/complete/saml/?SAMLRequest=<LogoutResponse>
with the appropriate LogoutResponse. In case of failure, the
LogoutResponse is expected to express that.
5. Zulip validates the LogoutResponse and if the response is a success
response, it executes the regular Zulip logout and the full flow is
finished.
2023-05-23 13:01:15 -07:00
Toyam Cox 650cdc474d docs: Also set X-Forwarded-Proto in proxies.
Django 4.0 and higher began checking the `Origin` header, which made
it important that Zulip know accurately if the request came over HTTPS
or HTTP; failure to do so would result in "CSRF verification failed"
errors.

For Zulip servers which are accessed via proxies, this means that
`X-Fowarded-Proto` must be set accurately.  Adjust the documentation
for the suggested configurations to add the header.

Fixes: #24599.

Co-authored-by: Alex Vandiver <alexmv@zulip.com>
2023-05-18 17:17:35 -04:00
Alex Vandiver a95b796a91 supervisor: Drop minfds back down from 1000000 to 40000.
1c76036c61 raised the number of `minfds` in Supervisor from 40k to
1M.  If Supervisor cannot guarantee that number of available file
descriptors, it will fail to start; `/etc/security/limits.conf` was
hence adjusted upwards as well.  However, on some virtualized
environments, including Proxmox LXC, setting
`/etc/security/limits.conf` may not be enough to raise the
system-level limits.  This causes `supervisord` with the larger
`minfds` to fail to start.

The limit of 1000000 was chosen to be arbitrarily high, assuming it
came without cost; it is not expected to ever be reached on any
deployment.  262b19346e already lowered one aspect of that
changeset, upon determining it did come with a cost.  Potentially
breaking virtualized deployments during upgrade is another cost of
that change.

Lower the `minfds` it back down to 40k, partially reverting
1c76036c61, but allow adjusting it upwards for extremely large
deployments.  We do not expect any except the largest deployments to
ever hit the 40k limit, and a frictionless deployment for the
vanishingly small number of huge deployments is not worth the
potential upgrade hiccups for the much more frequent smaller
deployments.
2023-05-18 13:04:33 -07:00
Anders Kaseorg 12310189ed install: Support Debian 12.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-05-18 11:52:22 -07:00
Alex Vandiver 6cb570f3f0 docs: Upgrade to zulip-cloud-current before exporting to Zulip Cloud.
Partial fix for #25482.
2023-05-15 14:44:11 -07:00
Alex Vandiver e0c05825ed docs: Importing from Zulip Cloud exports should use zulip-cloud-current.
Partial fix for #25482.
2023-05-15 14:44:11 -07:00
Alex Vandiver f4683de742 puppet: Switch the `rolling_restart` setting to use the bool values.
2c5fc1827c standardized which values are "true"; use them.
2023-05-11 15:54:15 -07:00
Alex Vandiver 857f79161f docs: Update documentation on compliance exports. 2023-05-11 12:01:54 -07:00
Alex Vandiver f11350f789 puppet: Add PostgreSQL 15 support.
Instead of copying over a mostly-unchanged `postgresql.conf`, we
transition to deploying a `conf.d/zulip.conf` which contains the
only material changes we made to the file, which were previously
appended to the end.

While shipping separate while `postgresql.conf` files for each
supported version is useful if there is large variety in supported
options between versions, there is not no such variation at current,
and the burden of overriding the entire default configuration is that
it must be keep up to date wit the package's version.
2023-05-10 14:06:02 -07:00
Alex Vandiver e5ae55637e install: Remove PostgreSQL 11 support.
Django 4.2 removes this support, so Zulip has not installed with
PostgreSQL 11 since 2c20028aa4.
2023-05-05 13:35:32 -07:00
Alex Vandiver 8a3236638a docs: Update sharding docs for single-org sharding option.
Co-authored-by: Tim Abbott <tabbott@zulip.com>
2023-05-01 11:28:08 -07:00
Alex Vandiver 510b96046a docs: Update production docs for local S3 caching. 2023-05-01 11:28:08 -07:00
Alex Vandiver b8a6de95d2 pg_backup_and_purge: Allow adjusting the backup concurrency.
SSDs are good at parallel random reads.
2023-04-26 10:54:51 -07:00
Alex Vandiver 19a11c9556 pg_backup_and_purge: Take backups on replicas, if present.
Taking backups on the database primary adds additional disk load,
which can impact the performance of the application.

Switch to taking backups on replicas, if they exist.  Some deployments
may have multiple replicas, and taking backups on all of them is
wasteful and potentially confusing; add a flag to inhibit taking
nightly snapshots on the host.

If the deployment is a single instance of PostgreSQL, with no
replicas, it takes backups as before, modulo the extra flag to allow
skipping taking them.
2023-04-26 10:54:51 -07:00
Alex Vandiver 7c023042cf puppet: Rotate access log files every day, not at 500M.
Since logrotate runs in a daily cron, this practically means "daily,
but only if it's larger than 500M."  For large installs with large
traffic, this is effectively daily for 10 days; for small installs, it
is an unknown amount of time.

Switch to daily logfiles, defaulting to 14 days to match nginx; this
can be overridden using a zulip.conf setting.  This makes it easier to
ensure that access logs are only kept for a bounded period of time.
2023-04-06 14:31:16 -04:00
Alex Vandiver a77c89f610 docs: Always suggest start-server, now that it is safer. 2023-04-04 10:58:56 -07:00
Alex Vandiver 5b9fb582e2 docs: Remove now-unnecessary reactivate_realm step after import.
113a8c4782 made this step unnecessary.
2023-04-04 10:58:56 -07:00
Mateusz Mandera 7ca08cb84b docs: Link to SCIM docs from SAML instructions. 2023-04-03 17:06:05 -07:00
Mateusz Mandera 1bfe48bce6 docs: Add ReadTheDocs documentation for SCIM. 2023-04-03 17:01:05 -07:00
Alya Abbott e136636715 docs: Clarify "Should I follow this installation guide?" instructions. 2023-03-30 09:08:48 -07:00
Alex Vandiver c686c5ed0f web: Save a needless 301 redirect from /plans to /plans/. 2023-03-24 14:51:01 -07:00
Alex Vandiver 262b19346e puppet: Decrease default nginx worker_connections.
Increasing worker_connections has a memory cost, unlike the rest of
the changes in 1c76036c61d8; setting it to 1 million caused nginx to
consume several GB of memory.

Reduce the default down to 10k, and allow deploys to configure it up
if necessary.  `worker_rlimit_nofile` is left at 1M, since it has no
impact on memory consumption.
2023-03-23 15:59:23 -07:00
Alex Vandiver 015a10637b docs: Document how to use SMTP without authentication.
This is the behaviour inherited from Django[^1].  While setting the
password to empty (`email_password = `) in
`/etc/zulip/zulip-secrets.conf` also would suffice, it's unclear what
the user would have been putting into `EMAIL_HOST_USER` in that
context.

Because we previously did not warn when `email_password` was not
present in `zulip-secrets.conf`, having the error message clarify the
correct configuration for disabling SMTP auth is important.

Fixes: #23938.

[^1]: https://docs.djangoproject.com/en/4.1/ref/settings/#std-setting-EMAIL_HOST_USER
2023-02-27 11:59:48 -08:00
Alex Vandiver 6969a6a92d docs: Update instructions for realm deletion to use management command.
This documentation was written in 9ece4c9f51, which predated the
`./manage delete_realm` command added in bff503feb4.
2023-02-26 17:11:07 -08:00
Alex Vandiver 8ede54fb1b docs: Add a link to installer flag docs from backup instructions.
The documentation for restoring backups referenced that it needed to
be to the same version of PostgreSQL, but did not explain how to do
that.

Link to the relevant section of the installer documentation, and name
the flag explicitly.

Fixes: #23691
2023-02-24 12:25:48 -05:00
David Rosa 3254023fa3 help: Update URLs to match "Restrict message editing and deletion" title.
Updates all references to the new URL and adds a URL redirect.

Follow up to #24329.
2023-02-10 15:56:16 -08:00
Alex Vandiver 3109d40b21 puppet: Add a sentry release class.
This installs the Sentry CLI, and uses it to send API events to Sentry
when a release is started and completed.
2023-02-10 15:53:10 -08:00
Alex Vandiver 840884ec89 upgrade-zulip: Provide directories to run hooks before/after upgrade.
These hooks are run immediately around the critical section of the
upgrade.  If the upgrade fails for preparatory reasons, the pre-deploy
hook may not be run; if it fails during the upgrade, the post-deploy
hook will not be run.  Hooks are called from the CWD of the new
deploy, with arguments of the old version and the new version.  If
they exit with non-0 exit code, the deploy aborts.
2023-02-10 15:53:10 -08:00
Alex Vandiver 7ab4fdf250 memcached: Allow overriding the max-item-size.
This is necessary for organizations with extremely large numbers of
members (20k+).
2023-02-09 12:04:29 -08:00
Mateusz Mandera d23b0a1f08 docs: Document how LDAP email address changes work (manually).
We will hopefully be able to just this in #16208 to document what
users need to configure in order to do this manually, but the content
here will be useful for anyone who hasn't set that up regardless.
2023-02-06 15:57:44 -08:00
Alessandro Toppi ff89590558 auth: Add JWT-based user API key fetch.
This adds a new endpoint /jwt/fetch_api_key that accepts a JWT and can
be used to fetch API keys for a certain user. The target realm is
inferred from the request and the user email is part of the JWT.

A JSON containing an user API key, delivery email and (optionally)
raw user profile data is returned in response.
The profile data in the response is optional and can be retrieved by
setting the POST param "include_profile" to "true" (default=false).

Co-authored-by: Mateusz Mandera <mateusz.mandera@zulip.com>
2023-02-03 15:23:35 -08:00
Alex Vandiver 68f4071873 puppet: Allow choice of timesync tool. 2023-01-31 14:20:41 -08:00
David Rosa a6abf959bb contributor docs: Improve the first sentence in "Upgrade Zulip". 2023-01-27 12:41:56 -08:00
David Rosa 538801c651 contributor docs: Rename "Customize Zulip" -> "Server configuration".
- Renames "Customize Zulip" to "Server configuration".
- Cross-links "Server configuration" with "System and deployment
configuration".

Fixes part of #23984.
2023-01-27 12:41:56 -08:00
David Rosa 08e9686cd2 contributor docs: Rename "Upgrade or modify Zulip" -> "Upgrade Zulip".
Fixes part of #23984.
2023-01-27 12:41:56 -08:00
David Rosa af39a1a554 contributor docs: Migrate "Modify Zulip" to its own page.
Splits /production/upgrade-or-modify.md to improve the organization
of production documentation.

Fixes #23984.
2023-01-27 12:41:56 -08:00
Tran Sang 3bea65b39c puppet: Set /etc/mailname based on postfix.mailname configuration.
The `postfix.mailname` setting in `/etc/zulip.conf` was previously
only used for incoming mail, to identify in Postfix configuration
which messages were "local."

Also set `/etc/mailname`, which is used by Postfix to set how it
identifies to other hosts when sending outgoing email.

Co-authored-by: Alex Vandiver <alexmv@zulip.com>
2023-01-27 15:08:22 -05:00
Alex Vandiver e19d4e5e0a docs: Mention probably needing to allow port 22 for SSH access. 2023-01-19 17:31:13 -08:00
Alex Vandiver 04cf68b45e uploads: Serve S3 uploads directly from nginx.
When file uploads are stored in S3, this means that Zulip serves as a
302 to S3.  Because browsers do not cache redirects, this means that
no image contents can be cached -- and upon every page load or reload,
every recently-posted image must be re-fetched.  This incurs extra
load on the Zulip server, as well as potentially excessive bandwidth
usage from S3, and on the client's connection.

Switch to fetching the content from S3 in nginx, and serving the
content from nginx.  These have `Cache-control: private, immutable`
headers set on the response, allowing browsers to cache them locally.

Because nginx fetching from S3 can be slow, and requests for uploads
will generally be bunched around when a message containing them are
first posted, we instruct nginx to cache the contents locally.  This
is safe because uploaded file contents are immutable; access control
is still mediated by Django.  The nginx cache key is the URL without
query parameters, as those parameters include a time-limited signed
authentication parameter which lets nginx fetch the non-public file.

This adds a number of nginx-level configuration parameters to control
the caching which nginx performs, including the amount of in-memory
index for he cache, the maximum storage of the cache on disk, and how
long data is retained in the cache.  The currently-chosen figures are
reasonable for small to medium deployments.

The most notable effect of this change is in allowing browsers to
cache uploaded image content; however, while there will be many fewer
requests, it also has an improvement on request latency.  The
following tests were done with a non-AWS client in SFO, a server and
S3 storage in us-east-1, and with 100 requests after 10 requests of
warm-up (to fill the nginx cache).  The mean and standard deviation
are shown.

|                   | Redirect to S3      | Caching proxy, hot  | Caching proxy, cold |
| ----------------- | ------------------- | ------------------- | ------------------- |
| Time in Django    | 263.0 ms ±  28.3 ms | 258.0 ms ±  12.3 ms | 258.0 ms ±  12.3 ms |
| Small file (842b) | 586.1 ms ±  21.1 ms | 266.1 ms ±  67.4 ms | 288.6 ms ±  17.7 ms |
| Large file (660k) | 959.6 ms ± 137.9 ms | 609.5 ms ±  13.0 ms | 648.1 ms ±  43.2 ms |

The hot-cache performance is faster for both large and small files,
since it saves the client the time having to make a second request to
a separate host.  This performance improvement remains at least 100ms
even if the client is on the same coast as the server.

Cold nginx caches are only slightly slower than hot caches, because
VPC access to S3 endpoints is extremely fast (assuming it is in the
same region as the host), and nginx can pool connections to S3 and
reuse them.

However, all of the 648ms taken to serve a cold-cache large file is
occupied in nginx, as opposed to the only 263ms which was spent in
nginx when using redirects to S3.  This means that to overall spend
less time responding to uploaded-file requests in nginx, clients will
need to find files in their local cache, and skip making an
uploaded-file request, at least 60% of the time.  Modeling shows a
reduction in the number of client requests by about 70% - 80%.

The `Content-Disposition` header logic can now also be entirely shared
with the local-file codepath, as can the `url_only` path used by
mobile clients.  While we could provide the direct-to-S3 temporary
signed URL to mobile clients, we choose to provide the
served-from-Zulip signed URL, to better control caching headers on it,
and greater consistency.  In doing so, we adjust the salt used for the
URL; since these URLs are only valid for 60s, the effect of this salt
change is minimal.
2023-01-09 18:23:58 -05:00
Alex Vandiver ed6d62a9e7 avatars: Serve /user_avatars/ through Django, which offloads to nginx.
Moving `/user_avatars/` to being served partially through Django
removes the need for the `no_serve_uploads` nginx reconfiguring when
switching between S3 and local backends.  This is important because a
subsequent commit will move S3 attachments to being served through
nginx, which would make `no_serve_uploads` entirely nonsensical of a
name.

Serve the files through Django, with an offload for the actual image
response to an internal nginx route.  In development, serve the files
directly in Django.

We do _not_ mark the contents as immutable for caching purposes, since
the path for avatar images is hashed only by their user-id and a salt,
and as such are reused when a user's avatar is updated.
2023-01-09 18:23:58 -05:00
Alya Abbott 358a0dda5b docs: Clarify --email installation option. 2023-01-05 16:10:34 -08:00
Mateusz Mandera b6067b63b8 docs: Mention the reset_authentication_attempt_count command.
The authenticate_by_username limit of 5 attempts per 30 minutes can get
annoying in some cases where the user really forgot their password and
should be allowed to keep trying with admin approvial - so we should
document the command that allows unblocking them.
2022-12-15 12:56:51 -08:00
David Rosa 0375dbc5e9 docs: Fix link to "Custom profile fields".
Follow-up to PR #23264.
2022-11-28 12:16:32 -08:00
Alex Vandiver b8ab02022c docs: Renumber unique IDs on policies for easier uniqueness. 2022-11-18 10:24:42 -08:00
Alex Vandiver 183653df9f docs: Include full policy for avatars and uploads buckets.
The documentation included the full policy for the file uploads
bucket, but only one additional statement for the avatars bucket; the
reader needed to assemble the full policy themselves.

Switch to explicitly providing the full policy for both.

Fixes #23110.
2022-11-18 10:24:42 -08:00
Alex Vandiver 68173d2212 docs: Remove a now-unused link reference.
031260573f removed the use of this.
2022-11-18 10:24:42 -08:00
Alex Vandiver bf00e44bde docs: Document the `export_search` compliance export tool. 2022-11-17 11:19:59 -08:00
Alex Vandiver 809246e1dc docs: Fix a typo in a link anchor. 2022-11-17 11:19:59 -08:00
Alex Vandiver ab71e97f1c docs: Remove an unused link anchor. 2022-11-17 11:19:59 -08:00
Tim Abbott 72b5af2260 docs: Clarify the http_only parameter.
Previously, the http_only parameter could have been misread as
allowing clients to connect to a Zulip server over HTTP directly.

Fixes #23506.
2022-11-09 17:05:15 -08:00
Mateusz Mandera 34a0139c2b rate_limit: Add interface to override rate limiting rules. 2022-11-08 08:56:42 -08:00
Mateusz Mandera d02b1f0ae6 docs: Move OPEN_REALM_CREATION doc section to multiple-organizations.md.
Since this is being moved to admin-facing documentation, also adds a
paragraph about the main concern with enabling this on a server that's
not zulip.com.
2022-10-31 17:56:45 -07:00
Anders Kaseorg 11a86ec328 install: Remove PostgreSQL 10 support.
PostgreSQL 10 reaches its upstream end of life in November, and is not
supported by Django 4.1.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-10-06 15:59:07 -07:00
Matt Keller 8d4931837d help: Document upgrading Ubuntu 20.04 to 22.04.
Fixes #22284
2022-09-27 15:51:57 -07:00
David Rosa 5cac44be06 docs: Reorganize sections and pages about contributing to Zulip.
We should rearrange Zulip's developer docs to make it easier to
find the documentation that new contributors need.

Name changes
Rename "Code contribution guide" section -> "Contributing to Zulip".
Rename "Contributing to Zulip" page -> "Contributing guide".

Organizational changes to the newly-named "Contributing to Zulip":
Move up "Contributing to Zulip", as the third link in sidebar index.
Move up renamed "Contributing guide" page to the top of this section.
Move up "Zulip code of Conduct", as the second link of this section.
Move down "Licensing", as the last link of this section.
Move "Accessibility" just below "HTML and CSS" in Subsystems section.

Update all links according to the changes above.
Redirects should be added as needed.

Fixes: #22517.
2022-08-25 11:24:57 -07:00
Tim Abbott b1435d1afd docs: Document changing subdomains on multiple organizations page. 2022-08-17 11:36:15 -07:00
Tim Abbott faf02b5c5e docs: Remove documentation for ROOT_DOMAIN_LANDING_PAGE.
This is not a feature intended to be used outside zulip.com, since it
just sets your server to have the zulip.com landing pages. I think
it's only been turned on by people who were confused by this text.
2022-08-17 11:36:15 -07:00
Tim Abbott 94e798fe55 docs: Clean up documentation for multiple hostnames.
The previous documentation did not spell out what to enter in the
realm creation form.
2022-08-17 11:36:15 -07:00
Alex Vandiver 526a04b4e6 restore-backup: Provide flags to leave settings.py and zulip.conf as-is. 2022-07-20 12:35:51 -07:00
Alex Vandiver 68c4b708a0 docs: Specify which CPU architectures are supported.
Fixes: #22310.
2022-06-29 17:28:05 -07:00
Alex Vandiver 4fd51cb5ad uwsgi: Increase request buffer size to 64k, from 8k default.
The default value in uwsgi is 4k; receiving more than this amount from
nginx leads to a 502 response (though, happily, the backend uwsgi does not
terminate).

ab18dbfde5 originally increased it from the unstated uwsgi default
of 4096, to 8192; b1da797955 made it configurable, in order to allow
requests from clients with many cookies, without causing 502's[1].

nginx defaults to a limitation of 1k, with 4 additional 8k header
lines allowed[2]; any request larger than that returns a response of
`400 Request Header Or Cookie Too Large`.  The largest header size
theoretically possible from nginx, by default, is thus 33k, though
that would require packing four separate headers to exactly 8k each.

Remove the gap between nginx's limit and uwsgi's, which could trigger
502s, by removing the uwsgi configurability, and setting a 64k size in
uwsgi (the max allowable), which is larger than nginx's default limit.

uWSGI's documentation of `buffer-size` ([3], [4]) also notes that "It
is a security measure too, so adapt to your app needs instead of
maxing it out."  Python has no security issues with buffers of 64k,
and there is no appreciable memory footprint difference to having a
larger buffer available in uwsgi.

[1]: https://chat.zulip.org/#narrow/stream/31-production-help/topic/works.20in.20Edge.20not.20Chrome/near/719523
[2]: https://nginx.org/en/docs/http/ngx_http_core_module.html#client_header_buffer_size
[3]: https://uwsgi-docs.readthedocs.io/en/latest/ThingsToKnow.html
[4]: https://uwsgi-docs.readthedocs.io/en/latest/Options.html#buffer-size
2022-06-28 16:14:24 -07:00
Alex Vandiver f9b7b8e5d9 docs: Document that loadbalancer.ips can be CIDR ranges. 2022-06-27 17:41:38 -07:00
Alex Vandiver 18230fcd99 docs: Correct and clarify wal-g backup documentation.
Backups are written every 16k of WAL archive, and by default do not
have an upper limit on how out of date they are, as `archive_timeout`
defaults to 0.

Also emphasize that these are streaming backups, not just one
point-in-time backup daily.

Fixes #21976.
2022-06-01 16:11:32 -07:00
Alex Vandiver 6bd7aac152 docs: Document the ./scripts/log-search tool. 2022-05-10 14:31:54 -07:00
Alex Vandiver 04d4ae9862 docs: Clarify nginx extension points. 2022-05-10 14:31:01 -07:00
Alex Vandiver 62642b899c docs: Update proxy docs.
Notable changes:
 - Describe `X-Forwarded-For` by name.
 - Switch each specific proxy to numbered steps.
 - Link back to the `X-Forwarded-For` section in each proxy
 - Default to using HTTPS, not HTTP, for the backend.
 - Include the HTTP-to-HTTPS redirect code for all proxies; it is
   important that it happen at the proxy, as the backend is unaware of
   it.
 - Call out Apache2 modules which are necessary.
 - Specify where the dhparam.pem file can be found.
 - Call out the `Host:` header forwarding necessary, and document
   `USE_X_FORWARDED_HOST` if that is not possible.
 - Standardize on 20 minutes of connection timeout.
2022-05-04 14:41:18 -07:00
Anders Kaseorg e952641013 install: Resupport Ubuntu 22.04.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-05-03 09:41:08 -07:00
Anders Kaseorg a543dcc8e3 Remove Debian 10 support.
As a consequence:

• Bump minimum supported Python version to 3.8.
• Move Vagrant environment to Ubuntu 20.04, which has Python 3.8.
• Move CI frontend tests to Ubuntu 20.04.
• Move production build test to Ubuntu 20.04.
• Move 3.4 upgrade test to Ubuntu 20.04.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-26 16:32:02 -07:00
Anders Kaseorg 63a1ef0e91 configure-rabbitmq: Remove use of sudo.
It already runs as root everywhere except in provision_inner, so move
the sudo there.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-19 12:36:31 -07:00
Anders Kaseorg cc30ed8ec7 actions: Delete zerver.lib.actions.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:38 -07:00
Mateusz Mandera 80a9cae0df docs: Fix incorrect path to SAML certs in SAML Keycloak instructions.
This was supposed to be /etc/zulip/saml/idps/
2022-04-13 15:53:03 -07:00
Alex Vandiver 488aaef9b7 docs: Fold FTS index updating into the upgrade step.
On the Debian 10 -> 11 upgrade, the server is running Zulip 4.x, which
lets us pass `--audit-fts-indexes` to `upgrade-zulip-stage-2` rather
than run the command as a separate step.
2022-04-06 11:01:23 -07:00
Alex Vandiver 1e3a6984a4 docs: Upgrade Zulip before trying to fix collations.
The reindex-textual-data tool needs the venv to be cable to run;
switch the order of the last two steps, making them now match the
Debian 9 -> 10 and 10 -> upgrades.

Ref #21296.
2022-04-06 11:01:23 -07:00
Alex Vandiver 5c8086bf90 docs: Fix typo.
We don't suggest self-hosing, unless via a sprinkler in warm weather.
2022-04-04 14:52:04 -07:00
Alex Vandiver 330f0649d7 docs: Remove a stray extra word. 2022-03-24 11:14:50 -07:00
Alex Vandiver 4f93b4b6e4 uploads: Skip the outgoing proxy if S3_KEY is unset.
When the credentials are provided by dint of being run on an EC2
instance with an assigned Role, we must be able to fetch the instance
metadata from IMDS -- which is precisely the type of internal-IP
request that Smokescreen denies.

While botocore supports a `proxies` argument to the `Config` object,
this is not actually respected when making the IMDS queries; only the
environment variables are read from.  See
https://github.com/boto/botocore/issues/2644

As such, implement S3_SKIP_PROXY by monkey-patching the
`botocore.utils.should_bypass_proxies` function, to allow requests to
IMDS to be made without Smokescreen impeding them.

Fixes #20715.
2022-03-24 10:21:35 -07:00
Alex Vandiver e2f4b284db docs: Remove teleport from example list of services.
This is not expected on generic Zulip servers.
2022-03-21 16:33:28 -07:00
Alex Vandiver f39ee5a16c docs: Remove references to supervisorctl (re)start all. 2022-03-21 16:33:28 -07:00
Alex Vandiver 086c0328bd docs: Be explicit about how to skip database creation. 2022-03-21 16:33:28 -07:00
Lorenzo Milesi 88e0d1b111 docs: Add more details on configuring LDAP group restriction.
Fixes #338.

Co-authored-by: Mateusz Mandera <mateusz.mandera@zulip.com>
2022-03-20 17:04:16 -07:00
rht 6be44a6971 docs: mobile-push-notifications.md: Fix grammar problems found by LanguageTool. 2022-03-18 15:28:16 -07:00
rht 74780d24d5 docs: management-commands.md: Fix grammar problems found by LanguageTool. 2022-03-18 15:28:16 -07:00
rht 507851c25a docs: requirements.md: Fix grammar problems found by LanguageTool. 2022-03-18 15:28:16 -07:00
Alex Vandiver c35a783c35 docs: Minor wording fixes to warm standby replication docs. 2022-03-17 12:53:26 -07:00
Alex Vandiver 2c26ad3714 docs: Break out and clarify wal-g backup configuration. 2022-03-17 12:53:26 -07:00
Alex Vandiver d17006da55 puppet: Support setting an `ssl_mode` verification level. 2022-03-15 12:43:50 -07:00
Alex Vandiver 253bef27f5 puppet: Support password-based PostgreSQL replication. 2022-03-15 12:43:50 -07:00
Alex Vandiver 6f5ae8d13d puppet: wal-g backups are required for replication.
Previously, it was possible to configure `wal-g` backups without
replication enabled; this resulted in only daily backups, not
streaming backups.  It was also possible to enable replication without
configuring the `wal-g` backups bucket; this simply failed to work.

Make `wal-g` backups always streaming, and warn loudly if replication
is enabled but `wal-g` is not configured.
2022-03-11 10:09:35 -08:00
Alex Vandiver 6496d43148 puppet: Only s3_backups_bucket is required for backups.
`s3_backups_key` / `s3_backups_secret_key` are optional, as the
permissions could come from the EC2 instance's role.
2022-03-11 10:09:35 -08:00
Alex Vandiver 19beed2709 puppet: Default s3_region to the current ec2 region. 2022-03-11 10:09:35 -08:00
Alex Vandiver bfdc547b00 docs: Document s3_region setting. 2022-03-11 10:09:35 -08:00
Anders Kaseorg 646e466341 install: Desupport Ubuntu 22.04 for now.
Ubuntu 22.04 pushed a post-feature-freeze update to Python 3.10,
breaking virtual environments in a Debian patch
(https://bugs.launchpad.net/ubuntu/+source/python3.10/+bug/1962791).
Also, our antique version of Tornado doesn’t work in 3.10, and we’ll
need to do some work to upgrade that.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-03-07 11:46:07 -08:00
Alya Abbott ba1f804518 docs: Update README and installation guide.
This is a general cleanup that also aims to link to the new
self-hosting page to provide added context.
2022-03-04 13:59:17 -08:00
Anders Kaseorg 75525f5b53 docs: Convert .html#fragment links to .md#fragment.
This uses the myst_heading_anchors option to automatically generate
header anchors and make Sphinx aware of them.  See
https://myst-parser.readthedocs.io/en/latest/syntax/optional.html#auto-generated-header-anchors.

Note: to be compatible with GitHub, MyST-Parser uses a slightly
different convention for .md fragment links than .html fragment links
when punctuation is involved.  This does not affect the generated
fragment links in the HTML output.

Fixes #13264.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-28 16:28:31 -08:00
Anders Kaseorg 17574f3689 docs: Replace manual fragment with automatic heading fragment.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-28 16:28:31 -08:00
Anders Kaseorg 894a50b5c9 install: Support Ubuntu 22.04.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-25 14:49:07 -08:00
Alex Vandiver 538287261d Revert "docs: Suggest running reindex-textual-data in the venv manually."
This reverts commit be7108ebca.
2022-02-25 14:04:27 -08:00
Anders Kaseorg c19d6fb3ef docs: Clean redundant relative links.
We previously had a convention of redundantly including the directory
in relative links to reduce mistakes when moving content from one file
to another.  However, these days we have a broken link checker in
test-documentation, and after #21237, MyST-Parser will check relative
links (including fragments) when you run build-docs.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-24 16:12:18 -08:00
Anders Kaseorg e3572894c5 docs: Clean redundant fragment links to the same page.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-24 16:12:18 -08:00
Anders Kaseorg 1490c91011 docs: Fix list item indentation mistake.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-24 15:31:01 -08:00
Anders Kaseorg b3260bd610 docs: Use Debian and Ubuntu version numbers over development codenames.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-23 12:04:24 -08:00
Anders Kaseorg 1fa2761790 upgrade-zulip-stage-2: Remove create_large_indexes optimization.
This was only used for upgrading from Zulip < 1.9.0, which is no
longer possible because Zulip < 2.1.0 had no common supported
platforms with current main.

If we ever want this optimization for a future migration, it would be
better implemented using Django merge migrations.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-23 11:59:45 -08:00
Anders Kaseorg e1f42c1ac5 docs: Add missing space to compound verbs “back up”, “log in”, etc.
Noun: backup, login, logout, lookup, setup.

Verb: back up, log in, log out, look up, set up.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-07 19:20:54 -08:00
Anders Kaseorg b0ce4f1bce docs: Fix many spelling mistakes.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-07 18:51:06 -08:00
Anders Kaseorg 3e159446f0 docs: Update “G Suite” to “Google Workspace”.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-07 18:51:06 -08:00
Alex Vandiver d61914e8e1 docs: Minor grammar fix in settings.py upgrade docs. 2022-02-04 15:44:48 -08:00
Alex Vandiver 487e7ccd11 docs: Fix the path to the py3 venv.
`/srv/zulip-py3-venv` only exists on development hosts; use the path
to the current venv.
2022-01-29 16:52:11 -08:00
Anders Kaseorg be7108ebca docs: Suggest running reindex-textual-data in the venv manually.
Until the previous commit makes its way into a release, we can
document this workaround.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-01-26 11:56:30 -08:00
Raghav Luthra 4b8cb0a8a9 docs: Uncapitalize the name for nginx.
This matches how nginx refers to itself on its own website and
documentation.

Fixes #20887.
2022-01-25 11:17:51 -08:00
Alex Vandiver 43d63bd5a1 puppet: Always set the RabbitMQ nodename to zulip@localhost.
This is required in order to lock down the RabbitMQ port to only
listen on localhost.  If the nodename is `rabbit@hostname`, in most
circumstances the hostname will resolve to an external IP, which the
rabbitmq port will not be bound to.

Installs which used `rabbit@hostname`, due to RabbitMQ having been
installed before Zulip, would not have functioned if the host or
RabbitMQ service was restarted, as the localhost restrictions in the
RabbitMQ configuration would have made rabbitmqctl (and Zulip cron
jobs that call it) unable to find the rabbitmq server.

The previous commit ensures that configure-rabbitmq is re-run after
the nodename has changed.  However, rabbitmq needs to be stopped
before `rabbitmq-env.conf` is changed; we use an `onlyif` on an `exec`
to print the warning about the node change, and let the subsequent
config change and notify of the service and configure-rabbitmq to
complete the re-configuration.
2022-01-25 01:48:02 +00:00
Alex Vandiver 694c4dfe8f puppet: Admit we leave epmd port 4369 open on all interfaces.
The Erlang `epmd` daemon listens on port 4369, and provides
information (without authentication) about which Erlang processes are
listening on what ports.  This information is not itself a
vulnerability, but may provide information for remote attackers about
what local Erlang services (such as `rabbitmq-server`) are running,
and where.

`epmd` supports an `ERL_EPMD_ADDRESS` environment variable to limit
which interfaces it binds on.  While this environment variable is set
in `/etc/default/rabbitmq-server`, Zulip unfortunately attempts to
start `epmd` using an explicit `exec` block, which ignores those
settings.

Regardless, this lack of `ERL_EPMD_ADDRESS` variable only controls
`epmd`'s startup upon first installation.  Upon reboot, there are two
ways in which `epmd` might be started, neither of which respect
`ERL_EPMD_ADDRESS`:

 - On Focal, an `epmd` service exists and is activated, which uses
   systemd's configuration to choose which interfaces to bind on, and
   thus `ERL_EPMD_ADDRESS` is irrelevant.

 - On Bionic (and Focal, due to a broken dependency from
   `rabbitmq-server` to `epmd@` instead of `epmd`, which may lead to
   the explicit `epmd` service losing a race), `epmd` is started by
   `rabbitmq-server` when it does not detect a running instance.
   Unfortunately, only `/etc/init.d/rabbitmq-server` would respects
   `/etc/default/rabbitmq-server` -- and it defers the actual startup
   to using systemd, which does not pass the environment variable
   down.  Thus, `ERL_EPMD_ADDRESS` is also irrelevant here.

We unfortunately cannot limit `epmd` to only listening on localhost,
due to a number of overlapping bugs and limitations:

 - Manually starting `epmd` with `-address 127.0.0.1` silently fails
   to start on hosts with IPv6 disabled, due to an Erlang bug ([1],
   [2]).

 - The dependencies of the systemd `rabbitmq-server` service can be
   fixed to include the `epmd` service, and systemd can be made to
   bind to `127.0.0.1:4369` and pass that socket to `epmd`, bypassing
   the above bug.  However, the startup of this service is not
   guaranteed, because it races with other sources of `epmd` (see
   below).

 - Any process that runs `rabbitmqctl` results in `epmd` being started
   if one is not currently running; these instances do not respect any
   environment variables as to which addresses to bind on.  This is
   also triggered by `service rabbitmq-server status`, as well as
   various Zulip cron jobs which inspect the rabbitmq queues.  As
   such, it is difficult-to-impossible to ensure that some other
   `epmd` process will not win the race and open the port on all
   interfaces.

Since the only known exposure from leaving port 4369 open is
information that rabbitmq is running on the host, and the complexity
of adjusting this to only bind on localhost is high, we remove the
setting which does not address the problem, and document that the port
is left open, and should be protected via system-level or
network-level firewalls.

[1]: https://bugs.launchpad.net/ubuntu/+source/erlang/+bug/1374109
[2]: https://github.com/erlang/otp/issues/4820
2022-01-25 01:46:51 +00:00
Anders Kaseorg a58a71ef43 Remove Ubuntu 18.04 support.
As a consequence:

• Bump minimum supported Python version to 3.7.
• Move Vagrant environment to Debian 10, which has Python 3.7.
• Move CI frontend tests to Debian 10.
• Move production build test to Debian 10.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-01-21 17:26:14 -08:00
Alex Vandiver be1c4c2bd8 docs: Mention Camo does not use a local Smokescreen in the proxies docs.
This documents the new behaviour in d328d3dd4d.
2022-01-21 15:57:27 -08:00
rht 42f46a78e9 docs: Fix grammar problems found by LanguageTool.
With tweaks to security-model.md by tabbott to expand the SSO acronym.

Ignored, but still needs discussion on whether we should exclude this
rule:

```
The word ‘install’ is not a noun.
  ✗ ...ble to connect to the client during the install process:  So you'll need to shut down a...
                                               ^^^^^^^
  ✓ ...ble to connect to the client during the installation process:  So you'll need to shut down a...
  A_INSTALL: a/the + install

The word ‘install’ is not a noun.
  ✗ ...detected at install time will cause the install to abort. If you already have PostgreSQ...
                                               ^^^^^^^
  ✓ ...detected at install time will cause the installation to abort. If you already have PostgreSQ...
  A_INSTALL: a/the + install
```
2022-01-21 14:02:14 -08:00
Alya Abbott 19154f81c0
docs: Clarify purpose of zulip-announce. 2022-01-19 15:34:24 -08:00
Alex Vandiver 5f237cb34e puppet: Document that upgrades from Git require 3GB.
The step of rebuilding static assets using webpack requires more than
2G of RAM.
2022-01-19 12:36:44 -08:00
Alex Vandiver d328d3dd4d puppet: Allow routing camo requests through an outgoing proxy.
Because Camo includes logic to deny access to private subnets, routing
its requests through Smokescreen is generally not necessary.  However,
it may be necessary if Zulip has configured a non-Smokescreen exit
proxy.

Default Camo to using the proxy only if it is not Smokescreen, with a
new `proxy.enable_for_camo` setting to override this behaviour if need
be.  Note that that setting is in `zulip.conf` on the host with Camo
installed -- not the Zulip frontend host, if they are different.

Fixes: #20550.
2022-01-07 12:08:10 -08:00
Alex Vandiver 2c5fc1827c puppet: Standardize what values are bools, and what true is.
For `no_serve_uploads`, `http_only`, which previously specified
"non-empty" to enable, this tightens what values are true.  For
`pgroonga` and `queue_workers_multiprocess`, this broadens the
possible values from `enabled`, and `true` respectively.
2022-01-07 12:08:10 -08:00
Alex Vandiver 6218ed91c2 puppet: Use lazy-apps and uwsgi control sockets for rolling reloads.
Restarting the uwsgi processes by way of supervisor opens a window
during which nginx 502's all responses.  uwsgi has a configuration
called "chain reloading" which allows for rolling restart of the uwsgi
processes, such that only one process at once in unavailable; see
uwsgi documentation ([1]).

The tradeoff is that this requires that the uwsgi processes load the
libraries after forking, rather than before ("lazy apps"); in theory
this can lead to larger memory footprints, since they are not shared.
In practice, as Django defers much of the loading, this is not as much
of an issue.  In a very basic test of memory consumption (measured by
total memory - free - caches - buffers; 6 uwsgi workers), both
immediately after restarting Django, and after requesting `/` 60 times
with 6 concurrent requests:

                      |  Non-lazy  |  Lazy app  | Difference
    ------------------+------------+------------+-------------
    Fresh             |  2,827,216 |  2,870,480 |   +43,264
    After 60 requests |  3,332,284 |  3,409,608 |   +77,324
    ..................|............|............|.............
    Difference        |   +505,068 |   +539,128 |   +34,060

That is, "lazy app" loading increased the footprint pre-requests by
43MB, and after 60 requests grew the memory footprint by 539MB, as
opposed to non-lazy loading, which grew it by 505MB.  Using wsgi "lazy
app" loading does increase the memory footprint, but not by a large
percentage.

The other effect is that processes may be served by either old or new
code during the restart window.  This may cause transient failures
when new frontend code talks to old backend code.

Enable chain-reloading during graceful, puppetless restarts, but only
if enabled via a zulip.conf configuration flag.

Fixes #2559.

[1]: https://uwsgi-docs.readthedocs.io/en/latest/articles/TheArtOfGracefulReloading.html#chain-reloading-lazy-apps
2022-01-05 14:48:52 -08:00
Alya Abbott aaf1258de2 developer docs: Tweak ToS for push notifications wording. 2021-12-14 14:13:34 -08:00
Tim Abbott ee77c6365a portico: Use /help/ style pages for displaying policies.
This replaces the TERMS_OF_SERVICE and PRIVACY_POLICY settings with
just a POLICIES_DIRECTORY setting, in order to support settings (like
Zulip Cloud) where there's more policies than just those two.

With minor changes by Eeshan Garg.
2021-12-10 17:56:12 -08:00
Alex Vandiver 01e8f752a8 puppet: Use certbot package timer, not our own cron job.
The certbot package installs its own systemd timer (and cron job,
which disabled itself if systemd is enabled) which updates
certificates.  This process races with the cron job which Zulip
installs -- the only difference being that Zulip respects the
`certbot.auto_renew` setting, and that it passes the deploy hook.
This means that occasionally nginx would not be reloaded, when the
systemd timer caught the expiration first.

Remove the custom cron job and `certbot-maybe-renew` script, and
reconfigure certbot to always reload nginx after deploying, using
certbot directory hooks.

Since `certbot.auto_renew` can't have an effect, remove the setting.
In turn, this removes the need for `--no-zulip-conf` to
`setup-certbot`.  `--deploy-hook` is similarly removed, as running
deploy hooks to restart nginx is now the default; pass
`--no-directory-hooks` in standalone mode to not attempt to reload
nginx.  The other property of `--deploy-hook`, of skipping symlinking
into place, is given its own flog.
2021-12-09 13:47:33 -08:00
Eeshan Garg 3bab91079f external links: Migrate the rest of /developer-community links.
We recently changed /developer-community to /development-community.
Now that this change is in production, we can also migrate the
external links in our ReadTheDocs documentation.
2021-12-09 12:14:26 -08:00
Alex Vandiver cb2d0ff32b postgresql: Support replication on PostgreSQL >= 11, document.
PostgreSQL 11 and below used a configuration file names
`recovery.conf` to manage replicas and standbys; support for this was
removed in PostgreSQL 12[1], and the configuration parameters were
moved into the main `postgresql.conf`.

Add `zulip.conf` settings for the primary server hostname and
replication username, so that the complete `postgresql.conf`
configuration on PostgreSQL 14 can continue to be managed, even when
replication is enabled.  For consistency, also begin writing out the
`recovery.conf` for PostgreSQL 11 and below.

In PostgreSQL 12 configuration and later, the `wal_level =
hot_standby` setting is removed, as `hot_standby` is equivalent to
`replica`, which is the default value[2].  Similarly, the
`hot_standby = on` setting is also the default[3].

Documentation is added for these features, and the commentary on the
"Export and Import" page referencing files under `puppet/zulip_ops/`
is removed, as those files no longer have any replication-specific
configuration.

[1]: https://www.postgresql.org/docs/current/recovery-config.html
[2]: https://www.postgresql.org/docs/12/runtime-config-wal.html#GUC-WAL-LEVEL
[3]: https://www.postgresql.org/docs/12/runtime-config-replication.html#GUC-HOT-STANDBY
2021-12-03 16:32:41 -08:00
Emilio López baea14ee57 docs: Clarify use of `loadbalancer.ips` when using a reverse proxy.
When Zulip is run behind one or more reverse proxies, you must
configure `loadbalancer.ips` so that Zulip respects the client IP
addresses found in the `X-Forwarded-For` header. This is not
immediately clear from the documentation, so this commit makes it more
clear and augments the existing examples to showcase this need.

Fixes: #19073
2021-12-03 13:59:31 -08:00