Commit Graph

1178 Commits

Author SHA1 Message Date
Alex Vandiver c9141785fd puppet: Use concat fragments to place port allows next to services.
This means that services will only open their ports if they are
actually run, without having to clutter rules.v4 with a log of `if`
statements.

This does not go as far as using `puppetlabs/firewall`[1] because that
would represent an additional DSL to learn; raw IPtables sections can
easily be inserted into the generated iptables file via
`concat::fragment` (either inline, or as a separate file), but config
can be centralized next to the appropriate service.

[1] https://forge.puppet.com/modules/puppetlabs/firewall
2021-05-27 21:14:48 -07:00
Alex Vandiver 4f79b53825 puppet: Factor out firewall config. 2021-05-27 21:14:48 -07:00
Alex Vandiver 87a109e3e0 puppet: Pull in pinned puppet modules.
Using puppet modules from the puppet forge judiciously will allow us
to simplify the configuration somewhat; this specifically pulls in the
stdlib module, which we were already using parts of.
2021-05-27 21:14:48 -07:00
Alex Vandiver f3eea72c2a setup: Merge multiple setup-apt-repo scripts into one.
This moves the `.asc` files into subdirectories, and writes out the
according `.list` files into them.  It moves from templates to
written-out `.list` files for clarity and ease of
implementation (Debian and Ubuntu need different templates for
`zulip`), and as a way of making explicit which releases are supported
for each list.  For the special-case of the PGroonga signing key, we
source an additional file within the directory.

This simplifies the process for adding another class of `.list` file.
2021-05-26 14:42:29 -07:00
Alex Vandiver 4f017614c5 nagios: Replace check_fts_update_log with a process_fts_updates flag.
This avoids having to duplicate the connection logic from
process_fts_updates.

Co-authored-by: Adam Birds <adam.birds@adbwebdesigns.co.uk>
2021-05-25 13:56:05 -07:00
Alex Vandiver ab130ceb35 nagios: Support arbitrary database user and dbname in replication check.
Co-authored-by: Adam Birds <adam.birds@adbwebdesigns.co.uk>
2021-05-25 13:56:05 -07:00
Alex Vandiver c17f502bb0 process_fts_updates: Support arbitrary database user and dbname.
Co-authored-by: Adam Birds <adam.birds@adbwebdesigns.co.uk>
2021-05-25 13:56:05 -07:00
Alex Vandiver 02fc0d3e1d db: Drop None and empty-string checking in arguments.
psycopg2 treats None and "" the same as not-provided:

```
assert connect(user="zulip", dbname="zulip")
assert connect(user="zulip", dbname="zulip", host="")
assert connect(user="zulip", dbname="zulip", host=None)
with Raises("no password supplied"):
    connect(user="zulip", dbname="zulip", host="localhost")

assert connect(user="zulip", dbname="zulip", port="")
assert connect(user="zulip", dbname="zulip", port=None)
assert connect(user="zulip", dbname="zulip", port=5432)
with Raises("could not connect to server"):
    connect(user="zulip", dbname="zulip", port=5000)

assert connect(dbname="zulip", host="localhost", password="right-password")
with Raises("no password supplied"):
    connect(dbname="zulip", host="localhost", password="")
with Raises("no password supplied"):
    connect(dbname="zulip", host="localhost", password=None)
with Raises("password authentication failed"):
    connect(dbname="zulip", host="localhost", password="wrong")
```

Co-authored-by: Adam Birds <adam.birds@adbwebdesigns.co.uk>
2021-05-25 13:46:58 -07:00
Alex Vandiver 9c652eb16b db: Use the pre-computed values from settings.
Rather than duplicate logic from `computed_settings`, use the values
that were computed therein.

Co-authored-by: Adam Birds <adam.birds@adbwebdesigns.co.uk>
2021-05-25 13:46:58 -07:00
Alex Vandiver 94d7c29d92 db: Use the same codepath for cases (2) and (3).
Using the second branch _only_ for case (3), of a PostgreSQL server on
a different host, leaves it untested in CI.  It also brings in an
unnecessary Django dependency.

Co-authored-by: Adam Birds <adam.birds@adbwebdesigns.co.uk>
2021-05-25 13:46:58 -07:00
Alex Vandiver add6971ad9 db: Make USING_PGROONGA logic clearer.
We only need to read the `zulip.conf` file to determine if we're using
PGROONGA if we are on the PostgreSQL machine, with no access to
Django.

Co-authored-by: Adam Birds <adam.birds@adbwebdesigns.co.uk>
2021-05-25 13:46:58 -07:00
Alex Vandiver 75bf19c9d9 db: Combine the `if "host" in pg_args` stanza with earlier clause.
The only way in which "host" could be set is in cases (1) or (2), when
it was potentially read from Django's settings.  In case (3), we
already know we are on the same host as the PostgreSQL server.

This unifies the two separated checks, which are actually the same
check.

Co-authored-by: Adam Birds <adam.birds@adbwebdesigns.co.uk>
2021-05-25 13:46:58 -07:00
Alex Vandiver 67fc8e84ea db: Clarify the 3 different cases that process_fts_updates must support.
Co-authored-by: Adam Birds <adam.birds@adbwebdesigns.co.uk>
2021-05-25 13:46:58 -07:00
Alex Vandiver 116e41f1da puppet: Move files out and back when mounting /srv.
Specifically, this affects /srv/zulip-aws-tools.
2021-05-23 13:29:23 -07:00
Alex Vandiver ea98549e88 puppet: Always install linux-image-virtual, for ksplice support. 2021-05-23 13:29:23 -07:00
Alex Vandiver 0b1dd27841 puppet: AWS mounts its extra disks with inconsistent names.
It is now /dev/nvme1n1, not /dev/nvme0n1; but it always has a
consistent major/minor node.  Source the file that defines these.
2021-05-23 13:29:23 -07:00
Alex Vandiver 82797dd53c settings: Standardize the name of the deliver_scheduled_messages logs.
This makes it match its command name, and other logfile name.
2021-05-18 12:39:28 -07:00
Alex Vandiver 343a1396af puppet: Rename logfile for deliver_scheduled_messages to be consistent. 2021-05-18 12:39:28 -07:00
Alex Vandiver ef6d0ec5ca puppet: Only run deliver_scheduled_messages and _emails on one server.
`deliver_scheduled_emails` and `deliver_scheduled_messages` use the
`ScheduledEmail` and `ScheduledMessage` tables as a queue,
effectively, pulling values off of them.  As noted in their comments,
this is not safe to run on multiple hosts at once.  As such, split out
the supervisor files for them.
2021-05-18 12:39:28 -07:00
Alex Vandiver 033a96aa5d puppet: Fix check_ssl_certificate check to check named host, not self. 2021-05-17 18:38:30 -07:00
Alex Vandiver a2b7a5ef4b puppet: Clarify 20m keepalive time from the LB is a max; it can be less. 2021-05-17 14:56:51 -07:00
Alex Vandiver 66a232e303 smokescreen: Bump version of Go and Smokescreen.
Move version pins to the latest versions of Go and Smokescreen.
2021-05-12 10:08:42 -10:00
Alex Vandiver feb7870db7 puppet: Adjust thresholds on autovac_freeze.
These thresholds are in relationship to the
`autovacuum_freeze_max_age`, *not* the XID wraparound, which happens
at 2^31-1.  As such, it is *perfectly normal* that they hit 100%, and
then autovacuum kicks in and brings it back down.  The unusual
condition is that PostgreSQL pushes past the point where an autovacuum
would be triggered -- therein lies the XID wraparound danger.

With the `autovacuum_freeze_max_age` set to 2000000000 in
`postgresql.conf`, XID wraparound happens at 107.3%.  Set the warning
and error thresholds to below this, but above 100% so this does not
trigger constantly.
2021-05-11 17:11:47 -07:00
Alex Vandiver 0f1611286d management: Rename the deliver_email command to deliver_scheduled_email.
This makes it parallel with deliver_scheduled_messages, and clarifies
that it is not used for simply sending outgoing emails (e.g. the
`email_senders` queue).

This also renames the supervisor job to match.
2021-05-11 13:07:29 -07:00
Anders Kaseorg 544bbd5398 docs: Fix capitalization mistakes.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-10 09:57:26 -07:00
Tim Abbott ad0be6cea1 puppet: Remove thumbor.conf nginx configuration.
This was missing in 405bc8dabf.
2021-05-07 16:57:29 -07:00
Anders Kaseorg 9d57fa9759 puppet: Use pgrep -x to avoid accidental matches.
Matching the full process name (-x without -f) or full command
line (-xf) is less prone to mistakes like matching a random substring
of some other command line or pgrep matching itself.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-07 08:54:41 -07:00
Anders Kaseorg 405bc8dabf requirements: Remove Thumbor.
Thumbor and tc-aws have been dragging their feet on Python 3 support
for years, and even the alphas and unofficial forks we’ve been running
don’t seem to be maintained anymore.  Depending on these projects is
no longer viable for us.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-06 20:07:32 -07:00
Alex Vandiver eda9ce2364 locale: Use `C.UTF-8` rather than `en_US.UTF-8`.
The `en_US.UTF-8` locale may not be configured or generated on all
installs; it also requires that the `locales` package be installed.
If users generate the `en_US.UTF-8` locale without adding it to the
permanent set of system locales, the generated `en_US.UTF-8` stops
working when the `locales` package is updated.

Switch to using `C.UTF-8` in all cases, which is guaranteed to be
installed.

Fixes #15819.
2021-05-04 08:51:46 -07:00
Alex Vandiver ddb9d16132 puppet: Install procps, for pgrep.
In puppet, we use pgrep in the collection stage, to see if rabbitmq is
running.  Sufficiently bare-bones systems will not have
`procps` (which provides `pgrep`) installed yet, which makes the
install abort when running `puppet` for the first time.

Just installing the `procps` package in Puppet is insufficient,
because the check in the `unless` block runs when Puppet is
determining which resources it needs to instantiate, and in what
order; any package installation has yet to happen.  As
`erlang-base` (which provides `epmd`) happens to have a dependency of
`procps`, any system without `pgrep` will also not have `epmd`
installed or running.  Regardless, it is safe to run `epmd -daemon`
even if one is already running, as the comment above notes.
2021-05-03 14:48:52 -07:00
Alex Vandiver 3577c6dbd4 puppet: `pgrep -f something` can match itself.
Using `pgrep -f epmd` to determine if `empd` is running is a race
condition with itself, since the pgrep is attempting to match the
"full process name" and its own full process name contains "epmd".
This leads to epmd not being started when it should be, which in turn
leads to rabbitmq-server failing to start.

Use the standard trick for this, namely a one-character character
class, to prevent self-matching.
2021-05-03 14:48:52 -07:00
Jennifer Hwang c9f5946239 puppet: Add override for queue_workers_multiprocess.
With tweaks to the documentation by tabbott.

This uses the following configuration option:

[application_server]
queue_workers_multiprocess = false
2021-04-20 14:37:15 -07:00
Tim Abbott bb676f1143 smokescreen: Move supervisor configuration to managed directory.
We've established the conf.d/zulip directory as the recommended path
for Zulip-managed configuration files, so this belongs there.
2021-04-16 14:05:42 -07:00
Gaurav Pandey 303e7b9701 ci: Add Debian bullseye to production test suite. 2021-04-15 21:38:31 -07:00
Gaurav Pandey feb720b463 install: Add beta support for debian bullseye for production.
This won't work on a real bullseye system until Bullseye actually
officially releases.

Fixes part of #17863.
2021-04-15 21:38:31 -07:00
Alex Vandiver 9de35d98d3 puppet: Ensure a snakeoil certificate, for Postfix and PostgreSQL.
We use the snakeoil TLS certificate for PostgreSQL and Postfix; some
VMs install the `ssl-cert` package but (reasonably) don't build the
snakeoil certs into the image.

Build them as needed.

Fixes #14955.
2021-04-15 21:37:55 -07:00
Anders Kaseorg b01d43f339 mypy: Fix strict_equality violations.
puppet/zulip/files/nagios_plugins/zulip_postgresql/check_postgresql_replication_lag:98: error: Non-overlapping equality check (left operand type: "List[List[str]]", right operand type: "Literal[0]")  [comparison-overlap]
zerver/tests/test_realm.py:650: error: Non-overlapping container check (element type: "Dict[str, Any]", container item type: "str")  [comparison-overlap]

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-04-13 09:18:18 -07:00
Alex Vandiver 93f3b41811 puppet: Also move avatars to the same nginx include file. 2021-04-09 08:28:42 -07:00
Alex Vandiver aae8f454ce puppet: Simplify uploads handling.
`uploads-route.noserve` and `uploads-route.internal` contained
identical location blocks for `/upload`, since differentiation was
necessary for Trusty until 33c941407b72; move the now-common sections
into `app`.

This the only differences between internal and S3 serving as a single
block which should be included or not based on config; move it to a
file which may or may not be placed in `app.d/`.
2021-04-09 08:28:42 -07:00
Alex Vandiver fb26c6b7ca puppet: Move uwsgi_pass setting into uwsgi_params.
We only ever call `uwsgi_pass django` in association with `include
uwsgi_params`; refactor it in.
2021-04-09 08:28:42 -07:00
Alex Vandiver 9cf9d5f2cf puppet: Move HTTP_X_REAL_IP setting into uwsgi_params.
This effectively also adds it to serving `/user_uploads`, where its
lack would cause failures to list the actual IP address.
2021-04-09 08:28:42 -07:00
Alex Vandiver 795517bd52 puppet: Only set X-Real-IP once.
07779ea879 added an additional `proxy_set_header` of `X-Real-IP` to
`puppet/zulip/files/nginx/zulip-include-common/proxy`; as noted in
that commit, Tornado longpoll proxies already included such a line.

Unfortunately, this equates to setting that header _twice_ for Tornado
ports, like so:

```
X-Real-Ip: 198.199.116.58
X-Real-Ip: 198.199.116.58
```

...which is represented, once parsed by Django, as an IP of
`198.199.116.58, 198.199.116.58`.  For IPv4, this odd "IP address" has
no problems, and appears in the access logs accordingly; for IPv6
addresses, however, its length is such that it overflows a call to
`getaddrinfo` when attempting to determine the validity of the IP.

Remove the now-duplicated inclusion of the header.
2021-04-09 08:28:42 -07:00
Alex Vandiver 07779ea879 middleware: Do not trust X-Forwarded-For; use X-Real-Ip, set from nginx.
The `X-Forwarded-For` header is a list of proxies' IP addresses; each
proxy appends the remote address of the host it received its request
from to the list, as it passes the request down.  A naïve parsing, as
SetRemoteAddrFromForwardedFor did, would thus interpret the first
address in the list as the client's IP.

However, clients can pass in arbitrary `X-Forwarded-For` headers,
which would allow them to spoof their IP address.  `nginx`'s behavior
is to treat the addresses as untrusted unless they match an allowlist
of known proxies.  By setting `real_ip_recursive on`, it also allows
this behavior to be applied repeatedly, moving from right to left down
the `X-Forwarded-For` list, stopping at the right-most that is
untrusted.

Rather than re-implement this logic in Django, pass the first
untrusted value that `nginx` computer down into Django via `X-Real-Ip`
header.  This allows consistent IP addresses in logs between `nginx`
and Django.

Proxied calls into Tornado (which don't use UWSGI) already passed this
header, as Tornado logging respects it.
2021-03-31 14:19:38 -07:00
Anders Kaseorg 29e4c71ec4 puppet: Reformat custom Ruby modules with Rufo.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-03-24 12:12:04 -07:00
Alex Vandiver 6ee74b3433 puppet: Check health of APT repository. 2021-03-23 19:27:42 -07:00
Alex Vandiver c01345d20c puppet: Add nagios check for long-lived certs that do not auto-renew. 2021-03-23 19:27:27 -07:00
Alex Vandiver 9ea86c861b puppet: Add a nagios alert configuration for smokescreen.
This verifies that the proxy is working by accessing a
highly-available website through it.  Since failure of this equates to
failures of Sentry notifications and Android mobile push
notifications, this is a paging service.
2021-03-18 10:11:15 -07:00
Anders Kaseorg 129ea6dd11 nginx: Consistently listen on IPv6 and with HTTP/2.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-03-17 17:46:32 -07:00
Alex Vandiver 15c58cce5a puppet: Create new nginx logfiles as the zulip user, not as www-data.
All of `/var/log/nginx/` is chown'd to `zulip` and the nginx processes
themselves run as `nginx`, and would thus (on their own) create new
logfiles as `zulip`.  Having `logrotate` create them as the package
default of `www-data` means that they are momentarily unreadable by
the `zulip` user just after rotation, which can cause problems with
logtail scripts.

Commit the standard `nginx` logrotate configuration, but with the
`zulip` user instead of the `www-data` user.
2021-03-16 14:45:13 -07:00
Alex Vandiver 3314fefaec puppet: Do not require a venv for zulip-puppet-apply.
0663b23d54 changed zulip-puppet-apply to
use the venv, because it began using `yaml` to parse the output of
puppet to determine if changes would happen.

However, not every install ends with a venv; notably, non-frontend
servers do not have one.  Attempting to run zulip-puppet-apply on them
hence now fails.

Remove this dependency on the venv, by installing a system
python3-yaml package -- though in reality, this package is already an
indirect dependency of the system.  Especially since pyyaml is quite
stable, we're not using it in any interesting way, and it does not
actually add to the dependencies, it is preferable to parsing the YAML
by hand in this instance.
2021-03-14 17:50:57 -07:00