Commit Graph

24 Commits

Author SHA1 Message Date
Alex Vandiver 6e981c18d5 puppet: Factor out cron job creation. 2024-04-24 14:40:28 -07:00
Alex Vandiver a4e6037dc4 kandra: Automate the second step of configuring database replicas.
If there is a replication primary configured, and no current database,
then we check all of the required secrets are in place, then pull down
the latest backup and trigger a PostgreSQL restart, which will pick up
downloading the remaining WAL logs to catch up, then start streaming
from the configured primary.
2024-04-17 17:31:49 -07:00
Alex Vandiver a8dbdd6d92 kandra: Ensure that pg_hba.conf is in place before starting PostgreSQL. 2024-04-17 17:31:49 -07:00
Alex Vandiver b0e3191434 puppet: Stop relying on "tidy" ordering, which ignores metaparams.
The `tidy` parameter is buggy, and ignores all ordering
metaparameters.  This is fixed in Puppet 7[^1], but it's helpful to
resolve it now.  Specifically, this fixes bugs with tidy running too
early, and deleting the old version of a package before its new
version is installed or symlinked, leaving a race condition if
anything tries to run the binary in this window.

This is mostly not a problem for Supervisor-managed processes, since
the binary is already running, and can continue to run if it is tidied
out from under the running process.  For stand-alone tools like wal-g,
which are run frequently by PostgreSQL, this may cause issues if
PostgreSQL tries to call them during a puppet run.

Remove all complicated uses of tidy, and replace them with an `exec`
which does the equivalent.  We also generate `file` resources for
binaries, making them easier (and clearer) to specify as dependencies.

[^1]: https://puppet.atlassian.net/browse/PUP-10688
2024-04-15 14:30:24 -07:00
Alex Vandiver 8d3120bf17 kandra: Remove zulip::static_asset_compiler include.
This was removed in 263212decf.
2024-04-12 15:00:33 -07:00
Alex Vandiver f4d109c289 puppet: Fix arrow alignment. 2024-04-05 09:18:04 -07:00
Alex Vandiver 235e2eefc8 puppet: Switch from top-level fact variables to facts dict, again.
These were somehow missed in 57f8b48ff9.
2024-04-05 09:18:04 -07:00
Alex Vandiver eaef12ce3a kandra: Ensure that the nagios known_hosts exist before running autossh. 2024-04-03 11:38:29 -07:00
Alex Vandiver f9805c9e1f kandra: rabbitmqctl may not exist when applying the initial catalog.
puppet hard-fails if it can't find the binary to run in `$PATH`, so we
need to make the `unless` short-circuit to false if puppet itself is
not installed yet (as during initial installation).
2024-03-29 16:25:37 -07:00
Alex Vandiver 3c2efd236c kandra: Skip prometheus database user creation. 2024-03-29 11:34:18 -07:00
Alex Vandiver da243e9fb7 kandra: Force disks to be set up before kernels are changed. 2024-03-29 11:34:18 -07:00
Alex Vandiver e4883cc2dc kandra: Move PostgreSQL data mountpoint to /srv/data.
We store a lot of other things in `/srv` now, and moving them mid-puppet
confuses things significantly.
2024-03-29 11:34:18 -07:00
Alex Vandiver 93c6fd4746 kandra: Pull instance storage device from nvme information. 2024-03-29 11:34:18 -07:00
Alex Vandiver 04f4e74709 kandra: Enable per-object metrics from rabbitmq.
These default to off, because in situations with thousands of queues,
consumers, and producers, they cause unreasonable overhead.  Our use
case has few enough queues that we do want to be able to inspect them
individually.

Enable per-object Prometheus metrics, per [1].

[1]: 78851828ec/deps/rabbitmq_prometheus (configuration)
2024-03-26 09:04:02 -07:00
Alex Vandiver e06b1794a8 kandra: Template and insert email credentials. 2024-03-21 16:14:44 -07:00
Alex Vandiver 75411b264e kandra: Remove unnecessary libapache2-mod-wsgi.
It does not exist in 22.04, and its last use was removed in
61666a9262.
2024-03-21 14:23:30 -07:00
Alex Vandiver 23504308fb puppet: Install the same version of postgres-client as the server.
We require a `pg_dump` whose version matches the version of the server
we are configured against (see 3a8b4b0205).  Installing the latest
`postgresql-client` does not guarantee that we have such a binary
present.
2024-03-21 12:34:34 -07:00
Alex Vandiver dfdaddea38 kandra: Add a memcached exporter using bmemcached. 2024-02-23 13:32:04 -08:00
Alex Vandiver fa6f4cc039 kandra: Skip hostnames which are undef.
If zulip.conf is not fully configured, do not error out if one of
the $hostname values is undef.
2024-02-07 10:42:12 -08:00
Alex Vandiver 3d63a87384 kandra: Puppet github.com keys to both root and zulip users.
We update to add the ecdsa-sha2-nistp256 key as well.
2024-02-07 10:42:12 -08:00
Alex Vandiver 927660a7b6 kandra: Remove munin. 2024-02-06 21:34:56 -08:00
Alex Vandiver 96d237d57e kandra: Pull in known_hosts to the nagios user.
This prevents failures when new hosts are initially added to the
configuration.
2024-02-07 00:02:40 -05:00
Alex Vandiver 2cbd9c2a47 kandra: Update default chrony configuration for 22.04. 2024-02-07 00:02:32 -05:00
Alex Vandiver b23d90ed62 puppet: Rename puppet/zulip_ops to puppet/kandra.
This makes for easier tab-completion, and also is a bit more explicit
about the expected consumer.
2024-02-06 17:56:27 -08:00