Commit Graph

33 Commits

Author SHA1 Message Date
Alex Vandiver a6d2112453 kandra: Run rabbitmq cron jobs as root.
This is a regression introduced in
f246b82f67.
2024-05-28 15:15:53 -07:00
Alex Vandiver f246b82f67 puppet: Factor out pattern of writing a nagios state file atomically. 2024-05-24 11:31:25 -07:00
Alex Vandiver 04e21044b9 check_send_receive_time: Default --site usefully.
This saves us the time of shelling out to a new python process,
loading all of Django, and printing one value we could just have read
in-process.  It is unclear why we ever did it this way.
2024-05-08 15:51:20 -07:00
Alex Vandiver 2bd60e8562 check_send_receive_time: Print no output on success. 2024-05-08 15:51:20 -07:00
Alex Vandiver 294fd914e1 kandra: Fix cron specification, to run once, not every minute from 7-8. 2024-05-08 12:16:36 -07:00
Alex Vandiver 908a805d3e kandra: Change the remaining check_send_receive_time to use zulip::cron.
This was mistakenly left off of 6e981c18d5.
2024-05-08 09:00:45 -07:00
Alex Vandiver e7511d43b8 kandra: Put the use_proxy override on the job it belongs on. 2024-05-08 09:00:45 -07:00
Alex Vandiver 572fafd6b9 cron: Set environment variables to use Smokescreen, if configured. 2024-04-24 14:40:28 -07:00
Alex Vandiver 2df91c70ef puppet: Move rabbitmq monitoring into kandra/, where it is used from. 2024-04-24 14:40:28 -07:00
Alex Vandiver 6e981c18d5 puppet: Factor out cron job creation. 2024-04-24 14:40:28 -07:00
Alex Vandiver a4e6037dc4 kandra: Automate the second step of configuring database replicas.
If there is a replication primary configured, and no current database,
then we check all of the required secrets are in place, then pull down
the latest backup and trigger a PostgreSQL restart, which will pick up
downloading the remaining WAL logs to catch up, then start streaming
from the configured primary.
2024-04-17 17:31:49 -07:00
Alex Vandiver a8dbdd6d92 kandra: Ensure that pg_hba.conf is in place before starting PostgreSQL. 2024-04-17 17:31:49 -07:00
Alex Vandiver b0e3191434 puppet: Stop relying on "tidy" ordering, which ignores metaparams.
The `tidy` parameter is buggy, and ignores all ordering
metaparameters.  This is fixed in Puppet 7[^1], but it's helpful to
resolve it now.  Specifically, this fixes bugs with tidy running too
early, and deleting the old version of a package before its new
version is installed or symlinked, leaving a race condition if
anything tries to run the binary in this window.

This is mostly not a problem for Supervisor-managed processes, since
the binary is already running, and can continue to run if it is tidied
out from under the running process.  For stand-alone tools like wal-g,
which are run frequently by PostgreSQL, this may cause issues if
PostgreSQL tries to call them during a puppet run.

Remove all complicated uses of tidy, and replace them with an `exec`
which does the equivalent.  We also generate `file` resources for
binaries, making them easier (and clearer) to specify as dependencies.

[^1]: https://puppet.atlassian.net/browse/PUP-10688
2024-04-15 14:30:24 -07:00
Alex Vandiver 8d3120bf17 kandra: Remove zulip::static_asset_compiler include.
This was removed in 263212decf.
2024-04-12 15:00:33 -07:00
Alex Vandiver f4d109c289 puppet: Fix arrow alignment. 2024-04-05 09:18:04 -07:00
Alex Vandiver 235e2eefc8 puppet: Switch from top-level fact variables to facts dict, again.
These were somehow missed in 57f8b48ff9.
2024-04-05 09:18:04 -07:00
Alex Vandiver eaef12ce3a kandra: Ensure that the nagios known_hosts exist before running autossh. 2024-04-03 11:38:29 -07:00
Alex Vandiver f9805c9e1f kandra: rabbitmqctl may not exist when applying the initial catalog.
puppet hard-fails if it can't find the binary to run in `$PATH`, so we
need to make the `unless` short-circuit to false if puppet itself is
not installed yet (as during initial installation).
2024-03-29 16:25:37 -07:00
Alex Vandiver 3c2efd236c kandra: Skip prometheus database user creation. 2024-03-29 11:34:18 -07:00
Alex Vandiver da243e9fb7 kandra: Force disks to be set up before kernels are changed. 2024-03-29 11:34:18 -07:00
Alex Vandiver e4883cc2dc kandra: Move PostgreSQL data mountpoint to /srv/data.
We store a lot of other things in `/srv` now, and moving them mid-puppet
confuses things significantly.
2024-03-29 11:34:18 -07:00
Alex Vandiver 93c6fd4746 kandra: Pull instance storage device from nvme information. 2024-03-29 11:34:18 -07:00
Alex Vandiver 04f4e74709 kandra: Enable per-object metrics from rabbitmq.
These default to off, because in situations with thousands of queues,
consumers, and producers, they cause unreasonable overhead.  Our use
case has few enough queues that we do want to be able to inspect them
individually.

Enable per-object Prometheus metrics, per [1].

[1]: 78851828ec/deps/rabbitmq_prometheus (configuration)
2024-03-26 09:04:02 -07:00
Alex Vandiver e06b1794a8 kandra: Template and insert email credentials. 2024-03-21 16:14:44 -07:00
Alex Vandiver 75411b264e kandra: Remove unnecessary libapache2-mod-wsgi.
It does not exist in 22.04, and its last use was removed in
61666a9262.
2024-03-21 14:23:30 -07:00
Alex Vandiver 23504308fb puppet: Install the same version of postgres-client as the server.
We require a `pg_dump` whose version matches the version of the server
we are configured against (see 3a8b4b0205).  Installing the latest
`postgresql-client` does not guarantee that we have such a binary
present.
2024-03-21 12:34:34 -07:00
Alex Vandiver dfdaddea38 kandra: Add a memcached exporter using bmemcached. 2024-02-23 13:32:04 -08:00
Alex Vandiver fa6f4cc039 kandra: Skip hostnames which are undef.
If zulip.conf is not fully configured, do not error out if one of
the $hostname values is undef.
2024-02-07 10:42:12 -08:00
Alex Vandiver 3d63a87384 kandra: Puppet github.com keys to both root and zulip users.
We update to add the ecdsa-sha2-nistp256 key as well.
2024-02-07 10:42:12 -08:00
Alex Vandiver 927660a7b6 kandra: Remove munin. 2024-02-06 21:34:56 -08:00
Alex Vandiver 96d237d57e kandra: Pull in known_hosts to the nagios user.
This prevents failures when new hosts are initially added to the
configuration.
2024-02-07 00:02:40 -05:00
Alex Vandiver 2cbd9c2a47 kandra: Update default chrony configuration for 22.04. 2024-02-07 00:02:32 -05:00
Alex Vandiver b23d90ed62 puppet: Rename puppet/zulip_ops to puppet/kandra.
This makes for easier tab-completion, and also is a bit more explicit
about the expected consumer.
2024-02-06 17:56:27 -08:00