Commit Graph

45 Commits

Author SHA1 Message Date
Alex Vandiver 6a811cb306 process_exporter: Use -recheck-with-time-limit to catch process renames. 2024-05-30 22:24:53 -07:00
Alex Vandiver a6d2112453 kandra: Run rabbitmq cron jobs as root.
This is a regression introduced in
f246b82f67.
2024-05-28 15:15:53 -07:00
Alex Vandiver d2516607bb puppet: Silence "needrestart" nags about kernel upgrades.
Ksplice keeps the kernel updated without restarts.
2024-05-28 14:34:26 -07:00
Alex Vandiver f246b82f67 puppet: Factor out pattern of writing a nagios state file atomically. 2024-05-24 11:31:25 -07:00
Alex Vandiver 230040caa9 puppet: Remove check_postgresql_backup.
We have replaced this monitoring with the black-box wal-g monitoring,
which is more accurate.
2024-05-24 11:27:59 -07:00
Alex Vandiver 04e21044b9 check_send_receive_time: Default --site usefully.
This saves us the time of shelling out to a new python process,
loading all of Django, and printing one value we could just have read
in-process.  It is unclear why we ever did it this way.
2024-05-08 15:51:20 -07:00
Alex Vandiver 2bd60e8562 check_send_receive_time: Print no output on success. 2024-05-08 15:51:20 -07:00
Alex Vandiver 294fd914e1 kandra: Fix cron specification, to run once, not every minute from 7-8. 2024-05-08 12:16:36 -07:00
Alex Vandiver 908a805d3e kandra: Change the remaining check_send_receive_time to use zulip::cron.
This was mistakenly left off of 6e981c18d5.
2024-05-08 09:00:45 -07:00
Alex Vandiver e7511d43b8 kandra: Put the use_proxy override on the job it belongs on. 2024-05-08 09:00:45 -07:00
Alex Vandiver 9dfaa83aa8 invites: Remove invites worker, make confirmation object in-process.
The "invites" worker exists to do two things -- make a Confirmation
object, and send the outgoing email.  Making the Confirmation object
in a background process from where the PreregistrationUser is created
temporarily leaves the PreregistrationUser in invalid state, and
results in 500's, and the user not immediately seeing the sent
invitation.  That the "invites" worker also wants to create the
Confirmation object means that "resending" an invite invalidates the
URL in the previous email, which can be confusing to the user.

Moving the Confirmation creation to the same transaction solves both
of these issues, and leaves the "invites" worker with nothing to do
but send the email; as such, we remove it entirely, and use the
existing "email_senders" worker to send the invites.  The volume of
invites is small enough that this will not affect other uses of that
worker.

Fixes: #21306
Fixes: #24275
2024-05-02 14:23:04 -07:00
Alex Vandiver 572fafd6b9 cron: Set environment variables to use Smokescreen, if configured. 2024-04-24 14:40:28 -07:00
Alex Vandiver 2df91c70ef puppet: Move rabbitmq monitoring into kandra/, where it is used from. 2024-04-24 14:40:28 -07:00
Alex Vandiver 6e981c18d5 puppet: Factor out cron job creation. 2024-04-24 14:40:28 -07:00
Alex Vandiver 49422c05c8 kandra: Add teleport services to "needsrestart" skip list.
These are often how one is connected to the node, and restarting them
would drop the connection one us actively using.
2024-04-19 09:55:17 -07:00
Alex Vandiver a4e6037dc4 kandra: Automate the second step of configuring database replicas.
If there is a replication primary configured, and no current database,
then we check all of the required secrets are in place, then pull down
the latest backup and trigger a PostgreSQL restart, which will pick up
downloading the remaining WAL logs to catch up, then start streaming
from the configured primary.
2024-04-17 17:31:49 -07:00
Alex Vandiver a8dbdd6d92 kandra: Ensure that pg_hba.conf is in place before starting PostgreSQL. 2024-04-17 17:31:49 -07:00
Alex Vandiver b0e3191434 puppet: Stop relying on "tidy" ordering, which ignores metaparams.
The `tidy` parameter is buggy, and ignores all ordering
metaparameters.  This is fixed in Puppet 7[^1], but it's helpful to
resolve it now.  Specifically, this fixes bugs with tidy running too
early, and deleting the old version of a package before its new
version is installed or symlinked, leaving a race condition if
anything tries to run the binary in this window.

This is mostly not a problem for Supervisor-managed processes, since
the binary is already running, and can continue to run if it is tidied
out from under the running process.  For stand-alone tools like wal-g,
which are run frequently by PostgreSQL, this may cause issues if
PostgreSQL tries to call them during a puppet run.

Remove all complicated uses of tidy, and replace them with an `exec`
which does the equivalent.  We also generate `file` resources for
binaries, making them easier (and clearer) to specify as dependencies.

[^1]: https://puppet.atlassian.net/browse/PUP-10688
2024-04-15 14:30:24 -07:00
Alex Vandiver 8d3120bf17 kandra: Remove zulip::static_asset_compiler include.
This was removed in 263212decf.
2024-04-12 15:00:33 -07:00
Alex Vandiver 87fb703e5b kandra: Log and timestamp autossh output.
By default, autossh writes to syslog; setting AUTOSSH_DEBUG is the
only way to produce output to STDERR.  Timestamp that and log that to
the logfile, making the logs perhaps useful.
2024-04-11 09:34:43 -07:00
Alex Vandiver f4d109c289 puppet: Fix arrow alignment. 2024-04-05 09:18:04 -07:00
Alex Vandiver 235e2eefc8 puppet: Switch from top-level fact variables to facts dict, again.
These were somehow missed in 57f8b48ff9.
2024-04-05 09:18:04 -07:00
Alex Vandiver eaef12ce3a kandra: Ensure that the nagios known_hosts exist before running autossh. 2024-04-03 11:38:29 -07:00
Alex Vandiver f9805c9e1f kandra: rabbitmqctl may not exist when applying the initial catalog.
puppet hard-fails if it can't find the binary to run in `$PATH`, so we
need to make the `unless` short-circuit to false if puppet itself is
not installed yet (as during initial installation).
2024-03-29 16:25:37 -07:00
Alex Vandiver 3c2efd236c kandra: Skip prometheus database user creation. 2024-03-29 11:34:18 -07:00
Alex Vandiver da243e9fb7 kandra: Force disks to be set up before kernels are changed. 2024-03-29 11:34:18 -07:00
Alex Vandiver e4883cc2dc kandra: Move PostgreSQL data mountpoint to /srv/data.
We store a lot of other things in `/srv` now, and moving them mid-puppet
confuses things significantly.
2024-03-29 11:34:18 -07:00
Alex Vandiver 93c6fd4746 kandra: Pull instance storage device from nvme information. 2024-03-29 11:34:18 -07:00
Alex Vandiver 64146b8704 kandra: Drop the munin port forward.
927660a7b6 removed the munin deploys; this stops tunneling port 4949
back to the nagios host.
2024-03-26 12:52:55 -07:00
Alex Vandiver 04f4e74709 kandra: Enable per-object metrics from rabbitmq.
These default to off, because in situations with thousands of queues,
consumers, and producers, they cause unreasonable overhead.  Our use
case has few enough queues that we do want to be able to inspect them
individually.

Enable per-object Prometheus metrics, per [1].

[1]: 78851828ec/deps/rabbitmq_prometheus (configuration)
2024-03-26 09:04:02 -07:00
Alex Vandiver e06b1794a8 kandra: Template and insert email credentials. 2024-03-21 16:14:44 -07:00
Alex Vandiver 75411b264e kandra: Remove unnecessary libapache2-mod-wsgi.
It does not exist in 22.04, and its last use was removed in
61666a9262.
2024-03-21 14:23:30 -07:00
Alex Vandiver 23504308fb puppet: Install the same version of postgres-client as the server.
We require a `pg_dump` whose version matches the version of the server
we are configured against (see 3a8b4b0205).  Installing the latest
`postgresql-client` does not guarantee that we have such a binary
present.
2024-03-21 12:34:34 -07:00
Anders Kaseorg 553f268b04 ruff: Fix RUF027 Possible f-string without an `f` prefix.
This is a preview rule, not yet enabled by default.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-03-01 09:30:04 -08:00
Anders Kaseorg 570f3dd447 python: Reformat with Ruff formatter.
https://docs.astral.sh/ruff/formatter/

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-02-29 17:07:16 -08:00
Alex Vandiver 802477ca1a memcached-exporter: Add metrics for max item size and max connections. 2024-02-27 10:16:00 -08:00
Alex Vandiver 11e0c448da memcached-exporter: Fix descriptions on read/written_bytes_total. 2024-02-27 10:16:00 -08:00
Alex Vandiver dfdaddea38 kandra: Add a memcached exporter using bmemcached. 2024-02-23 13:32:04 -08:00
Alex Vandiver fa6f4cc039 kandra: Skip hostnames which are undef.
If zulip.conf is not fully configured, do not error out if one of
the $hostname values is undef.
2024-02-07 10:42:12 -08:00
Alex Vandiver 3d63a87384 kandra: Puppet github.com keys to both root and zulip users.
We update to add the ecdsa-sha2-nistp256 key as well.
2024-02-07 10:42:12 -08:00
Alex Vandiver b1f899512a pack-local-script: Match mode of file outside of packing. 2024-02-07 10:42:12 -08:00
Alex Vandiver 927660a7b6 kandra: Remove munin. 2024-02-06 21:34:56 -08:00
Alex Vandiver 96d237d57e kandra: Pull in known_hosts to the nagios user.
This prevents failures when new hosts are initially added to the
configuration.
2024-02-07 00:02:40 -05:00
Alex Vandiver 2cbd9c2a47 kandra: Update default chrony configuration for 22.04. 2024-02-07 00:02:32 -05:00
Alex Vandiver b23d90ed62 puppet: Rename puppet/zulip_ops to puppet/kandra.
This makes for easier tab-completion, and also is a bit more explicit
about the expected consumer.
2024-02-06 17:56:27 -08:00