Commit Graph

1669 Commits

Author SHA1 Message Date
Alex Vandiver f9805c9e1f kandra: rabbitmqctl may not exist when applying the initial catalog.
puppet hard-fails if it can't find the binary to run in `$PATH`, so we
need to make the `unless` short-circuit to false if puppet itself is
not installed yet (as during initial installation).
2024-03-29 16:25:37 -07:00
Alex Vandiver 3c2efd236c kandra: Skip prometheus database user creation. 2024-03-29 11:34:18 -07:00
Alex Vandiver da243e9fb7 kandra: Force disks to be set up before kernels are changed. 2024-03-29 11:34:18 -07:00
Alex Vandiver e4883cc2dc kandra: Move PostgreSQL data mountpoint to /srv/data.
We store a lot of other things in `/srv` now, and moving them mid-puppet
confuses things significantly.
2024-03-29 11:34:18 -07:00
Alex Vandiver 93c6fd4746 kandra: Pull instance storage device from nvme information. 2024-03-29 11:34:18 -07:00
Prakhar Pratyush 3add31496f puppet: Add a daily cron-job to send zulip update announcements.
A daily cron job is configured to run the
'send_zulip_update_announcements' management command.
2024-03-27 11:43:08 -07:00
Alex Vandiver 80ef38757a puppet: Do not include postgresql_common for all PostgreSQL clients.
It pulls in the server.
2024-03-26 22:12:29 -07:00
Alex Vandiver 64146b8704 kandra: Drop the munin port forward.
927660a7b6 removed the munin deploys; this stops tunneling port 4949
back to the nagios host.
2024-03-26 12:52:55 -07:00
Alex Vandiver 04f4e74709 kandra: Enable per-object metrics from rabbitmq.
These default to off, because in situations with thousands of queues,
consumers, and producers, they cause unreasonable overhead.  Our use
case has few enough queues that we do want to be able to inspect them
individually.

Enable per-object Prometheus metrics, per [1].

[1]: 78851828ec/deps/rabbitmq_prometheus (configuration)
2024-03-26 09:04:02 -07:00
Alex Vandiver 08dd8a41e6 puppet: Fix arrow alignment. 2024-03-25 12:57:25 -07:00
Alex Vandiver 57f8b48ff9 puppet: Switch from top-level fact variables to facts dict.
This is linted by more recent puppet-lint.
2024-03-25 12:57:25 -07:00
Alex Vandiver e06b1794a8 kandra: Template and insert email credentials. 2024-03-21 16:14:44 -07:00
Alex Vandiver 75411b264e kandra: Remove unnecessary libapache2-mod-wsgi.
It does not exist in 22.04, and its last use was removed in
61666a9262.
2024-03-21 14:23:30 -07:00
Alex Vandiver 23504308fb puppet: Install the same version of postgres-client as the server.
We require a `pg_dump` whose version matches the version of the server
we are configured against (see 3a8b4b0205).  Installing the latest
`postgresql-client` does not guarantee that we have such a binary
present.
2024-03-21 12:34:34 -07:00
Alex Vandiver c129b1779f wal-g: Add support for incremental backups.
This only defaults to on for local-disk backups, since they are more
disk-size-sensitive, and local accesses are quite cheap compared to
loading multiple incremental backups from S3.
2024-03-19 09:58:58 -07:00
Alex Vandiver d726f87035 puppet: Remove backups cron job if they are no longer enabled. 2024-03-19 09:58:58 -07:00
Alex Vandiver 44ff1c24df wal-g: Provide a to-local-disk backup option. 2024-03-19 09:58:58 -07:00
Alex Vandiver c13e3dee24 katex: Replace subprocess call with minimal external service.
Replace a separate call to subprocess, starting `node` from scratch,
with an optional standalone node Express service which performs the
rendering.  In benchmarking, this reduces the overhead of a KaTeX call
from 120ms to 2.8ms.  This is notable because enough calls to KaTeX in
a single message would previously time out the whole message
rendering.

The service is optional because he majority of deployments do not use
enough LaTeX to merit the additional memory usage (60Mb).

Fixes: #17425.
2024-03-15 15:34:12 -07:00
Anders Kaseorg 553f268b04 ruff: Fix RUF027 Possible f-string without an `f` prefix.
This is a preview rule, not yet enabled by default.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-03-01 09:30:04 -08:00
Anders Kaseorg 570f3dd447 python: Reformat with Ruff formatter.
https://docs.astral.sh/ruff/formatter/

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-02-29 17:07:16 -08:00
Alex Vandiver 802477ca1a memcached-exporter: Add metrics for max item size and max connections. 2024-02-27 10:16:00 -08:00
Alex Vandiver 11e0c448da memcached-exporter: Fix descriptions on read/written_bytes_total. 2024-02-27 10:16:00 -08:00
Alex Vandiver be59ef9c02 runtornado: Swap deferred reload events to the default.
This makes no immediate reloads the default for runtornado, matching
the production configuration, and changes the development incantation
to be the one to specify the departure from the norm, with
--immediate-reloads.
2024-02-26 22:29:19 -08:00
Alex Vandiver 8673f87907 puppet: Allow enabling "stats sizes" in memcached. 2024-02-23 13:32:04 -08:00
Alex Vandiver dfdaddea38 kandra: Add a memcached exporter using bmemcached. 2024-02-23 13:32:04 -08:00
David Rosa e891c57d0e docs: Update links to new "Reverse proxies" page. 2024-02-20 21:03:32 -08:00
Alex Vandiver ec6f64f7b0 tornado: Perform rolling client restarts after servers are restarted.
Decouple the sending of client restart events from the restarting of
the servers.  Restarts use the new Tornado restart-clients endpoint to
inject "restart" events into queues of clients which were loaded from
the previous Tornado process.  The rate is controlled by the
`application_server.client_restart_rate`, in clients per minute, or a
flag to `restart-clients` which overrides it.  Note that a web client
will also spread its restart over 5 minutes, so artificially-slow
client restarts are generally not very necessary.

Restarts of clients are deferred to until after post-deploy hooks are
run, such that the pre- and post- deploy hooks are around the actual
server restarts, even if pushing restart events to clients takes
significant time.
2024-02-15 15:42:50 -08:00
Alex Vandiver fa6f4cc039 kandra: Skip hostnames which are undef.
If zulip.conf is not fully configured, do not error out if one of
the $hostname values is undef.
2024-02-07 10:42:12 -08:00
Alex Vandiver 3d63a87384 kandra: Puppet github.com keys to both root and zulip users.
We update to add the ecdsa-sha2-nistp256 key as well.
2024-02-07 10:42:12 -08:00
Alex Vandiver b1f899512a pack-local-script: Match mode of file outside of packing. 2024-02-07 10:42:12 -08:00
Alex Vandiver 927660a7b6 kandra: Remove munin. 2024-02-06 21:34:56 -08:00
Alex Vandiver 96d237d57e kandra: Pull in known_hosts to the nagios user.
This prevents failures when new hosts are initially added to the
configuration.
2024-02-07 00:02:40 -05:00
Alex Vandiver 2cbd9c2a47 kandra: Update default chrony configuration for 22.04. 2024-02-07 00:02:32 -05:00
Alex Vandiver b23d90ed62 puppet: Rename puppet/zulip_ops to puppet/kandra.
This makes for easier tab-completion, and also is a bit more explicit
about the expected consumer.
2024-02-06 17:56:27 -08:00
Alex Vandiver ccc106dca3 puppet: Add redis host to ~redistunnel/.ssh/known_hosts. 2024-02-06 11:34:09 -08:00
Alex Vandiver aaf3eb514d puppet: Allow controlling known_hosts.
Note that this uses `ssh-keyscan` to write in the currently-observed
host fingerprint; if DNS or network is untrusted during initial puppet
apply, this can allow attackers to write their own host key, obviating
the utility of known_hosts.

We do not view this as a likely attack mechanism, since in our
deployment the network and DNS is almost certainly trusted, and if
not, the timing attack to catch only initial configuration is likely
impossible.
2024-02-06 11:34:09 -08:00
Alex Vandiver fe1fd30a58 puppet: Create redistunnel group.
This was left off of 1fe5e7c7da.
2024-02-05 12:15:15 -08:00
Alex Vandiver dc950235e5 puppet: Raise net.core.somaxconn on redis. 2024-02-02 17:54:02 -08:00
Alex Vandiver ec59b4fd78 puppet: Enact redis' kernel config suggestions. 2024-02-02 17:54:02 -08:00
Alex Vandiver 069f262e7b puppet: Inline all sysctl settings, and always check for containers. 2024-02-02 17:54:02 -08:00
Alex Vandiver fd69f5f8e2 puppet: Call systemd reload after adding or changing a teleport service. 2024-02-02 17:54:02 -08:00
Alex Vandiver 1ac6e24eaa puppet: Rename exec resource of systemd_daemon_reload.
This makes it easier to put as a "notify" resource.
2024-02-02 17:54:02 -08:00
Alex Vandiver 4947753bfd puppet: Remove unused systemd_daemon_reload.
572443edc6 removed the callsite that triggered the exec in
`zulip::systemd_daemon_reload`, making its inclusion and ordering via
`require` moot.

Remove the call.
2024-02-02 17:54:02 -08:00
Alex Vandiver 6e9b25d993 puppet: Add key to allow prod to write ccache on zmirrorp. 2024-02-02 17:24:12 -08:00
Alex Vandiver 96b65cbeab install-ssh-authorized-keys: Merge multiple authorized_keys secrets. 2024-02-02 17:24:12 -08:00
Alex Vandiver 1567275959 install-ssh-authorized-keys: Use --check rather than another argument. 2024-02-02 17:24:12 -08:00
Alex Vandiver afa24923e5 install-ssh-authorized-keys: Enable -o pipefail to catch AWS errors. 2024-02-02 17:24:12 -08:00
Alex Vandiver a0347e76cb install-ssh-keys: Enable -o pipefail to catch AWS errors. 2024-02-02 17:24:12 -08:00
Alex Vandiver f8a9edc382 install-ssh-*: Pass -a to rsync to preserve rights, owners, and times. 2024-02-02 17:24:12 -08:00
Alex Vandiver 0bd1e2b434 puppet: Rename and limit production key distribution. 2024-02-02 17:24:12 -08:00