Commit Graph

72 Commits

Author SHA1 Message Date
Alex Vandiver 044ccdb334 chat.zulip.org: Enable Sentry hook. 2023-02-14 17:20:35 -05:00
Alex Vandiver e8123dfeea puppet: Match the `x` bits on directories to what puppet actually does.
Puppet _always_ sets the `+x` bit on directories if they have the `r`
bit set for that slot[^1]:

> When specifying numeric permissions for directories, Puppet sets the
> search permission wherever the read permission is set.

As such, for instance, `0640` is actually applied as `0750`.

Fix what we "want" to match what puppet is applying, by adding the `x`
bit.  In none of these cases did we actually intend the directory to
not be executable.

[1] https://www.puppet.com/docs/puppet/5.5/types/file.html#file-attribute-mode
2023-01-26 15:06:01 -08:00
Alex Vandiver d0de66b273 puppet: Remove "ensure => absent" rules which have all been applied. 2023-01-24 13:05:24 -08:00
Alex Vandiver 42f84a8cc7 puppet: Use existing autossh tunnels as OpenSSH "master" sockets.
A number of autossh connections are already left open for
port-forwarding Munin ports; autossh starts the connections and
ensures that they are automatically restarted if they are severed.

However, this represents a missed opportunity.  Nagios's monitoring
uses a large number of SSH connections to the remote hosts to run
commands on them; each of these connections requires doing a complete
SSH handshake and authentication, which can have non-trivial network
latency, particularly for hosts which may be located far away, in a
network topology sense (up to 1s for a no-op command!).

Use OpenSSH's ability to multiplex multiple connections over a single
socket, to reuse the already-established connection.  We leave an
explicit `ControlMaster no` in the general configuration, and not
`auto`, as we do not wish any of the short-lived Nagios connections to
get promoted to being a control socket if the autossh is not running
for some reason.

We enable protocol-level keepalives, to give a better chance of the
socket being kept open.
2022-11-01 22:24:40 -07:00
Alex Vandiver 9bd88a93e2 puppet: Tell needrestart to not default to restarting core services.
The `needrestart` tool added in 22.04 is useful in terms of listing
which services may need to be restarted to pick up updated libraries.
However, it prompts about the current state of services needing
restart for *every* subsequent `apt-get upgrade`, and defaulting core
services to restarting requires carefully manually excluding them
every time, at risk of causing an unscheduled outage.

Build a list of default-off services based on the list in
unattended-upgrades.
2022-07-19 17:51:18 -07:00
Alex Vandiver 775a084d0f nagios: Add a catchall "other" set. 2022-06-22 12:07:38 -07:00
Alex Vandiver 33472ee9ff nagios: Remove unused stats host set. 2022-06-22 12:07:38 -07:00
Alex Vandiver 7f6a77da31 puppet: Add a redis exporter. 2022-05-03 17:13:44 -07:00
Anders Kaseorg e9ba9b0e0d zulip-ec2-configure-interfaces: Remove.
Our current EC2 systems don’t have an interface named ‘eth0’, and if
they did, this script would do nothing but crash with ImportError
because we have never installed boto.utils for Python 3.

(The message of commit 2a4d851a7c made
an effort to document for future researchers why this script should
not have been blindly converted to Python 3.  However, commit
2dc6d09c2a (#14278) was evidently
unresearched and untested.)

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-05-03 02:25:59 -07:00
Anders Kaseorg 646a4d19a3 puppet: Remove quotes for enumerable values.
https://puppet.com/docs/puppet/7/style_guide.html#style_guide_module_design-quoting
“If a string is a value from an enumerable set of options, such as
present and absent, it SHOULD NOT be enclosed in quotes at all.”

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-29 22:06:46 -07:00
Alex Vandiver f6d27562fa puppet: Configure chrony to use AWS-local NTP sources.
This prevents hosts from spewing traffic to random hosts across the
Internet.
2022-03-25 17:07:53 -07:00
Alex Vandiver bdd2f35d05 puppet: Switch czo to using zulip_ops::app_frontend_monitoring.
This was clearly intended in f61ac4a28d
but never executed.
2022-03-20 16:12:11 -07:00
Alex Vandiver 17699bea44 puppet: postgresql_backups is auto-included if s3_backups_bucket is set.
Since 6496d43148.
2022-03-20 16:12:11 -07:00
Alex Vandiver bedc7c2986 puppet: Smokescreen is now auto-included in standalone.
Since c33562f0a8.
2022-03-20 16:12:11 -07:00
Alex Vandiver 788daa953b puppet: Factor out $::architecture case statement for golang. 2022-02-15 12:04:37 -08:00
Alex Vandiver 6bc5849ea8 puppet: Remove now-unused debathena apt repository. 2022-01-18 14:13:28 -08:00
Alex Vandiver b3f07cc98d puppet: Replace debathena zephyr package with equivalent puppet file. 2022-01-18 14:13:28 -08:00
Alex Vandiver a6d7539571 puppet: Replace debathena krb5 package with equivalent puppet file. 2022-01-18 14:13:28 -08:00
Alex Vandiver 75224ea5de puppet: python-dev is now purely virtual; install python2.7-dev. 2022-01-18 14:13:28 -08:00
Alex Vandiver 0b8a6a51b8 puppet: Remove all parts of AWS kernels.
Otherwise, we just uninstall the meta-package, and still restart into
the installed AWS kernel.
2022-01-12 15:52:19 -08:00
Alex Vandiver 1e80b844f4 puppet: Disable apparmor profile for msmtp.
As the nagios user, we want to read the msmtp configuration from
~nagios, which apparmor's profile does not allow msmtp to do.
2022-01-11 09:38:31 -08:00
Alex Vandiver 3c95ad82c6 puppet: Upgrade to nagios4.
This updates the puppeted nagios configuration file for the Nagios4
defaults.
2022-01-11 09:38:31 -08:00
Alex Vandiver 4a95967a33 puppet: Gather uwsgi stats from chat.zulip.org. 2022-01-03 21:26:57 -08:00
Anders Kaseorg 82748d45d8 install-yarn: Use test -ef in case /srv is a symlink.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-12-30 13:42:07 -08:00
Alex Vandiver c094867a74 puppet: Add aarch64 build hashes to external dependencies.
wal-g does not ship aarch64 binaries, currently; the compilation
process([1]) is somewhat complicated, so we defer the decision about
how to support wal-g for aarch64 until a later date.

[1]: https://github.com/wal-g/wal-g/blob/master/docs/PostgreSQL.md#installing
2021-12-29 16:35:15 -08:00
Alex Vandiver f166f9f7d6 puppet: Centralize versions and sha256 hashes of external dependencies.
This will make it easier to update versions of these dependencies.
2021-12-29 16:35:15 -08:00
Alex Vandiver 57662689a9 puppet: Provide a constant homedir for grafana user.
The homedir of a user cannot be changed if any processes are running
as them, so having it change over time as upgrades happen will break
puppet application, as the old grafana process under supervisor will
effectively lock changes to the user's homedir.

Unfortunately, that means that this change will thus fail to
puppet-apply unless `supervisorctl stop grafana` is run first, but
there's no way around that.
2021-12-29 16:35:15 -08:00
Alex Vandiver 6e55e52694 puppet: Pull out grafana $data_dir. 2021-12-29 16:35:15 -08:00
Alex Vandiver 1e4e6a09af puppet: Stop making resources for external binaries and directories.
In the event that extracting doesn't produce the binary we expected it
to, all this will do is create an _empty_ file where we expect the
binary to be.  This will likely muddle debugging.

Since the only reason the resourfce was made in the first place was to
make dependencies clear, switch to depending on the External_Dep
itself, when such a dependency is needed.
2021-12-29 16:35:15 -08:00
Alex Vandiver 3c163a7d5e puppet: Move slash out of $dir by convention. 2021-12-29 16:35:15 -08:00
Alex Vandiver bb5a2c8138 puppet: Move prometheus to external_dep. 2021-12-29 16:35:15 -08:00
Alex Vandiver e4b23daad7 puppet: Upgrade to Grafana 8.3.2, for CVE-2021-43813. 2021-12-10 14:00:11 -08:00
Alex Vandiver 291f688678 puppet: Use zulip::external_dep for grafana, template config.
Templating the config ensures that the service is restarted when it is
upgraded.
2021-12-08 20:58:10 -08:00
Alex Vandiver 3eae429ab4 puppet: Upgrade Grafana to 8.3.1, for CVE-2021-43798. 2021-12-08 20:58:10 -08:00
Alex Vandiver 7db146d0a9 puppet: Do not assume amd64 architecture. 2021-12-06 11:08:50 -08:00
Alex Vandiver 1806e0f45e puppet: Remove zulip.org configuration. 2021-08-26 17:21:31 -07:00
Alex Vandiver e46e862f2b puppet: Add a bare-bones zulipbot profile.
This sets up the firewalls appropriate for zulipbot, but does not
automate any of the configuration of zulipbot itself.
2021-08-24 16:05:58 -07:00
Alex Vandiver 5857dcd9b4 puppet: Configure ip6tables in parallel to ipv4.
Previously, IPv6 firewalls were left at the default all-open.

Configure IPv6 equivalently to IPv4.
2021-08-24 16:05:46 -07:00
Alex Vandiver 845509a9ec puppet: Be explicit that existing iptables are only ipv4. 2021-08-24 16:05:46 -07:00
Alex Vandiver e6bae4f1dd puppet: Remove zulip::nagios class.
93f62b999e removed the last file in
puppet/zulip/files/nagios_plugins/zulip_nagios_server, which means the
singular rule in zulip::nagios no longer applies cleanly.

Remove the `zulip::nagios` class, as it is no longer needed.
2021-07-09 17:29:41 -07:00
Anders Kaseorg 93f62b999e nagios: Replace check_website_response with standard check_http plugin.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-07-09 16:47:03 -07:00
Alex Vandiver 6c72698df2 puppet: Move zulip_ops supervisor config into /etc/supervisor/conf.d/zulip/.
This is similar cleanup to 3ab9b31d2f, but only affects zulip_ops
services; it serves to ensure that any of these services which are no
longer enabled are automatically removed from supervisor.

Note that this will cause a supervisor restart on all affected hosts,
which will restart all supervisor services.
2021-06-14 17:12:59 -07:00
Alex Vandiver c90ff80084 puppet: Bump grafana version to 8.0.1.
Most notably, this fixes an annoying bug with CloudWatch metrics being
repeated in graphs.
2021-06-10 15:49:08 -07:00
Alex Vandiver d905eb6131 puppet: Add a database teleport server.
Host-based md5 auth for 127.0.0.1 must be removed from `pg_hba.conf`,
otherwise password authentication is preferred over certificate-based
authentication for localhost.
2021-06-08 22:21:21 -07:00
Alex Vandiver 100a899d5d puppet: Add grafana server. 2021-06-08 22:21:00 -07:00
Alex Vandiver 459f37f041 puppet: Add prometheus server. 2021-06-08 22:21:00 -07:00
Alex Vandiver 19fb58e845 puppet: Add prometheus node exporter. 2021-06-08 22:21:00 -07:00
Alex Vandiver 61b6fc865c puppet: Add a label to teleport applications, to allow RBAC.
Roles can only grant or deny access based on labels; set one based on
the application name.
2021-06-08 15:19:04 -07:00
Alex Vandiver 359f37389a puppet: Remove in-nagios auth restrictions.
51b985b40d made nagios only accessible from localhost, or as proxied
via teleport.  Remove the HTTP-level auth requirements.
2021-06-07 16:17:45 -07:00
Alex Vandiver 2352fac6b5 puppet: Fix indentation. 2021-06-02 18:38:38 -07:00