Commit Graph

1369 Commits

Author SHA1 Message Date
Anders Kaseorg e9ba9b0e0d zulip-ec2-configure-interfaces: Remove.
Our current EC2 systems don’t have an interface named ‘eth0’, and if
they did, this script would do nothing but crash with ImportError
because we have never installed boto.utils for Python 3.

(The message of commit 2a4d851a7c made
an effort to document for future researchers why this script should
not have been blindly converted to Python 3.  However, commit
2dc6d09c2a (#14278) was evidently
unresearched and untested.)

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-05-03 02:25:59 -07:00
Alex Vandiver d891b9590a puppet: Fix non-replicated PostgreSQL 10 and 11 configuration.
6f5ae8d13d removed the `$replication` variable from the
configurations of PostgreSQL 12 and higher, but left it in the
templates for PostgreSQL 10 and 11.  Because `undef != ''`,
deployments on PostgreSQL 10 and 11 started trying to push to S3
backups, regardless of if they were configured, leaving frequent log
messages like:

```
2022-04-30 12:45:47.805 UTC [626d24ec.1f8db0]: [107-1] LOG: archiver process (PID 2086106) exited with exit code 1
2022-04-30 12:45:49.680 UTC [626d24ee.1f8dc3]: [18-1] LOG: checkpoint complete: wrote 19 buffers (0.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=1.910 s, sync=0.022 s, total=1.950 s; sync files=16, longest=0.018 s, average=0.002 s; distance=49 kB, estimate=373 kB
/usr/bin/timeout: failed to run command "/usr/local/bin/env-wal-g": No such file or directory
2022-04-30 12:46:17.852 UTC [626d2f99.1fd4e9]: [1-1] FATAL: archive command failed with exit code 127
2022-04-30 12:46:17.852 UTC [626d2f99.1fd4e9]: [2-1] DETAIL: The failed archive command was: /usr/bin/timeout 10m /usr/local/bin/env-wal-g wal-push pg_wal/000000010000000300000080
```

Switch the PostgreSQL 10 and 11 configuration to check
`s3_backups_bucket`, like the other versions.
2022-05-02 16:46:10 -07:00
Anders Kaseorg 646a4d19a3 puppet: Remove quotes for enumerable values.
https://puppet.com/docs/puppet/7/style_guide.html#style_guide_module_design-quoting
“If a string is a value from an enumerable set of options, such as
present and absent, it SHOULD NOT be enclosed in quotes at all.”

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-29 22:06:46 -07:00
Alex Vandiver c97162e485 puppet: Check that certbot certs are in use before fixing them.
It is possible to have previously installed certbot, but switched back
to using self-signed certificates -- in which case renewing them using
certbot may fail.

Verify that the certificate is a symlink into certbot's output
directory before running `fix-standalone-certbot`.
2022-04-27 16:01:15 -07:00
Anders Kaseorg 098a514599 python: Use Python 3.8 shlex.join function.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-27 12:57:49 -07:00
Alex Vandiver 35db1ee435 puppet: Only include "app_service" section if there are apps.
This works around gravitational/teleport#12256, but also produces config
files that are slightly cleaner.
2022-04-26 16:36:13 -07:00
Anders Kaseorg a7e6cb7705 puppet: ‘supervisorctl stop all’ before restarting Supervisor.
This fixes a failure of the 3.4 upgrade test running on Ubuntu 20.04
with Supervisor 4.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-26 16:32:02 -07:00
Alex Vandiver e5548ecba0 puppet: Upgrade external dependencies. 2022-04-21 13:54:14 -07:00
Alex Vandiver 1151118cc8 puppet: Upgrade Grafana to 8.4.6. 2022-04-12 16:41:45 -07:00
Alex Vandiver 572443edc6 puppet: Remove memcached SASL workaround.
https://bugs.launchpad.net/ubuntu/+source/memcached/+bug/1878721 was
fixed and released in Focal in 2020-06-24.

We don't bother with an `ensure => absent` because leaving this
in-place for existing installs does no harm.
2022-04-08 14:59:45 -07:00
Anders Kaseorg 935cb605a5 puppet: Do not ensure Chrony is running.
Commit f6d27562fa (#21564) tried to
ensure Chrony is running, which fails in containers where Chrony
doesn’t have permission to update the host clock.

The Debian package should still attempt to start it, and Puppet should
still restart it when chrony.conf is modified.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-03-30 11:37:54 -07:00
Alex Vandiver f6d27562fa puppet: Configure chrony to use AWS-local NTP sources.
This prevents hosts from spewing traffic to random hosts across the
Internet.
2022-03-25 17:07:53 -07:00
Alex Vandiver 5e128e7cad puppet: Extract the wal-g configuration from the backups.
This will allow it to be used for monitoring, to check the state in S3
rather than just trusting the backups when they said they ran.
2022-03-25 17:05:30 -07:00
Alex Vandiver d7b59c86ce puppet: Build wal-g from source for aarch64.
Since wal-g does not provide binaries for aarch64, build them from
source.  While building them from source for arm64 would better ensure
that build process is tested, the build process takes 7min and 700M of
temp files, which is an unacceptable cost; we thus only build on
aarch64.

Since the wal-g build process uses submodules, which are not in the
Github export, we clone the full wal-g repository.  Because the
repository is relatively small, we clone it anew on each new version,
rather than attempt to manage the remotes.

Fixes #21070.
2022-03-22 15:02:35 -07:00
Alex Vandiver 4d4c320a07 puppet: Switch from ntp to chrony.
Chrony is the recommended time server for Ubuntu since 18.04[1], and
is the default on Redhat; it is more accurate, and has lower-memory
usage, than ntp, which is only getting best-effort security
maintenance.

See:
- https://wiki.ubuntu.com/BionicBeaver/ReleaseNotes#Chrony
- https://chrony.tuxfamily.org/comparison.html
- https://engineering.fb.com/2020/03/18/production-engineering/ntp-service/
2022-03-22 13:07:27 -07:00
Alex Vandiver a2c8be9cd5 puppet: Increase download timeout from 5m to 10m.
The default timeout for `exec` commands in Puppet is 5 minutes[1].  On
slow connections, this may not be sufficient to download larger
downloads, such as the ~135MB golang tarball.

Increase the timeout to 10 minutes; this is a minimum download speed
of is ~225kB/s.

Fixes #21449.

[1]: https://puppet.com/docs/puppet/5.5/types/exec.html#exec-attribute-timeout
2022-03-21 15:47:04 -07:00
Alex Vandiver 9e850b08f3 puppet: Fix the PostgreSQL paths to recovery.conf / standby.conf. 2022-03-20 16:16:04 -07:00
Alex Vandiver 1bd5723cd2 puppet: Add a prometheus monitor for tornado processes. 2022-03-20 16:12:11 -07:00
Alex Vandiver 6b91652d9a puppet: Open the grok_exporter port.
The complete grok_exporter configuration is not ready to be committed,
but this at least prepares the way for it.
2022-03-20 16:12:11 -07:00
Alex Vandiver 6558655fc6 puppet: Add rabbitmq prometheus plugin, and open the firewall. 2022-03-20 16:12:11 -07:00
Alex Vandiver bdd2f35d05 puppet: Switch czo to using zulip_ops::app_frontend_monitoring.
This was clearly intended in f61ac4a28d
but never executed.
2022-03-20 16:12:11 -07:00
Alex Vandiver 17699bea44 puppet: postgresql_backups is auto-included if s3_backups_bucket is set.
Since 6496d43148.
2022-03-20 16:12:11 -07:00
Alex Vandiver bedc7c2986 puppet: Smokescreen is now auto-included in standalone.
Since c33562f0a8.
2022-03-20 16:12:11 -07:00
Alex Vandiver 6489c832a3 puppet: Upgrade third-party package versions. 2022-03-17 11:44:05 -07:00
Alex Vandiver d17006da55 puppet: Support setting an `ssl_mode` verification level. 2022-03-15 12:43:50 -07:00
Alex Vandiver 253bef27f5 puppet: Support password-based PostgreSQL replication. 2022-03-15 12:43:50 -07:00
Sahil Batra f0606b34ad user_groups: Add cron job for adding users to full members system group.
This commit adds a cron job which runs every hour to add the users to
full members system group if user is promoted to a full member.

This should ensure that full member status is available no more than
an hour after configuration suggests it should be.
2022-03-14 18:53:47 -07:00
Alex Vandiver 6f5ae8d13d puppet: wal-g backups are required for replication.
Previously, it was possible to configure `wal-g` backups without
replication enabled; this resulted in only daily backups, not
streaming backups.  It was also possible to enable replication without
configuring the `wal-g` backups bucket; this simply failed to work.

Make `wal-g` backups always streaming, and warn loudly if replication
is enabled but `wal-g` is not configured.
2022-03-11 10:09:35 -08:00
Alex Vandiver 6496d43148 puppet: Only s3_backups_bucket is required for backups.
`s3_backups_key` / `s3_backups_secret_key` are optional, as the
permissions could come from the EC2 instance's role.
2022-03-11 10:09:35 -08:00
Alex Vandiver 19beed2709 puppet: Default s3_region to the current ec2 region. 2022-03-11 10:09:35 -08:00
Anders Kaseorg b3260bd610 docs: Use Debian and Ubuntu version numbers over development codenames.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-23 12:04:24 -08:00
Anders Kaseorg 1629d6bfb3 python: Reformat with Black 22 (stable).
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-18 18:03:13 -08:00
Alex Vandiver c656d933fa puppet: Switch from $::memorysize_mb to non-legacy $::memory. 2022-02-15 12:04:37 -08:00
Alex Vandiver f2f4462e71 puppet: Switch from $::fqdn to non-legacy $::networking. 2022-02-15 12:04:37 -08:00
Alex Vandiver bb4c0799cc puppet: Switch to the canonical case for $::os['family'].
The == operator in Puppet is case-insensitive for ASCII
characters[1], which is potentially surprising.  Switch to the
canonical case that `$::os['family']` returns.

[1] https://puppet.com/docs/puppet/5.5/lang_expressions.html#string-encoding-and-comparisons
2022-02-15 12:04:37 -08:00
Alex Vandiver d4eefbbeea puppet: Switch from $::osfamily to non-legacy $::os. 2022-02-15 12:04:37 -08:00
Alex Vandiver a787ebe0e2 puppet: Switch from $::architecture to non-legacy $::os. 2022-02-15 12:04:37 -08:00
Alex Vandiver d7e8733705 puppet: Use goarch for wal-g.
wal-g does not currently provide pre-built binaries for
arm64/aarch64 (see #21070) but if they begin to, it will likely be
with the goarch names.
2022-02-15 12:04:37 -08:00
Alex Vandiver abdbe4ca83 puppet: Use goarch for go-camo. 2022-02-15 12:04:37 -08:00
Alex Vandiver be2f2a5bde puppet: Use goarch for golang.
Fixes: #21051.
2022-02-15 12:04:37 -08:00
Alex Vandiver 788daa953b puppet: Factor out $::architecture case statement for golang. 2022-02-15 12:04:37 -08:00
Anders Kaseorg f6a701090c setup-apt-repos: Don’t install lsb_release.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-14 16:38:53 -08:00
Anders Kaseorg 45f4db9702 puppet: Remove unused $release_name.
It would confuse a future Debian 15.10 release with Ubuntu 15.10, it
relies on the legacy fact $::operatingsystemrelease, the modern fact
$::os provides this information without extra logic, and it’s unused
as of commit 03bffd3938.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-14 16:38:53 -08:00
Alex Vandiver 291c5e87b6 puppet: Upgrade prometheus to 2.33.1. 2022-02-09 20:32:24 -08:00
Alex Vandiver 2d538c2356 puppet: Upgrade grafana to 8.3.6. 2022-02-09 20:32:24 -08:00
Alex Vandiver f2e66c0b20 puppet: Upgrade go-camo to 2.4.0. 2022-02-09 20:32:24 -08:00
Alex Vandiver 51a516384d puppet: Upgrade golang to 1.17.6. 2022-02-09 20:32:24 -08:00
Alex Vandiver 48263a01dd puppet: Upgrade puppet libraries. 2022-02-09 20:32:24 -08:00
Alex Vandiver e032b38661 puppet: Fix typo in uwsgi exporter dependency. 2022-02-08 15:17:17 -08:00
Alex Vandiver b3900bec7e puppet: Upgrade Grafana to 8.3.5.
https://grafana.com/docs/grafana/latest/release-notes/release-notes-8-3-5/
2022-02-08 11:13:40 -08:00