Commit Graph

765 Commits

Author SHA1 Message Date
rht 008879eb22 puppet: Add postgresql.conf path for CentOS. 2019-01-03 14:36:43 -08:00
rht dce43e1a0e puppet: Add CentOS-version of pg data path at pg_backup_and_purge. 2019-01-03 14:36:43 -08:00
rht 59993aea80 puppet: Abstract out path to postgresql.conf. 2019-01-03 14:36:43 -08:00
rht c189409ffd puppet: Initialize yum_repository.pp to wrap setup-yum-repo. 2019-01-03 14:36:43 -08:00
rht 1b02fb6d6d puppet: Add CentOS packages to rabbit.pp. 2019-01-03 14:36:43 -08:00
rht a3d67e52fe puppet: Add CentOS packages to redis.pp. 2019-01-03 14:36:43 -08:00
rht 788128f05c puppet: Add CentOS packages to nginx.pp. 2019-01-03 14:36:43 -08:00
rht 1965cc1491 puppet: Add CentOS packages to base.pp. 2019-01-03 14:36:42 -08:00
rht 0caaed4e1f puppet: Add CentOS packages to apache_sso.pp. 2019-01-03 14:36:41 -08:00
rht 9fd18a2c7f puppet: Detect CentOS for release_name. 2019-01-03 14:35:09 -08:00
Fabian Tribrunner 013b169469 check_email_deliverer_backlog: Switch down from root if required.
This now checks if the user is zulip, and if not, switches to the
zulip user, making it possible to run it as root.

Significantly modified by tabbott to not break existing behavior.
2019-01-02 10:16:56 -08:00
Fabian Tribrunner 4612a84a67 check_email_deliverer_process: Fix typo in process name.
This nagios check has never worked, since it had the wrong process
name from the beginning.
2018-12-30 09:41:42 -08:00
Anders Kaseorg 392175d6e8 Use #!/usr/bin/env for bash shebangs.
/bin/sh and /usr/bin/env are the only two binaries that NixOS provides
at a fixed path (outside a buildFHSUserEnv sandbox).

This discussion was split from #11004.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-12-17 17:21:08 -08:00
Tim Abbott 2558f101af docs: Add documentation for `if False` mypy pattern in scripts.
This should help make it clear what's going on with these scripts.
2018-12-17 11:12:53 -08:00
Tim Abbott bce90a3340 lint: Add lint rule for scripts importing typing improperly.
This is a common bug that users might be tempated to introduce.

And also fix two instances of this bug that were present in our
codebase, including an important one in our upgrade code path.
2018-12-17 10:46:37 -08:00
rht 43bedc0909 provision: Use vendored pg_hba.conf on CentOS. 2018-12-16 13:21:54 -08:00
rht c9d54f7854 puppet: Remove vendored puppetlabs apt and stdlibs dependencies.
This commit works by vendoring the couple functions we still use from
puppetlabs stdlib (join and range), but removing the rest of the
puppetlabs codebase, and of course cleaning up our linter rules in the
process.

Fixes #7423.
2018-12-11 13:03:26 -08:00
rht d2aa81858c puppet/zulip_ops: Replace apt::source with setup-apt-repo-debathena.
Tweaked by tabbott to use a clearer name.
2018-12-11 13:02:56 -08:00
rht 97766102df puppet/zulip: Replace apt::source and apt::ppa with setup-apt-repo. 2018-12-11 13:01:26 -08:00
Tim Abbott b218c2a70e loadbalancer: Use same certbot cert for zulipstaging.com.
This is a simple configuration improvement.
2018-12-07 13:43:21 -08:00
Tim Abbott 467694c1fa nginx: Enable http2 in external nginx configuration.
This should be a nice performance improvement for browsers that
support it.

We can't yet enabled this in the Zulip on-premise nginx configuration,
because that still has to support Trusty.
2018-12-07 13:43:02 -08:00
Tim Abbott e609e10229 puppet: Fix missing dependency of tsearch-extras on apt repository.
This isn't super required, in that we add these repositories via
`setup-apt-repo` in any case, but the previous code was wrong and
worth fixing in any case.
2018-11-30 10:45:04 -08:00
Tim Abbott ededdc512b nginx: Fix missing API authentication configuration.
This fixes a bug where our API routes for uploaded files (where we
need to use a consistent URL between session auth and API auth) were
not properly configured to pass through the API authentication headers
(and otherwise provide REST endpoint settings).

In particular, this prevented the Zulip mobile apps from being able to
access authenticated image files using these URLs.
2018-11-16 11:25:54 -08:00
Tim Abbott f62050212b tornado: Fix supervisord configuration for multiple processes.
Apparently, we can use the process group naming style of having dashes
in the names without using the explicit nun_procs feature of
supervisord configuration.

The new configuration is perfectly satisfactory, so there's no real
reason to prefer the old approach.
2018-11-06 17:56:06 -08:00
Tim Abbott 5abf4dee92 nagios: Add new host groups for Tornado processes.
We also move all the existing Tornado monitoring rules to the
singletornado_frontends rule.
2018-11-06 16:33:18 -08:00
Tim Abbott 5f3b79c9e7 nagios: Fix tab-based whitespace. 2018-11-06 16:30:29 -08:00
Tim Abbott 5e7aa27c29 puppet: Add supervisord support for multiple tornado processes. 2018-11-02 16:55:33 -07:00
Tim Abbott a5acbd51c3 settings: Add new zulip.conf setting for number of Tornado processes.
This will eventually be used to support Tornado sharding; for now,
it's just used to contain the code intended to support that feature.
2018-11-02 16:47:26 -07:00
Tim Abbott dc7d44a245 puppet: Don't run calculate-first-visible-message-id on most systems.
This should only be run on systems that are running zilencer, because
the cron job is part of the zilencer project.
2018-10-30 11:40:24 -07:00
Tim Abbott a4df001cef check_queue_worker_errors: Add support for running unprivileged.
Previously, this script needed access to Django settings, which in
turn required access to /etc/zulip/zulip-secrets.conf.  Since that
isn't world-readable, this meant that this couldn't run as an
unprivileged `nagios` user.

Fix that by just hardcoding the appropriate path under /var/log/.
2018-10-18 15:03:17 -07:00
Tim Abbott 98d89b676d pg_backup_and_purge: Fix incorrect conversion to use python3 types.
When using the Python 3 typing style, Python scripts can't import from
typing inside an `if False` (in contrast, one needs to import inside
an `if False` to support the Python 3 syntax without needing
python-typing installed).  So this was just incorrectly half-converted
from the Python 2 style to the Python 3 style.
2018-10-16 11:12:52 -07:00
Tim Abbott 2c7f9ce0fc puppet: Fix puppet-lint warnings in various manifests.
Apparently, `puppet-lint` on Ubuntu trusty throws warnings for certain
quoting patterns that are OK in modern `puppet-lint`.  I believe the
old Zulip code was actually correct (i.e. the old `puppet-lint`
implementation was the problem), but it seems worth changing anyway to
suppress the warnings.

We also exclude more of puppet-apt from linting, since it's
third-party code.
2018-08-28 13:46:31 -07:00
Tim Abbott b53a712856 nginx: Update configuration for using certbot certs everywhere. 2018-08-22 11:59:15 -07:00
Abhilash Verma 0e2322a322 logging: Show timestamp in UTC in non-django production scripts.
Done in pair programming with @aero31aero.

Fixes #9678.
2018-08-20 12:52:40 -07:00
Tim Abbott 5021f7b76f puppet: Fix accidental conflict on apache2 package.
Apparently, the work to force installation of the Python 3 version of
mod_wsgi was buggy and tried to force uninstall apache2.

Fixes #10318.
2018-08-16 14:15:35 -07:00
Tim Abbott 90828297e4 puppet-lint: Enforce double_quoted_strings check.
This makes our puppet codebase more consistent by using single-quoted
strings consistently.
2018-08-13 12:31:19 -07:00
Tim Abbott d0b51b70f4 puppet-lint: Enforce 2sp_soft_tables puppet-lint check.
This cleans up the puppet codebase's whitespace formatting to be more
consistent.
2018-08-13 12:31:16 -07:00
Tim Abbott b26e0a957d puppet-lint: Enforce arrow_alignment check.
This fixes all exceptions in our puppet codebase to this lint rule.
2018-08-13 12:30:57 -07:00
Tim Abbott 054c07b585 mypy: Fix run types in pg_backup_and_purge. 2018-08-09 12:57:53 -07:00
Tim Abbott db1f706d09 pg_backup_and_purge: Fix buggy recovery status parsing.
This was converted to Python 3 incorrectly, in a way that actually
completely broke the script (the .decode() that this adds is critical,
since 'f' != b'f').

We fix this, and also add an assert that makes the parsing code
safer against future refactors.
2018-08-09 11:48:48 -07:00
Aditya Bansal 5bfe24beef puppet-lint: Fix an error with defined type safepackage in base.pp.
We fix "ERROR: safepackage not in autoload module layout" error
which was caused by a defined type "safepackage" definitation
lying in the wrong place. We refactor to create the defined type
according to puppet guidelines. Link below:
https://docs.puppet.com/puppet/2.7/lang_defined_types.html
2018-08-07 10:03:40 -07:00
Aditya Bansal 710d4507de puppet-lint: Fix lines longer than 140 characters lint warnings.
We fix these by adding ignore statements in a bunch of files
where this error popped up. We target only specific lines using
the ignore statements and not the entire files.
2018-08-07 10:03:40 -07:00
Anders Kaseorg edfd5ef992 setup_disks.sh: Fix shellcheck warnings.
In puppet/zulip_ops/files/postgresql/setup_disks.sh line 15:
array_name=$(mdadm --examine --scan | sed 's/.*name=//')
^-- SC2034: array_name appears unused. Verify use (or export if used externally).

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-08-03 09:15:26 -07:00
Anders Kaseorg 5a0fecc2d5 munin_plugins: Fix shellcheck warnings.
In puppet/zulip_ops/files/munin-plugins/rabbitmq_connections line 66:
echo "connections.value $(HOME=$HOME rabbitmqctl list_connections | grep -v "^Listing" | grep -v "done.$" | wc -l)"
                                                                                         ^-- SC2126: Consider using grep -c instead of grep|wc -l.

In puppet/zulip_ops/files/munin-plugins/rabbitmq_consumers line 32:
VHOST=${vhost:-"/"}
^-- SC2034: VHOST appears unused. Verify use (or export if used externally).

In puppet/zulip_ops/files/munin-plugins/rabbitmq_messages line 32:
VHOST=${vhost:-"/"}
^-- SC2034: VHOST appears unused. Verify use (or export if used externally).

In puppet/zulip_ops/files/munin-plugins/rabbitmq_messages_unacknowledged line 32:
VHOST=${vhost:-"/"}
^-- SC2034: VHOST appears unused. Verify use (or export if used externally).

In puppet/zulip_ops/files/munin-plugins/rabbitmq_messages_uncommitted line 32:
VHOST=${vhost:-"/"}
^-- SC2034: VHOST appears unused. Verify use (or export if used externally).

In puppet/zulip_ops/files/munin-plugins/rabbitmq_queue_memory line 32:
VHOST=${vhost:-"/"}
^-- SC2034: VHOST appears unused. Verify use (or export if used externally).

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-08-03 09:15:08 -07:00
Anders Kaseorg 16ed5d5e79 env-wal-e: Fix shellcheck warnings.
In puppet/zulip/files/postgresql/env-wal-e line 6:
export AWS_ACCESS_KEY_ID=$(crudini --get "$ZULIP_SECRETS_CONF" secrets s3_backups_key)
       ^-- SC2155: Declare and assign separately to avoid masking return values.

In puppet/zulip/files/postgresql/env-wal-e line 7:
export AWS_SECRET_ACCESS_KEY=$(crudini --get "$ZULIP_SECRETS_CONF" secrets s3_backups_secret_key)
       ^-- SC2155: Declare and assign separately to avoid masking return values.

In puppet/zulip/files/postgresql/env-wal-e line 9:
if [ $? -ne 0 ]; then
     ^-- SC2181: Check exit code directly with e.g. 'if mycmd;', not indirectly with $?.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-08-03 09:13:07 -07:00
Anders Kaseorg d5bf4eed9a check_worker_memory: Fix shellcheck warnings.
In puppet/zulip/files/nagios_plugins/zulip_app_frontend/check_worker_memory line 12:
ps -o vsize,size,pid,user,command --sort -vsize $processes > "$datafile"
                                                ^-- SC2086: Double quote to prevent globbing and word splitting.

In puppet/zulip/files/nagios_plugins/zulip_app_frontend/check_worker_memory line 14:
top_worker=$(cat "$datafile" | head -n2 | tail -n1)
                 ^-- SC2002: Useless cat. Consider 'cmd < file | ..' or 'cmd file | ..' instead.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-08-03 08:42:08 -07:00
Anders Kaseorg eb4855b77b check_email_deliverer_process: Fix shellcheck warnings.
In puppet/zulip/files/nagios_plugins/zulip_app_frontend/check_email_deliverer_process line 16:
elif [ "$(echo "$STATUS" | egrep '(STOPPED)|(STARTING)|(BACKOFF)|(STOPPING)|(EXITED)|(FATAL)|(UNKNOWN)$')" ]
       ^-- SC2143: Use egrep -q instead of comparing output with [ -n .. ].
                           ^-- SC2196: egrep is non-standard and deprecated. Use grep -E instead.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-08-03 08:42:08 -07:00
Anders Kaseorg e3253a7a1b check_email_deliverer_backlog: Fix shellcheck warnings.
In puppet/zulip/files/nagios_plugins/zulip_app_frontend/check_email_deliverer_backlog line 8:
cd /home/zulip/deployments/current
^-- SC2164: Use 'cd ... || exit' or 'cd ... || return' in case cd fails.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-08-03 08:42:08 -07:00
Anders Kaseorg 510c97d861 scripts: Use shell quoting when displaying commands to be run.
This way, commands with arguments containing whitespace or
metacharacters are unambiguously readable.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-07-30 22:39:08 -07:00
Tim Abbott 7853254df7 puppet: Run thumbor by default on voyager systems.
With this change, all that one needs to do to start using thumbor in
production is to set the `THUMBOR_URL` setting.

Since without THUMBOR_URL enabled, the thumbor service doesn't
actually do anything, this is pretty safe.
2018-07-30 16:16:52 -07:00