Commit Graph

1301 Commits

Author SHA1 Message Date
Alex Vandiver 372bba4a8e puppet: Stop creating a /home/zulip/logs.
This was last really used in d7a3570c7e, in 2013, when it was
`/home/humbug/logs`.

Repoint the one obscure piece of tooling that writes there, and remove
the places that created it.
2023-01-26 15:06:01 -08:00
Anders Kaseorg 4eda29bd86 ruff: Fix RUF005 Consider spread instead of concatenation.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-26 10:16:30 -08:00
Anders Kaseorg 91b22cb1af ruff: Fix EXE001 Shebang is present but file is not executable.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-26 10:16:30 -08:00
Anders Kaseorg ff1971f5ad ruff: Fix SIM105 Use `contextlib.suppress` instead of try-except-pass.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-23 11:18:36 -08:00
Lalit 112df91fbd provision: Do not require that no other yarn precedes us in $PATH.
`check_version` in `install-yarn` had the rather careful check that
the yarn it installed into `/usr/bin/yarn` was the yarn which was
first in the user's `$PATH`.  This caused problems when the user had a
pre-existing `/usr/local/bin/yarn`; however, those problems are
limited to the `install-yarn` script itself, since the nearly all
calls to yarn from Zulip's code already hardcode the `/srv/zulip-yarn`
location, and do not depend on what is in `$PATH`.

Remove the checks in `install-yarn` that depend on the local `$PATH`,
and stop installing our `yarn` into it.  We also adjust the two
callsites which did not specify the full path to `yarn`, so use
`/srv/zulip-yarn`.

Fixes: #23993

Co-authored-by: Alex Vandiver <alexmv@zulip.com>
2023-01-19 17:51:52 -05:00
Alex Vandiver a1151aef8b create-database: Hide harmless "non-existant database" warnings.
During installation on a new host, `create-database` attempts to
verify that there isn't a bunch of data already in the database which
is it about to drop and recreate.  In the most common case, this
statement emits a scary-looking warning, since the database does not
exist yet:

```
+ /home/zulip/deployments/current/scripts/setup/create-database
+ POSTGRES_USER=postgres
++ crudini --get /etc/zulip/zulip.conf postgresql database_name
++ echo zulip
+ DATABASE_NAME=zulip
++ crudini --get /etc/zulip/zulip.conf postgresql database_user
++ echo zulip
+ DATABASE_USER=zulip
++ cd /
++ su postgres -c 'psql -v ON_ERROR_STOP=1 -Atc '\''SELECT COUNT(*) FROM zulip.zerver_message;'\'' zulip'
psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL:  database "zulip" does not exist
```

Because we are attempting to gracefully handle the case where the
database does not exist yet, we also continue (and drop the database)
in other, less expected cases -- for instance, if database contains a
schema we do not expect.

Explicitly check for the database existence first, and once we verify
that, allow any further failures in the `SELECT COUNT(*)` to abort
`create-database`.  This serves the dual purpose of hiding the "FATAL"
error for the common case when the database does not exist, as well as
preventing dropping the database if anything else goes awry.
2023-01-05 11:39:23 -08:00
Anders Kaseorg 2afdb46095 ruff: Enable new lints DTZ, ISC, PIE, PLW, Q, S, SIM.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-04 16:25:07 -08:00
Anders Kaseorg e1ed44907b ruff: Fix SIM118 Use `key in dict` instead of `key in dict.keys()`.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-04 16:25:07 -08:00
Anders Kaseorg a6442288cf ruff: Fix PLW0120 Else clause on loop without a break statement.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-04 16:25:07 -08:00
Anders Kaseorg 986bad8f90 install-shellcheck: Upgrade ShellCheck to 0.9.0.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-04 14:01:34 -08:00
Alex Vandiver 62562eedf7 upgrade-postgresql: Ensure the new PostgreSQL is running.
If a previous attempt at an upgrade failed for some reason, the new
PostgreSQL may be installed, and the conversion will succeed, but the
new PostgreSQL daemon will not be running (Puppet does not force it to
start).  This causes the upgrade to fail when analyzing statistics,
since the daemon isn't running.

Explicitly start the new PostgreSQL; this does nothing in most cases,
but will provider better resiliency when recovering from previous
partial upgrades.
2023-01-03 11:10:49 -08:00
Anders Kaseorg bd884c88ed Fix typos caught by typos.
https://github.com/crate-ci/typos

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-03 11:09:50 -08:00
Anders Kaseorg 89e954451b install-node: Upgrade Node.js from 18.10.0 to 18.12.1.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-12-07 09:55:05 -08:00
Alex Vandiver 2bbeef301a upgrade-postgresql: Force a known locale.
Some terminals (e.g. ssh from OS X) set an invalid locale, which
causes the `pg_upgradecluster` call late in the upgrade to fail.

Force a known locale, for consistency.  This mirrors the settings in
upgrade-zulip-stage-2, set in 11ab545f3b, and its subsequent
cleanups in 64c608a51a, ee0f4ca330, and eda9ce2364.
2022-12-06 09:47:30 -08:00
Anders Kaseorg f3f5dfb5aa ruff: Fix RUF004 exit() is only available in the interpreter.
‘exit’ is pulled in for the interactive interpreter as a side effect
of the site module; this can be disabled with python -S and shouldn’t
be relied on.

Also, use the NoReturn type where appropriate.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-12-04 22:11:24 -08:00
Alex Vandiver 89f20140c0 wal-g: Use pre-built aarch64 binary, rather than building from source.
Starting with wal-g 2.0.1, they provide `aarch64` assets[^1].
Effectively revert d7b59c86ce, and use
the pre-built binary for `aarch64` rather than spend a bunch of space
and time having to build it from source.

[^1]: https://github.com/wal-g/wal-g/releases/tag/v2.0.1
2022-11-30 12:13:47 -05:00
Anders Kaseorg 0258fba345 ruff: Fix N811 constant imported as non-constant.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-16 09:29:11 -08:00
Anders Kaseorg e5c26eeb86 tornado: Support sharding by user ID.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-15 17:27:01 -08:00
Anders Kaseorg fc9f23a85f scripts: Pass --retry 3 to curl.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-08 08:07:36 -08:00
Anders Kaseorg 7021852627 install-node: Silence expected “node: command not found” on first run.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-03 12:11:08 -07:00
Anders Kaseorg 9d2d6c8eb7 ruff: Fix M001 Unused `noqa` directive.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-03 12:10:15 -07:00
Alex Vandiver 6662a3bac4 teleport: Switch to the new apt host for Teleport.
The apt.releases.teleport.dev repository is deprecated as of the
release of Teleport 11, and has been replaced with
deb.releases.teleport.dev[1].

[1]: https://goteleport.com/docs/changelog/#deprecated-old-debrpm-repositories
2022-10-28 16:52:54 -07:00
Alex Vandiver f5f6a3789b restart-server: Default to running config and database checks.
If there is a syntax error in `settings.py`, `restart-server` should
provide a reasonable message about this.  It did so prior to
af08bcdb3f, becausde any invocation `./manage.py` without
`--skip-checks` will verify `settings.py`, among several other checks.
After af08bcdb3f, there are no `./manage.py` calls in most restarts,
which fa77be6e6c took further.

Add an explicit `./manage.py check` in the default case.
upgrade-zulip-stage-2 overrides this by passing `--skip-checks`, for
performance.  This also means that `upgrade-zulip-from-git` itself
picks up the same `--skip-checks` flag, since it inherits the same
flag parsing, though that is perhaps of dubious utility.
2022-10-14 13:10:46 -07:00
Anders Kaseorg afccebc1ee install-node: Upgrade Node.js from 16.17.0 to 18.10.0.
Although Node.js 18 is not the active LTS release for another 3 weeks,
the Node.js 16 end-of-life date was moved forward to September 2023,
(https://nodejs.org/en/blog/announcements/nodejs16-eol/), so it seems
prudent to switch now.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-10-11 10:50:57 -07:00
Anders Kaseorg 11a86ec328 install: Remove PostgreSQL 10 support.
PostgreSQL 10 reaches its upstream end of life in November, and is not
supported by Django 4.1.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-10-06 15:59:07 -07:00
Anders Kaseorg b267b17677 python: Use ‘not in’ for more negated membership tests.
Fixes “E713 Test for membership should be `not in`” found by ruff (now
that I’ve fixed it not to ignore scripts lacking a .py extension).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-09-26 12:09:46 -07:00
Anders Kaseorg 83bd709562 Revert "zulip-puppet-apply: Work around broken Puppet on Ubuntu 22.04."
This reverts commit 25c87cc7da (#21328).

This upstream Ubuntu bug was fixed.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-09-22 15:18:15 -07:00
Anders Kaseorg 403837e52d python: Use ‘not in’ for negated membership tests
Fixes “E713 Test for membership should be `not in`” found by
ruff (https://github.com/charliermarsh/ruff).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-09-17 11:48:33 -07:00
Anders Kaseorg 987ab741f9 sharding: Support Tornado sharding by regexes.
One should now be able to configure a regex by appending _regex to the
port number:

[tornado_sharding]
9802_regex = ^[l-p].*\.zulipchat\.com$

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-09-15 16:07:50 -07:00
Anders Kaseorg 7666ff603d sharding: Configure Tornado sharding with nginx map.
https://nginx.org/en/docs/http/ngx_http_map_module.html

Since Puppet doesn’t manage the contents of nginx_sharding.conf after
its initial creation, it needs to be renamed so we can give it
different default contents.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-09-15 16:07:50 -07:00
Anders Kaseorg ea6f18bb46 refresh-sharding-and-restart: Quote to prevent shell glob expansion.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-09-14 09:35:12 -07:00
Anders Kaseorg 5e4cec56cb install-node: Upgrade Node.js from 16.16.0 to 16.17.0.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-09-06 15:02:29 -07:00
Anders Kaseorg 5d77d50423 scripts: Help mypy resolve the psycopg2.connect overload.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-08-30 17:36:21 -07:00
Zixuan James Li 059d0e7be8 settings: Make SHARED_SECRET mandatory.
This implements get_mandatory_secret that ensures SHARED_SECRET is
set when we hit zerver.decorator.authenticate_notify. To avoid getting
ZulipSettingsError when setting up the secrets, we set an environment
variable DISABLE_MANDATORY_SECRET_CHECK to skip the check and default
its value to an empty string.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-08-25 12:13:03 -07:00
Anders Kaseorg 7da1586cbf install-node: Upgrade Node.js from 16.15.1 to 16.16.0.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-08-04 13:51:51 -07:00
Anders Kaseorg 443b974b3e python: Apply changes from pyupgrade.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-07-20 22:33:28 -07:00
Alex Vandiver 526a04b4e6 restore-backup: Provide flags to leave settings.py and zulip.conf as-is. 2022-07-20 12:35:51 -07:00
Alex Vandiver d8ae270899 restore-backup: Only extract /etc/zulip once.
This is already handled in the earlier block; there is no need to
extract it twice.
2022-07-19 17:56:40 -07:00
Alex Vandiver 1b57669771 restore-backup: Switch to run() to check exit codes. 2022-07-19 17:56:40 -07:00
Alex Vandiver c71c6187ea restore-backup: Ensure it is run as root. 2022-07-19 17:56:40 -07:00
Anders Kaseorg 81892df176 requirements: Upgrade to Django 4.0.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-07-13 16:07:17 -07:00
Anders Kaseorg 463fe515b8 install-yarn: Upgrade Yarn from 1.22.18 to 1.22.19.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-07-06 17:23:16 -07:00
Anders Kaseorg d104407531 log-search: Fix re.Match type annotations.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-07-05 12:55:03 -07:00
Alex Vandiver 2b02722d16 log-search: Add a filter to exclude all lines not explicitly wanted. 2022-06-28 15:59:31 -07:00
Alex Vandiver 180565d8d6 log-search: Fix copy/paste-o in filtering for presence. 2022-06-28 15:59:31 -07:00
Anders Kaseorg 3bf8ee2156 python: Unquote some unnecessarily quoted type annotations.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-06-26 17:37:41 -07:00
Alex Vandiver 41deef40cf nagios: Switch to generic check_cron_file for queues and consumers.
These share a common root; 91da4bd59b duplicated the code, but
didn't move the existing uses to the new utility.
2022-06-22 12:07:38 -07:00
Alex Vandiver e01a4242aa nagios: Sort queue consumer checks. 2022-06-22 12:07:38 -07:00
Alex Vandiver 27b63d0baf check-rabbitmq-consumers: Fix a misleading comment. 2022-06-22 12:07:38 -07:00
Alex Vandiver 4e06ee45c7 check-rabbitmq-consumers: Remove unused --min-threshold.
This has never actually been used  -- and does not make sense with the
check-all-queues-at-once model switched to in 88a123d5e0.  The
Tornado processes are the only ones we expect to be non-1, and since
they were added in 3f03dcdf5e the right number has been read from
config, not passed as an argument.
2022-06-22 12:07:38 -07:00