Tweaked provision script to run successfully in Fedora 38 and
included a script to build the groonga libs from source because
the packages in Fedora repos are outdated.
There is a major version jump from the last supported version (F34)
which is EOL so references and support for older versions were
removed.
Fixes: #20635
nginx sets the value of the `$http_host` variable to the empty string
when using http/3, as there is technically no `Host:` header sent:
https://github.com/nginx-quic/nginx-quic/issues/3
Users with a browser that support http/3 will send their first request
to nginx with http/2, and get an expected HTTP 200 -- but any
subsequent requests will fail with am HTTP 400, since the browser will
have upgraded to http/3, which has an empty `Host` header, which Zulip
rejects.
Switch to the `$host` variable, which works for all HTTP versions.
Co-authored-by: Alex Vandiver <alexmv@zulip.com>
Restore the default django.utils.log.AdminEmailHandler when
ERROR_REPORTING is enabled. Those with more sophisticated needs can
turn it off and use Sentry or a Sentry-compatible system.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
The claim in the comment from c8ec3dfcf6, that we can and should use
the current deploy's venv, misses one key case -- when upgrading the
operating system, the current deploy's venv is unworkable, since it
was configured for a previous version of Python. As such, any attempt
to load Django to verify the version of PostgreSQL it is talking to
must happen after the venv is configured.
Move the database version check into
`scripts/lib/check-database-compatibility`, which also moves it after
the new venv is configured.
Because we no longer reliably know, at `apt-get upgrade` time, what
version of PostgreSQL is installed, we hold all versions of the
pgroonga packages.
This ensures that the next `upgrade-zulip-from-git` has access to the
commit history of the initial install, if it was from a forked
repository. `/home/zulip/deployments/current` and `/srv/zulip.git`
are not quite organized into the steady-state that they will have
after one `upgrade-zulip-from-git`:
- `/home/zulip/deployments/current` is its own clone, not a worktree
- `/srv/zulip.git` has an origin of `/home/zulip/deployments/current`
- `remote.origin.mirror` is set on `/srv/zulip.git`
- `remote.origin.fetch` is `+refs/*:refs/*`
All but the first are automatically cleaned up by
`upgrade-zulip-from-git` when it is next run, using the code added in
30457ecd02. The additional complexity of making an existing
independent clone into a worktree seem not worth solving the first
point.
Updating the pgroonga package is not sufficient to upgrade the
extension in PostgreSQL -- an `ALTER EXTENSION pgroonga UPDATE` must
explicitly be run[^1]. Failure to do so can lead to unexpected behavior,
including crashes of PostgreSQL.
Expand on the existing `pgroonga_setup.sql.applied` file, to track
which version of the PostgreSQL extension has been configured. If the
file exists but is empty, we run `ALTER EXTENSION pgroonga UPDATE`
regardless -- if it is a no-op, it still succeeds with a `NOTICE`:
```
zulip=# ALTER EXTENSION pgroonga UPDATE;
NOTICE: version "3.0.8" of extension "pgroonga" is already installed
ALTER EXTENSION
```
The simple `ALTER EXTENSION` is sufficient for the
backwards-compatible case[^1] -- which, for our usage, is every
upgrade since 0.9 -> 1.0. Since version 1.0 was released in 2015,
before pgroonga support was added to Zulip in 2016, we can assume for
the moment that all pgroonga upgrades are backwards-compatible, and
not bother regenerating indexes.
Fixes: #25989.
[^1]: https://pgroonga.github.io/upgrade/
This was only necessary for PGroonga 1.x, and the `pgroonga` schema
will most likely be removed at some point inthe future, which will
make this statement error out.
Drop the unnecessary statement.
If the `postgresql.version` in `/etc/zulip/zulip.conf` is out of date
or wrong, upgrading to the actual current version would drop your
production database without prompting. While we do document taking a
Zulip backup (which includes a database backup) before running
`upgrade-postgresql`[^1], not everyone does so, with possibly
catastrophic consequences.
Do a true end-to-end check of the version in `/etc/zulip/zulip.conf`
by asking Django to query the database for its version, checking that
against the configured value, and aborting if there is any
disagreement.
[^1]: https://zulip.readthedocs.io/en/latest/production/upgrade.html#upgrading-postgresql
If `zulip-puppet-apply` is run during an upgrade, it will immediately
try to re-`stop-server` before running migrations; if the last step in
the puppet application was to restart `supervisor`, it may not be
listening on its UNIX socket yet. In such cases, `socket.connect()`
throws a `FileNotFoundError`:
```
Traceback (most recent call last):
File "./scripts/stop-server", line 53, in <module>
services = list_supervisor_processes(services, only_running=True)
File "./scripts/lib/supervisor.py", line 34, in list_supervisor_processes
processes = rpc().supervisor.getAllProcessInfo()
File "/usr/lib/python3.9/xmlrpc/client.py", line 1116, in __call__
return self.__send(self.__name, args)
File "/usr/lib/python3.9/xmlrpc/client.py", line 1456, in __request
response = self.__transport.request(
File "/usr/lib/python3.9/xmlrpc/client.py", line 1160, in request
return self.single_request(host, handler, request_body, verbose)
File "/usr/lib/python3.9/xmlrpc/client.py", line 1172, in single_request
http_conn = self.send_request(host, handler, request_body, verbose)
File "/usr/lib/python3.9/xmlrpc/client.py", line 1285, in send_request
self.send_content(connection, request_body)
File "/usr/lib/python3.9/xmlrpc/client.py", line 1315, in send_content
connection.endheaders(request_body)
File "/usr/lib/python3.9/http/client.py", line 1250, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/lib/python3.9/http/client.py", line 1010, in _send_output
self.send(msg)
File "/usr/lib/python3.9/http/client.py", line 950, in send
self.connect()
File "./scripts/lib/supervisor.py", line 10, in connect
self.sock.connect(self.host)
FileNotFoundError: [Errno 2] No such file or directory
```
Catch the `FileNotFoundError` and retry twice more, with backoff. If
it fails repeatedly, point to `service supervisor status` for further
debugging, as `FileNotFoundError` is rather misleading -- the file
exists, it simply is not accepting connections.
Instead of copying over a mostly-unchanged `postgresql.conf`, we
transition to deploying a `conf.d/zulip.conf` which contains the
only material changes we made to the file, which were previously
appended to the end.
While shipping separate while `postgresql.conf` files for each
supported version is useful if there is large variety in supported
options between versions, there is not no such variation at current,
and the burden of overriding the entire default configuration is that
it must be keep up to date wit the package's version.
pg_upgradecluster will start the cluster if the old cluster was
started before it ran, or if there are post-upgrade scripts to run.
Because neither of those are fully under our control, only attempt to
start the new cluster if it isn't already.
Installs which are upgrading to current `main`, and are upgrading for
the very first time from an install which was originally from git,
have a `/home/zulip/deployments/current` which, unlike all later
upgrades, is not a `git worktree` of `/srv/zulip.git`, but rather a
direct `git clone` of some arbitrary URL. As such, it does not have
an `upstream` remote, nor a cached `zulip-git-version` file.
This makes later attempts to determine the pre-upgrade revision of
git (for pre-deploy hooks) fail, as without a `zulip-git-version`
file, `ZULIP_VERSION` is insufficiently-specific (e.g. `6.1+git`), and
there is no guarantee the necessary tags exist either.
While we can make fresh git installs set up an `upstream` and run
`./tools/cache-zulip-git-version` going forward (see subsequent
commit), that does not address the issue for deploys which already
exist. For those, we must configure and fetch a `remote` in the old
checkout, followed by re-generating a cached `zulip-git-version`.
Fixes: #25076.
The existing `except subprocess.CalledProcessError` only catches if
there are syntax errors which prevent the `lastrun` file from being
written; it does not handle if there are properly-defined resources
which fail to evaluate (e.g. due to a missing dependency or file).
Check the `failed` resource count, and exit 2 if there are any such
resources. This will cause `zulip-puppet-apply --force --noop` (which
is used as a pre-flight check during upgrades) to properly detect and
signal on more types of invalid puppet configurations. In turn, this
will cause `upgrade-zulip` to not attempt to power through upgrades it
knows are destined to fail.
Since logrotate runs in a daily cron, this practically means "daily,
but only if it's larger than 500M." For large installs with large
traffic, this is effectively daily for 10 days; for small installs, it
is an unknown amount of time.
Switch to daily logfiles, defaulting to 14 days to match nginx; this
can be overridden using a zulip.conf setting. This makes it easier to
ensure that access logs are only kept for a bounded period of time.
Anders Kaseorg
shu.chen
David Rosa
n0099
Alex Vandiver