Installs which are upgrading to current `main`, and are upgrading for
the very first time from an install which was originally from git,
have a `/home/zulip/deployments/current` which, unlike all later
upgrades, is not a `git worktree` of `/srv/zulip.git`, but rather a
direct `git clone` of some arbitrary URL. As such, it does not have
an `upstream` remote, nor a cached `zulip-git-version` file.
This makes later attempts to determine the pre-upgrade revision of
git (for pre-deploy hooks) fail, as without a `zulip-git-version`
file, `ZULIP_VERSION` is insufficiently-specific (e.g. `6.1+git`), and
there is no guarantee the necessary tags exist either.
While we can make fresh git installs set up an `upstream` and run
`./tools/cache-zulip-git-version` going forward (see subsequent
commit), that does not address the issue for deploys which already
exist. For those, we must configure and fetch a `remote` in the old
checkout, followed by re-generating a cached `zulip-git-version`.
Fixes: #25076.
The existing `except subprocess.CalledProcessError` only catches if
there are syntax errors which prevent the `lastrun` file from being
written; it does not handle if there are properly-defined resources
which fail to evaluate (e.g. due to a missing dependency or file).
Check the `failed` resource count, and exit 2 if there are any such
resources. This will cause `zulip-puppet-apply --force --noop` (which
is used as a pre-flight check during upgrades) to properly detect and
signal on more types of invalid puppet configurations. In turn, this
will cause `upgrade-zulip` to not attempt to power through upgrades it
knows are destined to fail.
Since logrotate runs in a daily cron, this practically means "daily,
but only if it's larger than 500M." For large installs with large
traffic, this is effectively daily for 10 days; for small installs, it
is an unknown amount of time.
Switch to daily logfiles, defaulting to 14 days to match nginx; this
can be overridden using a zulip.conf setting. This makes it easier to
ensure that access logs are only kept for a bounded period of time.
Previously, we had an architecture where CSS inlining for emails was
done at provision time in inline_email_css.py. This was necessary
because the library we were using for this, Premailer, was extremely
slow, and doing the inlining for every outgoing email would have been
prohibitively expensive.
Now that we've migrated to a more modern library that inlines the
small amount of CSS we have into emails nearly instantly, we are able
to remove the complex architecture built to work around Premailer
being slow and just do the CSS inlining as the final step in sending
each individual email.
This has several significant benefits:
* Removes a fiddly provisioning step that made the edit/refresh cycle
for modifying email templates confusing; there's no longer a CSS
inlining step that, if you forget to do it, results in your testing a
stale variant of the email templates.
* Fixes internationalization problems related to translators working
with pre-CSS-inlined emails, and then Django trying to apply the
translators to the post-CSS-inlined version.
* Makes the send_custom_email pipeline simpler and easier to improve.
Signed-off-by: Daniil Fadeev <fadeevd@zulip.com>
While the previous commit handles the common case of all of the server
being started already, it still produces ERROR output lines from
supervisorctl when most of the server is already running. Take the
case where one worker is stopped:
```
$ supervisorctl stop zulip-workers:zulip_events_deferred_work
zulip-workers:zulip_events_deferred_work: stopped
$ ./scripts/start-server
2023-04-04 15:50:28,505 start-server: Running syntax and database checks
System check identified no issues (15 silenced).
2023-04-04 15:50:31,977 start-server: Starting Tornado process on port 9800
zulip-tornado:zulip-tornado-port-9800: ERROR (already started)
2023-04-04 15:50:32,283 start-server: Starting Tornado process on port 9801
zulip-tornado:zulip-tornado-port-9801: ERROR (already started)
2023-04-04 15:50:32,592 start-server: Starting django server
zulip-django: ERROR (already started)
2023-04-04 15:50:33,340 start-server: Starting workers
zulip-workers:zulip_events_deferred_work: started
zulip_deliver_scheduled_emails: ERROR (already started)
zulip_deliver_scheduled_messages: ERROR (already started)
process-fts-updates: ERROR (already started)
2023-04-04 15:50:34,659 start-server: Done!
Zulip started successfully!
```
More gracefully handle these cases:
```
$ ./scripts/start-server
2023-04-04 15:52:39,815 start-server: Running syntax and database checks
System check identified no issues (15 silenced).
2023-04-04 15:52:43,270 start-server: Starting Tornado process on port 9800
2023-04-04 15:52:43,287 start-server: zulip-tornado:zulip-tornado-port-9800 already started!
2023-04-04 15:52:43,287 start-server: Starting Tornado process on port 9801
2023-04-04 15:52:43,300 start-server: zulip-tornado:zulip-tornado-port-9801 already started!
2023-04-04 15:52:43,300 start-server: Starting django server
2023-04-04 15:52:43,316 start-server: zulip-django already started!
2023-04-04 15:52:43,793 start-server: Starting workers
zulip-workers:zulip_events_deferred_work: started
2023-04-04 15:52:45,111 start-server: Done!
Zulip started successfully!
```
Currently, the output from `start-server` if the server is already
running is potentially confusing, since it says ERROR several times:
```
$ ./scripts/start-server
2023-04-04 15:35:12,737 start-server: Running syntax and database checks
System check identified no issues (15 silenced).
2023-04-04 15:35:16,211 start-server: Starting Tornado process on port 9800
zulip-tornado:zulip-tornado-port-9800: ERROR (already started)
2023-04-04 15:35:16,528 start-server: Starting Tornado process on port 9801
zulip-tornado:zulip-tornado-port-9801: ERROR (already started)
2023-04-04 15:35:16,844 start-server: Starting django server
zulip-django: ERROR (already started)
2023-04-04 15:35:17,605 start-server: Starting workers
zulip_deliver_scheduled_emails: ERROR (already started)
zulip_deliver_scheduled_messages: ERROR (already started)
process-fts-updates: ERROR (already started)
2023-04-04 15:35:18,923 start-server: Done!
```
Catch the simple common case where all of the services are already
running, and output a clearer success message:
```
$ ./scripts/start-server
2023-04-04 15:39:52,367 start-server: Running syntax and database checks
System check identified no issues (15 silenced).
2023-04-04 15:39:55,857 start-server: Zulip is already started; nothing to do!
```
Primary goal of library replacement is improving execution speed.
This commit should not affect the functionality of the system
or make any changes to it.
On Docker for Mac with the gRPC FUSE or VirtioFS file sharing
implementations, we nondeterministically get errors like this from
pnpm install:
pnpm: ENOENT: no such file or directory, copyfile '/srv/zulip/.pnpm-store/v3/files/7d/6b44bb658625281b48194e5a3d3a07452bea1f256506dd16f7a21941ef3f0d259e1bcd0cc6202642bf1fd129bc187e6a3921d382d568d312bd83f3023979a0' -> '/srv/zulip/node_modules/.pnpm/regexpu-core@5.3.2/node_modules/_tmp_3227_7f867a9c510832f5f82601784e21e7be/LICENSE-MIT.txt'
Subcommand of ./lib/provision.py failed with exit status 1: /usr/local/bin/pnpm install --frozen-lockfile
Actual error output for the subcommand is just above this.
Work around this using --package-import-method=copy.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Ever since we started bundling the app with webpack, there’s been less
and less overlap between our ‘static’ directory (files belonging to
the frontend app) and Django’s interpretation of the ‘static’
directory (files served directly to the web).
Split the app out to its own ‘web’ directory outside of ‘static’, and
remove all the custom collectstatic --ignore rules. This makes it
much clearer what’s actually being served to the web, and what’s being
bundled by webpack. It also shrinks the release tarball by 3%.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
These hooks are run immediately around the critical section of the
upgrade. If the upgrade fails for preparatory reasons, the pre-deploy
hook may not be run; if it fails during the upgrade, the post-deploy
hook will not be run. Hooks are called from the CWD of the new
deploy, with arguments of the old version and the new version. If
they exit with non-0 exit code, the deploy aborts.
Corepack manages multiple per-project version of Yarn and PNPM, which
means we have to maintain less installation code, and could help us
switch away from Yarn 1 without making the system unusable for
development of other Yarn 1 projects.
https://nodejs.org/api/corepack.html
The Unicode spaces in the timerender test resulted from an ICU
upgrade: https://github.com/nodejs/node/pull/45068.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Black 23 enforces some slightly more specific rules about empty line
counts and redundant parenthesis removal, but the result is still
compatible with Black 22.
(This does not actually upgrade our Python environment to Black 23
yet.)
Signed-off-by: Anders Kaseorg <anders@zulip.com>
The `pg_upgrade` tool uses `pg_dump` as an internal step, and verifies
that the version of `pg_upgrade` is the same exactly the same as the
version of the PostgreSQL server it is upgrading to. A mismatch (even
in packaging versions) leads to it aborting:
```
/usr/lib/postgresql/14/bin/pg_upgrade -b /usr/lib/postgresql/13/bin -B /usr/lib/postgresql/14/bin -p 5432 -P 5435 -d /etc/postgresql/13/main -D /etc/postgresql/14/main --link
Finding the real data directory for the source cluster ok
Finding the real data directory for the target cluster ok
check for "/usr/lib/postgresql/14/bin/pg_dump" failed: incorrect version: found "pg_dump (PostgreSQL) 14.6 (Ubuntu 14.6-0ubuntu0.22.04.1)", expected "pg_dump (PostgreSQL) 14.6 (Ubuntu 14.6-1.pgdg22.04+1)"
Failure, exiting
```
Explicitly upgrade `postgresql-client` at the same time we upgrade
`postgresql` itself, so their versions match.
Fixes: #24192
This was last really used in d7a3570c7e, in 2013, when it was
`/home/humbug/logs`.
Repoint the one obscure piece of tooling that writes there, and remove
the places that created it.
`check_version` in `install-yarn` had the rather careful check that
the yarn it installed into `/usr/bin/yarn` was the yarn which was
first in the user's `$PATH`. This caused problems when the user had a
pre-existing `/usr/local/bin/yarn`; however, those problems are
limited to the `install-yarn` script itself, since the nearly all
calls to yarn from Zulip's code already hardcode the `/srv/zulip-yarn`
location, and do not depend on what is in `$PATH`.
Remove the checks in `install-yarn` that depend on the local `$PATH`,
and stop installing our `yarn` into it. We also adjust the two
callsites which did not specify the full path to `yarn`, so use
`/srv/zulip-yarn`.
Fixes: #23993
Co-authored-by: Alex Vandiver <alexmv@zulip.com>
During installation on a new host, `create-database` attempts to
verify that there isn't a bunch of data already in the database which
is it about to drop and recreate. In the most common case, this
statement emits a scary-looking warning, since the database does not
exist yet:
```
+ /home/zulip/deployments/current/scripts/setup/create-database
+ POSTGRES_USER=postgres
++ crudini --get /etc/zulip/zulip.conf postgresql database_name
++ echo zulip
+ DATABASE_NAME=zulip
++ crudini --get /etc/zulip/zulip.conf postgresql database_user
++ echo zulip
+ DATABASE_USER=zulip
++ cd /
++ su postgres -c 'psql -v ON_ERROR_STOP=1 -Atc '\''SELECT COUNT(*) FROM zulip.zerver_message;'\'' zulip'
psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL: database "zulip" does not exist
```
Because we are attempting to gracefully handle the case where the
database does not exist yet, we also continue (and drop the database)
in other, less expected cases -- for instance, if database contains a
schema we do not expect.
Explicitly check for the database existence first, and once we verify
that, allow any further failures in the `SELECT COUNT(*)` to abort
`create-database`. This serves the dual purpose of hiding the "FATAL"
error for the common case when the database does not exist, as well as
preventing dropping the database if anything else goes awry.
If a previous attempt at an upgrade failed for some reason, the new
PostgreSQL may be installed, and the conversion will succeed, but the
new PostgreSQL daemon will not be running (Puppet does not force it to
start). This causes the upgrade to fail when analyzing statistics,
since the daemon isn't running.
Explicitly start the new PostgreSQL; this does nothing in most cases,
but will provider better resiliency when recovering from previous
partial upgrades.
Anders Kaseorg
Alex Vandiver
Daniil Fadeev
Lalit