`setup_path()` previously only checked that some `zulip-py3-venv` was
the `sys.prefix`, not that it was the one associated with this
deployment. When `uwsgi` is started, it is started from `bin/uwsgi`
within a `zulip-py3-venv` virtualenv, and as such sets
`sys.executable` to that, resulting in uwsgi workers picking up the
library path of that virtualenv. On first start, `sys.path` thus
already matches the expected virtualenv, and the `setup_path` in
`zproject.wsgi` does nothing.
If a rolling restart was later done into a deployment with a different
virtualenv, the `zproject.wsgi` call to `setup_path()` did not change
`sys.path` to the new virtualenv, since it was already running within
_a_ virtualenv. This led to dependency version mismatches, and
potentially even more disastrous consequences if the old (but still
erroneously in use) virtualenv was later garbage-collected.
PR #26771 was a previous attempt to resolve this, but failed due to
not thinking of the uwsgi binary itself as possibly providing a
virtualenv path. We leave the `chdir` hooks from that in-place, since
it cannot hurt for the "master" uwsgi process to be chdir'd to `/`,
and the `hook-post-fork` `chdir` is reasonable as well.
Resolve the virtualenv in `setup_path()`, and activate it if it
differs from the one that is currently active. To be sure that no
other old virtualenvs are used, we also filter out any paths which
appear to be from other Zulip virtualenvs.
We need this check when switching between branches without `help-beta`
package. `node_modules` will be removed when working on a non `help-beta`
branch, but if `node_modules/.pnpm/lock.yaml` has not been updated by that
branch, we will end up in a situation where we might not have `node_modules`
even when we run the provision command.
We might not need this check when the `help-beta` initial folder
has been merged for a week or two, where almost all active PRs
would have been rebased upon main, making switching branches an ease.
There's no need for sharding, but this allows one to spend a bit of
extra memory to reduce image-processing latency when bursts of images
are uploaded at once.
A new table is created to track which path_id attachments are images,
and for those their metadata, and which thumbnails have been created.
Using path_id as the effective primary key lets us ignore if the
attachment is archived or not, saving some foreign key messes.
A new worker is added to observe events when rows are added to this
table, and to generate and store thumbnails for those images in
differing sizes and formats.
This adds `--automated` and `--no-automated` flags to all Zulip
management commands, whose default is based on if STDIN is a TTY.
This enables cron jobs and supervisor commands to continue to report
to Sentry, and manually-run commands (when reporting to Sentry does
not provide value, since the user can see them) to not.
Note that this only applies to Zulip commands -- core Django
commands (e.g. `./manage.py`) do not grow support for `--automated`
and will always report exceptions to Sentry.
`manage.py` subcommands in the `upgrade` and `restart-server` paths
are marked as `--automated`, since those may be run semi-unattended,
and they are useful to log to Sentry.
The "invites" worker exists to do two things -- make a Confirmation
object, and send the outgoing email. Making the Confirmation object
in a background process from where the PreregistrationUser is created
temporarily leaves the PreregistrationUser in invalid state, and
results in 500's, and the user not immediately seeing the sent
invitation. That the "invites" worker also wants to create the
Confirmation object means that "resending" an invite invalidates the
URL in the previous email, which can be confusing to the user.
Moving the Confirmation creation to the same transaction solves both
of these issues, and leaves the "invites" worker with nothing to do
but send the email; as such, we remove it entirely, and use the
existing "email_senders" worker to send the invites. The volume of
invites is small enough that this will not affect other uses of that
worker.
Fixes: #21306Fixes: #24275
Testing for it in Python means that we have to worry about keeping the
`upgrade-zulip-stage-2` backwards-compatible with all versions of
Python which we could ever be upgrading from -- which is all of them.
Factor out the "supported operating systems" check, and share it
between upgrade and install codepaths.
`--no-init-db` is used to silence the need for `--hostname` and
`--email` arguments; it is a proxy for "this is not a frontend host."
We would ideally like to use `has_class` to know if the user's
provided puppet classes are include an `app_frontend`, and thus
`--hostname` and `--email` are required -- but doing that requires
several other steps, and we would like this feedback to be immediate.
We make the presence of `--puppet-classes` equivalent to
`--no-init-db`, since nearly every configuration with
`--puppet-classes` does not install both a database and a frontend,
which is what is required to initialize a database.
While this could be done previously by calling
`upgrade-zulip-from-git --remote-url /srv/zulip.git`, the explicit
argument makes this more straightforward, and avoids churning the
`refs/remotes/origin/` namespace.
Anders Kaseorg
Alex Vandiver
Shubham Padia
Mateusz Mandera
Tim Abbott
Anders Kaseorg