With this change, we are now testing the production static asset
pipeline and installation process in a new testing job (and also run
the frontend/backend tests separately).
This means that changes that break the Zulip static asset pipeline or
production installation process are more likely to fail tests. The
testing is imperfect in that it does not have proper isolation -- we
build a complete Zulip development environment and then install a
Zulip production environment on top of it, so e.g. any apt
dependencies installed for Zulip development will still be available
for the Zulip production environment. But, it's better than nothing!
A good v2 of this would be to have the production setup process just
install the minimum stuff needed to run `build-release-tarball` and
then uninstall it / clean it up so that we can do a more clear
production installation, but that's more work.
Apparently it isn't supposed to work reliably with the container-based
infrastructure that we're using and empirically it's causing build
failures.
Thanks to @mijime for tracking this down.
This is a bit hackish in that ideally we'd use proper options parsing
in provision.py, but it works and I even ran the tests 100x for tests
for flakes and didn't get any, so it's definitely an improvement!
With this we'll be both testing the runtime and effectively the Dev VM
setup process, which is awesome; the additional thing I'd want to add
tests for is the production setup process...
Tim Abbott
Luke Faraone