Mirror
/
zulip
mirror of https://github.com/zulip/zulip.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
zulip/tools/test-install/prepare-base

63 lines
1.6 KiB
Plaintext
Raw Normal View History

Use #!/usr/bin/env for bash shebangs. /bin/sh and /usr/bin/env are the only two binaries that NixOS provides at a fixed path (outside a buildFHSUserEnv sandbox). This discussion was split from #11004. Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-12-18 02:08:53 +01:00
#!/usr/bin/env bash
test-install: Factor out booted-yet-p polling loop, use in prepare-base. Otherwise prepare-base is likely to fail when first run (but then succeed when rerun, because the container is left running), because the container isn't up yet when we try to operate in it. Also clean up the placement of `set -e` vs `set -x`.
2018-02-09 01:14:38 +01:00
set -e
install: Start on an LXC-based dev/test environment for the installer. In order to do development on the installer itself in a sane way, we need a reasonably fast and automatic way to get a fresh environment to try to run it in. This calls for some form of virtualization. Choices include * A public cloud, like EC2 or Digital Ocean. These could work, if we wrote some suitable scripts against their APIs, to manage appropriate base images (as AMIs or snapshots respectively) and to start fresh instances/droplets from a base image. There'd be some latency on starting a new VM, and this would also require the user to have an account on the relevant cloud with API access to create images and VMs. * A local whole-machine VM system (hypervisor) like VirtualBox or VMware, perhaps managing the configuration through Vagrant. These hypervisors can be unstable and painfully slow. They're often the only way to get development work done on a Mac or Windows machine, which is why we use them there for the normal Zulip development environment; but I don't really want to find out how their instability scales when constantly spawning fresh VMs from an image. * Containers. The new hotness, the name on everyone's lips, is Docker. But Docker is not designed for virtualizing a traditional Unix server, complete with its own init system and a fleet of processes with a shared filesystem -- in other words, the platform Zulip's installer and deployment system are for. Docker brings its own quite different model of deployment, and someday we may port Zulip from the traditional Unix server to the Docker-style deployment model, but for testing our traditional-Unix-server deployment we need a (virtualized) traditional Unix server. * Containers, with LXC. LXC provides containers that function as traditional Unix servers; because of the magic of containers, the overhead is quite low, and LXC offers handy snapshotting features so that we can quickly start up a fresh environment from a base image. Running LXC does require a Linux base system. For contributors whose local development machine isn't already Linux, the same solutions are available as for our normal development environment: the base system for running LXC could be e.g. a Vagrant-managed VirtualBox VM, or a machine in a public cloud. This commit adds a first version of such a thing, using LXC to manage a base image plus a fresh container for each test run. The test containers function as VMs: once installed, all the Zulip services run normally in them and can be managed in the normal production ways. This initial version has a shortage of usage messages or docs, and likely has some sharp edges. It also requires familiarity with the basics of LXC commands in order to make good use of the resulting containers: `lxc-ls -f`, `lxc-attach`, `lxc-stop`, and `lxc-start`, in particular.
2018-01-20 01:14:40 +01:00
if [ "$EUID" -ne 0 ]; then
echo "error: this script must be run as root" >&2
exit 1
fi
RELEASE="$1"
ARCH=amd64 # TODO: maybe i686 too
case "$RELEASE" in
test-install: Add xenial support.
2018-02-09 00:58:39 +01:00
xenial) extra_packages=(virtualenv postgresql-9.5)
;;
install: Start on an LXC-based dev/test environment for the installer. In order to do development on the installer itself in a sane way, we need a reasonably fast and automatic way to get a fresh environment to try to run it in. This calls for some form of virtualization. Choices include * A public cloud, like EC2 or Digital Ocean. These could work, if we wrote some suitable scripts against their APIs, to manage appropriate base images (as AMIs or snapshots respectively) and to start fresh instances/droplets from a base image. There'd be some latency on starting a new VM, and this would also require the user to have an account on the relevant cloud with API access to create images and VMs. * A local whole-machine VM system (hypervisor) like VirtualBox or VMware, perhaps managing the configuration through Vagrant. These hypervisors can be unstable and painfully slow. They're often the only way to get development work done on a Mac or Windows machine, which is why we use them there for the normal Zulip development environment; but I don't really want to find out how their instability scales when constantly spawning fresh VMs from an image. * Containers. The new hotness, the name on everyone's lips, is Docker. But Docker is not designed for virtualizing a traditional Unix server, complete with its own init system and a fleet of processes with a shared filesystem -- in other words, the platform Zulip's installer and deployment system are for. Docker brings its own quite different model of deployment, and someday we may port Zulip from the traditional Unix server to the Docker-style deployment model, but for testing our traditional-Unix-server deployment we need a (virtualized) traditional Unix server. * Containers, with LXC. LXC provides containers that function as traditional Unix servers; because of the magic of containers, the overhead is quite low, and LXC offers handy snapshotting features so that we can quickly start up a fresh environment from a base image. Running LXC does require a Linux base system. For contributors whose local development machine isn't already Linux, the same solutions are available as for our normal development environment: the base system for running LXC could be e.g. a Vagrant-managed VirtualBox VM, or a machine in a public cloud. This commit adds a first version of such a thing, using LXC to manage a base image plus a fresh container for each test run. The test containers function as VMs: once installed, all the Zulip services run normally in them and can be managed in the normal production ways. This initial version has a shortage of usage messages or docs, and likely has some sharp edges. It also requires familiarity with the basics of LXC commands in order to make good use of the resulting containers: `lxc-ls -f`, `lxc-attach`, `lxc-stop`, and `lxc-start`, in particular.
2018-01-20 01:14:40 +01:00
*)
echo "error: unsupported target release: $RELEASE" >&2
exit 1
;;
esac
test-install: Factor out booted-yet-p polling loop, use in prepare-base. Otherwise prepare-base is likely to fail when first run (but then succeed when rerun, because the container is left running), because the container isn't up yet when we try to operate in it. Also clean up the placement of `set -e` vs `set -x`.
2018-02-09 01:14:38 +01:00
THIS_DIR="$(dirname "$(readlink -f "$0")")"
set -x
install: Start on an LXC-based dev/test environment for the installer. In order to do development on the installer itself in a sane way, we need a reasonably fast and automatic way to get a fresh environment to try to run it in. This calls for some form of virtualization. Choices include * A public cloud, like EC2 or Digital Ocean. These could work, if we wrote some suitable scripts against their APIs, to manage appropriate base images (as AMIs or snapshots respectively) and to start fresh instances/droplets from a base image. There'd be some latency on starting a new VM, and this would also require the user to have an account on the relevant cloud with API access to create images and VMs. * A local whole-machine VM system (hypervisor) like VirtualBox or VMware, perhaps managing the configuration through Vagrant. These hypervisors can be unstable and painfully slow. They're often the only way to get development work done on a Mac or Windows machine, which is why we use them there for the normal Zulip development environment; but I don't really want to find out how their instability scales when constantly spawning fresh VMs from an image. * Containers. The new hotness, the name on everyone's lips, is Docker. But Docker is not designed for virtualizing a traditional Unix server, complete with its own init system and a fleet of processes with a shared filesystem -- in other words, the platform Zulip's installer and deployment system are for. Docker brings its own quite different model of deployment, and someday we may port Zulip from the traditional Unix server to the Docker-style deployment model, but for testing our traditional-Unix-server deployment we need a (virtualized) traditional Unix server. * Containers, with LXC. LXC provides containers that function as traditional Unix servers; because of the magic of containers, the overhead is quite low, and LXC offers handy snapshotting features so that we can quickly start up a fresh environment from a base image. Running LXC does require a Linux base system. For contributors whose local development machine isn't already Linux, the same solutions are available as for our normal development environment: the base system for running LXC could be e.g. a Vagrant-managed VirtualBox VM, or a machine in a public cloud. This commit adds a first version of such a thing, using LXC to manage a base image plus a fresh container for each test run. The test containers function as VMs: once installed, all the Zulip services run normally in them and can be managed in the normal production ways. This initial version has a shortage of usage messages or docs, and likely has some sharp edges. It also requires familiarity with the basics of LXC commands in order to make good use of the resulting containers: `lxc-ls -f`, `lxc-attach`, `lxc-stop`, and `lxc-start`, in particular.
2018-01-20 01:14:40 +01:00
CONTAINER_NAME=zulip-install-$RELEASE-base
if ! lxc-info -n "$CONTAINER_NAME" >/dev/null 2>&1; then
lxc-create -n "$CONTAINER_NAME" -t download -- -d ubuntu -r "$RELEASE" -a "$ARCH"
fi
lxc-start -n "$CONTAINER_NAME"
test-install: Factor out booted-yet-p polling loop, use in prepare-base. Otherwise prepare-base is likely to fail when first run (but then succeed when rerun, because the container is left running), because the container isn't up yet when we try to operate in it. Also clean up the placement of `set -e` vs `set -x`.
2018-02-09 01:14:38 +01:00
"$THIS_DIR"/lxc-wait -n "$CONTAINER_NAME"
install: Start on an LXC-based dev/test environment for the installer. In order to do development on the installer itself in a sane way, we need a reasonably fast and automatic way to get a fresh environment to try to run it in. This calls for some form of virtualization. Choices include * A public cloud, like EC2 or Digital Ocean. These could work, if we wrote some suitable scripts against their APIs, to manage appropriate base images (as AMIs or snapshots respectively) and to start fresh instances/droplets from a base image. There'd be some latency on starting a new VM, and this would also require the user to have an account on the relevant cloud with API access to create images and VMs. * A local whole-machine VM system (hypervisor) like VirtualBox or VMware, perhaps managing the configuration through Vagrant. These hypervisors can be unstable and painfully slow. They're often the only way to get development work done on a Mac or Windows machine, which is why we use them there for the normal Zulip development environment; but I don't really want to find out how their instability scales when constantly spawning fresh VMs from an image. * Containers. The new hotness, the name on everyone's lips, is Docker. But Docker is not designed for virtualizing a traditional Unix server, complete with its own init system and a fleet of processes with a shared filesystem -- in other words, the platform Zulip's installer and deployment system are for. Docker brings its own quite different model of deployment, and someday we may port Zulip from the traditional Unix server to the Docker-style deployment model, but for testing our traditional-Unix-server deployment we need a (virtualized) traditional Unix server. * Containers, with LXC. LXC provides containers that function as traditional Unix servers; because of the magic of containers, the overhead is quite low, and LXC offers handy snapshotting features so that we can quickly start up a fresh environment from a base image. Running LXC does require a Linux base system. For contributors whose local development machine isn't already Linux, the same solutions are available as for our normal development environment: the base system for running LXC could be e.g. a Vagrant-managed VirtualBox VM, or a machine in a public cloud. This commit adds a first version of such a thing, using LXC to manage a base image plus a fresh container for each test run. The test containers function as VMs: once installed, all the Zulip services run normally in them and can be managed in the normal production ways. This initial version has a shortage of usage messages or docs, and likely has some sharp edges. It also requires familiarity with the basics of LXC commands in order to make good use of the resulting containers: `lxc-ls -f`, `lxc-attach`, `lxc-stop`, and `lxc-start`, in particular.
2018-01-20 01:14:40 +01:00
run() {
lxc-attach -n "$CONTAINER_NAME" -- "$@"
}
run passwd -d root
run apt-get update
test-install: Do a dist-upgrade in prepare-base. This keeps the base tree up to date, saving the time we'd spend doing the same upgrades in each test install.
2018-03-03 00:36:01 +01:00
run apt-get dist-upgrade -y
install: Start on an LXC-based dev/test environment for the installer. In order to do development on the installer itself in a sane way, we need a reasonably fast and automatic way to get a fresh environment to try to run it in. This calls for some form of virtualization. Choices include * A public cloud, like EC2 or Digital Ocean. These could work, if we wrote some suitable scripts against their APIs, to manage appropriate base images (as AMIs or snapshots respectively) and to start fresh instances/droplets from a base image. There'd be some latency on starting a new VM, and this would also require the user to have an account on the relevant cloud with API access to create images and VMs. * A local whole-machine VM system (hypervisor) like VirtualBox or VMware, perhaps managing the configuration through Vagrant. These hypervisors can be unstable and painfully slow. They're often the only way to get development work done on a Mac or Windows machine, which is why we use them there for the normal Zulip development environment; but I don't really want to find out how their instability scales when constantly spawning fresh VMs from an image. * Containers. The new hotness, the name on everyone's lips, is Docker. But Docker is not designed for virtualizing a traditional Unix server, complete with its own init system and a fleet of processes with a shared filesystem -- in other words, the platform Zulip's installer and deployment system are for. Docker brings its own quite different model of deployment, and someday we may port Zulip from the traditional Unix server to the Docker-style deployment model, but for testing our traditional-Unix-server deployment we need a (virtualized) traditional Unix server. * Containers, with LXC. LXC provides containers that function as traditional Unix servers; because of the magic of containers, the overhead is quite low, and LXC offers handy snapshotting features so that we can quickly start up a fresh environment from a base image. Running LXC does require a Linux base system. For contributors whose local development machine isn't already Linux, the same solutions are available as for our normal development environment: the base system for running LXC could be e.g. a Vagrant-managed VirtualBox VM, or a machine in a public cloud. This commit adds a first version of such a thing, using LXC to manage a base image plus a fresh container for each test run. The test containers function as VMs: once installed, all the Zulip services run normally in them and can be managed in the normal production ways. This initial version has a shortage of usage messages or docs, and likely has some sharp edges. It also requires familiarity with the basics of LXC commands in order to make good use of the resulting containers: `lxc-ls -f`, `lxc-attach`, `lxc-stop`, and `lxc-start`, in particular.
2018-01-20 01:14:40 +01:00
# As an optimization, we install a bunch of packages the installer
# would install for itself.
run apt-get install -y --no-install-recommends \
install: Use readlink -f rather than realpath. It does exactly the same thing, though the name is less transparent; and it simplifies the script by avoiding an extra, early `apt-get install`.
2018-02-09 02:19:40 +01:00
xvfb parallel netcat unzip zip jq python3-pip wget curl eatmydata \
test-install: Pre-install a few more dependencies in base image.
2018-01-23 01:18:33 +01:00
git crudini openssl ssl-cert \
install: Start on an LXC-based dev/test environment for the installer. In order to do development on the installer itself in a sane way, we need a reasonably fast and automatic way to get a fresh environment to try to run it in. This calls for some form of virtualization. Choices include * A public cloud, like EC2 or Digital Ocean. These could work, if we wrote some suitable scripts against their APIs, to manage appropriate base images (as AMIs or snapshots respectively) and to start fresh instances/droplets from a base image. There'd be some latency on starting a new VM, and this would also require the user to have an account on the relevant cloud with API access to create images and VMs. * A local whole-machine VM system (hypervisor) like VirtualBox or VMware, perhaps managing the configuration through Vagrant. These hypervisors can be unstable and painfully slow. They're often the only way to get development work done on a Mac or Windows machine, which is why we use them there for the normal Zulip development environment; but I don't really want to find out how their instability scales when constantly spawning fresh VMs from an image. * Containers. The new hotness, the name on everyone's lips, is Docker. But Docker is not designed for virtualizing a traditional Unix server, complete with its own init system and a fleet of processes with a shared filesystem -- in other words, the platform Zulip's installer and deployment system are for. Docker brings its own quite different model of deployment, and someday we may port Zulip from the traditional Unix server to the Docker-style deployment model, but for testing our traditional-Unix-server deployment we need a (virtualized) traditional Unix server. * Containers, with LXC. LXC provides containers that function as traditional Unix servers; because of the magic of containers, the overhead is quite low, and LXC offers handy snapshotting features so that we can quickly start up a fresh environment from a base image. Running LXC does require a Linux base system. For contributors whose local development machine isn't already Linux, the same solutions are available as for our normal development environment: the base system for running LXC could be e.g. a Vagrant-managed VirtualBox VM, or a machine in a public cloud. This commit adds a first version of such a thing, using LXC to manage a base image plus a fresh container for each test run. The test containers function as VMs: once installed, all the Zulip services run normally in them and can be managed in the normal production ways. This initial version has a shortage of usage messages or docs, and likely has some sharp edges. It also requires familiarity with the basics of LXC commands in order to make good use of the resulting containers: `lxc-ls -f`, `lxc-attach`, `lxc-stop`, and `lxc-start`, in particular.
2018-01-20 01:14:40 +01:00
build-essential python3-dev \
closure-compiler memcached rabbitmq-server redis-server \
hunspell-en-us supervisor libssl-dev yui-compressor puppet \
deps: Replace libz-dev with zlib1g-dev since the former was renamed.
2018-02-09 22:28:20 +01:00
gettext libffi-dev libfreetype6-dev zlib1g-dev libjpeg-dev \
install: Start on an LXC-based dev/test environment for the installer. In order to do development on the installer itself in a sane way, we need a reasonably fast and automatic way to get a fresh environment to try to run it in. This calls for some form of virtualization. Choices include * A public cloud, like EC2 or Digital Ocean. These could work, if we wrote some suitable scripts against their APIs, to manage appropriate base images (as AMIs or snapshots respectively) and to start fresh instances/droplets from a base image. There'd be some latency on starting a new VM, and this would also require the user to have an account on the relevant cloud with API access to create images and VMs. * A local whole-machine VM system (hypervisor) like VirtualBox or VMware, perhaps managing the configuration through Vagrant. These hypervisors can be unstable and painfully slow. They're often the only way to get development work done on a Mac or Windows machine, which is why we use them there for the normal Zulip development environment; but I don't really want to find out how their instability scales when constantly spawning fresh VMs from an image. * Containers. The new hotness, the name on everyone's lips, is Docker. But Docker is not designed for virtualizing a traditional Unix server, complete with its own init system and a fleet of processes with a shared filesystem -- in other words, the platform Zulip's installer and deployment system are for. Docker brings its own quite different model of deployment, and someday we may port Zulip from the traditional Unix server to the Docker-style deployment model, but for testing our traditional-Unix-server deployment we need a (virtualized) traditional Unix server. * Containers, with LXC. LXC provides containers that function as traditional Unix servers; because of the magic of containers, the overhead is quite low, and LXC offers handy snapshotting features so that we can quickly start up a fresh environment from a base image. Running LXC does require a Linux base system. For contributors whose local development machine isn't already Linux, the same solutions are available as for our normal development environment: the base system for running LXC could be e.g. a Vagrant-managed VirtualBox VM, or a machine in a public cloud. This commit adds a first version of such a thing, using LXC to manage a base image plus a fresh container for each test run. The test containers function as VMs: once installed, all the Zulip services run normally in them and can be managed in the normal production ways. This initial version has a shortage of usage messages or docs, and likely has some sharp edges. It also requires familiarity with the basics of LXC commands in order to make good use of the resulting containers: `lxc-ls -f`, `lxc-attach`, `lxc-stop`, and `lxc-start`, in particular.
2018-01-20 01:14:40 +01:00
libldap2-dev libmemcached-dev python-dev python-pip \
test-install: Add xenial support.
2018-02-09 00:58:39 +01:00
python-six libxml2-dev libxslt1-dev libpq-dev \
"${extra_packages[@]}"
install: Start on an LXC-based dev/test environment for the installer. In order to do development on the installer itself in a sane way, we need a reasonably fast and automatic way to get a fresh environment to try to run it in. This calls for some form of virtualization. Choices include * A public cloud, like EC2 or Digital Ocean. These could work, if we wrote some suitable scripts against their APIs, to manage appropriate base images (as AMIs or snapshots respectively) and to start fresh instances/droplets from a base image. There'd be some latency on starting a new VM, and this would also require the user to have an account on the relevant cloud with API access to create images and VMs. * A local whole-machine VM system (hypervisor) like VirtualBox or VMware, perhaps managing the configuration through Vagrant. These hypervisors can be unstable and painfully slow. They're often the only way to get development work done on a Mac or Windows machine, which is why we use them there for the normal Zulip development environment; but I don't really want to find out how their instability scales when constantly spawning fresh VMs from an image. * Containers. The new hotness, the name on everyone's lips, is Docker. But Docker is not designed for virtualizing a traditional Unix server, complete with its own init system and a fleet of processes with a shared filesystem -- in other words, the platform Zulip's installer and deployment system are for. Docker brings its own quite different model of deployment, and someday we may port Zulip from the traditional Unix server to the Docker-style deployment model, but for testing our traditional-Unix-server deployment we need a (virtualized) traditional Unix server. * Containers, with LXC. LXC provides containers that function as traditional Unix servers; because of the magic of containers, the overhead is quite low, and LXC offers handy snapshotting features so that we can quickly start up a fresh environment from a base image. Running LXC does require a Linux base system. For contributors whose local development machine isn't already Linux, the same solutions are available as for our normal development environment: the base system for running LXC could be e.g. a Vagrant-managed VirtualBox VM, or a machine in a public cloud. This commit adds a first version of such a thing, using LXC to manage a base image plus a fresh container for each test run. The test containers function as VMs: once installed, all the Zulip services run normally in them and can be managed in the normal production ways. This initial version has a shortage of usage messages or docs, and likely has some sharp edges. It also requires familiarity with the basics of LXC commands in order to make good use of the resulting containers: `lxc-ls -f`, `lxc-attach`, `lxc-stop`, and `lxc-start`, in particular.
2018-01-20 01:14:40 +01:00
run ln -sf /usr/share/zoneinfo/Etc/UTC /etc/localtime
run locale-gen en_US.UTF-8 || true
echo "LC_ALL=en_US.UTF-8" | run tee /etc/default/locale
# TODO: on failure, either stop or print message
lxc-stop -n "$CONTAINER_NAME"