Mirror
/
zulip
mirror of https://github.com/zulip/zulip.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
zulip/tools/travis/production-helper

137 lines
9.0 KiB
Plaintext
Raw Normal View History

Expand testing done via Travis CI to cover production pipeline. With this change, we are now testing the production static asset pipeline and installation process in a new testing job (and also run the frontend/backend tests separately). This means that changes that break the Zulip static asset pipeline or production installation process are more likely to fail tests. The testing is imperfect in that it does not have proper isolation -- we build a complete Zulip development environment and then install a Zulip production environment on top of it, so e.g. any apt dependencies installed for Zulip development will still be available for the Zulip production environment. But, it's better than nothing! A good v2 of this would be to have the production setup process just install the minimum stuff needed to run `build-release-tarball` and then uninstall it / clean it up so that we can do a more clear production installation, but that's more work.
2015-10-15 01:47:42 +02:00
#!/bin/bash
set -e
set -x
travis: Fix prompting for user input in production-helper.
2015-12-08 05:31:53 +01:00
apt-get install -y openssl ssl-cert
production-helper: Use ln -nsf to install snakeoil symlinks. This fixes a problem where production-helper was not idempotent.
2016-08-12 06:50:16 +02:00
ln -nsf /etc/ssl/certs/ssl-cert-snakeoil.pem /etc/ssl/certs/zulip.combined-chain.crt
ln -nsf /etc/ssl/private/ssl-cert-snakeoil.key /etc/ssl/private/zulip.key
Expand testing done via Travis CI to cover production pipeline. With this change, we are now testing the production static asset pipeline and installation process in a new testing job (and also run the frontend/backend tests separately). This means that changes that break the Zulip static asset pipeline or production installation process are more likely to fail tests. The testing is imperfect in that it does not have proper isolation -- we build a complete Zulip development environment and then install a Zulip production environment on top of it, so e.g. any apt dependencies installed for Zulip development will still be available for the Zulip production environment. But, it's better than nothing! A good v2 of this would be to have the production setup process just install the minimum stuff needed to run `build-release-tarball` and then uninstall it / clean it up so that we can do a more clear production installation, but that's more work.
2015-10-15 01:47:42 +02:00
production-helper: Remove /root/zulip during setup process. This fixes a problem that caused production-helper to not be idempotent.
2016-08-12 06:50:44 +02:00
rm -rf /root/zulip
Expand testing done via Travis CI to cover production pipeline. With this change, we are now testing the production static asset pipeline and installation process in a new testing job (and also run the frontend/backend tests separately). This means that changes that break the Zulip static asset pipeline or production installation process are more likely to fail tests. The testing is imperfect in that it does not have proper isolation -- we build a complete Zulip development environment and then install a Zulip production environment on top of it, so e.g. any apt dependencies installed for Zulip development will still be available for the Zulip production environment. But, it's better than nothing! A good v2 of this would be to have the production setup process just install the minimum stuff needed to run `build-release-tarball` and then uninstall it / clean it up so that we can do a more clear production installation, but that's more work.
2015-10-15 01:47:42 +02:00
tar -xf zulip-server-travis.tar.gz
mv zulip-server-travis /root/zulip
Fix postgres errors in Travis CI again. Travis CI's model of installing every version of postgres on the test VM and then shutting all the versions other than the one requested down seems to not work very well with doing apt upgrades. It seems the best way to resolve this is to just uninstall the versions we don't need.
2016-01-19 20:44:21 +01:00
# Do an apt upgrade to start with an up-to-date machine
Expand testing done via Travis CI to cover production pipeline. With this change, we are now testing the production static asset pipeline and installation process in a new testing job (and also run the frontend/backend tests separately). This means that changes that break the Zulip static asset pipeline or production installation process are more likely to fail tests. The testing is imperfect in that it does not have proper isolation -- we build a complete Zulip development environment and then install a Zulip production environment on top of it, so e.g. any apt dependencies installed for Zulip development will still be available for the Zulip production environment. But, it's better than nothing! A good v2 of this would be to have the production setup process just install the minimum stuff needed to run `build-release-tarball` and then uninstall it / clean it up so that we can do a more clear production installation, but that's more work.
2015-10-15 01:47:42 +02:00
export APT_OPTIONS="-o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confold"
travis: Do an apt-get update before the apt upgrade. This should save several minutes off the Travis CI `production` suite's runtime, since previously we were doing the full apt upgrade process twice, resulting in things like multiple expensive rebuilds of the initramfs.
2016-05-02 23:34:49 +02:00
apt-get update
travis: hold expensive to upgrade packages in Travis CI. This should save a few minutes of time running the production test suite. This is part of solving #722.
2016-05-02 23:45:44 +02:00
# Hold upgrades to packages which are expensive to upgrade due to size
# or computational cost (e.g. initramfs rebuilds) and aren't really
# used by Zulip in production.
travis: Prevent upgrading oracle-java9-installer. This fixes a minor performance problem, and also avoids errors when Oracle's Java installer site is down.
2017-01-07 04:30:03 +01:00
apt-mark hold initramfs-tools initramfs-tools-bin oracle-java8-installer oracle-java9-installer udev linux-image-3.19.0-28-generic linux-image-generic-lts-vivid base-files linux-firmware chromium-browser google-chrome-stable g++-4.8 gcc-4.8 cpp-4.8 openjdk-6-jre-headless openjdk-7-jre-headless linux-image-generic-lts-xenial
production-helper: Hold tons of packages. This saves almost a minute doing apt upgrades in the production test suite.
2016-06-22 17:59:34 +02:00
# And hold tons more packages that aren't expensive to upgrade but
# there are a lot of. This is super ugly, but since Travis CI's
# machines never update, we can avoid years of package upgrades (takes
# ~60s to install) by doing this.
production-helper: Expand the apt-mark hold list.
2016-12-01 21:02:13 +01:00
apt-mark hold apport apt apt-transport-https apt-utils bash-completion bind9-host binutils binutils-doc bsdutils bzr cloud-guest-utils cloud-init coreutils cpio dnsutils docker-engine dosfstools dpkg dpkg-dev e2fslibs e2fsprogs firefox-locale-pt gcc-4.9-base git-core grub-common grub-pc grub-pc-bin grub2-common icedtea-6-plugin icedtea-7-plugin icedtea-netx icedtea-netx-common ifupdown imagemagick imagemagick-common initscripts irqbalance isc-dhcp-client isc-dhcp-common klibc-utils kpartx krb5-locales krb5-multidev libapt-inst1.5 libapt-pkg4.12 libarchive13 libbind9-90 libblkid1 libc-bin libcgmanager0 libcups2 libcupsfilters1 libcupsimage2 libcurl3-gnutls libcurl4-gnutls-dev libdns100 libdpkg-perl libdrm-dev libdrm-intel1 libdrm-nouveau2 libdrm-radeon1 libdrm2 libexpat1 libexpat1-dev libfreexl1 libgcc1 libgcrypt11 libgcrypt11-dev libgd3 libgl1-mesa-dev libgl1-mesa-dri libgl1-mesa-glx libglapi-mesa libgnutls-dev libgnutls-openssl27 libgnutls26 libgnutlsxx27 libgraphite2-3 libgssapi-krb5-2 libgssrpc4 libgstreamer-plugins-base1.0-0 libgtk2.0-0 libgtk2.0-bin libgtk2.0-common libgudev-1.0-0 libisc95 libisccc90 libisccfg90 libjasper-dev libjasper1 libk5crypto3 libkadm5clnt-mit9 libkadm5srv-mit9 libkdb5-7 libklibc libkrb5-3 libkrb5-dev libkrb5support0 liblcms2-2 liblcms2-dev liblwres90 liblxc1 libmagickcore-dev libmagickcore5 libmagickcore5-extra libmagickwand-dev libmagickwand5 libmount1 libmysqlclient-dev libmysqlclient18 libnettle4 libnl-3-200 libnl-genl-3-200 libnspr4 libnss3 libnss3-nssdb libnuma1 libpam-modules libpam-modules-bin libpam-runtime libpam-systemd libpam0g libpam0g-dev libpci3 libpcre3 libpcre3-dev libpcrecpp0 libpixman-1-0 libpixman-1-dev libpng12-0 libpng12-dev libpolkit-agent-1-0 libpolkit-backend-1-0 libpolkit-gobject-1-0 libpoppler44 libpython3.4 libpython3.4-dev libpython3.4-minimal libpython3.4-stdlib libsndfile1 libss2 libssl-dev libssl-doc libssl1.0.0 libsystemd-daemon0 libsystemd-login0 libtasn1-6 libtasn1-6-dev libtdb1 libtiff5 libtiff5-dev libtiffxx5 libuuid1 libxerces-c3.1 libxml2 libxml2-dev linux-libc-dev login lsb-base lshw lxc lxc-templates mesa-common-dev mount multiarch-support mysql-client mysql-client-5.5 mysql-client-core-5.5 mysql-common ntpdate openjdk-6-jre-lib openssh-client openssh-server openssh-sftp-server os-prober passwd pciutils perl perl-base perl-modules policykit-1 pollinate postgresql-9.1-postgis-scripts postgresql-9.2-postgis-scripts postgresql-client postgresql-client-9.1 postgresql-client-9.2 postgresql-client-9.4 postgresql-client-common postgresql-common python-apt python-apt-common python-bzrlib python-urllib3 python3-apport python3-apt python3-distupgrade python3-gdbm python3-lxc python3-problem-report python3-software-properties python3-update-manager python3.4 python3.4-dev python3.4-minimal rsync scons software-properties-common systemd-services sysv-rc sysvinit-utils tzdata tzdata-java ubuntu-release-upgrader-core uidmap unattended-upgrades unzip update-manager-core usbutils util-linux uuid-runtime accountsservice dbus libaccountsservice0 libdbus-1-3 tar
production-helper: Hold tons of packages. This saves almost a minute doing apt upgrades in the production test suite.
2016-06-22 17:59:34 +02:00
production-helper: use dist-upgrade to match install script. Previously, we were wasting time every time we installed packages, because `apt-get upgrade` would only install most of the packages `apt-get dist-upgrade` would.
2016-06-22 17:42:40 +02:00
apt-get dist-upgrade -y $APT_OPTIONS
travis: Remove rabbitmq nodename dependency on hostname. Because rabbitmq doesn't support changing the nodename of a running rabbitmq node, Zulip installations suffered a plague of issues where e.g. a Zulip server would reboot, the hostname would change, and suddenly the local rabbitmq instance being used by Zulip would stop working. We address this problem by using, by default, a fixed rabbitmq nodename, but providing server administrators the option to set the rabbitmq nodename used by Zulip however they choose. To upgrade an existing server to use this new configuration, one will need to add something like the following to /etc/zulip/zulip.conf: [rabbitmq] nodename = zulip@localhost However, I don't believe we have the puppet code in place to make this work correctly at initial installation without rabbitmq-server being already installed (but off), as we can easily setup in Travis CI but I haven't been willing to do for the installer. So for now, this just fixes our Travis CI problems. Fixes: #1579.
2016-08-10 03:40:07 +02:00
# Disable existing rabbitmq node so we can change it
service rabbitmq-server stop
rm -rf /var/lib/rabbitmq/mnesia/
travis: Workaround postgres 9.1 conflict issues on trusty. We ran into a bug with the Travis CI infrastructure where it postgres 9.1 is installed on the system, and so when we'd do an apt upgrade with a new version of 9.1, the 9.1 daemon would end up getting started and conflict with the 9.3 daemon we were trying to run.
2016-01-10 00:13:12 +01:00
# Install Zulip
Run a local camo server in voyager production environments. Camo is a caching image proxy, used in Zulip to avoid mixed-content warnings by proxying HTTP image content over HTTPS. We've been using it in zulip.com production for years; this change makes it available in standalone Zulip deployments.
2016-04-28 07:32:27 +02:00
env TRAVIS=1 /root/zulip/scripts/setup/install
Expand testing done via Travis CI to cover production pipeline. With this change, we are now testing the production static asset pipeline and installation process in a new testing job (and also run the frontend/backend tests separately). This means that changes that break the Zulip static asset pipeline or production installation process are more likely to fail tests. The testing is imperfect in that it does not have proper isolation -- we build a complete Zulip development environment and then install a Zulip production environment on top of it, so e.g. any apt dependencies installed for Zulip development will still be available for the Zulip production environment. But, it's better than nothing! A good v2 of this would be to have the production setup process just install the minimum stuff needed to run `build-release-tarball` and then uninstall it / clean it up so that we can do a more clear production installation, but that's more work.
2015-10-15 01:47:42 +02:00
cat >>/etc/zulip/settings.py <<EOF
# Travis CI override settings above
travis: Set prod EXTERNAL_HOST to resolve correctly. This is needed to use check_send_receive_time in the tests.
2016-05-08 19:36:29 +02:00
EXTERNAL_HOST = '127.0.0.1'
Expand testing done via Travis CI to cover production pipeline. With this change, we are now testing the production static asset pipeline and installation process in a new testing job (and also run the frontend/backend tests separately). This means that changes that break the Zulip static asset pipeline or production installation process are more likely to fail tests. The testing is imperfect in that it does not have proper isolation -- we build a complete Zulip development environment and then install a Zulip production environment on top of it, so e.g. any apt dependencies installed for Zulip development will still be available for the Zulip production environment. But, it's better than nothing! A good v2 of this would be to have the production setup process just install the minimum stuff needed to run `build-release-tarball` and then uninstall it / clean it up so that we can do a more clear production installation, but that's more work.
2015-10-15 01:47:42 +02:00
ZULIP_ADMINISTRATOR = 'zulip-travis-admin@travis.example.com'
AUTHENTICATION_BACKENDS = ( 'zproject.backends.EmailAuthBackend', )
NOREPLY_EMAIL_ADDRESS = 'noreply@travis.example.com'
DEFAULT_FROM_EMAIL = "Zulip <zulip@travis.example.com>"
settings: Improve ALLOWED_HOSTS defaults logic and docs. This removes the requirement for the user to put localhost/127.0.0.1 in their ALLOWED_HOSTS list, since it is now added automatically. Fixes: #1358.
2016-07-13 07:32:21 +02:00
ALLOWED_HOSTS = []
Expand testing done via Travis CI to cover production pipeline. With this change, we are now testing the production static asset pipeline and installation process in a new testing job (and also run the frontend/backend tests separately). This means that changes that break the Zulip static asset pipeline or production installation process are more likely to fail tests. The testing is imperfect in that it does not have proper isolation -- we build a complete Zulip development environment and then install a Zulip production environment on top of it, so e.g. any apt dependencies installed for Zulip development will still be available for the Zulip production environment. But, it's better than nothing! A good v2 of this would be to have the production setup process just install the minimum stuff needed to run `build-release-tarball` and then uninstall it / clean it up so that we can do a more clear production installation, but that's more work.
2015-10-15 01:47:42 +02:00
EOF
su zulip -c /home/zulip/deployments/current/scripts/setup/initialize-database
travis: Verify the server doesn't 500 in production test.
2016-05-08 03:19:48 +02:00
travis: Verify all supervisord jobs are running in production test. This requires a bit of complexity since supervisord automatically restarts failing jobs.
2016-05-08 03:49:43 +02:00
echo; echo "Now testing that the supervisord jobs are running properly"; echo
travis: Increase the supervisor sleep to 15s for now. The hope is this will help us investigate failures like https://travis-ci.org/zulip/zulip/jobs/203335213.
2017-02-20 08:41:26 +01:00
sleep 15 # Guaranteed to have a working supervisord process get an extra digit
travis: Verify all supervisord jobs are running in production test. This requires a bit of complexity since supervisord automatically restarts failing jobs.
2016-05-08 03:49:43 +02:00
if supervisorctl status | grep -vq RUNNING || supervisorctl status | sed 's/^.*uptime //' | grep -q 0:00:0; then
set +x
echo
echo "FAILURE: Supervisor output shows daemons are crashing:"
echo
supervisorctl status
echo
echo "DEBUG: printing Zulip server's error log:"
cat /var/log/zulip/errors.log
echo
echo "DEBUG: printing Zulip server's workers log:"
cat /var/log/zulip/workers.log
echo
echo "DEBUG: printing Zulip server's tornado log:"
cat /var/log/zulip/tornado.log
exit 1
fi
travis: Fix production suite flakiness. Previously, we were doing this request to the production server before waiting for all the supervisord processes to start; it's possible this could cause failures where we hit the server before the Django uwsgi processes are up. Hopefully fixes #2723.
2016-12-16 06:03:33 +01:00
# TODO: Ideally this would test actually logging in, but this is a start.
echo; echo "Now testing that the newly installed server's homepage loads"; echo
wget https://localhost -O /tmp/index.html --no-check-certificate -S 2> /tmp/wget-output || true # || true so we see errors.log if this 500s
grep -vi '\(Vary\|Content-Language\|expires\|issued by\|modified\|saved\|[.][.][.]\|Date\|[-][-]\)' /tmp/wget-output > /tmp/http-headers-processed
if ! diff -ur /tmp/http-headers-processed ~/success-http-headers.txt; then
set +x
echo
echo "FAILURE: The HTTP Headers returned from loading the homepage on the server do not match the contents of tools/travis/success-http-headers.txt. Typically, this means that the server threw a 500 when trying to load the homepage."
echo "Displaying the contents of the server's error log:"
echo
cat /var/log/zulip/errors.log
echo
echo "Displaying the contents of the main server log:"
echo
cat /var/log/zulip/server.log
exit 1
fi
travis: Add special check for queue processor lists matching.
2017-02-19 22:41:43 +01:00
# Start the RabbitMQ queue worker related section
echo; echo "Now confirming all the RabbitMQ queue processors are correctly registered!"; echo
# These hacky shell scripts just extract the sorted list of queue processors, running and expected
supervisorctl status | cut -f1 -dR | cut -f2- -d: | grep events | cut -f1 -d" " | cut -f3- -d_ | cut -f1 -d- | sort -u > /tmp/running_queue_processors.txt
nagios: Fix RabbitMQ Nagios checks running Django as root. This can cause problems by making the /var/log/zulip files owned by root (not zulip) and thus not writable by the Zulip user.
2017-02-21 06:47:56 +01:00
su zulip -c /home/zulip/deployments/current/scripts/lib/queue_workers.py | grep -v ^test$ > /tmp/expected_queue_processors.txt
travis: Fix various bugs in new queue worker test. * Now queue_workers.py sorts queue names and prints them on their own line. Previously it's output was nondeterministic. * Simplified grep strategy for removing the "test" worker.
2017-02-20 04:34:15 +01:00
if ! diff /tmp/expected_queue_processors.txt /tmp/running_queue_processors.txt >/dev/null; then
set +x
travis: Add special check for queue processor lists matching.
2017-02-19 22:41:43 +01:00
echo "FAILURE: Runnable queue processors declared in zerver/worker/queue_processors.py "
echo "do not match those in puppet/manifests/zulip/base.pp"
echo "See http://zulip.readthedocs.io/en/latest/queuing.html for details."
echo
diff -ur /tmp/expected_queue_processors.txt /tmp/running_queue_processors.txt
exit 1
fi
travis: Run various Nagios checks in production tests.
2016-05-08 03:49:56 +02:00
echo; echo "Now running RabbitMQ consumer Nagios tests"; echo
Fix excessive CPU usage by rabbitmq-numconsumers Nagios checks. The previous model for these Nagios checks was kinda crazy -- every minute, we'd run a full `rabbitmctl list_consumers` for each of the dozen+ consumers that we have, and then do the exact same parsing logic for each to determine whether the target queue has a running consumer to write out a state file. Because `rabbitmctl list_consumers` takes a small amount of resources, on systems where CPU is very limited (e.g. t2 style AWS instances), this minor CPU wastage could be problematic. Now we just do that `rabbitmqctl list_consumers` once per minute, and output all the state files from a single command. Further TODO items on this front include removing the hardcoded list of queues.
2016-08-12 23:09:36 +02:00
# First run the check that usually runs in cron and populates the state files
/home/zulip/deployments/current/scripts/nagios/check-rabbitmq-consumers
travis: Automate updates to production-helper Nagios test. This list was likely to end up out of date quickly, since it wasn't documented that you need to update it when adding a queue. The best solution is to just not require it to be updated.
2017-02-19 22:18:18 +01:00
# Then, compute the list of all Django queue workers to run Nagios checks against
queue_workers: Fix confusing --queue_type argument name.
2017-02-22 09:23:07 +01:00
consumer_list=$(/home/zulip/deployments/current/scripts/lib/queue_workers.py --queue-type=consumer)
travis: Automate updates to production-helper Nagios test. This list was likely to end up out of date quickly, since it wasn't documented that you need to update it when adding a queue. The best solution is to just not require it to be updated.
2017-02-19 22:18:18 +01:00
for consumer in $consumer_list; do
travis: Run various Nagios checks in production tests.
2016-05-08 03:49:56 +02:00
if ! /usr/lib/nagios/plugins/zulip_app_frontend/check_rabbitmq_consumers "$consumer"; then
set +x
echo
echo "FAILURE: Missing Nagios consumer for $consumer; displaying full consumer output:"
rabbitmqctl list_consumers
travis: Improve debuggability of rabbitmq errors.
2017-02-20 08:44:40 +01:00
supervisorctl status
travis: Add logs to rabbitmq consumer debug output.
2017-02-22 09:21:22 +01:00
echo "EVENTS LOGS"
echo
cat /var/log/zulip/events*.log
travis: Run various Nagios checks in production tests.
2016-05-08 03:49:56 +02:00
echo
exit 1
fi
done
echo; echo "Now running additional Nagios tests"; echo
if ! /usr/lib/nagios/plugins/zulip_app_frontend/check_queue_worker_errors || \
! su zulip -c /usr/lib/nagios/plugins/zulip_postgres_appdb/check_fts_update_log || \
nagios-plugins: Add websocket checking to nagios message sending test. - Add websocket client to create connection with SockJS websocket server. It contains callback method to launch after connection setup. - Add '--websocket' parameter to 'check_send_receive_time' script to check websocket connection. - Add testing websocket connection to production installation checking. - Add cronjob to launch websocket connection nagios test. This makes it possible for Zulip Nagios monitoring to check for problems impacting the websockets sending code path, which is what all web users use.
2016-12-29 12:02:44 +01:00
! su zulip -c "/usr/lib/nagios/plugins/zulip_app_frontend/check_send_receive_time --site=https://127.0.0.1/api --nagios --insecure" || \
! su zulip -c "/usr/lib/nagios/plugins/zulip_app_frontend/check_send_receive_time --site=https://127.0.0.1/api --nagios --websocket --insecure"; then
travis: Run various Nagios checks in production tests.
2016-05-08 03:49:56 +02:00
set +x
echo
echo "FAILURE: Nagios checks don't pass:"
echo
echo "DEBUG: printing Zulip server's error log:"
cat /var/log/zulip/errors.log
exit 1
fi
travis: Verify all supervisord jobs are running in production test. This requires a bit of complexity since supervisord automatically restarts failing jobs.
2016-05-08 03:49:43 +02:00
echo "Production installation test successful!"
exit 0