Commit Graph

184 Commits

Author SHA1 Message Date
Tim Abbott 759144232f upgrade-zulip-stage-2: Add --skip-puppet option. 2016-08-02 14:31:07 -07:00
Tim Abbott 8ce55b9789 upgrade-zulip-stage-2: Use argparse for options. 2016-08-02 14:31:07 -07:00
Tim Abbott 084d504c39 install: Remove unnecessary cd command at end. 2016-07-31 19:24:38 -07:00
Tim Abbott 63c757eac3 puppet: add zulipconf function for reading settings from zulip.conf.
This makes it convenient for us to have optional user-defined settings
in the main Zulip puppet configuration.
2016-07-30 21:23:12 -07:00
Tim Abbott af54edcaa7 scripts: Move apt repository setup to its own script. 2016-07-30 21:00:18 -07:00
Tim Abbott d07bcf060e install: Cleanup code for PPA setup. 2016-07-30 21:00:01 -07:00
Eklavya Sharma 51ea5c1602 scripts/: Make subprocess calls unicode-aware. 2016-07-26 12:06:41 -07:00
Alexander Trost 6e10236972 Disable auto-service-restart for docker installations. 2016-07-25 16:40:29 -07:00
Eklavya Sharma baa157344c Patch activate script only in development. 2016-07-21 14:11:41 -07:00
Eklavya Sharma e7813094d7 Patch activate script to set VIRTUAL_ENV correctly.
Patch a virtualenv's activate script to not resolve symlinks when
setting the environment variable VIRTUAL_ENV.

Fixes #1190.
2016-07-21 14:11:41 -07:00
Tim Abbott dd6e92a714 setup_venv: Fix missing dependencies on image libraries. 2016-07-20 14:55:45 -07:00
Eklavya Sharma 576a090499 Add libpq-dev to VENV_DEPENDENCIES. 2016-07-20 22:15:06 +05:30
Tim Abbott 35339f5117 Rename local_settings_template to prod_settings_template. 2016-07-19 20:59:59 -07:00
Tim Abbott 19b860ceec Rename local_settings.py symlink to prod_settings.py. 2016-07-19 20:59:59 -07:00
Tim Abbott 9c9b6176a9 install: Fix feature detection/set -e incompatibility.
Previously, the whole script would stop when a feature wasn't
available.
2016-07-19 20:12:27 -07:00
Tim Abbott 8eaa75b90f setup_venv: Remove unnecessary postgresql-server-dev-all dependency.
This package is a dependency for building tsearch_extras, but isn't
needed for building a Zulip virtualenv.
2016-07-19 19:19:42 -07:00
Umair Khan 9f42fda7f4 Use stripped down version of Talon. 2016-07-13 11:24:18 -07:00
Umair Khan 395e053ce3 Revert "Revert "Extract reply from email.""
This reverts commit f1ba3ded42.
2016-07-13 11:24:18 -07:00
Tim Abbott 883e991adf install: Improve support for non-default puppet rules.
Previously, the install script would fail if you passed various
non-default puppet rules, since the code to configure and restart
services that runs later on in the install script largely ran
unconditionally, regardless of whether the relevant service was
actually installed on the target system.

This should make the main install script reusable for installing
e.g. a dedicated Postgres server for use with Zulip.
2016-07-12 14:12:09 -07:00
Tim Abbott 80bf7e32a4 Add option to not create a virtualenv. 2016-07-12 14:12:05 -07:00
Eklavya Sharma f1ba3ded42 Revert "Extract reply from email."
This reverts commit f1f48f305e.

The use of sklearn unfortunately caused a substantial slowdown to the
Zulip provisioning process, which didn't seem worth it for a
relatively minor feature.
2016-07-10 11:30:30 -07:00
Umair Khan f1f48f305e Extract reply from email. 2016-07-08 10:58:25 -07:00
Eklavya Sharma 7e020e3dae scripts/lib/setup_venv.py: Use universal_newlines=True.
In python 3, subprocess uses bytes for input and output if
universal_newlines=False (the default).  It uses str for input and
output if universal_newlines=True.
Since we're dealing with strings here, add universal_newlines=True
to subprocess.check_output calls.
2016-07-06 13:50:55 -07:00
Eklavya Sharma f1b72c5f41 Upgrade wheel before installing venv requirements.
This is required because an older wheel version causes problems
when building wheels on python 3.
2016-07-02 11:53:26 -07:00
Eklavya Sharma 0b714ea6c6 Use python2-specific requirements files. 2016-07-02 11:53:26 -07:00
Tim Abbott f094123fd3 install: Add support for installing Xenial systems.
This isn't fully supported yet, but merging this makes it more
convenient to test Zulip on Ubuntu Xenial.
2016-06-28 23:05:38 -07:00
Tim Abbott 4aae4de294 Properly cache production virtualenvs in Travis CI.
Previously, the --travis argument wasn't available to the `setup_venv`
code when creating the production virtualenv.
2016-06-28 12:04:11 -07:00
Tim Abbott 7ed0ab8c4a Ensure Zulip virtualenvs are world-readable.
This is important for both ensuring the Nagios checks work correctly
in production, as well as making sure the `zulip` user can access the
virtualenv (owned by the `travis` user) in Travis CI.
2016-06-28 12:03:01 -07:00
Tim Abbott 4a7913bc22 setup_venv: Use correct VENV_CACHE_PATH for production suite. 2016-06-27 21:16:22 -07:00
Eklavya Sharma a9835c0ab2 Activate virtualenv in production Python code.
The manage.py change effectively switches the Zulip production server
to use the virtualenv, since all of our supervisord commands for the
various Python services go through manage.py.

Additionally, this migrates the production scripts and Nagios plugins
to use the virtualenv as well.
2016-06-27 19:55:35 -07:00
Eklavya Sharma 64affb83f9 Create a virtualenv when installing/upgrading production instances. 2016-06-27 19:55:11 -07:00
Tim Abbott dfd8fede4a setup_venv: Add missing build essential dependencies.
Apparently, c74a74dc74 introduced a bug
where we are no longer correctly depending on build-essential as part
of the Zulip development environment installation process.

Fixes #1111.
2016-06-24 09:11:10 -07:00
Eklavya Sharma 9482cdccf4 scripts/lib/setup-venv.py: Accept None as target_venv_path.
If target_venv_path in setup_virtualenv is None, create a virtualenv
named 'venv' but don't make a symlink to it.  Also return
cached_venv_path.
2016-06-23 14:17:17 -07:00
Eklavya Sharma 0365a4a9f2 Factor out apt dependencies for setting up virtualenv.
Move apt dependencies for creating a virtualenv from provision.py
to scripts/lib/setup_venv.py.
2016-06-23 14:17:17 -07:00
Tim Abbott d3f3046629 Add python-six to early installer dependencies.
Since we're now using python-six in zulip-puppet-apply, we need to
install python-six before calling into zulip-puppet-apply.
2016-06-22 08:11:34 -07:00
Eklavya Sharma 121a5f26fb Move hash_reqs.py from tools/ to scripts/lib/.
This is needed because hash_reqs.py is used to create a virtualenv.
Currently we only use virtualenv in development, but we will soon
start using it in production.  Scripts used in production should be
put in scripts/.
2016-06-22 18:12:08 +05:30
Eklavya Sharma fc8d4f9ef5 Factor out venv-creating code from provision.py.
Move setup_virtualenv and do_setup_virtualenv from provision.py to
scripts/lib/setup_venv.py.
2016-06-21 11:25:41 -07:00
Eklavya Sharma 149938d468 Change shebangs from python2.7 to python. 2016-05-29 05:03:08 -07:00
Tim Abbott a315849a9e Move bin/log-management-command to scripts/lib/.
We're in the process of eliminating the bin/ subdirectory in favor of
the scripts/ tree, and this one isn't user-facing.
2016-05-07 19:37:06 -07:00
Tim Abbott 52c1e8ac7d Run a local camo server in voyager production environments.
Camo is a caching image proxy, used in Zulip to avoid mixed-content
warnings by proxying HTTP image content over HTTPS.  We've been using
it in zulip.com production for years; this change makes it available
in standalone Zulip deployments.
2016-05-02 17:21:31 -07:00
Tim Abbott dc772518e7 Don't chown supervisor socket if it doesn't exist. 2016-04-26 15:27:35 -07:00
Tim Abbott 6a3c775842 install: Ensure prod-static/serve is created. 2016-04-26 15:27:35 -07:00
Tim Abbott bb25b6060e install: Avoid unnecessarily storing apt key under /root. 2016-04-26 15:07:53 -07:00
Tim Abbott e9416a9fb2 install: Add PUPPET_CLASSES variable. 2016-04-26 15:06:37 -07:00
Tim Abbott a9d86a3620 install: Add DEPLOYMENT_TYPE variable. 2016-04-26 15:04:32 -07:00
Tim Abbott 68c6d514e8 install: Add ADDITIONAL_PACKAGES option. 2016-04-26 15:02:28 -07:00
Tim Abbott ae48f6394b migrations: Disable prompting about content-type deletion.
The main function of prompting inside `manage.py migrate` is to ask
the user if they want to delete stale content-types, which is
unimportant and likely scary, so we disable doing so.
2016-04-13 13:19:02 -07:00
Tim Abbott 4352a022cd Add type: ignore to wildcard imports in pythonrc.py. 2016-04-03 15:40:24 -07:00
Tim Abbott f9222de83e Auto-load commonly used modules in manage.py shell.
This automatically loads settings, zerver.models.* and
zerver.lib.actions.* when you start `manage.py shell`, which should
save a bit of time basically every time someone uses it.

Fixes #275.
2016-03-19 11:32:49 -07:00
Eklavya Sharma c59185e119 Apply Python 3 futurize transform libfuturize.fixes.fix_print_with_import
Refer #256
2016-03-10 22:02:17 -08:00
Vladislav Manchev dfbea01c8f Add support for running OpenBSD in development environment. 2016-01-21 22:33:55 -08:00
Tim Abbott 26e9d55e16 deployments: Refactor locking libraries into zulip_tools.py.
The code in update-deployment and upgrade-zulip for managing the
deployment lock was nearly identical.
2016-01-11 21:36:42 -08:00
Tim Abbott f871090bb6 upgrade-zulip: Archive release tarballs at /home/zulip/archives.
A common issue when doing a Zulip upgrade is trying to pass
upgrade-zulip a tarball path under /root, which doesn't work because
the Zulip user doesn't have permission to read the tarball.  We
could fix this by just unpacking the tarballs as root, but it seemed
like a nicer approach would be to archive the release tarballs
somewhere readable by the Zulip user (/home/zulip/archives) and unpack
them from there.

Fixes #208.
2016-01-11 21:36:42 -08:00
Tim Abbott c101bf663d Run upgrade-zulip-stage-2 from an absolute path.
This should make it more obvious in tracebacks that we are running the
script from the version of Zulip we're upgrading to, not the old
version.
2016-01-11 21:36:42 -08:00
Tim Abbott 52d0423591 Document structure of upgrade-zulip-stage-2 more clearly. 2016-01-11 21:36:41 -08:00
Tim Abbott 186f563176 Fix deployment locks being leaked when a deployment fails.
The point of the lock is to prevent two deployments happening at the
same time and racing with each other, not to prevent doing any future
deployments after an error happens (which is what the current
implementation does in practice).

Addresses part of #208.
2016-01-11 21:36:41 -08:00
Tim Abbott e7353902df upgrade-zulip: Restart process-fts-updates if also a DB host.
Previoulsy, process-fts-updates wouldn't be restarted on a server
upgrade in Voyager configurations.
2016-01-09 16:59:43 -08:00
Tim Abbott 75b5a1b8da upgrade-zulip: Stop zulip-senders too when shutting down services. 2016-01-09 16:59:43 -08:00
Tim Abbott 7395003e6a Fix buggy #! lines using "/usr/bin/env python2.7 -u".
The #! line processing interpreted the argument to pass to `env` as
"python2.7 -u", which obviously isn't a real program.

We fix this by setting the PYTHONUNBUFFERED environment variable
inside the program, which has the same effect.

Thanks to Dan Fedele for the bug report and suggested solution!
2015-11-16 08:46:48 -08:00
Tim Abbott 6eb670097c Expand testing done via Travis CI to cover production pipeline.
With this change, we are now testing the production static asset
pipeline and installation process in a new testing job (and also run
the frontend/backend tests separately).

This means that changes that break the Zulip static asset pipeline or
production installation process are more likely to fail tests.  The
testing is imperfect in that it does not have proper isolation -- we
build a complete Zulip development environment and then install a
Zulip production environment on top of it, so e.g. any apt
dependencies installed for Zulip development will still be available
for the Zulip production environment.  But, it's better than nothing!

A good v2 of this would be to have the production setup process just
install the minimum stuff needed to run `build-release-tarball` and
then uninstall it / clean it up so that we can do a more clear
production installation, but that's more work.
2015-11-01 18:11:39 -08:00
Tim Abbott b7cd000af6 install: Check nginx configuration is valid.
It's better to fail here and have the user correct the issue than fail
later.
2015-11-01 18:06:59 -08:00
Steven Oud d5435fad1d Consistently use /usr/bin/env python2.7 in shebangs and commands. 2015-10-21 22:58:21 +00:00
Tim Abbott 92aebe595b Dramatically extend post-install documentation for production Zulip. 2015-09-30 09:04:14 -07:00
Tim Abbott 40ec59b93e install: Add nice error message for RabbitMQ not having started. 2015-09-29 18:41:31 -07:00
Tim Abbott 96bd1c38dc install: Make sure python is installed before using it.
This is relevant for completely bare Ubuntu systems which might only
have python3 installed.

Fixes #40.
2015-09-26 21:34:36 -07:00
Tim Abbott 9748780192 Remove unnecessary puppet.conf configuration.
Fixes #23.
2015-09-26 21:34:19 -07:00
Tim Abbott 00bf7b25b5 install: Remote the python-django-guardian hack.
This is no longer required since it's in the PPA now.
2015-09-26 18:01:18 -07:00
Andreas Litt aeb6a5df7c Use current working dir instead of /tmp 2015-09-25 22:11:59 +02:00
Andreas Litt 94c35d8fb0 Fix 'python-django-guardian' download
While the docu on https://www.zulip.org/server.html says:
```
cd /root/zulip
./scripts/setup/install
```
This script downloads the `python-django-guardian_1.3-1~zulip4_all.deb` file to current working dir (`/root/zulip` if you follow the docu), but tries to install it from /root/.
This fails obviously. So i changed the download location to /tmp/.
2015-09-25 22:02:13 +02:00
Tim Abbott b69c6228af Hackishly install a correct version of python-django-guardian in prod install.
(imported from commit 1027e05a1a9805a17fcb4b8dea2c2cf8cda280af)
2015-09-25 09:45:26 -07:00
Tim Abbott abb1a13e31 Switch to new Trusty PPA for Voyager install process.
(imported from commit 91fdc16f121938d36c389f5b285f0da942558759)
2015-09-25 09:45:26 -07:00
Tim Abbott 6540807761 install: Copy rather than moving local_settings_template.py.
(imported from commit 23e5af225738af32567c543df75375cd5ce378dc)
2015-09-24 22:41:00 -07:00
Tim Abbott 2fb9560476 puppet: enterprise=>voyager for single-server configuration.
(imported from commit 7cc47ac7c470239cd00f69cfd3137b39a0d06e3a)
2015-09-24 22:40:59 -07:00
Tim Abbott d281fc75fd Rename generate_voyager_secrets.py => generate_secrets.py.
(imported from commit c2f370c3b241601b4f6883d3953ceec1efda71be)
2015-08-21 10:33:36 -07:00
David Roe 3f7cb34b00 enterprise => voyager
(imported from commit 04be792bb480d5e5db1c91d296d1000cf1682571)
2015-08-21 10:33:35 -07:00
Jonathan Dahan e95dfd78ed Fix production deployment script.
(imported from commit 5b8398a0075dc297d3f86c65ce2ef43977e57ecb)
2015-08-20 20:50:41 -07:00
Luke Faraone 490af28958 Update URL for user APT key.
We're moving to S3, so apt.z.c no longer serves SSL.

(imported from commit a2d376337ddaf2f5fd4ddf193a1ff0732f89c78a)
2013-12-11 10:54:30 -05:00
Tim Abbott 3665a0e289 Fix permissions issues with upgrade-zulip from /root path.
(imported from commit b7fce92ac1b7317388debaae2bc3270fe3ae155e)
2013-11-25 17:16:15 -05:00
Tim Abbott 2e5eec5bd3 install: Remove apache restart from install process.
We don't use apache in the main app -- only for the SSO situation --
this code was just copied from our own install script.  And it caused
problems at CUSTOMER13 because they installed Apache in preparation for
the SSO integration, but restarting it failed.

(imported from commit 3f2961574134847c836e8b69736f60d9f8790201)
2013-11-15 15:04:13 -05:00
Zev Benjamin 974159ec94 Move apache2 restart for SSO sites to restart-server
(imported from commit f999e2b0591a11442c1d3fdba2393ecf6e78bad3)
2013-11-15 11:34:48 -05:00
Leo Franchi 68954ee89a Show info on how to remove lockdir in upgrade-deployment
(imported from commit 59984c9b3ce7ef133f0e929836533ae9d6d40707)
2013-11-14 12:00:03 -05:00
Tim Abbott af21f943a2 Fix paths to zulip_tools for relocated scripts.
(imported from commit 9a3664c0f7da1ce6dcb554e7e9ddef5bde6cad7e)
2013-11-14 11:16:16 -05:00
Tim Abbott f9707dd903 Clean up scripts/ by moving helper tools into lib/.
(imported from commit 31a99d5cc68524a1399daa50eb1fcda54383a317)
2013-11-14 08:47:11 -05:00
Zev Benjamin 87a6838e0e enterprise: Log the output of upgrade-zulip and install scripts
We may eventually want to rotate the log files, but this seems good
enough for now.

(imported from commit 9a54fa6b40bc62f68e52ef552c1a676856b21829)
2013-11-14 08:08:59 -05:00