Commit Graph

276 Commits

Author SHA1 Message Date
Tim Abbott 1d5edff927 setup_venv: Give up if virtualenv-clone isn't working.
virtualenv-clone can sometimes fail if the old virtualenv is
broken; in that case, we can just make a new one.
2017-01-09 11:21:42 -08:00
Tim Abbott 54acbc41ed prod: Ensure the Zulip version of node is installed. 2017-01-06 16:18:29 -08:00
Tim Abbott e4b065e03a Move tools/setup/install-node to scripts/setup/. 2017-01-06 16:18:29 -08:00
Tim Abbott 374e900432 upgrade-zulip: Fix bug in migrations detection. 2017-01-06 16:18:29 -08:00
Tim Abbott e5fbea1007 upgrade-zulip: Move static asset compilation before shutdown.
This saves about a minute of downtime when using
upgrade-zulip-from-git in the default configuration.

It should also save several seconds of downtime when upgrading to a
production release tarball as well.
2017-01-06 14:04:54 -08:00
Tim Abbott 90ee8d1207 upgrade-zulip: Only pause for migrations if not current.
This saves about 1s of downtime when doing a no-op upgrade with the
default settings.  Small, but worthwhile.
2017-01-06 14:03:55 -08:00
Saumya Rawat d3a7d0202f dev: Auto-create log directory in management command.
Check 'zulip/var/log' folder exists and create if not as FileHandler only checks for existence of log file.
2016-12-17 17:18:49 -08:00
Umair Khan 336a041ac0 Django 1.10: Use uWSGI.
Fixes: #1121

With some tweaks by tabbott to make the number of processes configurable.
2016-12-13 21:40:43 -08:00
Umair Khan fda0387695 Create Python version agnostic venv link for prod.
This link points to either 'zulip-venv' or 'zulip-py3-venv' and makes
uWSGI configration very easy.
2016-12-13 21:40:43 -08:00
Tim Abbott 95a49f6ea2 node_cache: Fix buggy production deployment code.
The previous version of the production deployment code for release
tarballs did not correctly install the node_modules directory.
2016-12-08 10:34:04 -08:00
anirudhjain75 beaa62cafa mypy: Convert several directories to use typing.Text.
Specifically, these directories are converted: [analytics/, scripts/,
tools/, zerver/management/, zilencer/, zproject/]
2016-12-07 20:51:05 -08:00
AZtheAsian c23f9e0df7 pep8: Fix E111 violations 2016-12-01 14:18:11 -08:00
Tim Abbott fd7cb10964 install: Check whether the system has at least 2GB RAM.
This should eliminate a common class of user error installing Zulip.

Fixes #2290, fixes #2320.
2016-11-30 16:07:57 -08:00
Tommy Ip e4091c6413 pep8: Fix E222 violations. 2016-11-30 21:49:02 +00:00
Tommy Ip 46b7d54b3e pep8: Fix E701 violations. 2016-11-30 20:45:09 +00:00
Igor Tokarev e6ae53cbff install: Add clear error message if upstart is installed on Xenial.
Fixes #2199.
2016-11-29 19:16:26 -08:00
Rafid Aslam 7a2282986a pep8: Fix E225 pep8 violations. 2016-11-28 15:21:15 -08:00
Anders Kaseorg 78d6c3d7e9 install: Fix RabbitMQ node name if RabbitMQ is not installed.
This indirectly causes the RabbitMQ node name for new Zulip
installations to default to zulip@localhost, which would eliminate the
persistent problems we have had

Fixes #194, #465, #1375, #1751.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2016-11-26 18:54:29 -08:00
Anders Kaseorg 207cf6302b Always start python via shebang lines.
This is preparation for supporting using Python 3 in production.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2016-11-26 14:46:37 -08:00
Anders Kaseorg 2d6525df04 install: Install python3, python3-six
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2016-11-26 13:26:18 -08:00
Anders Kaseorg 573ec14955 Remove shebang line from non-scripts
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2016-11-26 13:20:22 -08:00
Anders Kaseorg 712c98cb48 Use zulip-py3-venv when running on Python 3
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2016-11-26 13:20:22 -08:00
Tim Abbott f5935e81c7 install: Support being run not directly from /root/zulip.
This adds a dependency on the realpath package on trusty; we could try
to remove it if needed, but given that realpath is included in
coreutils on Xenial (and presumably anything else modern), I think
it's reasonable to add it.

Fixes #1797.
2016-11-18 19:56:58 -08:00
Tim Abbott 5b35aada7c zulip_tools: Fix run to not eat error output.
We fix this by just using `subprocess.check_call`.
2016-10-27 12:26:01 -07:00
Tim Abbott 8e82257444 Fix node_cache code to not require root.
The previous code caused problems using a system where the zulip user
doesn't have sudo rights.
2016-10-25 17:52:19 -07:00
Tim Abbott 41b0079ab2 node_cache: Fix success_stamp logic.
Previously, success_stamp was touched whenever we used a particular
node_modules version; it makes more sense to only touch it when the
node_modules directory has actually changed.
2016-10-25 17:52:17 -07:00
Umair Khan e428f3feda Run apt-get update only if sources.list has changed.
Fixes: #2025
2016-10-19 16:23:21 +05:00
Tim Abbott 4a4664d268 mypy: Remove a bunch of now-unnecessary type: ignore annotations.
Since mypy and typeshed have advanced a lot over the last several
months, we no longer need these `type: ignore` annotations.
2016-10-17 11:48:34 -07:00
Tim Abbott fc20c86d8d install: Move apt-get update into setup-apt-repo. 2016-10-16 01:13:50 -07:00
Tim Abbott fcde846b58 setup_venv: Fix handling of Git requirements.
get_package_names did not correctly strip the GitHub URLs from package
names, resulting in the "package names" for our dependencies installed
from Git being tracked with the complete sha1sum included in the name.
This meant that upgrading our virtualenvs incorrectly ended up
resorting to creating an entirely new virtualenv whenever we changed a
dependency that had previously been installed from GitHub URLs.
2016-10-16 01:10:43 -07:00
Tim Abbott 7c16172f9e node_cache: Fix buggy annotations for stdout/stderr. 2016-10-15 22:53:28 -07:00
Tim Abbott 8821b269bf flush-memcached: Replace 'set -x' with more clear output.
Similar to `terminate-psql-sessions`, this was just unnecessary spam
output.
2016-10-14 17:10:06 -07:00
Tim Abbott 382c8853f3 terminate-psql-sessions: Remove set -x.
Now that we're no longer actively debugging this tool, there's no need
to have it print everything it's doing.

This will make `test-backend` a lot nicer to use.
2016-10-14 17:08:05 -07:00
Tim Abbott 14f6e4c740 scripts: Stop using apt-add-repository.
Unfortunately, apt-add-repository is highly unreliable and was causing
problems both in Travis CI and with developers provisioning their
environment.
2016-10-11 22:10:36 -07:00
K.Kanakhin 14545d1647 create-production-venv: Add PY3 support to production venv creation script.
Create production python virtual env based on system python version.
2016-10-11 14:19:00 -07:00
Diptanshu8 d7253b144c generate-secrets: Refactor to make development/production explicit.
generate-secrets.py now requires --development for development environment
setup or --production for production environment setup (and one of these
options is mandatory).

This solves the problem that it was somewhat easy to accidentally run
generate-secrets.py without the `-d` option while doing manual development
environment setup.

Fixes: #1911.
2016-10-06 17:12:49 -07:00
Tim Abbott a2b91221d0 upgrade-zulip: Only remove zproject/local_settings.py if symlink.
This fixes a problem where if we're deploying from git,
local_settings.py ends up part of the diff in the deployed git
repository.
2016-10-04 20:41:17 -07:00
umkay d260a22637 Add a new statistics/analytics framework.
This is a first pass at building a framework for collecting various
stats about realms, users, streams, etc. Includes:
* New analytics tables for storing counts data
* Raw SQL queries for pulling data from zerver/models.py tables
* Aggregation functions for aggregating hourly stats into daily stats, and
  aggregating user/stream level stats into realm level stats
* A management command for pulling the data

Note that counts.py was added to the linter exclude list due to errors
around %%s.
2016-10-04 17:18:54 -07:00
Umair Khan b4214ec8cb Fix formatting of print in run function. 2016-09-30 10:42:52 -07:00
Tim Abbott c819a66e09 postgres-init-db: Use default bash to call flush-memcached. 2016-09-28 20:46:34 -07:00
Tim Abbott 9c0c6c0c23 node_cache: Don't make node_modules symlinks as root. 2016-09-28 00:36:40 -07:00
Sahil Dua 058587da77 Remove extra new lines at the ends of Zulip authoried files.
Fixes #1627.

[tweaked by tabbott to avoid patching third-party modules, for now]
2016-09-26 21:05:24 -07:00
umkay 798e6faa9e provision: Use NVM to install node and npm.
NVM takes a specific node version and installs the node package and
a corresponding compatible npm package.

We use it in a somewhat hackish way to install node/npm globally with
a pinned version, since that's how we actually want to consume node in
our development environment.

Other details:
- Travis CI now is configured to use the version of node installed by
provision; the easiest way to do this was to sabotage the existing node
installation.
- jsdom is upgraded to a current version, which both requires recent
node and also is required for the tests to pass with recent node.
This fixes running the node tests on Xenial.

Fixes #1498.

[tweaked by tabbott]
2016-09-23 14:34:44 -07:00
Steve Howell 13d6f52203 provision: Handle VENV_CACHE_PATH not existing.
If VENV_CACHE_PATH does not exist (which can happen if you destroy
your vagrant environment), then do a short circuit return in
try_to_copy_venv().
2016-09-20 06:55:08 -07:00
Umair Khan e3078b226a Optimize incremental virtualenv creation.
This adds a new system for copying packages from old virtualenvs that
are sufficiently similar to the new virtualenv required.

In practice, this results in a huge performance improvement for
re-provisioning Zulip development environments when the requirements
files have changed (which is the dominant performance problem with
provision today).

Fixes: #1507.
2016-09-19 22:35:32 -07:00
Umair Khan ff0f413d26 Use npm caching in upgrade-zulip-stage-2. 2016-09-16 12:40:48 -07:00
Umair Khan ec74974de3 Create node_cache module for npm caching. 2016-09-16 12:40:48 -07:00
Umair Khan 859b1fbb2c setup_venv.py: Remove comma from the end. 2016-09-08 14:14:28 -07:00
Christie Koehler 586b236375 upgrade: Create prod_settings symlink in step 2 if it doesn't exist.
Between releases 1.3.13 and 1.4.0, local_settings.py was renamed to
prod_settings.py. The upgrade scripts were adjusted to reflect this name
change. But because the first part of the upgrade script is run with the
currently installed version's code, the symlink to /etc/zulip/settings.py is
created with the old name. This was causing upgrade-zulip-stage-2 to fail.

Now upgrade-zulip-stage-2 creates the symlink at zproject/prod_settings.py
if it doesn't already exist.

Fixes #1731.
2016-09-06 14:51:16 -07:00
Tim Abbott 9818a760b5 Install pgroonga in development and (optionally) in prod.
This is preliminary work towards being able to merge support for using
the pgroonga full-text search solution for all languages in Zulip.
2016-08-25 18:03:55 -07:00