Commit Graph

142 Commits

Author SHA1 Message Date
Tim Abbott aa6567ee34 queue_workers: Fix confusing --queue_type argument name. 2017-02-22 00:23:26 -08:00
Tim Abbott 620f1e444e travis: Fix various bugs in new queue worker test.
* Now queue_workers.py sorts queue names and prints them on their own
  line.  Previously it's output was nondeterministic.
* Simplified grep strategy for removing the "test" worker.
2017-02-19 21:17:42 -08:00
Tim Abbott d6bbcd2737 travis: Automate updates to production-helper Nagios test.
This list was likely to end up out of date quickly, since it wasn't
documented that you need to update it when adding a queue.  The best
solution is to just not require it to be updated.
2017-02-19 16:19:53 -08:00
Tim Abbott 31d4f99573 upgrade: Stop trying to copy node_modules out of tarballs.
Now that we no longer use node_modules at all in production (it's only
used to generate static assets), we don't include `node_modules` in
the production tarballs, and thus we shouldn't attempt to copy
`node_modules` out of the production tarballs when installing.

Fixes a regression introduced in
d71f2e7b9b.
2017-02-07 10:39:31 -08:00
Tim Abbott 126b1c4edc upgrade: Don't require authors updates deploying from Git.
Fixes #3392.
2017-01-23 23:00:09 -08:00
Tim Abbott 22d1aa396b lint: Clean up W503 PEP-8 warning. 2017-01-23 20:50:04 -08:00
Ayush Goyal a85b539c4a zulip_tools: Improve color and copy for run() errors.
Tweaks to the text are edited by tabbott.
2017-01-17 14:37:15 -08:00
Tim Abbott 1d5edff927 setup_venv: Give up if virtualenv-clone isn't working.
virtualenv-clone can sometimes fail if the old virtualenv is
broken; in that case, we can just make a new one.
2017-01-09 11:21:42 -08:00
Tim Abbott 54acbc41ed prod: Ensure the Zulip version of node is installed. 2017-01-06 16:18:29 -08:00
Tim Abbott e4b065e03a Move tools/setup/install-node to scripts/setup/. 2017-01-06 16:18:29 -08:00
Tim Abbott 374e900432 upgrade-zulip: Fix bug in migrations detection. 2017-01-06 16:18:29 -08:00
Tim Abbott e5fbea1007 upgrade-zulip: Move static asset compilation before shutdown.
This saves about a minute of downtime when using
upgrade-zulip-from-git in the default configuration.

It should also save several seconds of downtime when upgrading to a
production release tarball as well.
2017-01-06 14:04:54 -08:00
Tim Abbott 90ee8d1207 upgrade-zulip: Only pause for migrations if not current.
This saves about 1s of downtime when doing a no-op upgrade with the
default settings.  Small, but worthwhile.
2017-01-06 14:03:55 -08:00
Saumya Rawat d3a7d0202f dev: Auto-create log directory in management command.
Check 'zulip/var/log' folder exists and create if not as FileHandler only checks for existence of log file.
2016-12-17 17:18:49 -08:00
Umair Khan fda0387695 Create Python version agnostic venv link for prod.
This link points to either 'zulip-venv' or 'zulip-py3-venv' and makes
uWSGI configration very easy.
2016-12-13 21:40:43 -08:00
Tim Abbott 95a49f6ea2 node_cache: Fix buggy production deployment code.
The previous version of the production deployment code for release
tarballs did not correctly install the node_modules directory.
2016-12-08 10:34:04 -08:00
AZtheAsian c23f9e0df7 pep8: Fix E111 violations 2016-12-01 14:18:11 -08:00
Tim Abbott fd7cb10964 install: Check whether the system has at least 2GB RAM.
This should eliminate a common class of user error installing Zulip.

Fixes #2290, fixes #2320.
2016-11-30 16:07:57 -08:00
Igor Tokarev e6ae53cbff install: Add clear error message if upstart is installed on Xenial.
Fixes #2199.
2016-11-29 19:16:26 -08:00
Rafid Aslam 7a2282986a pep8: Fix E225 pep8 violations. 2016-11-28 15:21:15 -08:00
Anders Kaseorg 78d6c3d7e9 install: Fix RabbitMQ node name if RabbitMQ is not installed.
This indirectly causes the RabbitMQ node name for new Zulip
installations to default to zulip@localhost, which would eliminate the
persistent problems we have had

Fixes #194, #465, #1375, #1751.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2016-11-26 18:54:29 -08:00
Anders Kaseorg 2d6525df04 install: Install python3, python3-six
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2016-11-26 13:26:18 -08:00
Anders Kaseorg 712c98cb48 Use zulip-py3-venv when running on Python 3
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2016-11-26 13:20:22 -08:00
Tim Abbott f5935e81c7 install: Support being run not directly from /root/zulip.
This adds a dependency on the realpath package on trusty; we could try
to remove it if needed, but given that realpath is included in
coreutils on Xenial (and presumably anything else modern), I think
it's reasonable to add it.

Fixes #1797.
2016-11-18 19:56:58 -08:00
Tim Abbott 5b35aada7c zulip_tools: Fix run to not eat error output.
We fix this by just using `subprocess.check_call`.
2016-10-27 12:26:01 -07:00
Tim Abbott 8e82257444 Fix node_cache code to not require root.
The previous code caused problems using a system where the zulip user
doesn't have sudo rights.
2016-10-25 17:52:19 -07:00
Tim Abbott 41b0079ab2 node_cache: Fix success_stamp logic.
Previously, success_stamp was touched whenever we used a particular
node_modules version; it makes more sense to only touch it when the
node_modules directory has actually changed.
2016-10-25 17:52:17 -07:00
Umair Khan e428f3feda Run apt-get update only if sources.list has changed.
Fixes: #2025
2016-10-19 16:23:21 +05:00
Tim Abbott 4a4664d268 mypy: Remove a bunch of now-unnecessary type: ignore annotations.
Since mypy and typeshed have advanced a lot over the last several
months, we no longer need these `type: ignore` annotations.
2016-10-17 11:48:34 -07:00
Tim Abbott fc20c86d8d install: Move apt-get update into setup-apt-repo. 2016-10-16 01:13:50 -07:00
Tim Abbott fcde846b58 setup_venv: Fix handling of Git requirements.
get_package_names did not correctly strip the GitHub URLs from package
names, resulting in the "package names" for our dependencies installed
from Git being tracked with the complete sha1sum included in the name.
This meant that upgrading our virtualenvs incorrectly ended up
resorting to creating an entirely new virtualenv whenever we changed a
dependency that had previously been installed from GitHub URLs.
2016-10-16 01:10:43 -07:00
Tim Abbott 7c16172f9e node_cache: Fix buggy annotations for stdout/stderr. 2016-10-15 22:53:28 -07:00
Tim Abbott 14f6e4c740 scripts: Stop using apt-add-repository.
Unfortunately, apt-add-repository is highly unreliable and was causing
problems both in Travis CI and with developers provisioning their
environment.
2016-10-11 22:10:36 -07:00
K.Kanakhin 14545d1647 create-production-venv: Add PY3 support to production venv creation script.
Create production python virtual env based on system python version.
2016-10-11 14:19:00 -07:00
Diptanshu8 d7253b144c generate-secrets: Refactor to make development/production explicit.
generate-secrets.py now requires --development for development environment
setup or --production for production environment setup (and one of these
options is mandatory).

This solves the problem that it was somewhat easy to accidentally run
generate-secrets.py without the `-d` option while doing manual development
environment setup.

Fixes: #1911.
2016-10-06 17:12:49 -07:00
Tim Abbott a2b91221d0 upgrade-zulip: Only remove zproject/local_settings.py if symlink.
This fixes a problem where if we're deploying from git,
local_settings.py ends up part of the diff in the deployed git
repository.
2016-10-04 20:41:17 -07:00
umkay d260a22637 Add a new statistics/analytics framework.
This is a first pass at building a framework for collecting various
stats about realms, users, streams, etc. Includes:
* New analytics tables for storing counts data
* Raw SQL queries for pulling data from zerver/models.py tables
* Aggregation functions for aggregating hourly stats into daily stats, and
  aggregating user/stream level stats into realm level stats
* A management command for pulling the data

Note that counts.py was added to the linter exclude list due to errors
around %%s.
2016-10-04 17:18:54 -07:00
Umair Khan b4214ec8cb Fix formatting of print in run function. 2016-09-30 10:42:52 -07:00
Tim Abbott 9c0c6c0c23 node_cache: Don't make node_modules symlinks as root. 2016-09-28 00:36:40 -07:00
Steve Howell 13d6f52203 provision: Handle VENV_CACHE_PATH not existing.
If VENV_CACHE_PATH does not exist (which can happen if you destroy
your vagrant environment), then do a short circuit return in
try_to_copy_venv().
2016-09-20 06:55:08 -07:00
Umair Khan e3078b226a Optimize incremental virtualenv creation.
This adds a new system for copying packages from old virtualenvs that
are sufficiently similar to the new virtualenv required.

In practice, this results in a huge performance improvement for
re-provisioning Zulip development environments when the requirements
files have changed (which is the dominant performance problem with
provision today).

Fixes: #1507.
2016-09-19 22:35:32 -07:00
Umair Khan ff0f413d26 Use npm caching in upgrade-zulip-stage-2. 2016-09-16 12:40:48 -07:00
Umair Khan ec74974de3 Create node_cache module for npm caching. 2016-09-16 12:40:48 -07:00
Umair Khan 859b1fbb2c setup_venv.py: Remove comma from the end. 2016-09-08 14:14:28 -07:00
Christie Koehler 586b236375 upgrade: Create prod_settings symlink in step 2 if it doesn't exist.
Between releases 1.3.13 and 1.4.0, local_settings.py was renamed to
prod_settings.py. The upgrade scripts were adjusted to reflect this name
change. But because the first part of the upgrade script is run with the
currently installed version's code, the symlink to /etc/zulip/settings.py is
created with the old name. This was causing upgrade-zulip-stage-2 to fail.

Now upgrade-zulip-stage-2 creates the symlink at zproject/prod_settings.py
if it doesn't already exist.

Fixes #1731.
2016-09-06 14:51:16 -07:00
Tim Abbott 9818a760b5 Install pgroonga in development and (optionally) in prod.
This is preliminary work towards being able to merge support for using
the pgroonga full-text search solution for all languages in Zulip.
2016-08-25 18:03:55 -07:00
Umair Khan 194cbf17a1 Allow run command to accept **kwargs. 2016-08-18 15:06:22 -07:00
Taranjeet Singh d606b95242 zulip_tools.py: Move zulip_tools.py in scripts/lib.
This commit moves zulip_tools.py as part of cleaning the root directory
and organizing proejct into better directory structure.
2016-08-15 16:44:50 -07:00
ukhan 180b438c44 Upgrade setuptools prior to updating mock 2016-08-12 17:56:23 -07:00
Tim Abbott 6496fe2a53 travis: Remove rabbitmq nodename dependency on hostname.
Because rabbitmq doesn't support changing the nodename of a running
rabbitmq node, Zulip installations suffered a plague of issues where
e.g. a Zulip server would reboot, the hostname would change, and
suddenly the local rabbitmq instance being used by Zulip would stop
working.

We address this problem by using, by default, a fixed rabbitmq
nodename, but providing server administrators the option to set the
rabbitmq nodename used by Zulip however they choose.

To upgrade an existing server to use this new configuration, one will
need to add something like the following to /etc/zulip/zulip.conf:

[rabbitmq]
nodename = zulip@localhost

However, I don't believe we have the puppet code in place to make this
work correctly at initial installation without rabbitmq-server being
already installed (but off), as we can easily setup in Travis CI but I
haven't been willing to do for the installer.  So for now, this just
fixes our Travis CI problems.

Fixes: #1579.
2016-08-12 09:38:23 -07:00