Commit Graph

241 Commits

Author SHA1 Message Date
Diptanshu8 d7253b144c generate-secrets: Refactor to make development/production explicit.
generate-secrets.py now requires --development for development environment
setup or --production for production environment setup (and one of these
options is mandatory).

This solves the problem that it was somewhat easy to accidentally run
generate-secrets.py without the `-d` option while doing manual development
environment setup.

Fixes: #1911.
2016-10-06 17:12:49 -07:00
Tim Abbott a2b91221d0 upgrade-zulip: Only remove zproject/local_settings.py if symlink.
This fixes a problem where if we're deploying from git,
local_settings.py ends up part of the diff in the deployed git
repository.
2016-10-04 20:41:17 -07:00
umkay d260a22637 Add a new statistics/analytics framework.
This is a first pass at building a framework for collecting various
stats about realms, users, streams, etc. Includes:
* New analytics tables for storing counts data
* Raw SQL queries for pulling data from zerver/models.py tables
* Aggregation functions for aggregating hourly stats into daily stats, and
  aggregating user/stream level stats into realm level stats
* A management command for pulling the data

Note that counts.py was added to the linter exclude list due to errors
around %%s.
2016-10-04 17:18:54 -07:00
Umair Khan b4214ec8cb Fix formatting of print in run function. 2016-09-30 10:42:52 -07:00
Tim Abbott c819a66e09 postgres-init-db: Use default bash to call flush-memcached. 2016-09-28 20:46:34 -07:00
Tim Abbott 9c0c6c0c23 node_cache: Don't make node_modules symlinks as root. 2016-09-28 00:36:40 -07:00
Sahil Dua 058587da77 Remove extra new lines at the ends of Zulip authoried files.
Fixes #1627.

[tweaked by tabbott to avoid patching third-party modules, for now]
2016-09-26 21:05:24 -07:00
umkay 798e6faa9e provision: Use NVM to install node and npm.
NVM takes a specific node version and installs the node package and
a corresponding compatible npm package.

We use it in a somewhat hackish way to install node/npm globally with
a pinned version, since that's how we actually want to consume node in
our development environment.

Other details:
- Travis CI now is configured to use the version of node installed by
provision; the easiest way to do this was to sabotage the existing node
installation.
- jsdom is upgraded to a current version, which both requires recent
node and also is required for the tests to pass with recent node.
This fixes running the node tests on Xenial.

Fixes #1498.

[tweaked by tabbott]
2016-09-23 14:34:44 -07:00
Steve Howell 13d6f52203 provision: Handle VENV_CACHE_PATH not existing.
If VENV_CACHE_PATH does not exist (which can happen if you destroy
your vagrant environment), then do a short circuit return in
try_to_copy_venv().
2016-09-20 06:55:08 -07:00
Umair Khan e3078b226a Optimize incremental virtualenv creation.
This adds a new system for copying packages from old virtualenvs that
are sufficiently similar to the new virtualenv required.

In practice, this results in a huge performance improvement for
re-provisioning Zulip development environments when the requirements
files have changed (which is the dominant performance problem with
provision today).

Fixes: #1507.
2016-09-19 22:35:32 -07:00
Umair Khan ff0f413d26 Use npm caching in upgrade-zulip-stage-2. 2016-09-16 12:40:48 -07:00
Umair Khan ec74974de3 Create node_cache module for npm caching. 2016-09-16 12:40:48 -07:00
Umair Khan 859b1fbb2c setup_venv.py: Remove comma from the end. 2016-09-08 14:14:28 -07:00
Christie Koehler 586b236375 upgrade: Create prod_settings symlink in step 2 if it doesn't exist.
Between releases 1.3.13 and 1.4.0, local_settings.py was renamed to
prod_settings.py. The upgrade scripts were adjusted to reflect this name
change. But because the first part of the upgrade script is run with the
currently installed version's code, the symlink to /etc/zulip/settings.py is
created with the old name. This was causing upgrade-zulip-stage-2 to fail.

Now upgrade-zulip-stage-2 creates the symlink at zproject/prod_settings.py
if it doesn't already exist.

Fixes #1731.
2016-09-06 14:51:16 -07:00
Tim Abbott 9818a760b5 Install pgroonga in development and (optionally) in prod.
This is preliminary work towards being able to merge support for using
the pgroonga full-text search solution for all languages in Zulip.
2016-08-25 18:03:55 -07:00
Tim Abbott fba93cb494 scripts: rename deploy-zulip-from-git to reference upgrading. 2016-08-25 10:48:59 -07:00
Vishnu Ks a7ead9e99d settings: Eliminate ADMIN_DOMAIN for creating initial realm.
We now use `./manage.py generate_realm_creation_link` as the flow flow
for creating one's first realm.
2016-08-25 09:37:33 -07:00
Umair Khan 194cbf17a1 Allow run command to accept **kwargs. 2016-08-18 15:06:22 -07:00
Tim Abbott e7c3a0c819 check-rabbitmq-consumers: Add missing tornado_return consumer.
I'd like to move this list to be automatically generated, but this
fixes the fact that it's missing for now.
2016-08-17 22:53:00 -07:00
Taranjeet Singh d606b95242 zulip_tools.py: Move zulip_tools.py in scripts/lib.
This commit moves zulip_tools.py as part of cleaning the root directory
and organizing proejct into better directory structure.
2016-08-15 16:44:50 -07:00
ukhan 180b438c44 Upgrade setuptools prior to updating mock 2016-08-12 17:56:23 -07:00
Tim Abbott 88a123d5e0 Fix excessive CPU usage by rabbitmq-numconsumers Nagios checks.
The previous model for these Nagios checks was kinda crazy -- every
minute, we'd run a full `rabbitmctl list_consumers` for each of the
dozen+ consumers that we have, and then do the exact same parsing
logic for each to determine whether the target queue has a running
consumer to write out a state file.

Because `rabbitmctl list_consumers` takes a small amount of resources,
on systems where CPU is very limited (e.g. t2 style AWS instances),
this minor CPU wastage could be problematic.

Now we just do that `rabbitmqctl list_consumers` once per minute, and
output all the state files from a single command.

Further TODO items on this front include removing the hardcoded list
of queues.
2016-08-12 14:09:36 -07:00
Tim Abbott 6496fe2a53 travis: Remove rabbitmq nodename dependency on hostname.
Because rabbitmq doesn't support changing the nodename of a running
rabbitmq node, Zulip installations suffered a plague of issues where
e.g. a Zulip server would reboot, the hostname would change, and
suddenly the local rabbitmq instance being used by Zulip would stop
working.

We address this problem by using, by default, a fixed rabbitmq
nodename, but providing server administrators the option to set the
rabbitmq nodename used by Zulip however they choose.

To upgrade an existing server to use this new configuration, one will
need to add something like the following to /etc/zulip/zulip.conf:

[rabbitmq]
nodename = zulip@localhost

However, I don't believe we have the puppet code in place to make this
work correctly at initial installation without rabbitmq-server being
already installed (but off), as we can easily setup in Travis CI but I
haven't been willing to do for the installer.  So for now, this just
fixes our Travis CI problems.

Fixes: #1579.
2016-08-12 09:38:23 -07:00
Alexander Trost 8d4896809a Deduplication of configuration: Database Initialization
Comment tweaked slightly by tabbott.
2016-08-05 13:33:22 -07:00
Tim Abbott 5bff72c385 Revert "Use apt-add-repository to setup Zulip PPA."
This reverts commit 3f95e567c1.

Apparently `apt-add-repository` fails periodically in CI.  I suspect
this is some sort of silly networking problem, but given that all
we're saving is a few lines of code, the old version was better if
this fails basically ever.
2016-08-05 13:29:17 -07:00
Tim Abbott 3f95e567c1 Use apt-add-repository to setup Zulip PPA. 2016-08-04 22:17:07 -07:00
Tim Abbott 4d02c6efef scripts: Purge old deployments on every upgrade.
Fixes: #554.
2016-08-04 17:02:48 -07:00
Tim Abbott 51f069aa18 scripts: Refactor deployment-finished logging. 2016-08-04 17:02:48 -07:00
Tim Abbott 91b1521578 purge-old-deployments: Avoid purging last/next deployments. 2016-08-04 17:02:48 -07:00
Tim Abbott 1158a86ae7 restart-server: Maintain a last symlink. 2016-08-04 17:02:48 -07:00
Tim Abbott 82ad32f058 deploy-zulip-from-git: Update next symlink. 2016-08-04 17:02:48 -07:00
Tim Abbott 0d39ed82d1 Annotate cron_file_helper. 2016-08-04 15:57:03 -07:00
Tim Abbott 3839a25c74 Annotate generate_secrets.py. 2016-08-04 15:53:23 -07:00
Tim Abbott a12909d0d3 deploy-zulip-from-git: Move migrations/puppet skipping options to config. 2016-08-02 14:31:07 -07:00
Tim Abbott c4d482e722 upgrade-zulip-stage-2: chdir to deploy_path explicitly.
This makes rerunning this manually if it fails much more convenient.
2016-08-02 14:31:07 -07:00
Tim Abbott 7e348df198 deploy-zulip-from-git: Cleanup git_repo_url computation.
Now we assume the main GitHub zulip.git if a repo isn't specified.
2016-08-02 14:31:07 -07:00
Tim Abbott dc4b89fb08 deploy-zulip-from-git: Add a cache directory. 2016-08-02 14:31:07 -07:00
Tim Abbott 0ee3178167 Move update-deployment to scripts/deploy-zulip-from-git. 2016-08-02 14:31:07 -07:00
Tim Abbott fef32af28c update-deployment: Switch to using update-zulip-part-2. 2016-08-02 14:31:07 -07:00
Tim Abbott 972b42ee7b upgrade-zulip-stage-2: Only stop services early when needed. 2016-08-02 14:31:07 -07:00
Tim Abbott 5886d3eeec upgrade-zulip-stage-2: Add --skip-migrations option. 2016-08-02 14:31:07 -07:00
Tim Abbott 759144232f upgrade-zulip-stage-2: Add --skip-puppet option. 2016-08-02 14:31:07 -07:00
Tim Abbott 8ce55b9789 upgrade-zulip-stage-2: Use argparse for options. 2016-08-02 14:31:07 -07:00
Tim Abbott 3b277c3b1f install: Give nice error message on installation failure. 2016-07-31 19:24:45 -07:00
Tim Abbott 084d504c39 install: Remove unnecessary cd command at end. 2016-07-31 19:24:38 -07:00
Tim Abbott 8b285ec0ff puppet: Read camo key from zulip-secrets.conf. 2016-07-31 00:23:24 -07:00
Tim Abbott 3239197fdb puppet: Silence puppet warnings about missing hiera.yaml. 2016-07-30 21:25:19 -07:00
Tim Abbott 63c757eac3 puppet: add zulipconf function for reading settings from zulip.conf.
This makes it convenient for us to have optional user-defined settings
in the main Zulip puppet configuration.
2016-07-30 21:23:12 -07:00
Tim Abbott af54edcaa7 scripts: Move apt repository setup to its own script. 2016-07-30 21:00:18 -07:00
Tim Abbott d07bcf060e install: Cleanup code for PPA setup. 2016-07-30 21:00:01 -07:00