Commit Graph

338 Commits

Author SHA1 Message Date
Harshit Bansal 8c9ea94878 scripts: Fix an issue in `purge-old-deployments` script.
We were not including the real path of the symlinks due to which we
were incorrectly deleting deployments pointed by last/current/next.
2017-10-30 23:09:51 -07:00
Harshit Bansal 1871d6fe1f minor: Remove unnecessary path juggling in `get_recent_deployments()`. 2017-10-29 14:38:20 -07:00
Tim Abbott 11ab545f3b install: Set the locale so our dependencies can install.
Many pip packages don't install properly without a US locale.
2017-10-29 11:49:08 -07:00
rht 8b6b4e043f install: Add option to get certs via certbot.
While this doesn't quite complete our plans for certbot support (it's
not documented, etc.), this is a great stride forward.
2017-10-27 17:19:34 -07:00
Shekh Ataul d239f77966 refactor: Replace mkdir_p functions with Python 3 builtin.
This didn't exist in Python 2, but it does in Python 3, so we get to
reap the rewards of dropping Python 2 support.

Fixes #7082.
2017-10-25 11:06:11 -07:00
Tim Abbott 6b1eb647e4 nginx: Fix bugs in new nginx configuration checks. 2017-10-24 14:29:36 -07:00
Tim Abbott 730c77c7df docs: Document scripts/setup/generate-self-signed-certs.
And more generally clean up our non-LetsEncrypt SSL docs.

This should make it a bit easier to setup a Zulip server.
2017-10-24 13:48:14 -07:00
Tim Abbott 1b653409f4 install: Provide a nicer error message for bad nginx configuration.
This also covers missing SSL configuration errors nicely.
2017-10-24 13:39:39 -07:00
rht c8090aafa4 lint: Remove several unused imports. 2017-10-18 10:56:11 -07:00
rht a603a4f9f5 Remove `from __future__ import absolute_import`.
Except in:
- docs/writing-bots-guide.md, because bots are supposed to be Python 2
  compatible
- puppet/zulip_ops/files/zulip-ec2-configure-interfaces, because this
  script is still on python2.7
- tools/lint
- tools/linter_lib
- tools/lister.py

For the latter two, because they might be yanked away to a separate repo
for general use with other FLOSS projects.
2017-10-17 22:59:42 -07:00
Tim Abbott 2ae2a94444 provision: Stop using shared var/ for caching apt state.
This didn't work at all when one did a `vagrant destroy` and then
`vagrant up`, because the cache state would be preserved even though
the machine is gone.

Fixes #5981.
2017-10-17 21:15:58 -07:00
rht 9ab54e5bd7 scripts/lib/install: Add flag to specify key settings.
This should make it easier to script the installation process, and
also conveniently are the options one would want for the --certbot
option.

Significantly modified by tabbott to have a sane right interface,
include --help, and avoid printing all the `set -x` garbage before the
usage notices.
2017-10-03 16:56:45 -07:00
Alexander Trost 6a7d34316c Add scripts to be used in docker-entrypoint.sh.
Based on #450, with commits
restructured by Rein Zustand.
Tweaks by Rein Zustand:
- Replace configure-cert with generate-self-signed-certs
- `mv scripts/lib/create-zulip-admin.sh scripts/lib/create-zulip-admin`
2017-09-30 09:55:48 -07:00
rht 71188d7b0a scripts: Remove import print_function. 2017-09-29 15:43:30 -07:00
Alexander Trost dea507f4b8 Add script to create admin from CLI. 2017-09-27 20:15:41 -07:00
Tim Abbott 358cb40ed1 cache: Add backwards compatibility for emoji cache.
This allows the emoji cache cleaning code to run against old emoji
caches.
2017-09-25 17:06:02 -07:00
Tim Abbott db982672b1 caches: Don't depend on yarn.lock existing.
This allows our cache cleaning code to run on servers that have copies
of Zulip 1.6 and older around.
2017-09-25 17:05:45 -07:00
Tim Abbott 0f19e501a6 caches: Suppress unnecessary output when cleaning caches.
This should make the cache cleaning process a lot less spammy.
2017-09-25 16:34:03 -07:00
Tim Abbott 703b87c116 caches: Move clean-unused-caches to scripts/lib/. 2017-09-25 16:34:03 -07:00
Tim Abbott 86a07baf40 zulip_tools: Skip the lock directory.
This is the one special directory that usually lives in deployments/
and is not a deployment.  Make sure we don't treat it as a deployment.
2017-09-25 15:15:32 -07:00
Harshit Bansal a6caf30ca7 scripts: Fix an issue in `get_recent_deployments()` due to relative paths.
We were checking for whether an item in the deployments directory
represents a directory but were using its relative path which was
causing a false value to be returned for all items irrespective of
their being a directory or not if the script was invoked from some
where other than the deployments directory.
2017-09-25 11:51:24 -07:00
Harshit Bansal 6ff7da04de emoji: Remove `NotoColorEmoji.ttf`.
We no longer use glyphs from `NotoColorEmoji.ttf` so removing this.
2017-09-24 04:51:33 -07:00
Harshit Bansal c8c1c8ef43 emoji: Remove `AndroidEmoji.ttf`. 2017-09-24 04:51:33 -07:00
Harshit Bansal 57161a92a1 scripts: Rearrange the arguments of `purge_unused_caches()`.
This commit re-arranges the arguments of `purge_unused_caches()`
function in order to remain consistent with other similar functions
in the library like `may_be_perform_caching()`.
2017-09-24 04:37:31 -07:00
Harshit Bansal df7ea375c1 scripts: Make default mode of cache-cleaning scripts much less verbose.
Print a detailed report only if `--verbose` flag is specified.

Fixes: #6632.
2017-09-24 04:37:31 -07:00
Harshit Bansal 3e8469a717 zulip_tools: Remove the now unused `GENERIC_CACHE_SCRIPT_PARSER`.
This has been replaced by `parse_cache_script_args()`.
2017-09-24 04:37:31 -07:00
Harshit Bansal 7f752f0942 scripts: Use `parse_cache_script_args()` in various cache cleaning scripts.
Instead of using `GENERIC_CACHE_SCRIPT_PARSER` and defining `parse_args()`
function in each script separately, use `parse_cache_script_args()`.
2017-09-24 04:37:31 -07:00
Harshit Bansal fe80330708 zulip_tools: Add `parse_cache_script_args()`.
This function will replace the repetitive definition of `parse_args()`
in various cache cleaning scripts. Also adds a `--verbose` argument
to the parser.
2017-09-24 04:37:31 -07:00
Harshit Bansal 4e6b68d02f zulip_tools: Change `purge_unused_caches()` API.
Instead of accepting individual arguments, accept `argparse.Namespace`
object as an argument.
2017-09-24 04:37:31 -07:00
Tim Abbott 0a91a5510c install: Fix check for whether update-prod-static is needed.
The previous version seems to be created without update-prod-static.
2017-09-22 19:52:40 -07:00
Tim Abbott 304bd86173 install: Support installing a Zulip server from a Git checkout.
Historically, one has needed to build a release tarball in order to
use/test the Zulip installer, but you could upgrade a Zulip server
from Git.  However, the only reason for that requirement was that we
didn't run `tools/update-prod-static` as part of the install script if
it's required.  A good test for that case is whether we're in a Git
repository, but a better one is to check whether the prod-static
content exists in the tarball paths.

Fixes #3704.
2017-09-22 15:47:42 -07:00
Juliana Bacelar 928dd06cc8 linter: Add lint rule banning 'import os.path' 2017-09-22 10:32:21 -07:00
julianasobreira df2d448f7b python: Add lint rule banning 'from os.path import'.
This enforces our use of a consistent style in how we access Python
modules; "from os.path import dirname" is a particularly popular
abbreviation inconsistent with our style, and so it deserves a lint
rule.

Commit message and error text tweaked by tabbott.

Fixes #6543.
2017-09-22 04:55:38 -07:00
Harshit Bansal 20f062f726 zulip_tools.py: Extract `may_be_perform_purging()` function.
Based on the `dry_run` flag, this function either purges the list
of directories passed to them or prints a listing of the directories
it would have purged/kept_back, had the `dry_run` flag been false.
2017-09-16 08:28:57 -07:00
Tim Abbott 600b164130 setup-apt-repo: Fix failure to rerun properly on provision failure.
Apparently, the refactoring to make this script only run when changes
are present was buggy, in that if `apt-get update` failed, running
provision against wouldn't rerun `apt-get update`, resulting in a
broken state that requires expertise to fix.  This closes that gap, by
using a stamp file to ensure we always successfully update apt before
proceeding.

It doesn't fix existing installations.
2017-09-12 07:08:54 -07:00
Tim Abbott 1a1df29053 get_recent_deployments: Skip uwsgi socket and friends.
This fixes an exception when running clean-venv-caches in production.
2017-08-27 18:18:53 -07:00
Tim Abbott e38ac00f3c caches: Move cache cleaning helper tools under scripts/lib.
This helps keep the root of scripts/ uncluttered.
2017-08-27 17:59:49 -07:00
Harshit Bansal facb5dbe85 zulip_tools.py: Extract `generate_sha1sum_emoji()`.
Given the path of a zulip installation, it returns a hash corresponding
to the emoji infrastructure of that installation.
2017-08-27 17:51:24 -07:00
Harshit Bansal 8e41bbe2b0 node_cache.py: Modify `generate_sha1sum_node_modules()`.
Modify `generate_sha1sum_node_modules()` such that it can calculate
the hash for a particular installation.

Tweaked by tabbott to use os.path.realpath in the setup_dir
calculation, to ensure it's consistent.
2017-08-27 17:51:24 -07:00
Harshit Bansal 36420ab636 zulip_tools.py: Add `purge_caches()` function.
This function can be used for purging unused cache directories.
2017-08-27 17:37:08 -07:00
Tim Abbott fa97dd1408 setup_path_on_import: Drop Python 2 support. 2017-08-23 19:21:50 -07:00
Tim Abbott 2424819749 install: Move upstart checks a bit earlier.
This should make it much more likely that users see this before
waiting a long time for other things to happen, since the `apt-get
dist-upgrade` step is really slow.  We can't move further to the top,
since this requires `lsb_release` to be installed.
2017-08-23 14:55:01 -07:00
Harshit Bansal 504abfce63 zulip_tools.py: Add `GENERIC_CACHE_SCRIPT_PARSER`.
This parser will act as a parent parser for all the cache cleaning scripts.
2017-08-23 00:00:34 -07:00
Harshit Bansal 6936bb1ba0 zulip_tools.py: Add `get_caches_to_be_purged()` function.
Given the path of directory containing all the caches, a list of
caches in use and threshold days, this function returns a list
of caches which can be removed safely.
2017-08-22 23:59:45 -07:00
Harshit Bansal e71f92b09e zulip_tools.py: Add `get_threshold_timestamp()` function.
Given `threshold_days` this function returns a timestamp corresponding
to the time before threshold number of days.
2017-08-22 23:57:20 -07:00
Harshit Bansal 8954605726 zulip_tools.py: Add `get_recent_deployments()` function.
This function returns a list of all the deployments directories
which are newer than some threshold number of days including the
`/root/zulip` directory if it exists.
2017-08-22 23:57:20 -07:00
Harshit Bansal 931e4752aa zulip_tools.py: Add `get_environment()` function.
This function can be used to determine the environment in which a
script is being executed.
2017-08-22 23:57:20 -07:00
rht 0a469fd4c8 requirements: Lock prod.txt instead of its py3_common dependency.
`py3_common.txt` is no longer locked.
2017-08-22 10:10:57 -07:00
rht 6a5869ec8d requirements: Rename py3_prod.txt to prod.txt. 2017-08-22 10:10:57 -07:00
Greg Price f73e898874 manage.py: Save an extra Django startup by converting one script to a library.
This saves us from spending 200-250ms of CPU time importing Django
again just to log that we're running a management command.  On
`scripts/restart-server`, this saves us from one thundering herd of
Django startups when all the queue workers are restarted; but there's
still the Django startup for the `manage.py` process itself for each
worker, so on a machine with e.g. 2 (virtual) cores the restart is
still painful.
2017-08-20 22:37:38 -07:00
Greg Price a099e698e2 py3: Switch almost all shebang lines to use `python3`.
This causes `upgrade-zulip-from-git`, as well as a no-option run of
`tools/build-release-tarball`, to produce a Zulip install running
Python 3, rather than Python 2.  In particular this means that the
virtualenv we create, in which all application code runs, is Python 3.

One shebang line, on `zulip-ec2-configure-interfaces`, explicitly
keeps Python 2, and at least one external ops script, `wal-e`, also
still runs on Python 2.  See discussion on the respective previous
commits that made those explicit.  There may also be some other
third-party scripts we use, outside of this source tree and running
outside our virtualenv, that still run on Python 2.
2017-08-16 17:54:43 -07:00
Tim Abbott d43f5ceeec database: Add database index for wildcard mentions. 2017-08-16 13:28:04 -07:00
Tim Abbott 8bb812c8a9 database: Add database index for alert words. 2017-08-16 12:39:01 -07:00
Steve Howell aedd433f7b Call create_large_migrations as part of upgrade.
We now call the create_large_migrations management command as part of
upgrade-zulip-stage-2 if needed, so that we can create large indexes
while the app is still up.
2017-08-16 12:39:00 -07:00
Anirudh Jain 28944b6c94 provision: Partially add zesty to supported systems.
We can't fully support it until we fix the tsearch_extras availability
issue, but for now, this is an improvement.

Tweaked by tabbott to cover the outstanding tsearch_extras issue.
2017-08-15 21:51:19 -07:00
Greg Price 4b58794fe2 provision: Install Python 3 versions of system dependencies.
Also make our dependency on `six` (for e.g. `replace-tarball-shebang`)
explicit -- we've been getting it via `python-pip`, but `python3-pip`
(on trusty) doesn't have that dependency for some reason.
2017-08-09 14:05:52 -07:00
Pweaver (Paul Weaver) 48815204e4 Changes node_modules to default to the same args for generate_sha1sum.
Since we can use both perfer_offline=True and False in a since build
prefer_offline shouldn't be used as a cache key or it will confuse the
cleanup script. Since yarn install (if successful) should be idempotent.
This will probably be ok.
2017-08-05 12:29:16 -07:00
Pweaver (Paul Weaver) 1afaa67c7c deps: Change npm to yarn for reliablity, security, and speed. 2017-08-05 12:29:06 -07:00
Pweaver (Paul Weaver) f444c68b62 Indent install-node code block in preperation for yarn migration. 2017-08-05 12:18:47 -07:00
Greg Price 2b146012e1 upgrade: Remove cosmetic-only half of pre-1.4.0 compatibility code.
If we do wind up with a symlink lying around at `local_settings.py`,
it won't do us any harm and shouldn't be materially more confusing
than the regular file we've long had there for almost all installs.
It'll also only last as long as the current deploy.  So just
let it be, and simplify the code a bit.

Also add a line to help the reader understand the remaining half of
this logic (which is essential so long as people might have pre-1.4.0
deploys lying around that they eventually get around to trying to
upgrade).  The fact that it's addressed to a situation which exists
only in the past of this tree, not in its present, makes a brief
comment potentially very helpful.
2017-07-31 21:27:32 -07:00
Greg Price ddb85c9cb1 upgrade: Fix str/bytes type error. 2017-07-31 15:39:13 -07:00
Tim Abbott b7211733e2 node_cache: Rename npm_cache to target_path.
This is cleaner and part of renaming these variables to refer to the
node_modules_cache.
2017-07-21 17:28:20 -07:00
Tim Abbott e1e5e15797 node_cache: Move npm_args inside the interface. 2017-07-21 17:28:20 -07:00
Tim Abbott b1944b5e1f node_cache: Fix buggy type annotations for copy_modules. 2017-07-21 17:28:20 -07:00
Tim Abbott 8523c24a17 node_modules: Move success stamp to root of cache dir.
This location makes more sense, since it's the entire directory we're
certifying, not just the `node_modules` subdirectory.
2017-07-21 17:28:20 -07:00
Pweaver (Paul Weaver) 1b7aee7cd6 scripts: Add cd_exec library helper. 2017-07-21 17:28:20 -07:00
Tim Abbott 59abefa616 node_cache: Extract cached_node_modules variable. 2017-07-21 17:28:20 -07:00
Tim Abbott 5d05cc7294 scripts: Rename NPM_CACHE_PATH to NODE_MODULES_CACHE_PATH. 2017-07-21 17:28:20 -07:00
Tim Abbott 1e2cc2e8ad third: Add yarn installer to Zulip repo. 2017-07-21 17:28:20 -07:00
Tim Abbott c1d2654f3a setup-apt-repo: Add pgroonga code for Debian stretch.
The Groonga apt repository for Debian has a slightly different
structure.
2017-07-14 17:22:20 -07:00
Tim Abbott f724900e68 install-node: Fix provisioning when node/npm don't exist.
Our recent performance changes to this script broke it in the case
where there was no previous version of node/npm installed.
2017-07-11 12:36:35 -07:00
neiljp (Neil Pilgrim) 8433d95d99 mypy: Make success_stamp parameter to do_npm_install be required. 2017-07-08 10:49:42 -07:00
Aditya Bansal a55cb919ff install-node: Fix errors in case node or npm is not installed. 2017-07-06 22:06:01 -07:00
Tim Abbott d7556eef17 install-node: Remove dependency on realpath. 2017-07-06 18:09:39 -07:00
sinwar c52dbd57f5 provision: Avoid spending 2s reinstalling node/npm.
Tweaked by tabbott to just check the versions.

Fixes #5184.
2017-07-06 17:57:22 -07:00
sinwar 9cab965601 install-node: hardcode the path for npm.
This replaces nvm in npm-wrapper by harcoding the path the way we do
with node.  The main benefit is that this saves a few hundred
milliseconds every time we invoke npm.
2017-07-06 17:44:28 -07:00
Tommy Ip 3b8864bafa linter: Create error printing library.
For performance reasons, we spawn each linter in a separate OS thread.
The downside of this is that all lints would end up in stdout without
much visual separation, resulting in confusing error log. This commit
introduce the `print_err` function, which shows which linter each line
of lint is from.
2017-07-06 13:46:10 +08:00
Umair Khan 908f099bb0 unpack-zulip: Do 2-step upgrade for version <= 1.3.10.
If the current version is less than or equal to 1.3.10, first
recommend an upgrade to the version 1.4.3 and then to the final
version.
2017-06-23 08:40:57 -04:00
Aditya Bansal cf503017d2 Use requirements file for pip/setuptools/wheels in venv setup.
Fixes: #5158.
2017-06-19 16:37:50 -04:00
Aditya Bansal e4af0caddb node_cache.py: Refactor to seperate out sha1sum generation.
Basically we just seperate out the sha1sum generation for the
node modules so that it can be reused later for cache clearance
logic. This is achieved by adding a function which returns the
sha1sum based HEX digest.
2017-06-19 07:55:23 -04:00
Vishnu Ks 7c5dd68414 Upgrade setuptools to 36.0.1. 2017-06-19 06:40:18 -04:00
Aditya Bansal f9cbb0936a setup_venv: Add wrapper to retry pip install if it fails first time. 2017-06-14 21:48:15 -07:00
Tim Abbott c37204c62a upgrade-zulip-stage-2: Remove an unused import. 2017-06-02 15:14:13 -07:00
Tim Abbott 9e0749ad82 email-mirror-postfix: Fix mypy errors. 2017-06-01 22:51:26 -07:00
K.Kanakhin 47ec9fbbe2 email-mirror: Rewrite email mirror script on pure python.
The Zulip email mirror script called by postfix had performance/load
issues, because it spent so much time on startup/import due to use of
the Zulip virtualenv.

The script was rewritten using pure python (no Django) to improve
performance.
2017-06-01 21:50:49 -07:00
Tim Abbott be814b940d setup_venv: Pin a version of setuptools for creating venvs.
This works around a bad recent setuptools release:

https://github.com/pypa/setuptools/issues/1042
2017-05-31 23:58:24 -07:00
Aditya Bansal ee369ceb0c pep8: Add compliance with rule E261 scripts/lib/email-mirror-postfix. 2017-05-31 17:07:15 -07:00
Aditya Bansal dee726f234 pep8: Add compliance with rule E261 scripts/lib/pythonrc.py. 2017-05-31 17:07:15 -07:00
theopen-institute 7eaa1fa0d0 create-production-venv: Fix symlink creation.
The install script was failing on 2nd+ attempts if the first attempt
was interrupted.

This failure happened because zulip-venv already existed at
`current_venv_path`. Changing the `ln` command's flags from `-s` to
`-nsf` should make this part of the script idempotent.
2017-05-20 21:31:50 -07:00
Tim Abbott 45a4aeac64 scripts: Run generate_secrets.py during the upgrade process.
Now that generate_secrets.py is idempotent, this allows us to
conveniently add new secrets whenever they are required.
2017-05-16 22:15:26 -07:00
rht 00e057bf44 install-node: bypass nvm wrapper for faster node startup.
This fixes a significant performance issue with LaTeX rendering (and
other things that invoked node) where starting up node took a few
hundred milliseconds due to nvm initialization.

Tweaked by tabbott to avoid copying the node binary itself, instead
using a tiny wrapper script.

This is important primarily because it's possible a future version of
node will expect to find libraries/dependencies/etc. installed via NVM
at some path related to the path of the node binary itself, and that's
more guaranteed with this new model.

Fixes #4618.
2017-05-09 09:17:54 -07:00
Aditya Bansal b3ad6a6d77 pep8: Add compliance with rule E261 to hash_reqs.py. 2017-05-07 23:21:50 -07:00
K.Kanakhin e3e52e7284 email-mirror: Move postfix email mirror integration to separate script.
This fixes a performance problem where we were previously starting up
a full Django process (~0.7s even on a fast machine) every time a new
email came in, potentially allowing users to accidentally DoS a Zulip
server.  Now, we just post over HTTPS, allowing the existing thread
pool support to do its job.

- Add script wrapper to communicate postfix pipe with django web server
  over HTTP(S). It uses shared_secret authentication mode.
- Add django view to process messages from email mirror server.
- Clean management command `email-mirror`. Left just functional
  for cron email processing.
- Add routes for new tornado view.
- Change pipe script in master process postfix config template
  based on updated script.
- Add tests.

Tweaked by tabbott to adjust the directory and set better defaults.

Fixes #2421.
2017-04-24 21:24:23 -07:00
Feorlen 6ff78ca0e8 Set umask 022 before starting prod upgrade.
Follow-on from #2373/ PR https://github.com/zulip/zulip/pull/4316, to set an
appropriate umask also when upgrading so files have appropriate permissions.

I've tested this by starting from a clean install, deleting /srv/* so new
files are downloaded, and then doing an upgrade. It worked starting with both
a current version from master and an older release installed with a less
restrictive umask and then the umask changed.

Fixes #2373.
2017-04-19 10:28:06 -07:00
Tim Abbott 5bf01fb7d4 create-production-venv: Fix issues with api/ relative path.
Fixes #4313.
2017-03-25 19:49:26 -07:00
Rishi Gupta 28d3af0965 Fix several new errors caught by mypy 0.501.
Clear out a bunch of easy to review errors, so we can focus on the more
complicated ones.
2017-03-03 14:12:52 -08:00
Tim Abbott aa6567ee34 queue_workers: Fix confusing --queue_type argument name. 2017-02-22 00:23:26 -08:00
Tim Abbott 620f1e444e travis: Fix various bugs in new queue worker test.
* Now queue_workers.py sorts queue names and prints them on their own
  line.  Previously it's output was nondeterministic.
* Simplified grep strategy for removing the "test" worker.
2017-02-19 21:17:42 -08:00
Tim Abbott d6bbcd2737 travis: Automate updates to production-helper Nagios test.
This list was likely to end up out of date quickly, since it wasn't
documented that you need to update it when adding a queue.  The best
solution is to just not require it to be updated.
2017-02-19 16:19:53 -08:00
Tim Abbott 31d4f99573 upgrade: Stop trying to copy node_modules out of tarballs.
Now that we no longer use node_modules at all in production (it's only
used to generate static assets), we don't include `node_modules` in
the production tarballs, and thus we shouldn't attempt to copy
`node_modules` out of the production tarballs when installing.

Fixes a regression introduced in
d71f2e7b9b.
2017-02-07 10:39:31 -08:00
Tim Abbott 126b1c4edc upgrade: Don't require authors updates deploying from Git.
Fixes #3392.
2017-01-23 23:00:09 -08:00
Tim Abbott 22d1aa396b lint: Clean up W503 PEP-8 warning. 2017-01-23 20:50:04 -08:00
Ayush Goyal a85b539c4a zulip_tools: Improve color and copy for run() errors.
Tweaks to the text are edited by tabbott.
2017-01-17 14:37:15 -08:00
Tim Abbott 1d5edff927 setup_venv: Give up if virtualenv-clone isn't working.
virtualenv-clone can sometimes fail if the old virtualenv is
broken; in that case, we can just make a new one.
2017-01-09 11:21:42 -08:00
Tim Abbott 54acbc41ed prod: Ensure the Zulip version of node is installed. 2017-01-06 16:18:29 -08:00
Tim Abbott e4b065e03a Move tools/setup/install-node to scripts/setup/. 2017-01-06 16:18:29 -08:00
Tim Abbott 374e900432 upgrade-zulip: Fix bug in migrations detection. 2017-01-06 16:18:29 -08:00
Tim Abbott e5fbea1007 upgrade-zulip: Move static asset compilation before shutdown.
This saves about a minute of downtime when using
upgrade-zulip-from-git in the default configuration.

It should also save several seconds of downtime when upgrading to a
production release tarball as well.
2017-01-06 14:04:54 -08:00
Tim Abbott 90ee8d1207 upgrade-zulip: Only pause for migrations if not current.
This saves about 1s of downtime when doing a no-op upgrade with the
default settings.  Small, but worthwhile.
2017-01-06 14:03:55 -08:00
Saumya Rawat d3a7d0202f dev: Auto-create log directory in management command.
Check 'zulip/var/log' folder exists and create if not as FileHandler only checks for existence of log file.
2016-12-17 17:18:49 -08:00
Umair Khan fda0387695 Create Python version agnostic venv link for prod.
This link points to either 'zulip-venv' or 'zulip-py3-venv' and makes
uWSGI configration very easy.
2016-12-13 21:40:43 -08:00
Tim Abbott 95a49f6ea2 node_cache: Fix buggy production deployment code.
The previous version of the production deployment code for release
tarballs did not correctly install the node_modules directory.
2016-12-08 10:34:04 -08:00
AZtheAsian c23f9e0df7 pep8: Fix E111 violations 2016-12-01 14:18:11 -08:00
Tim Abbott fd7cb10964 install: Check whether the system has at least 2GB RAM.
This should eliminate a common class of user error installing Zulip.

Fixes #2290, fixes #2320.
2016-11-30 16:07:57 -08:00
Igor Tokarev e6ae53cbff install: Add clear error message if upstart is installed on Xenial.
Fixes #2199.
2016-11-29 19:16:26 -08:00
Rafid Aslam 7a2282986a pep8: Fix E225 pep8 violations. 2016-11-28 15:21:15 -08:00
Anders Kaseorg 78d6c3d7e9 install: Fix RabbitMQ node name if RabbitMQ is not installed.
This indirectly causes the RabbitMQ node name for new Zulip
installations to default to zulip@localhost, which would eliminate the
persistent problems we have had

Fixes #194, #465, #1375, #1751.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2016-11-26 18:54:29 -08:00
Anders Kaseorg 2d6525df04 install: Install python3, python3-six
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2016-11-26 13:26:18 -08:00
Anders Kaseorg 712c98cb48 Use zulip-py3-venv when running on Python 3
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2016-11-26 13:20:22 -08:00
Tim Abbott f5935e81c7 install: Support being run not directly from /root/zulip.
This adds a dependency on the realpath package on trusty; we could try
to remove it if needed, but given that realpath is included in
coreutils on Xenial (and presumably anything else modern), I think
it's reasonable to add it.

Fixes #1797.
2016-11-18 19:56:58 -08:00
Tim Abbott 5b35aada7c zulip_tools: Fix run to not eat error output.
We fix this by just using `subprocess.check_call`.
2016-10-27 12:26:01 -07:00
Tim Abbott 8e82257444 Fix node_cache code to not require root.
The previous code caused problems using a system where the zulip user
doesn't have sudo rights.
2016-10-25 17:52:19 -07:00
Tim Abbott 41b0079ab2 node_cache: Fix success_stamp logic.
Previously, success_stamp was touched whenever we used a particular
node_modules version; it makes more sense to only touch it when the
node_modules directory has actually changed.
2016-10-25 17:52:17 -07:00
Umair Khan e428f3feda Run apt-get update only if sources.list has changed.
Fixes: #2025
2016-10-19 16:23:21 +05:00
Tim Abbott 4a4664d268 mypy: Remove a bunch of now-unnecessary type: ignore annotations.
Since mypy and typeshed have advanced a lot over the last several
months, we no longer need these `type: ignore` annotations.
2016-10-17 11:48:34 -07:00
Tim Abbott fc20c86d8d install: Move apt-get update into setup-apt-repo. 2016-10-16 01:13:50 -07:00
Tim Abbott fcde846b58 setup_venv: Fix handling of Git requirements.
get_package_names did not correctly strip the GitHub URLs from package
names, resulting in the "package names" for our dependencies installed
from Git being tracked with the complete sha1sum included in the name.
This meant that upgrading our virtualenvs incorrectly ended up
resorting to creating an entirely new virtualenv whenever we changed a
dependency that had previously been installed from GitHub URLs.
2016-10-16 01:10:43 -07:00
Tim Abbott 7c16172f9e node_cache: Fix buggy annotations for stdout/stderr. 2016-10-15 22:53:28 -07:00
Tim Abbott 14f6e4c740 scripts: Stop using apt-add-repository.
Unfortunately, apt-add-repository is highly unreliable and was causing
problems both in Travis CI and with developers provisioning their
environment.
2016-10-11 22:10:36 -07:00
K.Kanakhin 14545d1647 create-production-venv: Add PY3 support to production venv creation script.
Create production python virtual env based on system python version.
2016-10-11 14:19:00 -07:00
Diptanshu8 d7253b144c generate-secrets: Refactor to make development/production explicit.
generate-secrets.py now requires --development for development environment
setup or --production for production environment setup (and one of these
options is mandatory).

This solves the problem that it was somewhat easy to accidentally run
generate-secrets.py without the `-d` option while doing manual development
environment setup.

Fixes: #1911.
2016-10-06 17:12:49 -07:00
Tim Abbott a2b91221d0 upgrade-zulip: Only remove zproject/local_settings.py if symlink.
This fixes a problem where if we're deploying from git,
local_settings.py ends up part of the diff in the deployed git
repository.
2016-10-04 20:41:17 -07:00
umkay d260a22637 Add a new statistics/analytics framework.
This is a first pass at building a framework for collecting various
stats about realms, users, streams, etc. Includes:
* New analytics tables for storing counts data
* Raw SQL queries for pulling data from zerver/models.py tables
* Aggregation functions for aggregating hourly stats into daily stats, and
  aggregating user/stream level stats into realm level stats
* A management command for pulling the data

Note that counts.py was added to the linter exclude list due to errors
around %%s.
2016-10-04 17:18:54 -07:00
Umair Khan b4214ec8cb Fix formatting of print in run function. 2016-09-30 10:42:52 -07:00
Tim Abbott 9c0c6c0c23 node_cache: Don't make node_modules symlinks as root. 2016-09-28 00:36:40 -07:00
Steve Howell 13d6f52203 provision: Handle VENV_CACHE_PATH not existing.
If VENV_CACHE_PATH does not exist (which can happen if you destroy
your vagrant environment), then do a short circuit return in
try_to_copy_venv().
2016-09-20 06:55:08 -07:00
Umair Khan e3078b226a Optimize incremental virtualenv creation.
This adds a new system for copying packages from old virtualenvs that
are sufficiently similar to the new virtualenv required.

In practice, this results in a huge performance improvement for
re-provisioning Zulip development environments when the requirements
files have changed (which is the dominant performance problem with
provision today).

Fixes: #1507.
2016-09-19 22:35:32 -07:00
Umair Khan ff0f413d26 Use npm caching in upgrade-zulip-stage-2. 2016-09-16 12:40:48 -07:00
Umair Khan ec74974de3 Create node_cache module for npm caching. 2016-09-16 12:40:48 -07:00
Umair Khan 859b1fbb2c setup_venv.py: Remove comma from the end. 2016-09-08 14:14:28 -07:00
Christie Koehler 586b236375 upgrade: Create prod_settings symlink in step 2 if it doesn't exist.
Between releases 1.3.13 and 1.4.0, local_settings.py was renamed to
prod_settings.py. The upgrade scripts were adjusted to reflect this name
change. But because the first part of the upgrade script is run with the
currently installed version's code, the symlink to /etc/zulip/settings.py is
created with the old name. This was causing upgrade-zulip-stage-2 to fail.

Now upgrade-zulip-stage-2 creates the symlink at zproject/prod_settings.py
if it doesn't already exist.

Fixes #1731.
2016-09-06 14:51:16 -07:00
Tim Abbott 9818a760b5 Install pgroonga in development and (optionally) in prod.
This is preliminary work towards being able to merge support for using
the pgroonga full-text search solution for all languages in Zulip.
2016-08-25 18:03:55 -07:00
Umair Khan 194cbf17a1 Allow run command to accept **kwargs. 2016-08-18 15:06:22 -07:00
Taranjeet Singh d606b95242 zulip_tools.py: Move zulip_tools.py in scripts/lib.
This commit moves zulip_tools.py as part of cleaning the root directory
and organizing proejct into better directory structure.
2016-08-15 16:44:50 -07:00
ukhan 180b438c44 Upgrade setuptools prior to updating mock 2016-08-12 17:56:23 -07:00
Tim Abbott 6496fe2a53 travis: Remove rabbitmq nodename dependency on hostname.
Because rabbitmq doesn't support changing the nodename of a running
rabbitmq node, Zulip installations suffered a plague of issues where
e.g. a Zulip server would reboot, the hostname would change, and
suddenly the local rabbitmq instance being used by Zulip would stop
working.

We address this problem by using, by default, a fixed rabbitmq
nodename, but providing server administrators the option to set the
rabbitmq nodename used by Zulip however they choose.

To upgrade an existing server to use this new configuration, one will
need to add something like the following to /etc/zulip/zulip.conf:

[rabbitmq]
nodename = zulip@localhost

However, I don't believe we have the puppet code in place to make this
work correctly at initial installation without rabbitmq-server being
already installed (but off), as we can easily setup in Travis CI but I
haven't been willing to do for the installer.  So for now, this just
fixes our Travis CI problems.

Fixes: #1579.
2016-08-12 09:38:23 -07:00
Tim Abbott 5bff72c385 Revert "Use apt-add-repository to setup Zulip PPA."
This reverts commit 3f95e567c1.

Apparently `apt-add-repository` fails periodically in CI.  I suspect
this is some sort of silly networking problem, but given that all
we're saving is a few lines of code, the old version was better if
this fails basically ever.
2016-08-05 13:29:17 -07:00
Tim Abbott 3f95e567c1 Use apt-add-repository to setup Zulip PPA. 2016-08-04 22:17:07 -07:00
Tim Abbott 4d02c6efef scripts: Purge old deployments on every upgrade.
Fixes: #554.
2016-08-04 17:02:48 -07:00
Tim Abbott 51f069aa18 scripts: Refactor deployment-finished logging. 2016-08-04 17:02:48 -07:00