Commit Graph

182 Commits

Author SHA1 Message Date
Pweaver (Paul Weaver) 48815204e4 Changes node_modules to default to the same args for generate_sha1sum.
Since we can use both perfer_offline=True and False in a since build
prefer_offline shouldn't be used as a cache key or it will confuse the
cleanup script. Since yarn install (if successful) should be idempotent.
This will probably be ok.
2017-08-05 12:29:16 -07:00
Pweaver (Paul Weaver) 1afaa67c7c deps: Change npm to yarn for reliablity, security, and speed. 2017-08-05 12:29:06 -07:00
Pweaver (Paul Weaver) f444c68b62 Indent install-node code block in preperation for yarn migration. 2017-08-05 12:18:47 -07:00
Greg Price 2b146012e1 upgrade: Remove cosmetic-only half of pre-1.4.0 compatibility code.
If we do wind up with a symlink lying around at `local_settings.py`,
it won't do us any harm and shouldn't be materially more confusing
than the regular file we've long had there for almost all installs.
It'll also only last as long as the current deploy.  So just
let it be, and simplify the code a bit.

Also add a line to help the reader understand the remaining half of
this logic (which is essential so long as people might have pre-1.4.0
deploys lying around that they eventually get around to trying to
upgrade).  The fact that it's addressed to a situation which exists
only in the past of this tree, not in its present, makes a brief
comment potentially very helpful.
2017-07-31 21:27:32 -07:00
Greg Price ddb85c9cb1 upgrade: Fix str/bytes type error. 2017-07-31 15:39:13 -07:00
Tim Abbott b7211733e2 node_cache: Rename npm_cache to target_path.
This is cleaner and part of renaming these variables to refer to the
node_modules_cache.
2017-07-21 17:28:20 -07:00
Tim Abbott e1e5e15797 node_cache: Move npm_args inside the interface. 2017-07-21 17:28:20 -07:00
Tim Abbott b1944b5e1f node_cache: Fix buggy type annotations for copy_modules. 2017-07-21 17:28:20 -07:00
Tim Abbott 8523c24a17 node_modules: Move success stamp to root of cache dir.
This location makes more sense, since it's the entire directory we're
certifying, not just the `node_modules` subdirectory.
2017-07-21 17:28:20 -07:00
Pweaver (Paul Weaver) 1b7aee7cd6 scripts: Add cd_exec library helper. 2017-07-21 17:28:20 -07:00
Tim Abbott 59abefa616 node_cache: Extract cached_node_modules variable. 2017-07-21 17:28:20 -07:00
Tim Abbott 5d05cc7294 scripts: Rename NPM_CACHE_PATH to NODE_MODULES_CACHE_PATH. 2017-07-21 17:28:20 -07:00
Tim Abbott 1e2cc2e8ad third: Add yarn installer to Zulip repo. 2017-07-21 17:28:20 -07:00
Tim Abbott c1d2654f3a setup-apt-repo: Add pgroonga code for Debian stretch.
The Groonga apt repository for Debian has a slightly different
structure.
2017-07-14 17:22:20 -07:00
Tim Abbott f724900e68 install-node: Fix provisioning when node/npm don't exist.
Our recent performance changes to this script broke it in the case
where there was no previous version of node/npm installed.
2017-07-11 12:36:35 -07:00
neiljp (Neil Pilgrim) 8433d95d99 mypy: Make success_stamp parameter to do_npm_install be required. 2017-07-08 10:49:42 -07:00
Aditya Bansal a55cb919ff install-node: Fix errors in case node or npm is not installed. 2017-07-06 22:06:01 -07:00
Tim Abbott d7556eef17 install-node: Remove dependency on realpath. 2017-07-06 18:09:39 -07:00
sinwar c52dbd57f5 provision: Avoid spending 2s reinstalling node/npm.
Tweaked by tabbott to just check the versions.

Fixes #5184.
2017-07-06 17:57:22 -07:00
sinwar 9cab965601 install-node: hardcode the path for npm.
This replaces nvm in npm-wrapper by harcoding the path the way we do
with node.  The main benefit is that this saves a few hundred
milliseconds every time we invoke npm.
2017-07-06 17:44:28 -07:00
Tommy Ip 3b8864bafa linter: Create error printing library.
For performance reasons, we spawn each linter in a separate OS thread.
The downside of this is that all lints would end up in stdout without
much visual separation, resulting in confusing error log. This commit
introduce the `print_err` function, which shows which linter each line
of lint is from.
2017-07-06 13:46:10 +08:00
Umair Khan 908f099bb0 unpack-zulip: Do 2-step upgrade for version <= 1.3.10.
If the current version is less than or equal to 1.3.10, first
recommend an upgrade to the version 1.4.3 and then to the final
version.
2017-06-23 08:40:57 -04:00
Aditya Bansal cf503017d2 Use requirements file for pip/setuptools/wheels in venv setup.
Fixes: #5158.
2017-06-19 16:37:50 -04:00
Aditya Bansal e4af0caddb node_cache.py: Refactor to seperate out sha1sum generation.
Basically we just seperate out the sha1sum generation for the
node modules so that it can be reused later for cache clearance
logic. This is achieved by adding a function which returns the
sha1sum based HEX digest.
2017-06-19 07:55:23 -04:00
Vishnu Ks 7c5dd68414 Upgrade setuptools to 36.0.1. 2017-06-19 06:40:18 -04:00
Aditya Bansal f9cbb0936a setup_venv: Add wrapper to retry pip install if it fails first time. 2017-06-14 21:48:15 -07:00
Tim Abbott c37204c62a upgrade-zulip-stage-2: Remove an unused import. 2017-06-02 15:14:13 -07:00
Tim Abbott 9e0749ad82 email-mirror-postfix: Fix mypy errors. 2017-06-01 22:51:26 -07:00
K.Kanakhin 47ec9fbbe2 email-mirror: Rewrite email mirror script on pure python.
The Zulip email mirror script called by postfix had performance/load
issues, because it spent so much time on startup/import due to use of
the Zulip virtualenv.

The script was rewritten using pure python (no Django) to improve
performance.
2017-06-01 21:50:49 -07:00
Tim Abbott be814b940d setup_venv: Pin a version of setuptools for creating venvs.
This works around a bad recent setuptools release:

https://github.com/pypa/setuptools/issues/1042
2017-05-31 23:58:24 -07:00
Aditya Bansal ee369ceb0c pep8: Add compliance with rule E261 scripts/lib/email-mirror-postfix. 2017-05-31 17:07:15 -07:00
Aditya Bansal dee726f234 pep8: Add compliance with rule E261 scripts/lib/pythonrc.py. 2017-05-31 17:07:15 -07:00
theopen-institute 7eaa1fa0d0 create-production-venv: Fix symlink creation.
The install script was failing on 2nd+ attempts if the first attempt
was interrupted.

This failure happened because zulip-venv already existed at
`current_venv_path`. Changing the `ln` command's flags from `-s` to
`-nsf` should make this part of the script idempotent.
2017-05-20 21:31:50 -07:00
Tim Abbott 45a4aeac64 scripts: Run generate_secrets.py during the upgrade process.
Now that generate_secrets.py is idempotent, this allows us to
conveniently add new secrets whenever they are required.
2017-05-16 22:15:26 -07:00
rht 00e057bf44 install-node: bypass nvm wrapper for faster node startup.
This fixes a significant performance issue with LaTeX rendering (and
other things that invoked node) where starting up node took a few
hundred milliseconds due to nvm initialization.

Tweaked by tabbott to avoid copying the node binary itself, instead
using a tiny wrapper script.

This is important primarily because it's possible a future version of
node will expect to find libraries/dependencies/etc. installed via NVM
at some path related to the path of the node binary itself, and that's
more guaranteed with this new model.

Fixes #4618.
2017-05-09 09:17:54 -07:00
Aditya Bansal b3ad6a6d77 pep8: Add compliance with rule E261 to hash_reqs.py. 2017-05-07 23:21:50 -07:00
K.Kanakhin e3e52e7284 email-mirror: Move postfix email mirror integration to separate script.
This fixes a performance problem where we were previously starting up
a full Django process (~0.7s even on a fast machine) every time a new
email came in, potentially allowing users to accidentally DoS a Zulip
server.  Now, we just post over HTTPS, allowing the existing thread
pool support to do its job.

- Add script wrapper to communicate postfix pipe with django web server
  over HTTP(S). It uses shared_secret authentication mode.
- Add django view to process messages from email mirror server.
- Clean management command `email-mirror`. Left just functional
  for cron email processing.
- Add routes for new tornado view.
- Change pipe script in master process postfix config template
  based on updated script.
- Add tests.

Tweaked by tabbott to adjust the directory and set better defaults.

Fixes #2421.
2017-04-24 21:24:23 -07:00
Feorlen 6ff78ca0e8 Set umask 022 before starting prod upgrade.
Follow-on from #2373/ PR https://github.com/zulip/zulip/pull/4316, to set an
appropriate umask also when upgrading so files have appropriate permissions.

I've tested this by starting from a clean install, deleting /srv/* so new
files are downloaded, and then doing an upgrade. It worked starting with both
a current version from master and an older release installed with a less
restrictive umask and then the umask changed.

Fixes #2373.
2017-04-19 10:28:06 -07:00
Tim Abbott 5bf01fb7d4 create-production-venv: Fix issues with api/ relative path.
Fixes #4313.
2017-03-25 19:49:26 -07:00
Rishi Gupta 28d3af0965 Fix several new errors caught by mypy 0.501.
Clear out a bunch of easy to review errors, so we can focus on the more
complicated ones.
2017-03-03 14:12:52 -08:00
Tim Abbott aa6567ee34 queue_workers: Fix confusing --queue_type argument name. 2017-02-22 00:23:26 -08:00
Tim Abbott 620f1e444e travis: Fix various bugs in new queue worker test.
* Now queue_workers.py sorts queue names and prints them on their own
  line.  Previously it's output was nondeterministic.
* Simplified grep strategy for removing the "test" worker.
2017-02-19 21:17:42 -08:00
Tim Abbott d6bbcd2737 travis: Automate updates to production-helper Nagios test.
This list was likely to end up out of date quickly, since it wasn't
documented that you need to update it when adding a queue.  The best
solution is to just not require it to be updated.
2017-02-19 16:19:53 -08:00
Tim Abbott 31d4f99573 upgrade: Stop trying to copy node_modules out of tarballs.
Now that we no longer use node_modules at all in production (it's only
used to generate static assets), we don't include `node_modules` in
the production tarballs, and thus we shouldn't attempt to copy
`node_modules` out of the production tarballs when installing.

Fixes a regression introduced in
d71f2e7b9b.
2017-02-07 10:39:31 -08:00
Tim Abbott 126b1c4edc upgrade: Don't require authors updates deploying from Git.
Fixes #3392.
2017-01-23 23:00:09 -08:00
Tim Abbott 22d1aa396b lint: Clean up W503 PEP-8 warning. 2017-01-23 20:50:04 -08:00
Ayush Goyal a85b539c4a zulip_tools: Improve color and copy for run() errors.
Tweaks to the text are edited by tabbott.
2017-01-17 14:37:15 -08:00
Tim Abbott 1d5edff927 setup_venv: Give up if virtualenv-clone isn't working.
virtualenv-clone can sometimes fail if the old virtualenv is
broken; in that case, we can just make a new one.
2017-01-09 11:21:42 -08:00
Tim Abbott 54acbc41ed prod: Ensure the Zulip version of node is installed. 2017-01-06 16:18:29 -08:00
Tim Abbott e4b065e03a Move tools/setup/install-node to scripts/setup/. 2017-01-06 16:18:29 -08:00