Commit Graph

365 Commits

Author SHA1 Message Date
Greg Price a099e698e2 py3: Switch almost all shebang lines to use `python3`.
This causes `upgrade-zulip-from-git`, as well as a no-option run of
`tools/build-release-tarball`, to produce a Zulip install running
Python 3, rather than Python 2.  In particular this means that the
virtualenv we create, in which all application code runs, is Python 3.

One shebang line, on `zulip-ec2-configure-interfaces`, explicitly
keeps Python 2, and at least one external ops script, `wal-e`, also
still runs on Python 2.  See discussion on the respective previous
commits that made those explicit.  There may also be some other
third-party scripts we use, outside of this source tree and running
outside our virtualenv, that still run on Python 2.
2017-08-16 17:54:43 -07:00
Tim Abbott d43f5ceeec database: Add database index for wildcard mentions. 2017-08-16 13:28:04 -07:00
Tim Abbott 8bb812c8a9 database: Add database index for alert words. 2017-08-16 12:39:01 -07:00
Steve Howell aedd433f7b Call create_large_migrations as part of upgrade.
We now call the create_large_migrations management command as part of
upgrade-zulip-stage-2 if needed, so that we can create large indexes
while the app is still up.
2017-08-16 12:39:00 -07:00
Anirudh Jain 28944b6c94 provision: Partially add zesty to supported systems.
We can't fully support it until we fix the tsearch_extras availability
issue, but for now, this is an improvement.

Tweaked by tabbott to cover the outstanding tsearch_extras issue.
2017-08-15 21:51:19 -07:00
Greg Price 0debc3f7ea install: Update a comment for our post-/root/zulip world.
We may not necessarily be running out of /root/zulip or any particular path,
but the point this comment was really trying to make in the first place stands.
Make it more clearly and still-accurately.
2017-08-15 17:41:07 -07:00
Greg Price 4b58794fe2 provision: Install Python 3 versions of system dependencies.
Also make our dependency on `six` (for e.g. `replace-tarball-shebang`)
explicit -- we've been getting it via `python-pip`, but `python3-pip`
(on trusty) doesn't have that dependency for some reason.
2017-08-09 14:05:52 -07:00
Pweaver (Paul Weaver) 48815204e4 Changes node_modules to default to the same args for generate_sha1sum.
Since we can use both perfer_offline=True and False in a since build
prefer_offline shouldn't be used as a cache key or it will confuse the
cleanup script. Since yarn install (if successful) should be idempotent.
This will probably be ok.
2017-08-05 12:29:16 -07:00
Pweaver (Paul Weaver) 1afaa67c7c deps: Change npm to yarn for reliablity, security, and speed. 2017-08-05 12:29:06 -07:00
Pweaver (Paul Weaver) f444c68b62 Indent install-node code block in preperation for yarn migration. 2017-08-05 12:18:47 -07:00
Greg Price 1c7054e7e5 upgrade: Reorder steps slightly for clarity. 2017-07-31 21:27:32 -07:00
Greg Price 5002d9ecb9 upgrade: Remove disused hack for `zulip-dropbox.png`.
I believe this hack is no longer being triggered; so cut it out.
2017-07-31 21:27:32 -07:00
Greg Price 2b146012e1 upgrade: Remove cosmetic-only half of pre-1.4.0 compatibility code.
If we do wind up with a symlink lying around at `local_settings.py`,
it won't do us any harm and shouldn't be materially more confusing
than the regular file we've long had there for almost all installs.
It'll also only last as long as the current deploy.  So just
let it be, and simplify the code a bit.

Also add a line to help the reader understand the remaining half of
this logic (which is essential so long as people might have pre-1.4.0
deploys lying around that they eventually get around to trying to
upgrade).  The fact that it's addressed to a situation which exists
only in the past of this tree, not in its present, makes a brief
comment potentially very helpful.
2017-07-31 21:27:32 -07:00
Greg Price 2cd6ff62d9 upgrade: Remove disused local_settings.py case.
The local_settings.py file in the repo has been getting stale
for some time now, and I'm pretty sure this codepath is
never used.  Delete it.
2017-07-31 21:27:32 -07:00
Greg Price ddb85c9cb1 upgrade: Fix str/bytes type error. 2017-07-31 15:39:13 -07:00
Tim Abbott 53e4d8562b lint: Add a lint check for bare `type: ignore`s. 2017-07-27 16:31:55 -07:00
Tim Abbott b7211733e2 node_cache: Rename npm_cache to target_path.
This is cleaner and part of renaming these variables to refer to the
node_modules_cache.
2017-07-21 17:28:20 -07:00
Tim Abbott e1e5e15797 node_cache: Move npm_args inside the interface. 2017-07-21 17:28:20 -07:00
Tim Abbott b1944b5e1f node_cache: Fix buggy type annotations for copy_modules. 2017-07-21 17:28:20 -07:00
Tim Abbott 8523c24a17 node_modules: Move success stamp to root of cache dir.
This location makes more sense, since it's the entire directory we're
certifying, not just the `node_modules` subdirectory.
2017-07-21 17:28:20 -07:00
Pweaver (Paul Weaver) 1b7aee7cd6 scripts: Add cd_exec library helper. 2017-07-21 17:28:20 -07:00
Tim Abbott 59abefa616 node_cache: Extract cached_node_modules variable. 2017-07-21 17:28:20 -07:00
Tim Abbott 5d05cc7294 scripts: Rename NPM_CACHE_PATH to NODE_MODULES_CACHE_PATH. 2017-07-21 17:28:20 -07:00
Tim Abbott 1e2cc2e8ad third: Add yarn installer to Zulip repo. 2017-07-21 17:28:20 -07:00
rht 614e3bb1ef scripts: Add script to autogenerate a self-signed SSL cert.
This will simplify step 1 of prod-install instruction to reduce
suffering in testing/experimenting production environments.

Attribution: the scripts/setup/configure-certs is based on @galexrt's
5c0daf6211

Further tweaked by tabbott to rename the script and edit the messages.
2017-07-16 17:22:32 -07:00
Tim Abbott c1d2654f3a setup-apt-repo: Add pgroonga code for Debian stretch.
The Groonga apt repository for Debian has a slightly different
structure.
2017-07-14 17:22:20 -07:00
EdOverflow 86cfed3e81 Update generate_secrets.py 2017-07-11 12:43:46 -07:00
Tim Abbott f724900e68 install-node: Fix provisioning when node/npm don't exist.
Our recent performance changes to this script broke it in the case
where there was no previous version of node/npm installed.
2017-07-11 12:36:35 -07:00
neiljp (Neil Pilgrim) 8433d95d99 mypy: Make success_stamp parameter to do_npm_install be required. 2017-07-08 10:49:42 -07:00
Aditya Bansal a55cb919ff install-node: Fix errors in case node or npm is not installed. 2017-07-06 22:06:01 -07:00
Tim Abbott d7556eef17 install-node: Remove dependency on realpath. 2017-07-06 18:09:39 -07:00
sinwar c52dbd57f5 provision: Avoid spending 2s reinstalling node/npm.
Tweaked by tabbott to just check the versions.

Fixes #5184.
2017-07-06 17:57:22 -07:00
sinwar 9cab965601 install-node: hardcode the path for npm.
This replaces nvm in npm-wrapper by harcoding the path the way we do
with node.  The main benefit is that this saves a few hundred
milliseconds every time we invoke npm.
2017-07-06 17:44:28 -07:00
Tommy Ip 3b8864bafa linter: Create error printing library.
For performance reasons, we spawn each linter in a separate OS thread.
The downside of this is that all lints would end up in stdout without
much visual separation, resulting in confusing error log. This commit
introduce the `print_err` function, which shows which linter each line
of lint is from.
2017-07-06 13:46:10 +08:00
Greg Price e1ce3daaff scripts/upgrade-zulip-from-git: Don't cache the repo URL, only the contents
We document the `deployment.git_repo_url` setting in `/etc/zulip/zulip.conf`
to control where this script fetches from, and don't say that it's
only read on the first such upgrade and cached thereafter.  The documented
behavior seems like the right behavior.  So use the currently configured
URL every time, by writing it anew into the config of our cache repo.
2017-06-28 15:49:57 -07:00
Umair Khan 908f099bb0 unpack-zulip: Do 2-step upgrade for version <= 1.3.10.
If the current version is less than or equal to 1.3.10, first
recommend an upgrade to the version 1.4.3 and then to the final
version.
2017-06-23 08:40:57 -04:00
Aditya Bansal cf503017d2 Use requirements file for pip/setuptools/wheels in venv setup.
Fixes: #5158.
2017-06-19 16:37:50 -04:00
Aditya Bansal e4af0caddb node_cache.py: Refactor to seperate out sha1sum generation.
Basically we just seperate out the sha1sum generation for the
node modules so that it can be reused later for cache clearance
logic. This is achieved by adding a function which returns the
sha1sum based HEX digest.
2017-06-19 07:55:23 -04:00
Vishnu Ks 7c5dd68414 Upgrade setuptools to 36.0.1. 2017-06-19 06:40:18 -04:00
Aditya Bansal f9cbb0936a setup_venv: Add wrapper to retry pip install if it fails first time. 2017-06-14 21:48:15 -07:00
Tim Abbott b70986469c generate_secrets: Fix handling of missing trailing newlines.
When we added support for automatically adding new secrets in
generate_secrets.py, we failed to account for the possibility that a
human editor might have let the secrets file without a trailing
newline.

We address this by adding a leading newline before our new secret.

Fixes #5209.
2017-06-03 23:17:04 -07:00
Tim Abbott c37204c62a upgrade-zulip-stage-2: Remove an unused import. 2017-06-02 15:14:13 -07:00
Tim Abbott 9e0749ad82 email-mirror-postfix: Fix mypy errors. 2017-06-01 22:51:26 -07:00
K.Kanakhin 47ec9fbbe2 email-mirror: Rewrite email mirror script on pure python.
The Zulip email mirror script called by postfix had performance/load
issues, because it spent so much time on startup/import due to use of
the Zulip virtualenv.

The script was rewritten using pure python (no Django) to improve
performance.
2017-06-01 21:50:49 -07:00
Tim Abbott be814b940d setup_venv: Pin a version of setuptools for creating venvs.
This works around a bad recent setuptools release:

https://github.com/pypa/setuptools/issues/1042
2017-05-31 23:58:24 -07:00
Aditya Bansal ee369ceb0c pep8: Add compliance with rule E261 scripts/lib/email-mirror-postfix. 2017-05-31 17:07:15 -07:00
Aditya Bansal 807fee68d6 pep8: Add compliance with rule E261 nagios/check-rabbitmq-consumers. 2017-05-31 17:07:15 -07:00
Aditya Bansal dee726f234 pep8: Add compliance with rule E261 scripts/lib/pythonrc.py. 2017-05-31 17:07:15 -07:00
Elliott Jin 0ec9e54954 bots: Add queue and QueueProcessingWorker for embedded bots. 2017-05-25 15:00:51 -07:00
theopen-institute 7eaa1fa0d0 create-production-venv: Fix symlink creation.
The install script was failing on 2nd+ attempts if the first attempt
was interrupted.

This failure happened because zulip-venv already existed at
`current_venv_path`. Changing the `ln` command's flags from `-s` to
`-nsf` should make this part of the script idempotent.
2017-05-20 21:31:50 -07:00