Commit Graph

390 Commits

Author SHA1 Message Date
Tim Abbott 600b164130 setup-apt-repo: Fix failure to rerun properly on provision failure.
Apparently, the refactoring to make this script only run when changes
are present was buggy, in that if `apt-get update` failed, running
provision against wouldn't rerun `apt-get update`, resulting in a
broken state that requires expertise to fix.  This closes that gap, by
using a stamp file to ensure we always successfully update apt before
proceeding.

It doesn't fix existing installations.
2017-09-12 07:08:54 -07:00
Tim Abbott 6d5dac96c6 zulip-puppet-apply: Fix flushing of stdout for Python 3.
Apparently, this UI worked just fine with Python 2, but failed to
actually print usefully with Python 3.
2017-08-27 18:36:39 -07:00
Tim Abbott 1a1df29053 get_recent_deployments: Skip uwsgi socket and friends.
This fixes an exception when running clean-venv-caches in production.
2017-08-27 18:18:53 -07:00
Tim Abbott e38ac00f3c caches: Move cache cleaning helper tools under scripts/lib.
This helps keep the root of scripts/ uncluttered.
2017-08-27 17:59:49 -07:00
Harshit Bansal f90f9c7404 scripts: Add `clean-unused-caches` script.
This script can be used for running all the three cache cleaning scripts
in one go.
2017-08-27 17:51:24 -07:00
Harshit Bansal e5361b4403 clean_emoji_cache: Expand `clean-emoji-cache`.
Expands `clean-emoji-cache` so that it can be used in production
environment as well. Also moves it to `scripts/` from `tools/`.
2017-08-27 17:51:24 -07:00
Harshit Bansal facb5dbe85 zulip_tools.py: Extract `generate_sha1sum_emoji()`.
Given the path of a zulip installation, it returns a hash corresponding
to the emoji infrastructure of that installation.
2017-08-27 17:51:24 -07:00
Harshit Bansal 948cf54ee3 clean-npm-cache: Expand `clean-npm-cache`.
Expands `clean-npm-cache` so that it can be used in production
environment as well. Also moves it to `scripts/` from `tools/`.
2017-08-27 17:51:24 -07:00
Harshit Bansal 8e41bbe2b0 node_cache.py: Modify `generate_sha1sum_node_modules()`.
Modify `generate_sha1sum_node_modules()` such that it can calculate
the hash for a particular installation.

Tweaked by tabbott to use os.path.realpath in the setup_dir
calculation, to ensure it's consistent.
2017-08-27 17:51:24 -07:00
Harshit Bansal 8aadbc258d clean-venv-cache: Don't remove the currently active cache in dev.
In dev always include the currently active cache in order not to break
current installation in case dependencies are updated with bumping the
provision version.
2017-08-27 17:51:24 -07:00
Harshit Bansal 5e2b54446f clean-venv-cache: Directly import functions from 'hash_reqs.py'.
Instead of running the 'hash_reqs.py' as a script, directly import
functions from it to calculate the hash. This will speed up the
script.
2017-08-27 17:51:24 -07:00
Harshit Bansal ecbc75e00f clean-venv-cache: Migrate to use `purge_unused_caches()` function. 2017-08-27 17:51:23 -07:00
Harshit Bansal 36420ab636 zulip_tools.py: Add `purge_caches()` function.
This function can be used for purging unused cache directories.
2017-08-27 17:37:08 -07:00
Tim Abbott b8e7369dee mypy: Remove type: ignores not needed in Python 3. 2017-08-25 11:04:20 -07:00
Tim Abbott fa97dd1408 setup_path_on_import: Drop Python 2 support. 2017-08-23 19:21:50 -07:00
Tim Abbott 2424819749 install: Move upstart checks a bit earlier.
This should make it much more likely that users see this before
waiting a long time for other things to happen, since the `apt-get
dist-upgrade` step is really slow.  We can't move further to the top,
since this requires `lsb_release` to be installed.
2017-08-23 14:55:01 -07:00
Harshit Bansal 3534e328c5 clean-venv-cache: Expand `clean-venv-cache` tool.
Expands `clean-venv-cache` tool so that it can be used in prod.
Also moves it from `tools/` to `scripts/`.
2017-08-23 00:03:25 -07:00
Harshit Bansal 504abfce63 zulip_tools.py: Add `GENERIC_CACHE_SCRIPT_PARSER`.
This parser will act as a parent parser for all the cache cleaning scripts.
2017-08-23 00:00:34 -07:00
Harshit Bansal 6936bb1ba0 zulip_tools.py: Add `get_caches_to_be_purged()` function.
Given the path of directory containing all the caches, a list of
caches in use and threshold days, this function returns a list
of caches which can be removed safely.
2017-08-22 23:59:45 -07:00
Harshit Bansal e71f92b09e zulip_tools.py: Add `get_threshold_timestamp()` function.
Given `threshold_days` this function returns a timestamp corresponding
to the time before threshold number of days.
2017-08-22 23:57:20 -07:00
Harshit Bansal 8954605726 zulip_tools.py: Add `get_recent_deployments()` function.
This function returns a list of all the deployments directories
which are newer than some threshold number of days including the
`/root/zulip` directory if it exists.
2017-08-22 23:57:20 -07:00
Harshit Bansal 931e4752aa zulip_tools.py: Add `get_environment()` function.
This function can be used to determine the environment in which a
script is being executed.
2017-08-22 23:57:20 -07:00
rht 0a469fd4c8 requirements: Lock prod.txt instead of its py3_common dependency.
`py3_common.txt` is no longer locked.
2017-08-22 10:10:57 -07:00
rht 6a5869ec8d requirements: Rename py3_prod.txt to prod.txt. 2017-08-22 10:10:57 -07:00
Greg Price f73e898874 manage.py: Save an extra Django startup by converting one script to a library.
This saves us from spending 200-250ms of CPU time importing Django
again just to log that we're running a management command.  On
`scripts/restart-server`, this saves us from one thundering herd of
Django startups when all the queue workers are restarted; but there's
still the Django startup for the `manage.py` process itself for each
worker, so on a machine with e.g. 2 (virtual) cores the restart is
still painful.
2017-08-20 22:37:38 -07:00
Greg Price a099e698e2 py3: Switch almost all shebang lines to use `python3`.
This causes `upgrade-zulip-from-git`, as well as a no-option run of
`tools/build-release-tarball`, to produce a Zulip install running
Python 3, rather than Python 2.  In particular this means that the
virtualenv we create, in which all application code runs, is Python 3.

One shebang line, on `zulip-ec2-configure-interfaces`, explicitly
keeps Python 2, and at least one external ops script, `wal-e`, also
still runs on Python 2.  See discussion on the respective previous
commits that made those explicit.  There may also be some other
third-party scripts we use, outside of this source tree and running
outside our virtualenv, that still run on Python 2.
2017-08-16 17:54:43 -07:00
Tim Abbott d43f5ceeec database: Add database index for wildcard mentions. 2017-08-16 13:28:04 -07:00
Tim Abbott 8bb812c8a9 database: Add database index for alert words. 2017-08-16 12:39:01 -07:00
Steve Howell aedd433f7b Call create_large_migrations as part of upgrade.
We now call the create_large_migrations management command as part of
upgrade-zulip-stage-2 if needed, so that we can create large indexes
while the app is still up.
2017-08-16 12:39:00 -07:00
Anirudh Jain 28944b6c94 provision: Partially add zesty to supported systems.
We can't fully support it until we fix the tsearch_extras availability
issue, but for now, this is an improvement.

Tweaked by tabbott to cover the outstanding tsearch_extras issue.
2017-08-15 21:51:19 -07:00
Greg Price 0debc3f7ea install: Update a comment for our post-/root/zulip world.
We may not necessarily be running out of /root/zulip or any particular path,
but the point this comment was really trying to make in the first place stands.
Make it more clearly and still-accurately.
2017-08-15 17:41:07 -07:00
Greg Price 4b58794fe2 provision: Install Python 3 versions of system dependencies.
Also make our dependency on `six` (for e.g. `replace-tarball-shebang`)
explicit -- we've been getting it via `python-pip`, but `python3-pip`
(on trusty) doesn't have that dependency for some reason.
2017-08-09 14:05:52 -07:00
Pweaver (Paul Weaver) 48815204e4 Changes node_modules to default to the same args for generate_sha1sum.
Since we can use both perfer_offline=True and False in a since build
prefer_offline shouldn't be used as a cache key or it will confuse the
cleanup script. Since yarn install (if successful) should be idempotent.
This will probably be ok.
2017-08-05 12:29:16 -07:00
Pweaver (Paul Weaver) 1afaa67c7c deps: Change npm to yarn for reliablity, security, and speed. 2017-08-05 12:29:06 -07:00
Pweaver (Paul Weaver) f444c68b62 Indent install-node code block in preperation for yarn migration. 2017-08-05 12:18:47 -07:00
Greg Price 1c7054e7e5 upgrade: Reorder steps slightly for clarity. 2017-07-31 21:27:32 -07:00
Greg Price 5002d9ecb9 upgrade: Remove disused hack for `zulip-dropbox.png`.
I believe this hack is no longer being triggered; so cut it out.
2017-07-31 21:27:32 -07:00
Greg Price 2b146012e1 upgrade: Remove cosmetic-only half of pre-1.4.0 compatibility code.
If we do wind up with a symlink lying around at `local_settings.py`,
it won't do us any harm and shouldn't be materially more confusing
than the regular file we've long had there for almost all installs.
It'll also only last as long as the current deploy.  So just
let it be, and simplify the code a bit.

Also add a line to help the reader understand the remaining half of
this logic (which is essential so long as people might have pre-1.4.0
deploys lying around that they eventually get around to trying to
upgrade).  The fact that it's addressed to a situation which exists
only in the past of this tree, not in its present, makes a brief
comment potentially very helpful.
2017-07-31 21:27:32 -07:00
Greg Price 2cd6ff62d9 upgrade: Remove disused local_settings.py case.
The local_settings.py file in the repo has been getting stale
for some time now, and I'm pretty sure this codepath is
never used.  Delete it.
2017-07-31 21:27:32 -07:00
Greg Price ddb85c9cb1 upgrade: Fix str/bytes type error. 2017-07-31 15:39:13 -07:00
Tim Abbott 53e4d8562b lint: Add a lint check for bare `type: ignore`s. 2017-07-27 16:31:55 -07:00
Tim Abbott b7211733e2 node_cache: Rename npm_cache to target_path.
This is cleaner and part of renaming these variables to refer to the
node_modules_cache.
2017-07-21 17:28:20 -07:00
Tim Abbott e1e5e15797 node_cache: Move npm_args inside the interface. 2017-07-21 17:28:20 -07:00
Tim Abbott b1944b5e1f node_cache: Fix buggy type annotations for copy_modules. 2017-07-21 17:28:20 -07:00
Tim Abbott 8523c24a17 node_modules: Move success stamp to root of cache dir.
This location makes more sense, since it's the entire directory we're
certifying, not just the `node_modules` subdirectory.
2017-07-21 17:28:20 -07:00
Pweaver (Paul Weaver) 1b7aee7cd6 scripts: Add cd_exec library helper. 2017-07-21 17:28:20 -07:00
Tim Abbott 59abefa616 node_cache: Extract cached_node_modules variable. 2017-07-21 17:28:20 -07:00
Tim Abbott 5d05cc7294 scripts: Rename NPM_CACHE_PATH to NODE_MODULES_CACHE_PATH. 2017-07-21 17:28:20 -07:00
Tim Abbott 1e2cc2e8ad third: Add yarn installer to Zulip repo. 2017-07-21 17:28:20 -07:00
rht 614e3bb1ef scripts: Add script to autogenerate a self-signed SSL cert.
This will simplify step 1 of prod-install instruction to reduce
suffering in testing/experimenting production environments.

Attribution: the scripts/setup/configure-certs is based on @galexrt's
5c0daf6211

Further tweaked by tabbott to rename the script and edit the messages.
2017-07-16 17:22:32 -07:00