zulip/tools
Greg Price bf5f1b5f20 install: Start on an LXC-based dev/test environment for the installer.
In order to do development on the installer itself in a sane way,
we need a reasonably fast and automatic way to get a fresh environment
to try to run it in.

This calls for some form of virtualization.  Choices include

 * A public cloud, like EC2 or Digital Ocean.  These could work, if we
   wrote some suitable scripts against their APIs, to manage
   appropriate base images (as AMIs or snapshots respectively) and to
   start fresh instances/droplets from a base image.  There'd be some
   latency on starting a new VM, and this would also require the user
   to have an account on the relevant cloud with API access to create
   images and VMs.

 * A local whole-machine VM system (hypervisor) like VirtualBox or
   VMware, perhaps managing the configuration through Vagrant.  These
   hypervisors can be unstable and painfully slow.  They're often the
   only way to get development work done on a Mac or Windows machine,
   which is why we use them there for the normal Zulip development
   environment; but I don't really want to find out how their
   instability scales when constantly spawning fresh VMs from an image.

 * Containers.  The new hotness, the name on everyone's lips, is Docker.
   But Docker is not designed for virtualizing a traditional Unix server,
   complete with its own init system and a fleet of processes with a
   shared filesystem -- in other words, the platform Zulip's installer
   and deployment system are for.  Docker brings its own quite
   different model of deployment, and someday we may port Zulip from
   the traditional Unix server to the Docker-style deployment model,
   but for testing our traditional-Unix-server deployment we need a
   (virtualized) traditional Unix server.

 * Containers, with LXC.  LXC provides containers that function as
   traditional Unix servers; because of the magic of containers, the
   overhead is quite low, and LXC offers handy snapshotting features
   so that we can quickly start up a fresh environment from a base
   image.  Running LXC does require a Linux base system.  For
   contributors whose local development machine isn't already Linux,
   the same solutions are available as for our normal development
   environment: the base system for running LXC could be e.g. a
   Vagrant-managed VirtualBox VM, or a machine in a public cloud.

This commit adds a first version of such a thing, using LXC to manage
a base image plus a fresh container for each test run.  The test
containers function as VMs: once installed, all the Zulip services run
normally in them and can be managed in the normal production ways.

This initial version has a shortage of usage messages or docs, and
likely has some sharp edges.  It also requires familiarity with the
basics of LXC commands in order to make good use of the resulting
containers: `lxc-ls -f`, `lxc-attach`, `lxc-stop`, and `lxc-start`,
in particular.
2018-01-19 17:27:04 -08:00
..
circleci/images circleci: Set up locale en_US.UTF-8. 2017-12-20 19:18:29 -08:00
documentation_crawler docs: Reorganize developer docs to improve navigation. 2017-11-16 09:45:08 -08:00
droplets droplets: Update snapshot id. 2017-11-29 14:24:08 -08:00
lib reminder_bot: Add infra for adding reminder bot to every realm. 2018-01-19 11:33:11 -05:00
linter_lib billing: Integrate Stripe, using Stripe Checkout. 2018-01-17 16:43:54 -08:00
setup provision: Remove transitional Python 2->3 logic in setup_venvs. 2017-12-18 21:59:51 -08:00
test-install install: Start on an LXC-based dev/test environment for the installer. 2018-01-19 17:27:04 -08:00
tests lint: Fix tools tests. 2017-11-16 13:59:40 -08:00
travis circleci: Skip checking links in documentation for now. 2018-01-09 10:53:13 -08:00
zulip-export zulip-export: Convert to using argparse from optparse. 2017-11-10 15:54:02 -08:00
README.md docs: Update links to ReadTheDocs to always use https. 2017-11-16 10:59:24 -08:00
__init__.py
build-docs tools: Add wrapper tool 'build-docs' for building documentation. 2017-06-14 20:16:02 -07:00
build-release-tarball build-release-tarball: Improve help/error output. 2017-11-16 22:19:58 -08:00
check-capitalization docs: Update links from codebase to point to ReadTheDocs. 2017-11-16 10:53:49 -08:00
check-css css linter: Fix error reporting. 2018-01-03 13:26:55 -05:00
check-frontend-i18n tools: Revert to Python 2 typing syntax for now. 2017-12-13 10:38:15 -08:00
check-issue-labels refactor: Remove six.moves.configparser import. 2017-11-07 10:51:44 -08:00
check-provision check-provision: Replace optparse with argparse. 2017-09-30 09:32:46 -07:00
check-swagger lint: Ban non-unique operationIds in Swagger. 2017-06-26 22:02:26 -04:00
check-templates tools: Revert to Python 2 typing syntax for now. 2017-12-13 10:38:15 -08:00
check-urls tools: Remove print_function. 2017-09-29 15:44:56 -07:00
clean-branches tools: Update clean-branches to clean review branches. 2017-06-01 21:55:33 -07:00
clean-repo clean-repo: Revert optimization to .pyc removal. 2017-07-30 17:24:55 -07:00
commit-message-lint docs: Fix spelling of Travis CI. 2017-07-31 09:36:22 -07:00
commit-msg lint: Check if commit message is not empty before running gitlint. 2017-06-25 11:29:14 -04:00
compile-handlebars-templates tools: Revert to Python 2 typing syntax for now. 2017-12-13 10:38:15 -08:00
conf.ini-template Add tool for scanning issues without area labels. 2017-02-03 09:59:35 -08:00
coveragerc coverage: Update pattern for __str__ for new typing syntax. 2017-11-22 12:16:58 -08:00
create-test-api-docs tools: Revert to Python 2 typing syntax for now. 2017-12-13 10:38:15 -08:00
deploy-branch
deployment-lock-ctl
diagnose requirements: Rename requirements files. 2017-11-21 02:38:26 +05:30
django-template-graph
do-destroy-rebuild-database test_fixtures: Move test_db_status state under UUID_VAR_PATH. 2017-10-17 21:15:58 -07:00
do-destroy-rebuild-test-database Remove extra new lines at the ends of Zulip authoried files. 2016-09-26 21:05:24 -07:00
fetch-pull-request tools: Add script to see PRs as authors intended. 2017-06-01 21:55:08 -07:00
fetch-rebase-pull-request git tools: Make error messages more clear. 2017-05-08 10:51:59 -07:00
find-add-class tools: Revert to Python 2 typing syntax for now. 2017-12-13 10:38:15 -08:00
find-unused-css
generate-custom-icon-webfont Generate custom-icon-webfont on each provision or update-prod-static. 2017-11-20 16:36:49 -08:00
get-handlebar-vars tools: Revert to Python 2 typing syntax for now. 2017-12-13 10:38:15 -08:00
html-grep tools: Revert to Python 2 typing syntax for now. 2017-12-13 10:38:15 -08:00
inline-email-css Text-wrap the remaining long lines exceeding 110. 2017-11-15 10:58:03 -08:00
js-dep-visualizer.py tools: Remove print_function. 2017-09-29 15:44:56 -07:00
lint linter: Re-enable gitlint. 2017-12-18 09:36:47 -05:00
lister.py mypy: Use Python 3 type syntax in tools/lister.py. 2017-12-05 16:32:18 -08:00
minify-js tools: Remove print_function. 2017-09-29 15:44:56 -07:00
munge-postgres-logs tools: Remove `.pl` extension from user script. 2017-10-11 12:52:36 -07:00
optimize-svg tools: Add optimize-svg tool for checking for unoptimized SVG files. 2017-05-30 11:15:56 -07:00
pre-commit linter: Replace --pep8 with new --frontend/--backend options. 2017-08-27 13:50:53 -07:00
pretty-print-html tools: Remove print_function. 2017-09-29 15:44:56 -07:00
provision vagrant: Skip the shell virtualenv warning on provision. 2017-10-27 11:44:47 -07:00
release-tarball-exclude.txt build-release-tarball: Stop using .gitattributes. 2017-10-11 21:51:42 -07:00
renumber-migrations refactor: Remove six.moves.input import. 2017-11-07 10:51:44 -08:00
replace-tarball-shebang tools: Text-wrap long lines exceeding 110. 2017-11-10 16:24:09 -08:00
replacer replacer: Run as script with python 3. 2017-10-05 10:07:54 -07:00
reset-to-pull-request git tools: Make error messages more clear. 2017-05-08 10:51:59 -07:00
review tools: Remove print_function. 2017-09-29 15:44:56 -07:00
run-dev-queue-processors tools: Remove absolute_import in most tools. 2017-09-29 12:28:43 -07:00
run-dev.py schedulemessages: Add delivery system for scheduled message. 2018-01-10 09:18:02 -05:00
run-mypy requirements: Upgrade mypy to 0.550. 2017-11-25 10:06:27 -08:00
send-github-payloads json: Replace most use of simplejson with json. 2017-10-11 22:55:35 -07:00
setup-git-repo lint: Add checks for commit messages using gitlint. 2017-04-21 13:45:12 -07:00
show-profile-results mypy: Remove ignores for a few typeshed bugs fixed upstream. 2017-10-27 17:09:00 -07:00
start-dockers
stop-run-dev tools: Consistently use hyphen in user script names. 2017-10-11 12:52:36 -07:00
sync-translations tools: Add new script to sync translations. 2017-10-05 23:07:16 -07:00
test-all tools: Remove `.py` extensions from user scripts. 2017-10-11 12:52:36 -07:00
test-all-docker
test-api test-api: Fix tests to work with subdomains. 2017-10-02 13:31:22 -07:00
test-backend Slack importer: Add the slack to zulip data conversion files. 2017-12-27 07:50:53 -05:00
test-documentation circleci: Skip checking links in documentation for now. 2018-01-09 10:53:13 -08:00
test-help-documentation test-help-documentation: Pass use_db=True. 2017-11-16 13:44:53 -08:00
test-js-with-casper
test-js-with-node node tests: Add 100% test coverage for user_groups.js. 2018-01-06 11:50:52 -05:00
test-locked-requirements tools: Rewrite `test-locked-requirements` to be more performant. 2017-11-20 15:35:47 -08:00
test-migrations two_factor: Add configuration and URLs. 2017-11-28 15:21:40 -08:00
test-queue-worker-reload test-queue-worker-reload: Add wait to avoid race condition. 2018-01-05 08:24:57 -05:00
test-run-dev test-run-dev: Redirect stderr to STDOUT. 2018-01-01 09:42:11 -05:00
test-slack-importer slack importer: Refactor checking for availability of realm subdomain. 2018-01-17 09:38:53 -05:00
test-tools tools: Revert to Python 2 typing syntax for now. 2017-12-13 10:38:15 -08:00
update-authors-json /team: Fetch zulip-js repository data. 2018-01-16 12:23:19 -05:00
update-locked-requirements requirements: Rename requirements files. 2017-11-21 02:38:26 +05:30
update-prod-static datetimepicker: Add flatpickr lib as dependancy. 2018-01-19 11:33:11 -05:00
webpack tools: Revert to Python 2 typing syntax for now. 2017-12-13 10:38:15 -08:00
webpack.assets.json help: Merge api.js and help.js bundles to fix code blocks. 2017-10-30 22:53:17 -07:00
webpack.config.ts webpack: jQuery and underscore should be compilied normally to make errors readable. 2017-08-05 19:53:56 -07:00
zanitizer
zanitizer_config.pm.sample

README.md

This directory contains scripts that are used in building, managing, testing, and other forms of work in a Zulip development environment. Note that tools that are also useful in production belong in scripts/ or should be Django management commands.

For more details, see https://zulip.readthedocs.io/en/latest/overview/directory-structure.html.