zulip/tools
Steve Howell 7eb6d32d59 provision: Let build_emoji build its own cache.
We no longer need to maintain duplicate code
related to where we set up the emoji
cache directory.

And we no longer need two extra steps for
people doing advanced (i.e. manual) setup.

There was no clear benefit to having provision
build the cache directory for `build_emoji`,
when it was easy to make `build_emoji` more
self-sufficient.  The `build_emoji` tool
was already importing the library that has
`run_as_root`, and it was already responsible
for 99% of the create-directory kind of tasks.

(We always call `build_emoji` unconditionally from
`provision`, so there's no rationale in terms
of avoiding startup time or something.)

ASIDE:

Its not completely clear to me why we need
to put this directory in "/srv", instead of
somewhere more local (like we already do for
Travis), but maybe it's just to be like
its siblings in "/srv":

    node_modules
    yarn.lock
    zulip-emoji-cache
    zulip-npm-cache
    zulip-py3-venv
    zulip-thumbor-venv
    zulip-venv-cache
    zulip-yarn

I guess the caches that we keep in var are
dev-only, although I think some of what's under
`zulip-emoji-cache` is also dev-only in nature?

    ./var/webpack-cache
    ./var/mypy-cache

In `docs/subsystems/emoji.md` we say this:

```
The `build_emoji` tool generates the set of files under
`static/generated/emoji` (or really, it generates the
`/srv/zulip-emoji-cache/<sha1>/emoji` tree, and
`static/generated/emoji` is a symlink to that tree;we do this in
order to cache old versions to make provisioning and production
deployments super fast in the common case that we haven't changed the
emoji tooling). [...]
```

I don't really understand that rationale for the development
case, since `static/generated` is as much ignored by `git` as
'/srv' is, without the complications of needing `sudo` to create it.

And in production, I'm not sure how much time we're really saving,
as it takes me about 1.4s to fully rebuild the cache in dev, not to
mention we're taking on upgrade risk by sharing files between versions.
2020-04-17 09:53:26 -07:00
..
ci nginx: Set X-XSS-Protection: 1; mode=block. 2020-04-05 16:13:53 -07:00
circleci circleci: Add base image for Ubuntu Focal. 2020-03-25 16:09:59 -07:00
documentation_crawler python: Modernize legacy Python 2 syntax with pyupgrade. 2020-04-09 16:43:22 -07:00
droplets python: Modernize legacy Python 2 syntax with pyupgrade. 2020-04-09 16:43:22 -07:00
i18n python: Modernize legacy Python 2 syntax with pyupgrade. 2020-04-09 16:43:22 -07:00
lib provision: Let build_emoji build its own cache. 2020-04-17 09:53:26 -07:00
linter_lib emails: Add tests for sending custom emails. 2020-04-14 10:50:43 -07:00
setup provision: Let build_emoji build its own cache. 2020-04-17 09:53:26 -07:00
test-install test-install: Use lxc-destroy -f instead of lxc-stop. 2019-12-18 03:48:39 -08:00
tests check-rabbitmq-queue: Add a simple algorithm to analyze queue stats. 2020-04-09 13:41:01 -07:00
wsl tools: Add script to start database services for WSL. 2020-04-11 12:05:33 -07:00
zulip-export python: Modernize legacy Python 2 syntax with pyupgrade. 2020-04-09 16:43:22 -07:00
README.md cleanup: Delete trailing newlines. 2019-08-06 23:29:11 -07:00
__init__.py
build-docs docs: Replace Makefile based on output of newer sphinx-quickstart. 2019-12-02 18:39:33 -08:00
build-release-tarball build-release-tarball: Run with zulip-py-venv symlink present. 2019-07-21 18:43:52 -07:00
cache-zulip-git-version version: Only let `git describe` match tags beginning with a digit. 2019-10-24 14:54:45 -07:00
check-capitalization i18n: Move static/locale back to locale. 2019-07-02 14:57:55 -07:00
check-frontend-i18n i18n: Move static/locale back to locale. 2019-07-02 14:57:55 -07:00
check-issue-labels lint: Fix code that evaded our lint checks for string % non-tuple. 2019-04-23 15:21:37 -07:00
check-openapi js: Convert a.indexOf(…) !== -1 to a.includes(…). 2020-02-10 14:08:12 -08:00
check-provision tools: Extract get_provisioning_status check logic. 2019-06-23 21:55:02 -07:00
check-templates linter_lib: Fix mypy errors. 2019-08-09 17:22:45 -07:00
clean-branches text: Fix some typos (most of them found and fixed by codespell). 2020-03-27 17:25:56 -07:00
clean-repo
commit-message-lint commit-message-lint: Detect the upstream remote name. 2020-02-04 13:57:51 -08:00
commit-msg Use #!/usr/bin/env for bash shebangs. 2018-12-17 17:21:08 -08:00
conf.ini-template
coveragerc test-backend: Remove rate_limiter from not_yet_fully_covered. 2020-04-15 11:20:37 -07:00
create-test-api-docs api_docs: Replaced cgi.escape with html.escape. 2020-03-23 10:28:21 -07:00
debug-require-webpack-plugin.ts webpack: Generalize debug-require-webpack-plugin interface. 2020-02-25 14:01:06 -08:00
debug-require.js webpack: Generalize debug-require-webpack-plugin interface. 2020-02-25 14:01:06 -08:00
deploy-branch Use #!/usr/bin/env for bash shebangs. 2018-12-17 17:21:08 -08:00
diagnose typing: Remove now-unnecessary conditional import. 2019-07-29 15:18:22 -07:00
django-template-graph django-template-graph: Fix shellcheck warnings. 2018-10-17 17:38:56 -07:00
do-destroy-rebuild-database flush-memcached: Respect MEMCACHED_LOCATION; handle errors. 2019-10-01 16:05:55 -07:00
do-destroy-rebuild-test-database do-destroy-rebuild-test-database: Fix shellcheck warnings. 2018-08-03 09:15:26 -07:00
documentation.vnufilter test-documentation: Validate HTML with vnu.jar. 2019-06-27 14:53:21 -07:00
duplicate_commits.json tools: Move duplicate_commits.json file to tools directory. 2020-04-07 16:06:29 -07:00
fetch-contributor-data fetch-contributor-data: Fix running in production. 2020-04-12 12:14:07 -07:00
fetch-pull-request tools: Use require_clean_work_tree everywhere. 2020-04-06 17:52:10 -07:00
fetch-rebase-pull-request tools: Use require_clean_work_tree everywhere. 2020-04-06 17:52:10 -07:00
find-unused-css Use #!/usr/bin/env for bash shebangs. 2018-12-17 17:21:08 -08:00
generate-integration-docs-screenshot generate-integration-docs-screenshot: Allow sending custom headers. 2020-04-17 09:41:55 -07:00
get-handlebar-vars python: Modernize legacy Python 2 syntax with pyupgrade. 2020-04-09 16:43:22 -07:00
html-grep templates: Rename *.handlebars ↦ *.hbs and - ↦ _. 2019-07-12 21:11:03 -07:00
js-dep-visualizer.py tools: Add TypeScript to the dependency visualizer. 2019-04-12 11:14:42 -07:00
lint python: Modernize legacy Python 2 syntax with pyupgrade. 2020-04-09 16:43:22 -07:00
message-screenshot.js message-screenshot: Create image directory if it doesn't exist. 2020-04-17 09:41:55 -07:00
pre-commit lint: Add --skip arg to replace --no-gitlint/mypy. 2019-06-18 11:32:04 -07:00
pretty-print-html python: Modernize legacy Python 2 syntax with pyupgrade. 2020-04-09 16:43:22 -07:00
provision install, provision: Treat all nonzero exit codes as failure, not just 1. 2019-07-19 11:22:11 -07:00
push-to-pull-request reset-to-pull-request: Add a usage message. 2020-02-11 14:45:17 -08:00
release-tarball-exclude.txt
renumber-migrations renumber-migrations: Exclude migration 0209. 2019-12-11 16:08:32 -08:00
replacer zulint: Move lister.py to tools/zulint. 2018-08-04 19:53:53 -07:00
reset-to-pull-request tools: Use require_clean_work_tree everywhere. 2020-04-06 17:52:10 -07:00
review tools/review: Don’t pretend to emulate shell=True either. 2018-07-30 22:39:08 -07:00
run-dev.py run-dev: Automatically set EXTERNAL_HOST for droplet dev servers. 2020-04-10 16:50:13 -04:00
run-mypy lint: Run mypy with --no-error-summary. 2019-11-13 13:26:02 -08:00
run-tsc tsconfig: Move to top level. 2019-11-04 18:12:11 -08:00
run-yarn-deduplicate yarn.lock: Share duplicate packages with yarn-deduplicate. 2019-09-02 19:30:09 -07:00
setup-git-repo Use #!/usr/bin/env for bash shebangs. 2018-12-17 17:21:08 -08:00
sgrep.yml sgrep: Install syntactic code search tool as an external linter. 2020-04-01 15:08:34 -07:00
show-profile-results
stop-run-dev python: Modernize legacy Python 2 syntax with pyupgrade. 2020-04-09 16:43:22 -07:00
test-all lint: Use --groups to specify specific groups to run. 2019-06-23 22:23:15 -07:00
test-api populate_db, tests: Restrict emails in zulip realm. 2020-03-19 16:21:31 -07:00
test-backend test-backend: Remove rate_limiter from not_yet_fully_covered. 2020-04-15 11:20:37 -07:00
test-documentation test-documentation: Invoke sphinx-build via make instead of directly. 2019-12-02 18:39:33 -08:00
test-emoji-name-scripts test-emoji-name-scripts: Avoid hardcoded paths in /var/tmp. 2019-01-15 16:05:50 -08:00
test-help-documentation test-help-documentation: Bind vnu.jar to 127.0.0.1. 2019-06-28 18:18:29 -07:00
test-js-with-casper
test-js-with-node test-js-with-node: Show correct coverage link in color. 2020-04-11 07:05:04 -04:00
test-js-with-puppeteer test-js-with-puppeteer: Extract code to prepare for puppeteer run. 2020-04-17 09:41:55 -07:00
test-locked-requirements test-locked-requirements: Cache only one copy of each valid hash. 2020-04-05 13:45:25 -07:00
test-migrations requirements: Upgrade django-otp from 0.5.2 to 0.6.0. 2019-07-07 22:28:54 -07:00
test-queue-worker-reload queue: Remove missedmessage_email_senders. 2020-01-31 12:13:51 -08:00
test-run-dev Revert "test-run-dev: Use a pipe rather than polling a log file." 2020-03-17 14:16:18 -07:00
test-tools
update-locked-requirements requirements: Fork pip.txt to pip2.txt for Python 2. 2020-02-05 12:38:10 -08:00
update-prod-static scripts: Rename inline-email-css to inline_email_css.py. 2020-04-10 15:29:47 -07:00
update-zuliprc-api-field python: Modernize legacy Python 2 syntax with pyupgrade. 2020-04-09 16:43:22 -07:00
webpack webpack: Enable code splitting and deduplication. 2019-10-28 15:53:15 -07:00
webpack-helpers.ts js: Automatically convert var to let and const in more files. 2019-11-20 14:10:47 -08:00
webpack.assets.json emoji: Resolve emoji sprite sheets and stylesheets through Webpack. 2020-02-25 14:43:46 -08:00
webpack.config.ts static: Serve webpack bundles from the root domain. 2020-04-10 00:48:02 -07:00
zanitizer docs: Avoid hardcoded /tmp paths in miscellaneous documentation. 2019-01-15 16:16:12 -08:00
zanitizer_config.pm.sample zanitizer_config.pm.sample: keep_file → scrub_filename 2018-09-23 20:42:27 -07:00

README.md

This directory contains scripts that are used in building, managing, testing, and other forms of work in a Zulip development environment. Note that tools that are also useful in production belong in scripts/ or should be Django management commands.

For more details, see https://zulip.readthedocs.io/en/latest/overview/directory-structure.html.