provision: Let build_emoji build its own cache.

We no longer need to maintain duplicate code
related to where we set up the emoji
cache directory.

And we no longer need two extra steps for
people doing advanced (i.e. manual) setup.

There was no clear benefit to having provision
build the cache directory for `build_emoji`,
when it was easy to make `build_emoji` more
self-sufficient.  The `build_emoji` tool
was already importing the library that has
`run_as_root`, and it was already responsible
for 99% of the create-directory kind of tasks.

(We always call `build_emoji` unconditionally from
`provision`, so there's no rationale in terms
of avoiding startup time or something.)

ASIDE:

Its not completely clear to me why we need
to put this directory in "/srv", instead of
somewhere more local (like we already do for
Travis), but maybe it's just to be like
its siblings in "/srv":

    node_modules
    yarn.lock
    zulip-emoji-cache
    zulip-npm-cache
    zulip-py3-venv
    zulip-thumbor-venv
    zulip-venv-cache
    zulip-yarn

I guess the caches that we keep in var are
dev-only, although I think some of what's under
`zulip-emoji-cache` is also dev-only in nature?

    ./var/webpack-cache
    ./var/mypy-cache

In `docs/subsystems/emoji.md` we say this:

```
The `build_emoji` tool generates the set of files under
`static/generated/emoji` (or really, it generates the
`/srv/zulip-emoji-cache/<sha1>/emoji` tree, and
`static/generated/emoji` is a symlink to that tree;we do this in
order to cache old versions to make provisioning and production
deployments super fast in the common case that we haven't changed the
emoji tooling). [...]
```

I don't really understand that rationale for the development
case, since `static/generated` is as much ignored by `git` as
'/srv' is, without the complications of needing `sudo` to create it.

And in production, I'm not sure how much time we're really saving,
as it takes me about 1.4s to fully rebuild the cache in dev, not to
mention we're taking on upgrade risk by sharing files between versions.
This commit is contained in:
Steve Howell 2020-04-17 12:08:55 +00:00 committed by Tim Abbott
parent bf3decfd0c
commit 7eb6d32d59
3 changed files with 10 additions and 16 deletions

View File

@ -224,8 +224,6 @@ Now run these commands:
```
sudo ./scripts/lib/install-node
yarn install
sudo mkdir /srv/zulip-emoji-cache
sudo chown -R `whoami`: /srv/zulip-emoji-cache
./tools/setup/emoji/build_emoji
./scripts/setup/inline_email_css.py
./tools/setup/build_pygments_data

View File

@ -8,7 +8,7 @@ import shutil
ZULIP_PATH = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
sys.path.append(ZULIP_PATH)
from scripts.lib.zulip_tools import run, run_as_root, OKBLUE, ENDC, \
from scripts.lib.zulip_tools import run, OKBLUE, ENDC, \
get_dev_uuid_var_path, file_or_package_hash_updated
from version import PROVISION_VERSION
@ -16,15 +16,6 @@ from version import PROVISION_VERSION
from tools.setup.generate_zulip_bots_static_files import generate_zulip_bots_static_files
VENV_PATH = "/srv/zulip-py3-venv"
is_travis = 'TRAVIS' in os.environ
# TODO: De-duplicate this with emoji_dump.py
EMOJI_CACHE_PATH = "/srv/zulip-emoji-cache"
if is_travis:
# In Travis CI, we don't have root access
EMOJI_CACHE_PATH = "/home/travis/zulip-emoji-cache"
UUID_VAR_PATH = get_dev_uuid_var_path()
def create_var_directories() -> None:
@ -156,9 +147,6 @@ def main(options: argparse.Namespace) -> int:
# The `build_emoji` script requires `emoji-datasource` package
# which we install via npm; thus this step is after installing npm
# packages.
if not os.access(EMOJI_CACHE_PATH, os.W_OK):
run_as_root(["mkdir", "-p", EMOJI_CACHE_PATH])
run_as_root(["chown", "%s:%s" % (os.getuid(), os.getgid()), EMOJI_CACHE_PATH])
run(["tools/setup/emoji/build_emoji"])
# copy over static files from the zulip_bots package

View File

@ -17,7 +17,7 @@ from emoji_names import EMOJI_NAME_MAPS
ZULIP_PATH = os.path.join(os.path.dirname(os.path.abspath(__file__)), '../../../')
sys.path.append(ZULIP_PATH)
from scripts.lib.zulip_tools import generate_sha1sum_emoji
from scripts.lib.zulip_tools import generate_sha1sum_emoji, run_as_root
TARGET_EMOJI_DUMP = os.path.join(ZULIP_PATH, 'static', 'generated', 'emoji')
TARGET_EMOJI_STYLES = os.path.join(ZULIP_PATH, 'static', 'generated', 'emoji-styles')
@ -77,6 +77,14 @@ if 'TRAVIS' in os.environ:
EMOJI_CACHE_PATH = "/home/travis/zulip-emoji-cache"
def main() -> None:
if not os.access(EMOJI_CACHE_PATH, os.W_OK):
# Note: In production, this block will fail, since we don't
# assume sudo access; but it should never run in production
# anyway, because EMOJI_CACHE_PATH is created by puppet before
# build_emoji would be run.
run_as_root(["mkdir", "-p", EMOJI_CACHE_PATH])
run_as_root(["chown", "%s:%s" % (os.getuid(), os.getgid()), EMOJI_CACHE_PATH])
sha1_hexdigest = generate_sha1sum_emoji(ZULIP_PATH)
source_emoji_dump = os.path.join(EMOJI_CACHE_PATH, sha1_hexdigest, 'emoji')
success_stamp = os.path.join(source_emoji_dump, '.success-stamp')