Mirror
/
zulip
mirror of https://github.com/zulip/zulip.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
zulip/scripts/lib/clean_node_cache.py

49 lines
1.7 KiB
Python
Raw Normal View History

clean-npm-cache: Expand `clean-npm-cache`. Expands `clean-npm-cache` so that it can be used in production environment as well. Also moves it to `scripts/` from `tools/`.
2017-08-19 09:26:52 +02:00
#!/usr/bin/env python3
import argparse
import os
import sys
typing: Remove now-unnecessary conditional import. As a result of dropping support for trusty, we can remove our old pattern of putting `if False` before importing the typing module, which was essential for Python 3.4 support, but not required and maybe harmful on newer versions. cron_file_helper check_rabbitmq_consumers hash_reqs check_zephyr_mirror check_personal_zephyr_mirrors check_cron_file zulip_tools check_postgres_replication_lag api_test_helpers purge-old-deployments setup_venv node_cache clean_venv_cache clean_node_cache clean_emoji_cache pg_backup_and_purge restore-backup generate_secrets zulip-ec2-configure-interfaces diagnose check_user_zephyr_mirror_liveness
2019-07-23 23:58:11 +02:00
from typing import Set
clean-npm-cache: Expand `clean-npm-cache`. Expands `clean-npm-cache` so that it can be used in production environment as well. Also moves it to `scripts/` from `tools/`.
2017-08-19 09:26:52 +02:00
caches: Move cache cleaning helper tools under scripts/lib. This helps keep the root of scripts/ uncluttered.
2017-08-28 02:59:49 +02:00
ZULIP_PATH = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
clean-npm-cache: Expand `clean-npm-cache`. Expands `clean-npm-cache` so that it can be used in production environment as well. Also moves it to `scripts/` from `tools/`.
2017-08-19 09:26:52 +02:00
sys.path.append(ZULIP_PATH)
scripts: Remove unused imports. Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-02-02 23:53:23 +01:00
from scripts.lib.zulip_tools import \
scripts: Use `parse_cache_script_args()` in various cache cleaning scripts. Instead of using `GENERIC_CACHE_SCRIPT_PARSER` and defining `parse_args()` function in each script separately, use `parse_cache_script_args()`.
2017-09-23 20:54:19 +02:00
get_environment, get_recent_deployments, parse_cache_script_args, \
clean-npm-cache: Expand `clean-npm-cache`. Expands `clean-npm-cache` so that it can be used in production environment as well. Also moves it to `scripts/` from `tools/`.
2017-08-19 09:26:52 +02:00
purge_unused_caches
ENV = get_environment()
NODE_MODULES_CACHE_PATH = "/srv/zulip-npm-cache"
python: Convert function type annotations to Python 3 style. Generated by com2ann (slightly patched to avoid also converting assignment type annotations, which require Python 3.6), followed by some manual whitespace adjustment, and six fixes for runtime issues: - def __init__(self, token: Token, parent: Optional[Node]) -> None: + def __init__(self, token: Token, parent: "Optional[Node]") -> None: -def main(options: argparse.Namespace) -> NoReturn: +def main(options: argparse.Namespace) -> "NoReturn": -def fetch_request(url: str, callback: Any, **kwargs: Any) -> Generator[Callable[..., Any], Any, None]: +def fetch_request(url: str, callback: Any, **kwargs: Any) -> "Generator[Callable[..., Any], Any, None]": -def assert_server_running(server: subprocess.Popen[bytes], log_file: Optional[str]) -> None: +def assert_server_running(server: "subprocess.Popen[bytes]", log_file: Optional[str]) -> None: -def server_is_up(server: subprocess.Popen[bytes], log_file: Optional[str]) -> bool: +def server_is_up(server: "subprocess.Popen[bytes]", log_file: Optional[str]) -> bool: - method_kwarg_pairs: List[FuncKwargPair], + method_kwarg_pairs: "List[FuncKwargPair]", Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-19 03:48:37 +02:00
def get_caches_in_use(threshold_days: int) -> Set[str]:
python: Modernize legacy Python 2 syntax with pyupgrade. Generated by `pyupgrade --py3-plus --keep-percent-format` on all our Python code except `zthumbor` and `zulip-ec2-configure-interfaces`, followed by manual indentation fixes. Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-09 21:51:58 +02:00
setups_to_check = {ZULIP_PATH}
clean-npm-cache: Expand `clean-npm-cache`. Expands `clean-npm-cache` so that it can be used in production environment as well. Also moves it to `scripts/` from `tools/`.
2017-08-19 09:26:52 +02:00
caches_in_use = set()
if ENV == "prod":
setups_to_check |= get_recent_deployments(threshold_days)
if ENV == "dev":
# In dev always include the currently active cache in order
# not to break current installation in case dependencies
# are updated with bumping the provision version.
CURRENT_CACHE = os.path.dirname(os.path.realpath(os.path.join(ZULIP_PATH, "node_modules")))
caches_in_use.add(CURRENT_CACHE)
for setup_dir in setups_to_check:
clean-npm-cache: Fix buggy garbage-collection logic. We saw issues with /srv/zulip_npm_cache being cleaned incorrectly by this tool in production (more correctly, we noticed broken symlinks to those directories, even from the current deployment). Print-debugging showed that indeed older deployments were being ignored, because the logic for `get_caches_in_use` was totally broken (this was sorta masked because we also keep the last week's deployments). The specific bug here turned out to be that we weren't passing the `production` argument to generate_sha1sum_node_modules, but the broader problem is that this logic isn't robust to changes in the hashing algorithm. Fix this by replacing the broken logic for trying to compute the correct hash for that deployment with just checking the symlink inside the deployment to let it self-report. We can't easily do this same change for clean-venv-cache, because we use multiple virtualenvs there. But a similar change could be useful for the emoji cache as well. Fixes #8116.
2018-03-29 00:32:47 +02:00
node_modules_link_path = os.path.join(setup_dir, "node_modules")
clean-npm-cache: Use islink to check node_modules. Very old Zulip deployments might have node_modules be an actual directory, which we should ignore for the purposes of garbage-collection.
2018-03-29 19:42:55 +02:00
if not os.path.islink(node_modules_link_path):
clean-npm-cache: Expand `clean-npm-cache`. Expands `clean-npm-cache` so that it can be used in production environment as well. Also moves it to `scripts/` from `tools/`.
2017-08-19 09:26:52 +02:00
# If 'package.json' file doesn't exist then no node_modules
# cache is associated with this setup.
clean-npm-cache: Fix buggy garbage-collection logic. We saw issues with /srv/zulip_npm_cache being cleaned incorrectly by this tool in production (more correctly, we noticed broken symlinks to those directories, even from the current deployment). Print-debugging showed that indeed older deployments were being ignored, because the logic for `get_caches_in_use` was totally broken (this was sorta masked because we also keep the last week's deployments). The specific bug here turned out to be that we weren't passing the `production` argument to generate_sha1sum_node_modules, but the broader problem is that this logic isn't robust to changes in the hashing algorithm. Fix this by replacing the broken logic for trying to compute the correct hash for that deployment with just checking the symlink inside the deployment to let it self-report. We can't easily do this same change for clean-venv-cache, because we use multiple virtualenvs there. But a similar change could be useful for the emoji cache as well. Fixes #8116.
2018-03-29 00:32:47 +02:00
continue
scripts: Fix incorrect garbage-collection of emoji/node caches. Apparently, we were incorrectly expressing the paths in the caches_in_use data structures for these two cache-cleaning algorithms, resulting in the default threshhold_days algorithm controlling which caches could be garbage-collected. While the emoji one was just a performance optimization for upgrade-zulip-from-git, it was possible for the main `node_modules` cache in use in production to be GCed, resulting in LaTeX rendering being broken.
2018-12-03 20:59:08 +01:00
# The actual cache path doesn't include the /node_modules
caches_in_use.add(os.path.dirname(os.readlink(node_modules_link_path)))
clean-npm-cache: Expand `clean-npm-cache`. Expands `clean-npm-cache` so that it can be used in production environment as well. Also moves it to `scripts/` from `tools/`.
2017-08-19 09:26:52 +02:00
return caches_in_use
clean-unused-caches: Optimize performance. This saves about 400ms when running clean-unused-caches, basically by calling its sub-rountines by import (rather than `subprocess.check_call()`). The performance optimization seems well worth it. Fixes #9766.
2018-06-18 16:28:34 +02:00
def main(args: argparse.Namespace) -> None:
clean-npm-cache: Expand `clean-npm-cache`. Expands `clean-npm-cache` so that it can be used in production environment as well. Also moves it to `scripts/` from `tools/`.
2017-08-19 09:26:52 +02:00
caches_in_use = get_caches_in_use(args.threshold_days)
purge_unused_caches(
scripts: Rearrange the arguments of `purge_unused_caches()`. This commit re-arranges the arguments of `purge_unused_caches()` function in order to remain consistent with other similar functions in the library like `may_be_perform_caching()`.
2017-09-23 21:21:55 +02:00
NODE_MODULES_CACHE_PATH, caches_in_use, "node modules cache", args)
clean-npm-cache: Expand `clean-npm-cache`. Expands `clean-npm-cache` so that it can be used in production environment as well. Also moves it to `scripts/` from `tools/`.
2017-08-19 09:26:52 +02:00
if __name__ == "__main__":
clean-unused-caches: Optimize performance. This saves about 400ms when running clean-unused-caches, basically by calling its sub-rountines by import (rather than `subprocess.check_call()`). The performance optimization seems well worth it. Fixes #9766.
2018-06-18 16:28:34 +02:00
args = parse_cache_script_args("This script cleans unused zulip npm caches.")
main(args)