Mirror
/
zulip
mirror of https://github.com/zulip/zulip.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
zulip/scripts/lib/clean_emoji_cache.py

46 lines
1.6 KiB
Python
Raw Normal View History

clean_emoji_cache: Expand `clean-emoji-cache`. Expands `clean-emoji-cache` so that it can be used in production environment as well. Also moves it to `scripts/` from `tools/`.
2017-08-19 19:55:42 +02:00
#!/usr/bin/env python3
import argparse
import os
import sys
typing: Remove now-unnecessary conditional import. As a result of dropping support for trusty, we can remove our old pattern of putting `if False` before importing the typing module, which was essential for Python 3.4 support, but not required and maybe harmful on newer versions. cron_file_helper check_rabbitmq_consumers hash_reqs check_zephyr_mirror check_personal_zephyr_mirrors check_cron_file zulip_tools check_postgres_replication_lag api_test_helpers purge-old-deployments setup_venv node_cache clean_venv_cache clean_node_cache clean_emoji_cache pg_backup_and_purge restore-backup generate_secrets zulip-ec2-configure-interfaces diagnose check_user_zephyr_mirror_liveness
2019-07-23 23:58:11 +02:00
from typing import Set
clean_emoji_cache: Expand `clean-emoji-cache`. Expands `clean-emoji-cache` so that it can be used in production environment as well. Also moves it to `scripts/` from `tools/`.
2017-08-19 19:55:42 +02:00
caches: Move cache cleaning helper tools under scripts/lib. This helps keep the root of scripts/ uncluttered.
2017-08-28 02:59:49 +02:00
ZULIP_PATH = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
clean_emoji_cache: Expand `clean-emoji-cache`. Expands `clean-emoji-cache` so that it can be used in production environment as well. Also moves it to `scripts/` from `tools/`.
2017-08-19 19:55:42 +02:00
sys.path.append(ZULIP_PATH)
scripts: Remove unused imports. Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-02-02 23:53:23 +01:00
from scripts.lib.zulip_tools import \
get_environment, get_recent_deployments, \
scripts: Use `parse_cache_script_args()` in various cache cleaning scripts. Instead of using `GENERIC_CACHE_SCRIPT_PARSER` and defining `parse_args()` function in each script separately, use `parse_cache_script_args()`.
2017-09-23 20:54:19 +02:00
parse_cache_script_args, purge_unused_caches
clean_emoji_cache: Expand `clean-emoji-cache`. Expands `clean-emoji-cache` so that it can be used in production environment as well. Also moves it to `scripts/` from `tools/`.
2017-08-19 19:55:42 +02:00
ENV = get_environment()
EMOJI_CACHE_PATH = "/srv/zulip-emoji-cache"
python: Convert function type annotations to Python 3 style. Generated by com2ann (slightly patched to avoid also converting assignment type annotations, which require Python 3.6), followed by some manual whitespace adjustment, and six fixes for runtime issues: - def __init__(self, token: Token, parent: Optional[Node]) -> None: + def __init__(self, token: Token, parent: "Optional[Node]") -> None: -def main(options: argparse.Namespace) -> NoReturn: +def main(options: argparse.Namespace) -> "NoReturn": -def fetch_request(url: str, callback: Any, **kwargs: Any) -> Generator[Callable[..., Any], Any, None]: +def fetch_request(url: str, callback: Any, **kwargs: Any) -> "Generator[Callable[..., Any], Any, None]": -def assert_server_running(server: subprocess.Popen[bytes], log_file: Optional[str]) -> None: +def assert_server_running(server: "subprocess.Popen[bytes]", log_file: Optional[str]) -> None: -def server_is_up(server: subprocess.Popen[bytes], log_file: Optional[str]) -> bool: +def server_is_up(server: "subprocess.Popen[bytes]", log_file: Optional[str]) -> bool: - method_kwarg_pairs: List[FuncKwargPair], + method_kwarg_pairs: "List[FuncKwargPair]", Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-19 03:48:37 +02:00
def get_caches_in_use(threshold_days: int) -> Set[str]:
python: Modernize legacy Python 2 syntax with pyupgrade. Generated by `pyupgrade --py3-plus --keep-percent-format` on all our Python code except `zthumbor` and `zulip-ec2-configure-interfaces`, followed by manual indentation fixes. Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-09 21:51:58 +02:00
setups_to_check = {ZULIP_PATH}
clean_emoji_cache: Expand `clean-emoji-cache`. Expands `clean-emoji-cache` so that it can be used in production environment as well. Also moves it to `scripts/` from `tools/`.
2017-08-19 19:55:42 +02:00
caches_in_use = set()
if ENV == "prod":
setups_to_check |= get_recent_deployments(threshold_days)
if ENV == "dev":
CACHE_SYMLINK = os.path.join(ZULIP_PATH, "static", "generated", "emoji")
CURRENT_CACHE = os.path.dirname(os.path.realpath(CACHE_SYMLINK))
caches_in_use.add(CURRENT_CACHE)
for setup_dir in setups_to_check:
clean-emoji-cache: Stop recalculating paths for other deployments. This is the analog of 7b2c9223e77a9a70120678aac02c125cdb1f6314 for the emoji cache; the only difference is that the existing code was working correctly. It's still worth changing for improved robustness.
2018-03-29 00:48:29 +02:00
emoji_link_path = os.path.join(setup_dir, "static/generated/emoji")
if not os.path.islink(emoji_link_path):
# This happens for a deployment directory extracted from a
# tarball, which just has a copy of the emoji data, not a symlink.
clean_emoji_cache: Expand `clean-emoji-cache`. Expands `clean-emoji-cache` so that it can be used in production environment as well. Also moves it to `scripts/` from `tools/`.
2017-08-19 19:55:42 +02:00
continue
scripts: Fix incorrect garbage-collection of emoji/node caches. Apparently, we were incorrectly expressing the paths in the caches_in_use data structures for these two cache-cleaning algorithms, resulting in the default threshhold_days algorithm controlling which caches could be garbage-collected. While the emoji one was just a performance optimization for upgrade-zulip-from-git, it was possible for the main `node_modules` cache in use in production to be GCed, resulting in LaTeX rendering being broken.
2018-12-03 20:59:08 +01:00
# The actual cache path doesn't include the /emoji
caches_in_use.add(os.path.dirname(os.readlink(emoji_link_path)))
clean_emoji_cache: Expand `clean-emoji-cache`. Expands `clean-emoji-cache` so that it can be used in production environment as well. Also moves it to `scripts/` from `tools/`.
2017-08-19 19:55:42 +02:00
return caches_in_use
clean-unused-caches: Optimize performance. This saves about 400ms when running clean-unused-caches, basically by calling its sub-rountines by import (rather than `subprocess.check_call()`). The performance optimization seems well worth it. Fixes #9766.
2018-06-18 16:28:34 +02:00
def main(args: argparse.Namespace) -> None:
clean_emoji_cache: Expand `clean-emoji-cache`. Expands `clean-emoji-cache` so that it can be used in production environment as well. Also moves it to `scripts/` from `tools/`.
2017-08-19 19:55:42 +02:00
caches_in_use = get_caches_in_use(args.threshold_days)
zulip_tools.py: Extract `may_be_perform_purging()` function. Based on the `dry_run` flag, this function either purges the list of directories passed to them or prints a listing of the directories it would have purged/kept_back, had the `dry_run` flag been false.
2017-08-30 23:58:00 +02:00
purge_unused_caches(
scripts: Rearrange the arguments of `purge_unused_caches()`. This commit re-arranges the arguments of `purge_unused_caches()` function in order to remain consistent with other similar functions in the library like `may_be_perform_caching()`.
2017-09-23 21:21:55 +02:00
EMOJI_CACHE_PATH, caches_in_use, "emoji cache", args)
clean_emoji_cache: Expand `clean-emoji-cache`. Expands `clean-emoji-cache` so that it can be used in production environment as well. Also moves it to `scripts/` from `tools/`.
2017-08-19 19:55:42 +02:00
if __name__ == "__main__":
clean-unused-caches: Optimize performance. This saves about 400ms when running clean-unused-caches, basically by calling its sub-rountines by import (rather than `subprocess.check_call()`). The performance optimization seems well worth it. Fixes #9766.
2018-06-18 16:28:34 +02:00
args = parse_cache_script_args("This script cleans unused zulip emoji caches.")
main(args)