Mirror
/
zulip
mirror of https://github.com/zulip/zulip.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
zulip/scripts/lib/node_cache.py

90 lines
3.5 KiB
Python
Raw Normal View History

Create node_cache module for npm caching.
2016-08-18 13:51:25 +02:00
import hashlib
node cache: Don't shell out to yarn to get the version. This can easily be done by just reading the package.json file and extracting the version value, just like we now do in install-node.
2018-12-07 23:43:52 +01:00
import json
python: Sort imports with isort. Fixes #2665. Regenerated by tabbott with `lint --fix` after a rebase and change in parameters. Note from tabbott: In a few cases, this converts technical debt in the form of unsorted imports into different technical debt in the form of our largest files having very long, ugly import sequences at the start. I expect this change will increase pressure for us to split those files, which isn't a bad thing. Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-11 00:54:34 +02:00
import os
node_cache: Avoid shelling out for rm, ln, mkdir, cp, cd, touch. Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-13 06:56:46 +02:00
import shutil
python: Sort imports with isort. Fixes #2665. Regenerated by tabbott with `lint --fix` after a rebase and change in parameters. Note from tabbott: In a few cases, this converts technical debt in the form of unsorted imports into different technical debt in the form of our largest files having very long, ugly import sequences at the start. I expect this change will increase pressure for us to split those files, which isn't a bad thing. Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-11 00:54:34 +02:00
from typing import List, Optional
Create node_cache module for npm caching.
2016-08-18 13:51:25 +02:00
python: Sort imports with isort. Fixes #2665. Regenerated by tabbott with `lint --fix` after a rebase and change in parameters. Note from tabbott: In a few cases, this converts technical debt in the form of unsorted imports into different technical debt in the form of our largest files having very long, ugly import sequences at the start. I expect this change will increase pressure for us to split those files, which isn't a bad thing. Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-11 00:54:34 +02:00
from scripts.lib.zulip_tools import run, subprocess_text_output
Create node_cache module for npm caching.
2016-08-18 13:51:25 +02:00
python: Add lint rule banning 'from os.path import'. This enforces our use of a consistent style in how we access Python modules; "from os.path import dirname" is a particularly popular abbreviation inconsistent with our style, and so it deserves a lint rule. Commit message and error text tweaked by tabbott. Fixes #6543.
2017-09-22 08:15:01 +02:00
ZULIP_PATH = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
deps: Change npm to yarn for reliablity, security, and speed.
2017-07-27 23:22:52 +02:00
ZULIP_SRV_PATH = "/srv"
Create node_cache module for npm caching.
2016-08-18 13:51:25 +02:00
deps: Change npm to yarn for reliablity, security, and speed.
2017-07-27 23:22:52 +02:00
NODE_MODULES_CACHE_PATH = os.path.join(ZULIP_SRV_PATH, 'zulip-npm-cache')
YARN_BIN = os.path.join(ZULIP_SRV_PATH, 'zulip-yarn/bin/yarn')
node cache: Don't shell out to yarn to get the version. This can easily be done by just reading the package.json file and extracting the version value, just like we now do in install-node.
2018-12-07 23:43:52 +01:00
YARN_PACKAGE_JSON = os.path.join(ZULIP_SRV_PATH, 'zulip-yarn/package.json')
deps: Change npm to yarn for reliablity, security, and speed.
2017-07-27 23:22:52 +02:00
Changes node_modules to default to the same args for generate_sha1sum. Since we can use both perfer_offline=True and False in a since build prefer_offline shouldn't be used as a cache key or it will confuse the cleanup script. Since yarn install (if successful) should be idempotent. This will probably be ok.
2017-07-28 00:18:23 +02:00
DEFAULT_PRODUCTION = False
python: Convert function type annotations to Python 3 style. Generated by com2ann (slightly patched to avoid also converting assignment type annotations, which require Python 3.6), followed by some manual whitespace adjustment, and six fixes for runtime issues: - def __init__(self, token: Token, parent: Optional[Node]) -> None: + def __init__(self, token: Token, parent: "Optional[Node]") -> None: -def main(options: argparse.Namespace) -> NoReturn: +def main(options: argparse.Namespace) -> "NoReturn": -def fetch_request(url: str, callback: Any, **kwargs: Any) -> Generator[Callable[..., Any], Any, None]: +def fetch_request(url: str, callback: Any, **kwargs: Any) -> "Generator[Callable[..., Any], Any, None]": -def assert_server_running(server: subprocess.Popen[bytes], log_file: Optional[str]) -> None: +def assert_server_running(server: "subprocess.Popen[bytes]", log_file: Optional[str]) -> None: -def server_is_up(server: subprocess.Popen[bytes], log_file: Optional[str]) -> bool: +def server_is_up(server: "subprocess.Popen[bytes]", log_file: Optional[str]) -> bool: - method_kwarg_pairs: List[FuncKwargPair], + method_kwarg_pairs: "List[FuncKwargPair]", Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-19 03:48:37 +02:00
def get_yarn_args(production: bool) -> List[str]:
Changes node_modules to default to the same args for generate_sha1sum. Since we can use both perfer_offline=True and False in a since build prefer_offline shouldn't be used as a cache key or it will confuse the cleanup script. Since yarn install (if successful) should be idempotent. This will probably be ok.
2017-07-28 00:18:23 +02:00
if production:
yarn_args = ["--prod"]
else:
yarn_args = []
return yarn_args
python: Convert function type annotations to Python 3 style. Generated by com2ann (slightly patched to avoid also converting assignment type annotations, which require Python 3.6), followed by some manual whitespace adjustment, and six fixes for runtime issues: - def __init__(self, token: Token, parent: Optional[Node]) -> None: + def __init__(self, token: Token, parent: "Optional[Node]") -> None: -def main(options: argparse.Namespace) -> NoReturn: +def main(options: argparse.Namespace) -> "NoReturn": -def fetch_request(url: str, callback: Any, **kwargs: Any) -> Generator[Callable[..., Any], Any, None]: +def fetch_request(url: str, callback: Any, **kwargs: Any) -> "Generator[Callable[..., Any], Any, None]": -def assert_server_running(server: subprocess.Popen[bytes], log_file: Optional[str]) -> None: +def assert_server_running(server: "subprocess.Popen[bytes]", log_file: Optional[str]) -> None: -def server_is_up(server: subprocess.Popen[bytes], log_file: Optional[str]) -> bool: +def server_is_up(server: "subprocess.Popen[bytes]", log_file: Optional[str]) -> bool: - method_kwarg_pairs: List[FuncKwargPair], + method_kwarg_pairs: "List[FuncKwargPair]", Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-19 03:48:37 +02:00
def generate_sha1sum_node_modules(
python: Use trailing commas consistently. Automatically generated by the following script, based on the output of lint with flake8-comma: import re import sys last_filename = None last_row = None lines = [] for msg in sys.stdin: m = re.match( r"\x1b\[35mflake8 \|\x1b\[0m \x1b\[1;31m(.+):(\d+):(\d+): (\w+)", msg ) if m: filename, row_str, col_str, err = m.groups() row, col = int(row_str), int(col_str) if filename == last_filename: assert last_row != row else: if last_filename is not None: with open(last_filename, "w") as f: f.writelines(lines) with open(filename) as f: lines = f.readlines() last_filename = filename last_row = row line = lines[row - 1] if err in ["C812", "C815"]: lines[row - 1] = line[: col - 1] + "," + line[col - 1 :] elif err in ["C819"]: assert line[col - 2] == "," lines[row - 1] = line[: col - 2] + line[col - 1 :].lstrip(" ") if last_filename is not None: with open(last_filename, "w") as f: f.writelines(lines) Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-10 05:23:40 +02:00
setup_dir: Optional[str] = None, production: bool = DEFAULT_PRODUCTION,
python: Convert function type annotations to Python 3 style. Generated by com2ann (slightly patched to avoid also converting assignment type annotations, which require Python 3.6), followed by some manual whitespace adjustment, and six fixes for runtime issues: - def __init__(self, token: Token, parent: Optional[Node]) -> None: + def __init__(self, token: Token, parent: "Optional[Node]") -> None: -def main(options: argparse.Namespace) -> NoReturn: +def main(options: argparse.Namespace) -> "NoReturn": -def fetch_request(url: str, callback: Any, **kwargs: Any) -> Generator[Callable[..., Any], Any, None]: +def fetch_request(url: str, callback: Any, **kwargs: Any) -> "Generator[Callable[..., Any], Any, None]": -def assert_server_running(server: subprocess.Popen[bytes], log_file: Optional[str]) -> None: +def assert_server_running(server: "subprocess.Popen[bytes]", log_file: Optional[str]) -> None: -def server_is_up(server: subprocess.Popen[bytes], log_file: Optional[str]) -> bool: +def server_is_up(server: "subprocess.Popen[bytes]", log_file: Optional[str]) -> bool: - method_kwarg_pairs: List[FuncKwargPair], + method_kwarg_pairs: "List[FuncKwargPair]", Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-19 03:48:37 +02:00
) -> str:
node_cache.py: Modify `generate_sha1sum_node_modules()`. Modify `generate_sha1sum_node_modules()` such that it can calculate the hash for a particular installation. Tweaked by tabbott to use os.path.realpath in the setup_dir calculation, to ensure it's consistent.
2017-08-19 11:06:45 +02:00
if setup_dir is None:
setup_dir = os.path.realpath(os.getcwd())
PACKAGE_JSON_FILE_PATH = os.path.join(setup_dir, 'package.json')
YARN_LOCK_FILE_PATH = os.path.join(setup_dir, 'yarn.lock')
Create node_cache module for npm caching.
2016-08-18 13:51:25 +02:00
sha1sum = hashlib.sha1()
node_cache.py: Modify `generate_sha1sum_node_modules()`. Modify `generate_sha1sum_node_modules()` such that it can calculate the hash for a particular installation. Tweaked by tabbott to use os.path.realpath in the setup_dir calculation, to ensure it's consistent.
2017-08-19 11:06:45 +02:00
sha1sum.update(subprocess_text_output(['cat', PACKAGE_JSON_FILE_PATH]).encode('utf8'))
caches: Don't depend on yarn.lock existing. This allows our cache cleaning code to run on servers that have copies of Zulip 1.6 and older around.
2017-09-26 01:52:27 +02:00
if os.path.exists(YARN_LOCK_FILE_PATH):
# For backwards compatibility, we can't assume yarn.lock exists
sha1sum.update(subprocess_text_output(['cat', YARN_LOCK_FILE_PATH]).encode('utf8'))
python: Modernize legacy Python 2 syntax with pyupgrade. Generated by `pyupgrade --py3-plus --keep-percent-format` on all our Python code except `zthumbor` and `zulip-ec2-configure-interfaces`, followed by manual indentation fixes. Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-09 21:51:58 +02:00
with open(YARN_PACKAGE_JSON) as f:
python: Convert json.loads(f.read()) to json.load(f). Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-03-20 02:12:02 +01:00
yarn_version = json.load(f)['version']
node cache: Don't shell out to yarn to get the version. This can easily be done by just reading the package.json file and extracting the version value, just like we now do in install-node.
2018-12-07 23:43:52 +01:00
sha1sum.update(yarn_version.encode("utf8"))
Create node_cache module for npm caching.
2016-08-18 13:51:25 +02:00
sha1sum.update(subprocess_text_output(['node', '--version']).encode('utf8'))
Changes node_modules to default to the same args for generate_sha1sum. Since we can use both perfer_offline=True and False in a since build prefer_offline shouldn't be used as a cache key or it will confuse the cleanup script. Since yarn install (if successful) should be idempotent. This will probably be ok.
2017-07-28 00:18:23 +02:00
yarn_args = get_yarn_args(production=production)
sha1sum.update(''.join(sorted(yarn_args)).encode('utf8'))
node_cache.py: Refactor to seperate out sha1sum generation. Basically we just seperate out the sha1sum generation for the node modules so that it can be reused later for cache clearance logic. This is achieved by adding a function which returns the sha1sum based HEX digest.
2017-06-17 20:05:41 +02:00
return sha1sum.hexdigest()
python: Convert function type annotations to Python 3 style. Generated by com2ann (slightly patched to avoid also converting assignment type annotations, which require Python 3.6), followed by some manual whitespace adjustment, and six fixes for runtime issues: - def __init__(self, token: Token, parent: Optional[Node]) -> None: + def __init__(self, token: Token, parent: "Optional[Node]") -> None: -def main(options: argparse.Namespace) -> NoReturn: +def main(options: argparse.Namespace) -> "NoReturn": -def fetch_request(url: str, callback: Any, **kwargs: Any) -> Generator[Callable[..., Any], Any, None]: +def fetch_request(url: str, callback: Any, **kwargs: Any) -> "Generator[Callable[..., Any], Any, None]": -def assert_server_running(server: subprocess.Popen[bytes], log_file: Optional[str]) -> None: +def assert_server_running(server: "subprocess.Popen[bytes]", log_file: Optional[str]) -> None: -def server_is_up(server: subprocess.Popen[bytes], log_file: Optional[str]) -> bool: +def server_is_up(server: "subprocess.Popen[bytes]", log_file: Optional[str]) -> bool: - method_kwarg_pairs: List[FuncKwargPair], + method_kwarg_pairs: "List[FuncKwargPair]", Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-19 03:48:37 +02:00
def setup_node_modules(
production: bool = DEFAULT_PRODUCTION,
prefer_offline: bool = False,
) -> None:
Changes node_modules to default to the same args for generate_sha1sum. Since we can use both perfer_offline=True and False in a since build prefer_offline shouldn't be used as a cache key or it will confuse the cleanup script. Since yarn install (if successful) should be idempotent. This will probably be ok.
2017-07-28 00:18:23 +02:00
yarn_args = get_yarn_args(production=production)
deps: Change npm to yarn for reliablity, security, and speed.
2017-07-27 23:22:52 +02:00
if prefer_offline:
yarn_args.append("--prefer-offline")
Changes node_modules to default to the same args for generate_sha1sum. Since we can use both perfer_offline=True and False in a since build prefer_offline shouldn't be used as a cache key or it will confuse the cleanup script. Since yarn install (if successful) should be idempotent. This will probably be ok.
2017-07-28 00:18:23 +02:00
sha1sum = generate_sha1sum_node_modules(production=production)
node_cache: Rename npm_cache to target_path. This is cleaner and part of renaming these variables to refer to the node_modules_cache.
2017-07-22 01:21:01 +02:00
target_path = os.path.join(NODE_MODULES_CACHE_PATH, sha1sum)
cached_node_modules = os.path.join(target_path, 'node_modules')
success_stamp = os.path.join(target_path, '.success-stamp')
Create node_cache module for npm caching.
2016-08-18 13:51:25 +02:00
# Check if a cached version already exists
if not os.path.exists(success_stamp):
deps: Change npm to yarn for reliablity, security, and speed.
2017-07-27 23:22:52 +02:00
do_yarn_install(target_path,
yarn_args,
update-prod-static: Stop hiding output in update-prod-static.log. Now that we've cleaned up this tool's output, there's no reason to use an awkward mechanism to hide its output; we can just print it out like a normal program. Fixes #14644; resolves #14701.
2020-04-24 22:24:11 +02:00
success_stamp)
Create node_cache module for npm caching.
2016-08-18 13:51:25 +02:00
python: Convert percent formatting to "".format in certain files. These files can’t use f-strings yet because they need to run in Python 2 or Python 3.5. Generated by pyupgrade. Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-14 04:36:06 +02:00
print("Using cached node modules from {}".format(cached_node_modules))
node_cache: Avoid shelling out for rm, ln, mkdir, cp, cd, touch. Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-13 06:56:46 +02:00
if os.path.islink('node_modules'):
os.remove('node_modules')
elif os.path.isdir('node_modules'):
shutil.rmtree('node_modules')
os.symlink(cached_node_modules, 'node_modules')
Create node_cache module for npm caching.
2016-08-18 13:51:25 +02:00
python: Convert function type annotations to Python 3 style. Generated by com2ann (slightly patched to avoid also converting assignment type annotations, which require Python 3.6), followed by some manual whitespace adjustment, and six fixes for runtime issues: - def __init__(self, token: Token, parent: Optional[Node]) -> None: + def __init__(self, token: Token, parent: "Optional[Node]") -> None: -def main(options: argparse.Namespace) -> NoReturn: +def main(options: argparse.Namespace) -> "NoReturn": -def fetch_request(url: str, callback: Any, **kwargs: Any) -> Generator[Callable[..., Any], Any, None]: +def fetch_request(url: str, callback: Any, **kwargs: Any) -> "Generator[Callable[..., Any], Any, None]": -def assert_server_running(server: subprocess.Popen[bytes], log_file: Optional[str]) -> None: +def assert_server_running(server: "subprocess.Popen[bytes]", log_file: Optional[str]) -> None: -def server_is_up(server: subprocess.Popen[bytes], log_file: Optional[str]) -> bool: +def server_is_up(server: "subprocess.Popen[bytes]", log_file: Optional[str]) -> bool: - method_kwarg_pairs: List[FuncKwargPair], + method_kwarg_pairs: "List[FuncKwargPair]", Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-19 03:48:37 +02:00
def do_yarn_install(
target_path: str,
yarn_args: List[str],
python: Use trailing commas consistently. Automatically generated by the following script, based on the output of lint with flake8-comma: import re import sys last_filename = None last_row = None lines = [] for msg in sys.stdin: m = re.match( r"\x1b\[35mflake8 \|\x1b\[0m \x1b\[1;31m(.+):(\d+):(\d+): (\w+)", msg ) if m: filename, row_str, col_str, err = m.groups() row, col = int(row_str), int(col_str) if filename == last_filename: assert last_row != row else: if last_filename is not None: with open(last_filename, "w") as f: f.writelines(lines) with open(filename) as f: lines = f.readlines() last_filename = filename last_row = row line = lines[row - 1] if err in ["C812", "C815"]: lines[row - 1] = line[: col - 1] + "," + line[col - 1 :] elif err in ["C819"]: assert line[col - 2] == "," lines[row - 1] = line[: col - 2] + line[col - 1 :].lstrip(" ") if last_filename is not None: with open(last_filename, "w") as f: f.writelines(lines) Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-10 05:23:40 +02:00
success_stamp: str,
python: Convert function type annotations to Python 3 style. Generated by com2ann (slightly patched to avoid also converting assignment type annotations, which require Python 3.6), followed by some manual whitespace adjustment, and six fixes for runtime issues: - def __init__(self, token: Token, parent: Optional[Node]) -> None: + def __init__(self, token: Token, parent: "Optional[Node]") -> None: -def main(options: argparse.Namespace) -> NoReturn: +def main(options: argparse.Namespace) -> "NoReturn": -def fetch_request(url: str, callback: Any, **kwargs: Any) -> Generator[Callable[..., Any], Any, None]: +def fetch_request(url: str, callback: Any, **kwargs: Any) -> "Generator[Callable[..., Any], Any, None]": -def assert_server_running(server: subprocess.Popen[bytes], log_file: Optional[str]) -> None: +def assert_server_running(server: "subprocess.Popen[bytes]", log_file: Optional[str]) -> None: -def server_is_up(server: subprocess.Popen[bytes], log_file: Optional[str]) -> bool: +def server_is_up(server: "subprocess.Popen[bytes]", log_file: Optional[str]) -> bool: - method_kwarg_pairs: List[FuncKwargPair], + method_kwarg_pairs: "List[FuncKwargPair]", Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-19 03:48:37 +02:00
) -> None:
node_cache: Avoid shelling out for rm, ln, mkdir, cp, cd, touch. Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-13 06:56:46 +02:00
os.makedirs(target_path, exist_ok=True)
shutil.copy('package.json', target_path)
shutil.copy("yarn.lock", target_path)
.yarnrc: Set ignore-scripts true. Follow up to #13065, to keep manual yarn invocations consistent with our automated ones. Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-28 00:32:07 +02:00
shutil.copy(".yarnrc", target_path)
node_cache: Extract cached_node_modules variable.
2017-07-22 00:53:58 +02:00
cached_node_modules = os.path.join(target_path, 'node_modules')
node_cache: Remove unused copy_modules parameter. Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-13 07:04:28 +02:00
print("Cached version not found! Installing node modules.")
deps: Change npm to yarn for reliablity, security, and speed.
2017-07-27 23:22:52 +02:00
node_cache: Remove unused copy_modules parameter. Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-13 07:04:28 +02:00
# Copy the existing node_modules to speed up install
node_cache: Don’t retry copying node_modules; let yarn do its thing. `copytree` throws an error if the target already exists, and we don’t really want to rerun the copy anyway. Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-10-29 00:49:23 +01:00
if os.path.exists("node_modules") and not os.path.exists(cached_node_modules):
node_cache: Preserve symlinks when copying an old node_modules tree. Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-10-06 21:53:15 +02:00
shutil.copytree("node_modules/", cached_node_modules, symlinks=True)
node_cache: Remove unused copy_modules parameter. Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-13 07:04:28 +02:00
if os.environ.get('CUSTOM_CA_CERTIFICATES'):
update-prod-static: Stop hiding output in update-prod-static.log. Now that we've cleaned up this tool's output, there's no reason to use an awkward mechanism to hide its output; we can just print it out like a normal program. Fixes #14644; resolves #14701.
2020-04-24 22:24:11 +02:00
run([YARN_BIN, "config", "set", "cafile", os.environ['CUSTOM_CA_CERTIFICATES']])
python: Replace list literal concatenation with * unpacking. Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-02 06:59:07 +02:00
run([YARN_BIN, "install", "--non-interactive", "--frozen-lockfile", *yarn_args],
update-prod-static: Stop hiding output in update-prod-static.log. Now that we've cleaned up this tool's output, there's no reason to use an awkward mechanism to hide its output; we can just print it out like a normal program. Fixes #14644; resolves #14701.
2020-04-24 22:24:11 +02:00
cwd=target_path)
node_cache: Avoid shelling out for rm, ln, mkdir, cp, cd, touch. Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-13 06:56:46 +02:00
with open(success_stamp, 'w'):
pass