Commit Graph

40 Commits

Author SHA1 Message Date
arpit551 7f769512aa travis: Remove Travis unwanted code.
Since in travis we don't have root access so we used to add different
srv path. As now we shifted our production suites to Circle CI
we don't need that code so removed it.

Also we used a hacky code in commit-lint-message for travis which is
now of no use.
2020-04-28 11:11:23 -07:00
Tim Abbott 7e0eeb20a3 update-prod-static: Stop hiding output in update-prod-static.log.
Now that we've cleaned up this tool's output, there's no reason to use
an awkward mechanism to hide its output; we can just print it out like
a normal program.

Fixes #14644; resolves #14701.
2020-04-25 15:39:03 -07:00
Anders Kaseorg 5901e7ba7e python: Convert function type annotations to Python 3 style.
Generated by com2ann (slightly patched to avoid also converting
assignment type annotations, which require Python 3.6), followed by
some manual whitespace adjustment, and six fixes for runtime issues:

-    def __init__(self, token: Token, parent: Optional[Node]) -> None:
+    def __init__(self, token: Token, parent: "Optional[Node]") -> None:

-def main(options: argparse.Namespace) -> NoReturn:
+def main(options: argparse.Namespace) -> "NoReturn":

-def fetch_request(url: str, callback: Any, **kwargs: Any) -> Generator[Callable[..., Any], Any, None]:
+def fetch_request(url: str, callback: Any, **kwargs: Any) -> "Generator[Callable[..., Any], Any, None]":

-def assert_server_running(server: subprocess.Popen[bytes], log_file: Optional[str]) -> None:
+def assert_server_running(server: "subprocess.Popen[bytes]", log_file: Optional[str]) -> None:

-def server_is_up(server: subprocess.Popen[bytes], log_file: Optional[str]) -> bool:
+def server_is_up(server: "subprocess.Popen[bytes]", log_file: Optional[str]) -> bool:

-    method_kwarg_pairs: List[FuncKwargPair],
+    method_kwarg_pairs: "List[FuncKwargPair]",

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-18 20:42:48 -07:00
Anders Kaseorg c734bbd95d python: Modernize legacy Python 2 syntax with pyupgrade.
Generated by `pyupgrade --py3-plus --keep-percent-format` on all our
Python code except `zthumbor` and `zulip-ec2-configure-interfaces`,
followed by manual indentation fixes.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-09 16:43:22 -07:00
Anders Kaseorg 39f9abeb3f python: Convert json.loads(f.read()) to json.load(f).
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-03-24 10:46:32 -07:00
Anders Kaseorg d6377b00c0 node_cache: Don’t retry copying node_modules; let yarn do its thing.
`copytree` throws an error if the target already exists, and we don’t
really want to rerun the copy anyway.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-10-29 12:30:28 -07:00
Anders Kaseorg 9182293d50 node_cache: Preserve symlinks when copying an old node_modules tree.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-10-06 15:19:53 -07:00
Anders Kaseorg 9e481e353a .yarnrc: Set ignore-scripts true.
Follow up to #13065, to keep manual yarn invocations consistent with
our automated ones.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-28 16:15:54 -07:00
Anders Kaseorg 43c8c720ef node_cache: Run yarn with --ignore-scripts.
As predicted in https://www.kb.cert.org/vuls/id/319816/, a malicious
worm is beginning to spread across the npm ecosystem through package
postinstall scripts.  Only instead of direct self-replicating code,
the replication vector is the temptation to monetize postinstall
scripts by polluting the console logs with paid advertisements.  The
effect will be the same unless we all put a stop to this while we
still can.

Apply the recommended VU#319816 workaround, which is to disable
lifecycle scripts when installing npm packages.  The only fallout is:

* node-sass can’t run because it uses compiled native code; we replace
  it with Dart Sass.
* phantomjs-prebuilt doesn’t download the binary at install time; we
  tell it to download it in run-casper.
* ttf2woff2 transparently falls back from native code to an Emscripten
  build.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-26 14:36:03 -07:00
Anders Kaseorg fa913b7f5d node_cache: Avoid shelling out for rm, ln, mkdir, cp, cd, touch.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-17 12:59:09 -07:00
Anders Kaseorg d761bbc2ed node_cache: Remove unused copy_modules parameter.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-14 14:39:04 -07:00
Anders Kaseorg becef760bf cleanup: Delete leading newlines.
Previous cleanups (mostly the removals of Python __future__ imports)
were done in a way that introduced leading newlines.  Delete leading
newlines from all files, except static/assets/zulip-emoji/NOTICE,
which is a verbatim copy of the Apache 2.0 license.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-06 23:29:11 -07:00
Wyatt Hoodes a109508e34 typing: Remove now-unnecessary conditional import.
As a result of dropping support for trusty, we can remove our old
pattern of putting `if False` before importing the typing module,
which was essential for Python 3.4 support, but not required and maybe
harmful on newer versions.

cron_file_helper
check_rabbitmq_consumers
hash_reqs
check_zephyr_mirror
check_personal_zephyr_mirrors
check_cron_file
zulip_tools
check_postgres_replication_lag
api_test_helpers
purge-old-deployments
setup_venv
node_cache
clean_venv_cache
clean_node_cache
clean_emoji_cache
pg_backup_and_purge
restore-backup
generate_secrets
zulip-ec2-configure-interfaces
diagnose
check_user_zephyr_mirror_liveness
2019-07-29 15:18:22 -07:00
Anders Kaseorg d267884cc1 node_cache: Run yarn install with --frozen-lockfile.
This way a yarn.lock inconsistent with package.json can’t slip through
unnoticed.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-05-20 19:23:37 -07:00
Anders Kaseorg e984107966 scripts: Remove unused imports.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-02-02 17:02:58 -08:00
Tim Abbott 2558f101af docs: Add documentation for `if False` mypy pattern in scripts.
This should help make it clear what's going on with these scripts.
2018-12-17 11:12:53 -08:00
Tim Abbott 9423f8e31c node cache: Don't shell out to yarn to get the version.
This can easily be done by just reading the package.json file and
extracting the version value, just like we now do in install-node.
2018-12-07 14:44:52 -08:00
xificurC 9e053c74cf scripts: Allow configuring a custom CA bundle for build process.
For building Zulip in an environment where a custom CA certificate is
required to access the public Internet, one needs to be able to
specify that CA certificate for all network access done by the Zulip
installer/build process.  This change allows configuring that via the
environment.
2018-08-13 15:45:28 -07:00
Aditya Bansal e14974ff2c scripts: Change use of typing.Text to str. 2018-05-10 14:19:49 -07:00
neiljp (Neil Pilgrim) 221be326a9 mypy: Set type parameters for IO to Any in scripts/lib/node_cache.py. 2017-11-04 19:47:44 -07:00
rht 71188d7b0a scripts: Remove import print_function. 2017-09-29 15:43:30 -07:00
Tim Abbott db982672b1 caches: Don't depend on yarn.lock existing.
This allows our cache cleaning code to run on servers that have copies
of Zulip 1.6 and older around.
2017-09-25 17:05:45 -07:00
julianasobreira df2d448f7b python: Add lint rule banning 'from os.path import'.
This enforces our use of a consistent style in how we access Python
modules; "from os.path import dirname" is a particularly popular
abbreviation inconsistent with our style, and so it deserves a lint
rule.

Commit message and error text tweaked by tabbott.

Fixes #6543.
2017-09-22 04:55:38 -07:00
Harshit Bansal 8e41bbe2b0 node_cache.py: Modify `generate_sha1sum_node_modules()`.
Modify `generate_sha1sum_node_modules()` such that it can calculate
the hash for a particular installation.

Tweaked by tabbott to use os.path.realpath in the setup_dir
calculation, to ensure it's consistent.
2017-08-27 17:51:24 -07:00
Pweaver (Paul Weaver) 48815204e4 Changes node_modules to default to the same args for generate_sha1sum.
Since we can use both perfer_offline=True and False in a since build
prefer_offline shouldn't be used as a cache key or it will confuse the
cleanup script. Since yarn install (if successful) should be idempotent.
This will probably be ok.
2017-08-05 12:29:16 -07:00
Pweaver (Paul Weaver) 1afaa67c7c deps: Change npm to yarn for reliablity, security, and speed. 2017-08-05 12:29:06 -07:00
Tim Abbott b7211733e2 node_cache: Rename npm_cache to target_path.
This is cleaner and part of renaming these variables to refer to the
node_modules_cache.
2017-07-21 17:28:20 -07:00
Tim Abbott e1e5e15797 node_cache: Move npm_args inside the interface. 2017-07-21 17:28:20 -07:00
Tim Abbott b1944b5e1f node_cache: Fix buggy type annotations for copy_modules. 2017-07-21 17:28:20 -07:00
Tim Abbott 8523c24a17 node_modules: Move success stamp to root of cache dir.
This location makes more sense, since it's the entire directory we're
certifying, not just the `node_modules` subdirectory.
2017-07-21 17:28:20 -07:00
Tim Abbott 59abefa616 node_cache: Extract cached_node_modules variable. 2017-07-21 17:28:20 -07:00
Tim Abbott 5d05cc7294 scripts: Rename NPM_CACHE_PATH to NODE_MODULES_CACHE_PATH. 2017-07-21 17:28:20 -07:00
neiljp (Neil Pilgrim) 8433d95d99 mypy: Make success_stamp parameter to do_npm_install be required. 2017-07-08 10:49:42 -07:00
Aditya Bansal e4af0caddb node_cache.py: Refactor to seperate out sha1sum generation.
Basically we just seperate out the sha1sum generation for the
node modules so that it can be reused later for cache clearance
logic. This is achieved by adding a function which returns the
sha1sum based HEX digest.
2017-06-19 07:55:23 -04:00
Tim Abbott 95a49f6ea2 node_cache: Fix buggy production deployment code.
The previous version of the production deployment code for release
tarballs did not correctly install the node_modules directory.
2016-12-08 10:34:04 -08:00
Tim Abbott 8e82257444 Fix node_cache code to not require root.
The previous code caused problems using a system where the zulip user
doesn't have sudo rights.
2016-10-25 17:52:19 -07:00
Tim Abbott 41b0079ab2 node_cache: Fix success_stamp logic.
Previously, success_stamp was touched whenever we used a particular
node_modules version; it makes more sense to only touch it when the
node_modules directory has actually changed.
2016-10-25 17:52:17 -07:00
Tim Abbott 7c16172f9e node_cache: Fix buggy annotations for stdout/stderr. 2016-10-15 22:53:28 -07:00
Tim Abbott 9c0c6c0c23 node_cache: Don't make node_modules symlinks as root. 2016-09-28 00:36:40 -07:00
Umair Khan ec74974de3 Create node_cache module for npm caching. 2016-09-16 12:40:48 -07:00