Commit Graph

3453 Commits

Author SHA1 Message Date
Anders Kaseorg 096ef1445f parse_os_release: Use /etc/os-release always; remove DISTRIB_FAMILY.
To replace DISTRIB_FAMILY, there’s now an os_families function using
the standard ID and ID_LIKE information in /etc/os-release.

Fixes #13070; fixes #13071.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-29 17:30:20 -07:00
Anders Kaseorg d312d04510 styles: Replace Sass with PostCSS.
It’s about as fast as node-sass (faster, according to their
benchmarks) and more flexible.  Autoprefixer is neat: we can now go
delete all our -moz-, -webkit-, etc. lines and have them autogenerated
as necessary based on .browserslistrc.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-29 16:35:51 -07:00
Anders Kaseorg e0d122217b provision: Use pre-built pgroonga on Debian 10.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-29 16:35:29 -07:00
Tim Abbott 8ef610f8c8 provision: Remove now-duplicate extra Ubuntu section. 2019-08-29 12:53:10 -07:00
Anders Kaseorg db44d61aab setup-apt-repo: Remove PPA and packagecloud repository.
We no longer use tsearch_extras, and the camo patch is irrelevant on
systemd systems (Xenial and newer).  So we no longer need to
provide/install a PPA at all.

Closes #13027.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-29 12:53:04 -07:00
Anders Kaseorg 6701c4463c search: Remove now unnecessary tsearch_extra dependency.
Now that we're implemented tsearch_extras in pure postgres, we no
longer need a custom extension.  This should help us considerably, as
it means we no longer need to ship custom apt packages at all.

Fixes #467.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-29 12:49:26 -07:00
rht 07808e35be parse_lsb_release: Use /etc/os-release instead of /etc/lsb-release. 2019-08-28 17:53:27 -07:00
Tim Abbott 83a012eaeb lint: Improve over-aggressive pattern botserver->Botserver pattern.
This is still not a great pattern, but this is sufficient to avoid it
firing on URLs of the form zulipbotserver.example.com.
2019-08-28 16:18:16 -07:00
Vinit Singh d09a80260b lint: Replace local variables named 'msgid' with 'message_id'.
Follow up of commit 2a1305d. Replace all local variables named 'msgid'
with 'message_id' in all JS and HTML files, and adds a linter rule for
it as well.

Resolves #12952.
2019-08-28 15:19:30 -07:00
Anders Kaseorg 37ece9b8e2 webpack: Replace node-sass with Dart Sass.
It doesn't require scripts to install, allowing us to migrate yarn to
the more secure --ignore-scripts option.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-26 14:35:26 -07:00
Anders Kaseorg f1b91e577e requirements: Include packages that pip-tools considers unsafe.
It’s unclear why pip-tools considers these packages unsafe, and
excluding them from being pinned has resulted in nondeterministic
output that makes our test suite unhappy.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-25 22:32:31 -07:00
Anders Kaseorg a3b0af44bc run-mypy: Remove --quick option.
mypy no longer has a `--quick` option.  Its argument parser
autocompletes `--quick` to `--quickstart-file`, leading to a confusing
error message.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-25 15:04:12 -07:00
Anders Kaseorg 99904527f5 mypy: Remove daemon mode.
mypy in daemon mode takes some 400 MiB of memory, and cannot follow
imports of type-annotated third-party packages; meanwhile, non-daemon
mode is no longer nearly as slow as it once was.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-25 15:04:12 -07:00
Anders Kaseorg 9c49049c9e update-locked-requirements: Use pip-compile --no-header.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-25 15:03:20 -07:00
Anders Kaseorg 6822634d22 requirements: Remove editable flag from *.in.
Fixes #12374.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-25 15:03:20 -07:00
Anders Kaseorg fb9e9b76a8 requirements: When removing future requirement, leave it commented.
futures is no longer there to be removed.  Be clear about why we’re
removing future (it was never a “pip-tools bug”), and leave evidence
behind to help indicate how long that will be needed.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-25 15:03:20 -07:00
Anders Kaseorg bd7ef1be51 test-locked-requirements: Improve logged output.
Run pip-compile with --quiet so we don’t have to redirect its stderr;
then we can see any exceptions it might throw.  Print any resulting
diff in the right order and without extra newlines separating each
line.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-25 15:03:20 -07:00
Anders Kaseorg a053a925b3 requirements: Upgrade pip-tools to 3.8.0; downgrade pip to 19.1.1.
These are not the latest versions, but pip-tools 3.9.0 or 4.0.0 fails
to resolve dependencies from Git URLs:

pip._internal.exceptions.DistributionNotFound: No matching distribution found for zulip==0.6.1_git (from -r requirements/common.in (line 135))

while pip 19.2 breaks pip-tools 3.8.0:

TypeError: __init__() got an unexpected keyword argument 'find_links'

Fixes #10802.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-25 15:03:20 -07:00
Tim Abbott d0d16ed81c Revert "test_mattermost_importer: Fix missing coverage."
This reverts commit c48e53f587.

This new test code had nondeterministic failures; reverting so we can
debug without disruption.
2019-08-21 11:20:20 -07:00
Anders Kaseorg 349f22524e provision: Don’t rely on uid == gid assumption when invoking chown.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-17 12:59:09 -07:00
Anders Kaseorg 88793e80ea provision: Skip some rm, mkdir, chown commands if possible.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-17 12:59:09 -07:00
okmanl 2a1305de9f lint: Add a rule to avoid msgid as a Python variable name.
This is for consistency with our usual patterns, see #12995.  We will
need a similar commit for JavaScript to complete #12995.
2019-08-17 12:47:13 -07:00
Hemanth V. Alluri 243d8ffc51 openapi: Add a test to ensure generated curl examples work.
This new test runs each generated curl example against the Zulip API,
checking whether it returns successfully without errors.

Significantly modified by tabbott for simplicity.
2019-08-17 11:35:08 -07:00
Tim Abbott e8b4e80d45 test-api: Fix double use of django.setup.
django.setup is already called (with different/better environment
variables) inside test_server_running; we shouldn't be calling it just
before that to make imports work.

I discovered this because imports done at the wrong time would
potentially incorrectly have `testserver` as the EXTERNAL_HOST.
2019-08-17 11:35:08 -07:00
Priyank Patel 463ecb375f refactor: Add get_stream_by_narrow_operand_access_unchecked.
This let's us clean up the linter that excludes the use of get_stream
and by adding the access_unchecked in the name we make it clear that
it should be used with caution.

Refactoring idea by Tim Abbott.
2019-08-17 11:10:00 -07:00
Anders Kaseorg fc6a8396ed mypy: In non-daemon mode, follow package imports.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-16 14:13:40 -07:00
Anders Kaseorg 6bca996e2e webpack: Chill. Out. When watching for changes.
Polling for changes every 100 milliseconds was burning enough CPU to
set mid-2015 MacBooks on fire.  Use the default inotify watching,
except on filesystems where that’s known not to work (nfs, vboxsf), in
which case polling once per second is more than enough for even the
fastest typers.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-16 11:05:21 -07:00
Anders Kaseorg ead13c11e5 run-dev: Don’t rewrite the Content-Length header.
A HEAD response has a Content-Length but no body; it’s not correct in
that case to let Tornado default Content-Length to 0.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-12 16:47:41 -07:00
Anders Kaseorg b0859f4b1e linter_lib: Fix mypy errors.
tools/linter_lib/pyflakes.py:35: error: Argument 3 to "run_pyflakes" has incompatible type "List[Tuple[bytes, bytes]]"; expected "List[Tuple[str, str]]"
    tools/linter_lib/custom_check.py:110: error: Argument "rules" to "RuleList" has incompatible type "List[Dict[str, Any]]"; expected "List[Rule]"
    tools/linter_lib/custom_check.py:214: error: Argument "rules" to "RuleList" has incompatible type "List[Dict[str, Any]]"; expected "List[Rule]"
    tools/linter_lib/custom_check.py:214: error: Argument "shebang_rules" to "RuleList" has incompatible type "List[Dict[str, Any]]"; expected "List[Rule]"
    tools/linter_lib/custom_check.py:502: error: Argument "rules" to "RuleList" has incompatible type "List[Dict[str, Any]]"; expected "List[Rule]"
    tools/linter_lib/custom_check.py:502: error: Argument "shebang_rules" to "RuleList" has incompatible type "List[Dict[str, Any]]"; expected "List[Rule]"
    tools/linter_lib/custom_check.py:519: error: Argument "rules" to "RuleList" has incompatible type "List[Dict[str, Any]]"; expected "List[Rule]"
    tools/linter_lib/custom_check.py:706: error: Argument "rules" to "RuleList" has incompatible type "List[Dict[str, Any]]"; expected "List[Rule]"
    tools/linter_lib/custom_check.py:728: error: Argument "rules" to "RuleList" has incompatible type "List[Dict[str, Any]]"; expected "List[Rule]"
    tools/linter_lib/custom_check.py:738: error: Argument "rules" to "RuleList" has incompatible type "List[Dict[str, Any]]"; expected "List[Rule]"
    tools/linter_lib/custom_check.py:779: error: Argument "rules" to "RuleList" has incompatible type "List[Dict[str, Any]]"; expected "List[Rule]"
    tools/linter_lib/custom_check.py:779: error: Argument "length_exclude" to "RuleList" has incompatible type "Set[str]"; expected "List[str]"
    tools/linter_lib/custom_check.py:803: error: Argument "length_exclude" to "RuleList" has incompatible type "Set[str]"; expected "List[str]"
    tools/linter_lib/custom_check.py:805: error: Unsupported operand types for + ("List[Rule]" and "List[Dict[str, Any]]")
    tools/linter_lib/custom_check.py:819: error: Argument "rules" to "RuleList" has incompatible type "List[Dict[str, Any]]"; expected "List[Rule]"

These were missed the `zulint` package was missing PEP 561 type
annotation markers, and if it’d had them, mypy daemon mode would’ve
required us to set `follow_imports = skip` for it.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 17:22:45 -07:00
Anders Kaseorg 17acf95437 update-authors-json: Clean up type ignores.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 16:39:16 -07:00
Anders Kaseorg cd1306c8d9 test-install: Add bionic.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 16:27:03 -07:00
Anders Kaseorg 671844d667 provision: Add default-jre-headless for vnu-jar.
This was previously pulled in by yui-compressor.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-08 17:28:20 -07:00
Tim Abbott de0a41bc9c provision: Fix missing dependency on unzip.
Because this is often installed by default, we hadn't noticed that our
Slack importer doesn't run without it.

Thanks to Ray Kraesig for the report.
2019-08-08 10:49:20 -07:00
Tim Abbott d9e5becb0e tools: Fix running check-openapi locally. 2019-08-07 14:18:27 -07:00
Anders Kaseorg 0962393933 cleanup: Delete trailing newlines.
Delete trailing newlines from all files, except
tools/ci/success-http-headers.txt and tools/setup/dev-motd, where they
are significant, and static/third, where we want to stay close to
upstream.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-06 23:29:11 -07:00
Anders Kaseorg becef760bf cleanup: Delete leading newlines.
Previous cleanups (mostly the removals of Python __future__ imports)
were done in a way that introduced leading newlines.  Delete leading
newlines from all files, except static/assets/zulip-emoji/NOTICE,
which is a verbatim copy of the Apache 2.0 license.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-06 23:29:11 -07:00
Wyatt Hoodes 8c26183c65 mypy: Assign warn_unreachable in mypy.ini.
Instead of doing this rather clumsily in `run_mypy`,
we configure the option in `mypy.ini`.
2019-08-06 12:43:29 -07:00
Hemanth V. Alluri f280e9cf84 lib: Rename lib/api_test_helpers.py to openapi/python_examples.py
This will make the contained code easier to find.
2019-08-05 21:06:19 -07:00
Hemanth V. Alluri c90056bdb2 tools: Move check-swagger to check-openapi and make it executable. 2019-08-05 21:06:19 -07:00
Anders Kaseorg 68dd8e4ec8 mypy: Migrate from mypy_extensions to typing_extensions.
This gives us access to typing_extensions.Deque, which was not added
to typing until 3.5.4.

(PROVISION_VERSION is not bumped because the transitive dependency set
in dev.txt hasn’t changed.)

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-05 17:24:09 -07:00
Wyatt Hoodes e6ae160a0c mypy: Run --warn-unreachable by default. 2019-07-31 16:18:24 -07:00
Wyatt Hoodes f80402c64d tools: Use mypy_extensions for NoReturn.
This is the right way to access it with support for older Python
versions where it might not be in typing yet.
2019-07-31 16:18:24 -07:00
Wyatt Hoodes 4beec5c6b9 typing: Use TYPE_CHECKING when dealing with cyclic dependencies. 2019-07-31 12:19:39 -07:00
Tim Abbott 97b256d1f0 css: Extract rendered_markdown.scss.
This moves our main CSS for rendered Zulip message content into an
external file, which may be reusable but in any case should make it
easier to find this content.
2019-07-31 12:08:17 -07:00
Wyatt Hoodes a109508e34 typing: Remove now-unnecessary conditional import.
As a result of dropping support for trusty, we can remove our old
pattern of putting `if False` before importing the typing module,
which was essential for Python 3.4 support, but not required and maybe
harmful on newer versions.

cron_file_helper
check_rabbitmq_consumers
hash_reqs
check_zephyr_mirror
check_personal_zephyr_mirrors
check_cron_file
zulip_tools
check_postgres_replication_lag
api_test_helpers
purge-old-deployments
setup_venv
node_cache
clean_venv_cache
clean_node_cache
clean_emoji_cache
pg_backup_and_purge
restore-backup
generate_secrets
zulip-ec2-configure-interfaces
diagnose
check_user_zephyr_mirror_liveness
2019-07-29 15:18:22 -07:00
Aman 2183a74040 zulint: Use zulint from the extracted repository.
zulint will be added as a "third-party" dependency in zulip from now
on.  See the new project at https://github.com/zulip/zulint for more
details.
2019-07-26 11:35:43 -07:00
Wyatt Hoodes a920f0396d webpack: Comment why NoReturn can't import normally. 2019-07-25 17:48:33 -07:00
Wyatt Hoodes 927feb615e provision: Comment why NoReturn can't import normally. 2019-07-25 17:48:20 -07:00
Wyatt Hoodes 3060bb2208 requirements: Upgrade to mypy 0.720.
We also add the option of the newly added `--warn_unreachable`
flag.
2019-07-25 17:41:10 -07:00
Anders Kaseorg a32663dc85 lint: Do not forbid ES6 const.
Our pipeline was upgraded in commit
f54a63e2f9 (#12838).

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-25 16:12:48 -07:00