Commit Graph

4911 Commits

Author SHA1 Message Date
Zixuan James Li f88530575a test-backend: Cast test runner to the custom Runner type.
Mypy does not know the acccurate return type of `get_runner` that is
determined by the `TEST_RUNNER` setting. We need to cast it to the
correct type to use methods like `get_shallow_tested_templates`. Note
that we import conditionally to avoid adding runtime dependency on
`zerver`.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-28 09:48:16 -07:00
Anders Kaseorg a2f43beef4 commit-message-lint: Take release branches into account.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-07-22 10:39:05 -07:00
Matt Keller 44e93613fb tools: Add executable bit to release tool. 2022-07-21 19:08:10 -07:00
evykassirer afab7002bf emoji: Add script to generate emoji_names.py, only with custom names.
This script pulls from our previously custom-written emoji strings
to prepare to fill in the rest from CLDR.

This commit has no user-facing changes.
2022-07-21 13:22:07 -07:00
evykassirer 190c0737a7 emoji: Store customized emoji in file separate from emoji_names.py.
Emoji that we specifically choose names or aliases for will be
stored in this new file so that we can generate emoji_names.py
from the custom names and also CLDR (for the rest).

This new file isn't used for anything yet (no user facing change);
it will be used in future commits.
2022-07-21 13:22:07 -07:00
Anders Kaseorg 412e90f601 provision: Avoid distutils; keep PROVISION_VERSION as a tuple.
distutils is deprecated in Python 3.10 and will be removed in Python
3.12.  We don’t need a full-powered version parser for this anyway.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-07-20 22:03:30 -07:00
Alex Vandiver 58962f0b7b release: Use the right variable to build predictable tarball path.
`tools/build-release-tarball` respects `OUTPUT_DIR`, not `OUTDIR`.
2022-07-19 17:29:03 -07:00
Alex Vandiver 39efe6f312 ci: Remove references to CircleCI, which we no longer use. 2022-07-16 10:43:40 -07:00
Alex Vandiver f8e2d652e1 ci: Test upgrades from the minimum of each major version, not the max. 2022-07-16 10:43:40 -07:00
Zixuan James Li 3404282927 tail-ses: Add IdentityArgsDict.
This fixes having incompatible types of different items in
the dictionary, and makes `**identity_args` type-safe.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-15 13:49:11 -07:00
Anders Kaseorg c520890f54 node_tests: Remove low-hanging uses of __Rewire__.
When we were preparing the conversion to ES modules in 2019, the
primary obstacle was that the Node tests extensively relied on the
ability to reach into modules and mutate their CommonJS exports in
order to mock things.  ES module bindings are not mutable, so in
commit 173c9cee42 we added
babel-plugin-rewire-ts as a kludgy transpilation-based workaround for
this to unblock the conversion.

However, babel-plugin-rewire-ts is slow, buggy, nonstandard,
confusing, and unmaintained.  It’s incompatible with running our ES
modules as native ES modules, and prevents us from taking advantage of
modern tools for ES modules.  So we want to excise all use of
__Rewire__ (and the disallow_rewire, override_rewire helper functions
that rely on it) from the tests and remove babel-plugin-rewire-ts.

Commits 64abdc199e and
e17ba5260a (#20730) prepared for this by
letting us see where __Rewire__ is being used.  Now we go through and
remove most of the uses that are easy to remove without modifying the
production code at all.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-07-13 16:27:30 -07:00
Anders Kaseorg 81892df176 requirements: Upgrade to Django 4.0.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-07-13 16:07:17 -07:00
Ganesh Pawar c6d636ef48 realm_domains_modal: Migrate modal to dialog_widget. 2022-07-07 14:56:45 -07:00
Alex Vandiver c915400101 release: Write a tool to automate final release checks and steps. 2022-07-07 13:19:00 -07:00
Alex Vandiver d9958618a9 build-release-tarball: Use OUTPUT_DIR environment variable, if set. 2022-07-07 13:19:00 -07:00
Alex Vandiver 2d132bf342 build-release-tarball: Verify full releases are tagged and pushed.
This ensures that anything that looks like a full release will
identify itself as such.  It requires that the version have a tag at
the current commit (though it doesn't need to be pushed yet), as well
as the commits exist in `main` or a release branch in the remote.
2022-07-07 13:19:00 -07:00
Zixuan James Li b3f27b1843 mypy: Enable local_partial_types.
dmypy is innately incompatible with `local_partial_types` being
`False` as it uses fined-grained incremental mode for caching.

We need to enable `local_partial_types` for mypy as well so that the two
will behave the same way.

This requires us to add additional type annotation in certain siuations
but that's fine.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-06 17:33:13 -07:00
Zixuan James Li bf9f9c8b5d tools: Support running mypy daemon for better performance.
mypy daemon performs significantly better than running the regular
mypy cli tool when we type check the entire codebase multiple
times locally.

This adds running mypy daemon as an option for both
`tools/run-mypy` and `tools/lint`.

To ensure daemon messages like "Daemon started", "Daemon stopped"
won't get printed we filter any output that starts with "Daemon".

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-06 17:33:13 -07:00
Zixuan James Li ea3478e377 tools: Extract ROOT_DIR.
Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-07-06 17:33:13 -07:00
Anders Kaseorg e4e51b0eb4 install-shfmt: Upgrade shfmt from 3.5.0 to 3.5.1.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-07-06 17:23:16 -07:00
Anders Kaseorg feff1d0411 requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-07-05 17:54:17 -07:00
Alex Vandiver aabbb2ad10 bootstrap-awscli: Leave CWD unchanged. 2022-06-28 09:39:31 -04:00
Alex Vandiver 1b5e058d3b install-aws-server: Add a --debug-key to allow logins during bootstrap. 2022-06-28 09:39:31 -04:00
Alex Vandiver 1a1061e77e install-aws-server: Switch to getopt. 2022-06-28 09:39:31 -04:00
Alex Vandiver 1be9ab2690 install-aws-server: Assume zulip_ops::profile:: prefix on all roles.
This will require that any profile-specific sections of
`$HOME/.zulip-install-server.conf` be renamed to their short form.
2022-06-28 09:39:31 -04:00
Alex Vandiver 9a3be2af64 install-aws-server: Fix default_branch config name. 2022-06-28 09:39:31 -04:00
Alex Vandiver f5d4dea2f0 bootstrap-aws-installer: Leave `ubuntu` user deletion until the last.
This makes it easier to log in and figure out what went wrong if
something failed during the bootstrapping process.
2022-06-28 09:39:31 -04:00
Anders Kaseorg ef3510fa6d nginx: Remove legacy X-XSS-Protection header.
Support for this header was removed in Chrome 78, Safari 15.4, and
Edge 17.  It was never supported in Firefox.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-06-27 17:38:18 -07:00
Anders Kaseorg 25f6b53e90 docs: Capitalize Handlebars consistently.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-06-27 13:35:10 -07:00
Anders Kaseorg d5c302feef run-dev: Remove unnecessary finish calls.
Tornado finishes the request automatically.  Avoids this error, hidden
until commit 81f7192ca3 (#22301), when
the browser tab is closed:

Traceback (most recent call last):
  File "/srv/zulip-py3-venv/lib/python3.8/site-packages/tornado/web.py", line 1683, in _execute
    result = await result
  File "tools/run-dev.py", line 280, in prepare
    await self.finish()
tornado.iostream.StreamClosedError: Stream is closed

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-06-26 17:38:43 -07:00
Anders Kaseorg 0dfde9db94 python: Simplify unnecessary getattr calls.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-06-26 17:38:22 -07:00
Anders Kaseorg 3bf8ee2156 python: Unquote some unnecessarily quoted type annotations.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-06-26 17:37:41 -07:00
Anders Kaseorg e069330084 check-thirdparty: License under GPLv2+.
Since python-debian is GPL licensed, our script that imports it should
arguably be GPL licensed as well.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-06-26 17:33:11 -07:00
Alex Vandiver 6cd4d907ba upload-release: Only copy full releases to zulip-server-latest.tar.gz.
`zulip-server-6.0-pre0.tar.gz` should not be copied to
`zulip-server-latest.tar.gz`.
2022-06-23 19:24:54 -07:00
Anders Kaseorg 81f7192ca3 tornado: Add missing await for finish calls.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-06-23 19:22:12 -07:00
Anders Kaseorg c944adfcc6 mypy: Enable redundant-expr errors.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-06-23 19:22:12 -07:00
Alex Vandiver 41deef40cf nagios: Switch to generic check_cron_file for queues and consumers.
These share a common root; 91da4bd59b duplicated the code, but
didn't move the existing uses to the new utility.
2022-06-22 12:07:38 -07:00
Alex Vandiver 91379fd67e ci: Update upgrade test to 5.3, from 5.2. 2022-06-21 17:40:33 -07:00
Alex Vandiver a35af3f38b install/upgrade: Allow new packages during `apt-get upgrade`.
`postgresql-14.4` is a notable upgrade in the PostgreSQL series, as it
fixes potential database corruption from `CREATE INDEX CONCURRENTLY`
statements which are run while rows are modified[1].  However, it also
requires an upgrade from `libllvm9` to `libllvm10`, which means it is
not installed by a mere `apt-get upgrade`.

Add the `--with-new-pkgs` flag to all of the potentially relevant
`apt-get upgrade` calls, so that this (and similar) packages are
upgraded successfully.

[1]: https://www.postgresql.org/docs/release/14.4/
2022-06-21 11:21:49 -07:00
Anders Kaseorg a7f9c4f958 logging: Pass more format arguments to logging.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-06-03 12:27:23 -07:00
Anders Kaseorg 79842fd228 install-shfmt: Upgrade shfmt from 3.4.3 to 3.5.0.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-06-02 12:03:49 -07:00
Adam Sah ba5cf331a2 testing: 100% coverage for zerver/tests/test_digest.py. 2022-06-01 16:09:13 -07:00
Anders Kaseorg ecb900bd03 requirements: Update zulint with multiline pattern support.
https://github.com/zulip/zulint/pull/28

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-06-01 14:34:58 -07:00
Zixuan James Li 63e9ae8389 typing: Apply trivial fixes to adjust edge cases in typing.
Add none-checks, rename variables (to avoid redefinition of
the same variable with different types error), add necessary
type annotations.

This is a part of #18777.

Signed-off-by: Zixuan James Li <359101898@qq.com>
2022-05-30 12:03:51 -07:00
Alex Vandiver fb338f22d7 ci: Stop rabbitmq when making CI images.
The production CI image starts `rabbitmq-server` but does not stop it,
which leaves a stale `/var/run/rabbitmq/pid` file in the image.

`rabbitmqctl wait --timeout 600 /var/run/rabbitmq/pid`, which is run
after starting the rabbitmq node, reads the PID file and waits for the
PID to be running, and for rabbitmq's port to be responding to pings.

If it reads an old PID file before the new PID is written, it
aborts (all but the first and last lines are output from `rabbitmqctl
wait` that is hidden by `/etc/init.d/rabbitmq-server`):

```
 * Starting RabbitMQ Messaging Server rabbitmq-server
Waiting for pid file '/var/run/rabbitmq/pid' to appear
pid is 341
Waiting for erlang distribution on node 'rabbit@fc8f64d6acdb' while OS process '341' is running
Error:
process_not_running
 * FAILED - check /var/log/rabbitmq/startup_\{log, _err\}
 ```

If it failed, the `production-upgrade` script tried to start
`rabbitmq` again -- despite it already still starting in the
background.  These two attempts conflicted, and often one or both
failed.

Stop `rabbitmq-server` when building the image, which removes the
stale PID file.
2022-05-19 13:30:59 -07:00
Adam Sah 04ccd8c6d8 testing: 100% code coverage for zerver/lib/outgoing_webhook.py. 2022-05-17 13:51:00 -07:00
Alex Vandiver c93024cd5b oneclick: Fail if the fab command fails. 2022-05-17 13:41:12 -07:00
Alex Vandiver ff647dff03 oneclick: Do not use a stale Zulip client.
Initializing the Zulip client opens a long-lived TCP connection due to
connection pooling in urllib3.  In Github Actions, the network kills
such requests after ~270s, making the later `send_message` call fail.

Use a singular call to `zulip.Client()` early on to verify the
credentials, and do not cache the resulting client object.  Instead,
re-create it during the final step when it is needed, so we do not run
afoul of bad TCP connection state.

This would ideally be fixed via connection keepalive or retry at the
level of the Zulip module.
2022-05-17 13:41:12 -07:00
Adam Sah 534754442a test-backend: Improve performance by disabling XML report.
We don't use the XML report ourselves. We add options to make this
easy to control if specific circumstances indicate doing so.
2022-05-16 16:38:02 -07:00
Adam Sah 492272d597 test-backend: Display test function in HTML coverage reports. 2022-05-16 16:38:02 -07:00
Alex Vandiver 9e42cf435b tail-ses: Add a tool to tail SES delivery/bounce/complaint queues. 2022-05-06 17:49:00 -07:00
Ganesh Pawar 59e810790f user_groups: Convert inline form to modal.
Fixes part of #21298.
2022-05-04 17:46:30 -07:00
Raghav Luthra fb89c14f34 lint: Improve data-tippy-allowHTML error message.
The error message a user gets from the linter when using the
data-tippy-allowHTML attribute now conveys the fact that the
<template> tag is supposed to hold the tooltip content. This
might make understanding the correct workflow easier for
someone who encounters this error.
2022-05-04 17:38:30 -07:00
Adam Sah a9ce02348b testing: Improve backend code coverage for semaphore webhook. 2022-05-04 16:13:24 -07:00
Alex Vandiver bf562f8fff ci: Update upgrade test to 5.2, from 5.1. 2022-05-04 11:37:15 -07:00
Tim Abbott 25999b7056 lint: Ban Help Center relative links without leading /.
Such links would likely end up broken if we change our URL scheme to
no longer be based around URL fragments.
2022-05-04 11:18:27 -07:00
patricklauer d86b7c4ef7
provision: Add missing ENDC to avoid colouring all output. 2022-05-03 12:41:43 -07:00
Anders Kaseorg e952641013 install: Resupport Ubuntu 22.04.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-05-03 09:41:08 -07:00
Anders Kaseorg 7acb642fa5 requirements: Upgrade to Tornado 6.
Fixes #8913.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-05-02 17:41:49 -07:00
Anders Kaseorg f23bfe91c0 run-dev: Fix types.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-05-02 17:41:49 -07:00
Anders Kaseorg c9faefd50e cache: Instantiate only one BMemcached cache backend.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-05-02 17:41:49 -07:00
Anders Kaseorg 0ef9309e92 run-dev: Switch to asyncio event loop.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-05-02 17:41:49 -07:00
Anders Kaseorg bded7180f7 tornado: Unfork tornado.autoreload.
We previously forked tornado.autoreload to work around a problem where
it would crash if you introduce a syntax error and not recover if you
fix it (https://github.com/tornadoweb/tornado/issues/2398).

A much more maintainable workaround for that issue, at least in
current Tornado, is to use tornado.autoreload as the main module.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-05-02 17:41:49 -07:00
Anders Kaseorg 1c7954b452 run-dev: Avoid deprecated tornado.gen.engine.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-05-02 17:41:49 -07:00
Anders Kaseorg 080a806d60 build-pgroonga: Update PGroonga to 2.3.6.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-29 16:02:45 -07:00
Tim Abbott cd011b1a94 ci: Verify production servers using /api/v1/server_settings.
The previous commit changed our response for the root domain on a
self-hosted Zulip server containing no realms to be a 404.

This made the HTTP headers verification done here less robust than
would be ideal, in that one could imagine a bug that makes every URL
404 passing that test.

So we instead change the check to verify the /server_settings API,
which doesn't have that problem.
2022-04-28 12:34:59 -07:00
Aman Agrawal 4e08c737ca home: For web public realms, skip login for spectators.
To provide a smoother experience of accessing a web public stream,
we don't ask user to login unless user directly requests a
`/login` URL.

Fixes #21690.
2022-04-28 12:34:29 -07:00
Alex Vandiver 4325bf0fc9 droplets: Add a tool to clean up old hostnames in DNS. 2022-04-28 11:30:47 -07:00
Alex Vandiver b5bcc5f1b4 droplets: Switch to Ubuntu 20.04 base image. 2022-04-28 11:30:47 -07:00
Alex Vandiver 702807829f droplets: Add IPv6 addresses to droplets. 2022-04-28 11:30:47 -07:00
Alex Vandiver 738d21f24b droplets: Allow specifying the region for the droplet. 2022-04-28 11:30:47 -07:00
Alex Vandiver b18b9d25e1 droplets: Add a script to do the provisioning. 2022-04-28 11:30:47 -07:00
Aman Agrawal 44efc8eb37 web_public_view: Redirect to current narrow after login.
Add current hash as `next` URL to all login buttons
a spectator can access while in app.
2022-04-28 10:47:49 -07:00
Anders Kaseorg a2825e5984 python: Use Python 3.8 typing.{Protocol,TypedDict}.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-27 12:57:49 -07:00
Anders Kaseorg 098a514599 python: Use Python 3.8 shlex.join function.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-27 12:57:49 -07:00
Anders Kaseorg bd072d79a4 test-backend: Respect --parallel=N when running specific tests.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-26 18:34:14 -07:00
Anders Kaseorg a543dcc8e3 Remove Debian 10 support.
As a consequence:

• Bump minimum supported Python version to 3.8.
• Move Vagrant environment to Ubuntu 20.04, which has Python 3.8.
• Move CI frontend tests to Ubuntu 20.04.
• Move production build test to Ubuntu 20.04.
• Move 3.4 upgrade test to Ubuntu 20.04.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-26 16:32:02 -07:00
Anders Kaseorg 63a1ef0e91 configure-rabbitmq: Remove use of sudo.
It already runs as root everywhere except in provision_inner, so move
the sudo there.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-19 12:36:31 -07:00
Anders Kaseorg cc30ed8ec7 actions: Delete zerver.lib.actions.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:38 -07:00
Anders Kaseorg 59f6b090c7 actions: Split out zerver.actions.realm_settings.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:37 -07:00
Anders Kaseorg ec174dfb47 actions: Split out zerver.actions.bots.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:36 -07:00
Anders Kaseorg cbad5739ab actions: Split out zerver.actions.create_user.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:35 -07:00
Anders Kaseorg 5fcbc412cf actions: Split out zerver.actions.streams.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:35 -07:00
Anders Kaseorg 975066e3f0 actions: Split out zerver.actions.message_send.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:34 -07:00
Anders Kaseorg ec6355389a actions: Split out zerver.actions.user_settings.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:34 -07:00
Anders Kaseorg d7981dad62 actions: Split out zerver.actions.users.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:34 -07:00
Anders Kaseorg a29f1b39da actions: Move part into zerver.lib.streams.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:32 -07:00
Anders Kaseorg dd8b1aaba6 actions: Add zerver/actions directory.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 13:48:35 -07:00
Anders Kaseorg a682530fd4 node_tests: Enforce 100% coverage for test files.
This makes it easier to find obsolete parts of the tests that should
be cleaned up.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-10 19:31:56 -04:00
Alex Vandiver e2a3fe0930 ci: Test upgrades from 3.x, 4.x and 5.x. 2022-04-08 17:10:03 -07:00
Alex Vandiver e6e4b7b3ef timeout: Add test coverage. 2022-04-07 17:26:01 -07:00
Alex Vandiver 35e27aef4a migrations: Remove the possibly-duplicated emoji re-uploading.
In 85e531e377, we duplicated this block
of migration code to fix a bug, but moving it (aka deleting the
original copy) is a cleaner solution.
2022-04-01 17:51:00 -07:00
Tim Abbott 85e531e377 migrations: Repeat part of migration 0376.
The blockquote explains the motivation for this change in detail.

Fixes #21608.
2022-04-01 15:20:43 -07:00
Tim Abbott d9bf8baca1 tools: Add per-repository commit counts in contributions tool.
This makes the output nice enough to include in the blog post.
2022-03-29 14:13:17 -07:00
Tim Abbott 13da2c2fb7 total-contributions: Clone repositories if already available.
This makes it more convenient to manage.
2022-03-28 23:01:58 -07:00
Tim Abbott a7f13ba723 total-contributions: Support all repositories and improve output. 2022-03-28 23:01:58 -07:00
Tim Abbott 7cbe9665b6 total-contributions: Exclude dependabot commits. 2022-03-28 23:01:58 -07:00
Tim Abbott a234fe9c4c total-contributions: Use commits by date, not by release. 2022-03-28 23:01:58 -07:00
Tim Abbott 6b5ce3579c total-contributions: Do a fetch for zulip/zulip as well.
This avoids producing stale output by having a consistent algorithm
for all repositories.
2022-03-28 23:01:58 -07:00
Anders Kaseorg 2762121162 python: Convert last type comments to annotations.
We had skipped these in #14693 so we could keep generating a friendly
error on Python 3.5, but we gave that up in #19801.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-03-24 20:32:39 -07:00
Abhishek Reddypalle cab37b4aca tools: Add tool to count contributions across all major repositories.
This will allow Zulip release announcements to credit contributions
made to Zulip projects beyond the server in our release announcements.

Fixes #19044.
2022-03-23 14:20:30 -07:00
Tim Abbott 9d460a513e tools: Fix typo in previous commit. 2022-03-18 16:57:51 -07:00
Tim Abbott a9d7a15ce2 tools: Add missing exclude to check-capitalization.
This tool needs improvement, but this fixes main failing CI.
2022-03-18 16:41:48 -07:00
Anders Kaseorg e2e645a183 browserslist: Stop transpiling for pre-ES2019 browsers.
Specifically, this desupports:

android 4.4.3-4.4.4
baidu 7.12
ie 11
kaios 2.5
op_mini all

although we’ve already been blocking IE 11 since 3.0 (#14662).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-03-17 15:24:46 -07:00
Alex Vandiver 6b0876063f bootstrap-awscli: Upgrade awscli version. 2022-03-17 11:44:05 -07:00
Alex Vandiver d150236217 ci: Test upgrades from 4.11. 2022-03-15 16:00:02 -07:00
Alex Vandiver 7650b5a972 session: Enforce that changes cannot happen in a transaction. 2022-03-15 13:52:15 -07:00
Anders Kaseorg 29ecf415fc validator: Add WildValue class for enforcing JSON type checking.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-03-15 13:02:02 -07:00
Steve Howell 8dcb7c571f node tests: Test get_convos directly.
In order to simplify our node tests, we are willing
to sacrifice coverage on UI-oriented glue modules.
2022-03-15 11:33:16 -07:00
Alex Vandiver 2fc7054a09 droplet: Fix printed instructions to have the right username/hostname. 2022-03-10 12:25:05 -08:00
Alex Vandiver 72b10937fc droplet: Factor out common droplet_domain_name. 2022-03-10 12:25:05 -08:00
Alex Vandiver 5086241361 droplet: Set a secure erlang cookie at startup. 2022-03-10 12:25:05 -08:00
Alex Vandiver aa9039d83e droplet: Switch to a new Debian 10 template snapshot. 2022-03-10 12:25:05 -08:00
Alex Vandiver d8c77eafb4 droplet: Always create with the "dev" tag.
This allows the firewall to be enforced on new hosts.
2022-03-10 12:25:05 -08:00
Alex Vandiver 25d753889b droplet: Allow overriding the subdomain. 2022-03-10 12:25:05 -08:00
Steve Howell c43d48b22f stream create: Overhaul create-stream add-subscribers UI.
The most notable change here is that when you are adding
subscribers to a stream as part of creating the stream,
you can now use the same essential pill-based UI for
adding users as we do when you edit subscribers for an
existing stream.

We don't try to exactly mimic the edit-stream UI or
implementation, since when you are adding subscribers
during create-stream, we are just updating a list in
memory, whereas in the edit-stream UI, we immediately
send info to the server.

Fixes #20499
2022-03-07 16:58:58 -08:00
Tim Abbott 92cc771392 test-all: Pass --skip-external-links to documentation tests.
This is what we already do in CI.  The external links often cause
these tests to fail, and it's not helpful for test-all to not match
CI.
2022-03-07 11:54:01 -08:00
Lauryn Menard 58f21fc748 edit_history: Remove `LEGACY_PREV_TOPIC` constant from code base.
Removes `LEGACY_PREV_TOPIC` which is no longer needed due to the
message edit history migration.

Also remove additions to the linter exclude list that were added
earlier in this commit series.
2022-03-04 10:25:48 -08:00
Tim Abbott 5e819cdebc edit_history: Support prev_subject not being present in database.
This commit serves as a checkpoint that can be deployed to production
before running the big Message format migration in the next commit.
2022-03-04 10:25:48 -08:00
Tim Abbott 85222b790d edit_history: Refactor to use FormattedEditHistoryEvent type.
We fix the mutation of caller and other bad patterns, as well as
adding explicit typing to make the code readable.

We also update the OpenAPI documentation for previously
undocumented `prev_strem` field in the `/get-message-history`
endpoint for API validation testing.

Co-authored-by: Lauryn Menard <lauryn.menard@gmail.com>
2022-03-04 10:25:48 -08:00
Tim Abbott f1e5ed91a1 types: Add EditHistoryEvent and APIEditHistoryEvent types.
These types will help make iteration on this code easier.

Note that `user_id` can be null due to the fact that
edit history entries before March 2017 did not log
the user that made the edit, which was years after
supporting topic edits (discovered in test deployment
of migration on chat.zulip.org).

Co-authored-by: Lauryn Menard <lauryn.menard@gmail.com>
2022-03-04 10:25:48 -08:00
Dinesh 74989d8171 tooling: test-js-with-puppeteer: Accept full relative file path.
Changes in a529dc8 to raise exception for invalid file name
has removed support for passing full file paths.
This commit fixes it.

Thanks to Steve Howell (showell) for reporting this.
2022-03-04 04:48:03 -05:00
Anders Kaseorg 44767dd653 tippyjs: Avoid unsafe allowHTML API in favor of <template> elements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-03-03 10:13:24 -08:00
Dinesh 36cc322525 tooling: Add --loop option to test-js-with-puppeteer. 2022-03-02 12:57:09 -05:00
Dinesh 02cf8e831f refactor: Extract code of running a single puppeteer test into a function. 2022-03-02 12:57:09 -05:00
Dinesh a529dc8c76 tooling: Make test-js-with-puppeteer fail fast on passing invalid file names.
fixes: #20970
2022-03-02 12:57:09 -05:00
Anders Kaseorg bff2741499 install-shfmt: Upgrade shfmt from 3.4.2 to 3.4.3.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-03-01 23:09:46 -08:00
Anders Kaseorg 503127c2e9 docs: Fix Git capitalization.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-28 16:04:35 -08:00
Alex Vandiver 62f4f3435f ci: Test upgrades from 4.10. 2022-02-25 16:28:33 -08:00
Anders Kaseorg 894a50b5c9 install: Support Ubuntu 22.04.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-25 14:49:07 -08:00
Anders Kaseorg 21cd1c10b3 docs: Add missing space in “time zone”.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-24 14:05:12 -08:00
Anders Kaseorg 076b0f06a2 capitalization: Prune unused phrases from IGNORED_PHRASES.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-23 12:08:20 -08:00
Anders Kaseorg cb9630e0db capitalization: Escape regex metacharacters in IGNORED_PHRASES.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-23 12:08:20 -08:00
Anders Kaseorg b3260bd610 docs: Use Debian and Ubuntu version numbers over development codenames.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-23 12:04:24 -08:00
Steve Howell ca17a452fd subscribers: Rename file to stream_edit_subscribers.js.
We are going to move to this code organization for
managing streams:

    stream_create.js
        stream_create_subscribers.js
    stream_edit.js
        stream_edit_subscribers.js

The modules stream_create.js and stream_edit.js historically
manage the entire process of creating and editing stream
data (respectively).

Going forward both will delegate most of the subscriber-specific
pieces to either stream_create_subscribers or stream_edit_subscribers.

The stream_*_subscribers modules will be somewhat similar in
nature, but the way that we manage subscribers at creation time
is a bit different than how we manage subscribers at edit time.
2022-02-22 16:29:36 -08:00
Steve Howell ebda1dfc8b manage streams: Extract add_subscribers_pill.js.
This is mostly a pure code move.  A few small tweaks:

    * The create() function is new.
    * The new module doesn't assume a `pill_widget`
      global.

This module represents the truly re-usable code
that can be shared during these two user actions:

    * edit-stream subscribers (now)
    * create-stream subscribers (future)

In both situations the input pill has (or will have)
essentially the same behavior, and the next commit
will tighten up the abstraction.

(The two processes will both also use fairly similar
ListWidgets, but the mechanics of managing the list
are going to be different, so we do not intend
to keep around stream_subscribers_ui in its current
name. More on that later.)
2022-02-22 16:29:36 -08:00
Steve Howell 65b51ae3bd subscribers: Extract subscriber_api.
This simplifies some of our dependencies.

As an example, we really don't want compose.js
to depend on stream_subscribers_ui.js, since
the former doesn't use any actual UI code from
the latter.

We also rename the two functions here:

    invite_user_to_stream -> add_user_ids_to_stream
    remove_user_from_stream -> remove_user_id_from_stream

(The notion of "inviting" somebody to a stream is
somewhat misleading, since there is really no invitation
mechanism; you just add them.)

Apart from naming changes this is a verbatim code move.

Finally, we eliminate a little bit of test cruft--the
`override` helper already ensures that a function gets
called at least once during a test.
2022-02-22 16:29:36 -08:00
Steve Howell 57398c9933 node tests: Remove stream_edit tests.
These tests have been historically difficult to maintain.

We have pretty good direct test coverage on the
components used by stream_edit.

The code tested here was mostly glue code and jQuery
code, which the node tests are particularly poorly
suited for testing.

Note that we lose 100% line coverage on
stream_settings_containers.js, but that module
is literally a single-line function to describe
a jQuery container, and the node tests for that
would be more convoluted than helpful.
2022-02-22 16:29:36 -08:00
Anders Kaseorg 1629d6bfb3 python: Reformat with Black 22 (stable).
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-18 18:03:13 -08:00
Aman Agrawal 9b622b7d25 gear_menu: Allow user to configure preferred theme.
We save the preferred theme in localstorage so that user doesn't
have to re-select the theme on every reload. Users on slow
computers might see flash of a theme change, if it happens.
2022-02-18 11:29:48 -08:00
Anders Kaseorg 924df5aaf5 provision: Binary-patch OpenSSL ARM64 assembly bug.
Fixes #20910.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-18 10:54:02 -08:00
Anders Kaseorg f6a701090c setup-apt-repos: Don’t install lsb_release.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-14 16:38:53 -08:00
Anders Kaseorg 7213116dd3 provision: Use apt-get --allow-downgrades.
Needed for commit 9c8d2b7be3 (#21115).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-14 16:02:35 -08:00
Anders Kaseorg 961633fcec timezone: List only canonical timezone identifiers.
For aliases that will no longer be listed, see the third column of

grep '^L ' zulip-py3-venv/lib/python3.*/site-packages/pytz/zoneinfo/tzdata.zi

Time zones previously set to an alias will be canonicalized on demand.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-11 17:38:57 -08:00
Tim Abbott fdabf0b357 pgroonga: Disable PGroonga in development to fix CI.
A recent Postgres upstream release appears to have broken PGroonga.
While we wait for https://github.com/pgroonga/pgroonga/issues/203 to
be resolved, disable PGroonga in our automated tests so that Zulip
CI passes.
2022-02-11 11:35:43 -08:00
Mateusz Mandera 30ac291eba emoji: Add migration to reupload all RealmEmoji and ensure .author.
Fixes #19732.
2022-02-10 17:45:31 -08:00
Aman Agrawal 7614f2203a pricing: Replace "Zulip Standard" with "Zulip Cloud Standard".
Case sensitive replace.
2022-02-09 11:00:24 -08:00
Tim Abbott 729c09074a js: Avoid href="#" for buttons with a JavaScript click handler.
As seen in
https://chat.zulip.org/#narrow/stream/9-issues/topic/edit.20history.20bug/near/1320430,
clicking such a link takes you to the user's default view if the click
handler throws an exception before doing preventDefault().

There hrefs also have the negative effect of having your browser claim
that clicking the link will navigate you to the default view, which it
won't.

Comes with a linter rule to prevent future instances, since it seems
there are some recently added ones, though they are likely the result
of copy/paste.
2022-02-08 17:38:12 -08:00
Anders Kaseorg b0ce4f1bce docs: Fix many spelling mistakes.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-07 18:51:06 -08:00
Anders Kaseorg 3e159446f0 docs: Update “G Suite” to “Google Workspace”.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-07 18:51:06 -08:00
Lauryn Menard 6f7e53c517 lint: Remove custom python rule for json_success without parameter.
As a preparatory step to refactoring json_success to accept
request as a parameter, removes custom lint python check for
calling json_success without a parameter.
2022-02-04 15:16:55 -08:00
Austin Riba 5f83bc5cfe lightbox: Replace lightbox_canvas with PanZoom library.
This PR changes how the Pan & Zoom feature of images displayed in the
attachment lightbox are handled.

The existing method of using a canvas element is replaced by the Panzoom
library (timmywil/panzoom). This library is lightweight and has 0
transitive dependencies.

This fixes #20759 where the issue is that the viewport of a zoomed image
was not expanding to fill the available space on the page. Switching to
this new library also solves several other UX issues:

    * Images are no longer blurred when in Pan & Zoom mode.
    * The zoom behavior itself uses focal point zooming: zooming occurs
      where the cursor is on the image instead of at the center of the
      image, reducing the need for extra panning.
    * CSS transitions are used for a more visually pleasing experience
      when switching images, toggling zoom off, etc.
    * The library has the potential to open other file types which
      leaves that option open for us in the future.
2022-02-04 14:58:36 -08:00
Puneeth Chaganti 3d55a1a5bf emoji: Make large square emoji names consistent. 2022-02-03 15:08:34 -08:00
Puneeth Chaganti d55c137277 emoji: Add yellow_large_square and green_large_square emojis.
Wordle has recently become a thing and it uses green, yellow and white (or
black in dark mode) large square unicode characters to let people share their
gameplay. Zulip converts the white and black large square unicode characters to
emojis, but not the green and yellow ones. This causes the Wordle grid to be
misaligned when shared on Zulip.

This commit adds green and yellow large square emojis to our emoji list to fix
the problem.
2022-02-02 16:26:31 -08:00
Aman Agrawal 0a614fe985 bookend: Move conditional bookend content inside the template.
We move the stream subscribed/unsubscribed bookend info from
js files to bookend handlebar.

Tweaked by tabbott to override the check-templates indentation logic.
2022-01-31 16:32:09 -08:00
Aman Agrawal 98fa5c9ae2 flatpickr: Move `show_flatpickr` function in a new module.
This would avoid cyclic imports to access `show_flatpickr`
function.
2022-01-28 11:43:07 -08:00
Anders Kaseorg 48024097c4 install-shfmt: Upgrade shfmt from 3.4.1 to 3.4.2.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-01-24 15:55:38 -08:00
Anders Kaseorg 97e4e9886c python: Replace universal_newlines with text.
This is supported in Python ≥ 3.7.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-01-23 22:16:01 -08:00
Anders Kaseorg 5fb935bf60 python: Replace deprecated urllib3 BACKOFF_MAX.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-01-23 22:14:17 -08:00
Anders Kaseorg ee71650908 python: Replace requests.packages.urllib3 alias with urllib3.
requests stopped vendoring urllib3 in 2.16.0 (2017-05-26).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-01-23 22:14:17 -08:00
Anders Kaseorg a58a71ef43 Remove Ubuntu 18.04 support.
As a consequence:

• Bump minimum supported Python version to 3.7.
• Move Vagrant environment to Debian 10, which has Python 3.7.
• Move CI frontend tests to Debian 10.
• Move production build test to Debian 10.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-01-21 17:26:14 -08:00
Anders Kaseorg d035efd467 ci: Test upgrade-postgresql on Ubuntu 20.04.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-01-21 17:26:14 -08:00
Aman Agrawal c4efc97d5a compose: Extend empty compose navigation logic to recipient boxes.
Navigation key presses like `Up` and `PageUp` with an empty recipient
boxes will now close the compose and propagate the keypress to the message
list or recent topics, depending upon the active view.

This extends behavior we've had for a long time with focus in the
compose box itself.
2022-01-21 16:40:13 -08:00
Anders Kaseorg 1178e015d1 provision: Install non-PGDG PGroonga package in development environment.
The development environment installs PostgreSQL from the OS, not PGDG,
so we should install the non-PGDG PGroonga package to match.  This is
required on Debian 10 where postgresql-12-pgdg-pgroonga does not exist.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-01-18 16:06:25 -08:00
Alex Vandiver 7b95f38854 install-aws-server: Link to run-instances documentation. 2022-01-12 15:52:19 -08:00
Alex Vandiver 44b8321721 install-aws-server: Shell hygene. 2022-01-12 15:52:19 -08:00
Alex Vandiver 1522eeaebf install-aws-server: Configurable disk size. 2022-01-12 15:52:19 -08:00
Alex Vandiver 0288964884 install-aws-server: Default to encrypted gp3 disks. 2022-01-12 15:52:19 -08:00
Alex Vandiver f52ee561ae install-aws-server: Configurable availability zone. 2022-01-12 15:52:19 -08:00
Alex Vandiver 8e89097dc1 install-aws-server: Intentionally space-split security groups.
This allows for multiple security groups to be specified,
space-separated, in the config.
2022-01-12 15:52:19 -08:00
Alex Vandiver c66fd6a01a install-aws-server: Configurable IAM profile for hosts. 2022-01-12 15:52:19 -08:00
Alex Vandiver d9d3368553 install-aws-server: Remove ASG logic. 2022-01-12 15:52:19 -08:00
Anders Kaseorg 82a048004a test-locked-requirements: Rename may_be_setup_cache to maybe_set_up_cache.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-01-12 13:21:35 -08:00
Steve Howell dcffa17e29 test-js-with-node: Simplify failures for --coverage.
When you use nyc, its code instrumentation transforms
the code so that line numbers and columns no longer
make sense, and the long stack trace is likely to cause
more confusion than convenience.

We want to encourage a workflow where you debug your
node tests using the normal (and much quicker mode)
before running `--coverage`.
2022-01-08 10:10:57 -05:00
Mateusz Mandera 5939329485 do_delete_user: Add migration to fix bugged UserProfiles.
do_delete_users had two bugs:
1. Creating the replacement dummy users
with active=True
2. Creating the replacement dummy users with email domain set to
realm.uri, which may not be a valid email domain.
Prior commits fixed the bugs, and this migration fixes the pre-existing
objects.
2022-01-04 15:42:03 -08:00
Alex Vandiver 0628c5d817 bootstrap-awscli: Set -eu, for if this is run standalone.
The existing callsites of this are via `source` or being inline'd into
the startup of a new host; in both of these cases, the surrounding
script is already `set -eu`.  However, if run as a standalone tool, it
should also configure itself to catch checksum failures and other
problems.
2021-12-31 11:14:40 -08:00
Alex Vandiver 7fcd173ec3 aws: Upgrade awscli version. 2021-12-31 11:14:40 -08:00
Steve Howell cb21506ea7 node tests: Sort list of exempt files. 2021-12-30 11:36:52 -08:00
Steve Howell 0777cd73bb stream edit: Extract stream_subscribers_ui.js.
This is a fairly straightforward extraction.

It's good to test this with Iago, and then go into
Manage Streams and add/remove subscribers for a stream
like devel.

I copy/pasted two small functions that will soon
diverge from stream_edit. The get_stream_id function
will either use a module variable (since we're
generally only editing subscribers for one stream, and
we already have the singleton assumption with
`input_pill`) or a more strict CSS selector.  And then
get_sub_for_target depends on get_stream_id. We may not
always need full subs, anyway, and when we adapt some
of this code for creating streams, things are likely to
change.

I stopped exporting a couple functions that have no
callers outside of this module.

The main entry point for the module is
enable_subscriber_management.

We continue to export invite_user_to_stream and
remove_user_from_stream, which should possibly be just
pulled into their own module to lessen some
dependencies, but they don't have too much baggage,
since they just wrap channel calls.
2021-12-30 11:36:52 -08:00
Anders Kaseorg f9271098bf upload-release: Avoid appending to bytes in a loop.
Appending to bytes in a loop leads to a quadratic slowdown since
Python doesn’t optimize this for bytes like it does for str.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-12-29 16:50:08 -08:00
Alex Vandiver 74b804880e bootstrap-aws: Support aarch64. 2021-12-29 16:35:15 -08:00
Mateusz Mandera c5c3ab66d6 remote_server: Migrate RemoteZulipServer.uuid to be UUIDField.
Given that these values are uuids, it's better to use UUIDField which is
meant for exactly that, rather than an arbitrary CharField.

This requires modifying some tests to use valid uuids.
2021-12-28 10:11:34 -08:00
Anders Kaseorg 702ce071f4 python: Accept Optional[FrameType] in signal handlers.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-12-28 09:31:55 -08:00
Anders Kaseorg 2ff3b51e03 test-documentation: Suppress Unicode Private Use Area validator warning.
sphinx_rtd_theme references the Font Awesome fa-link icon directly
with U+F0C1.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-12-22 16:55:31 -08:00
Alex Vandiver 71b56f7c1c puppet: process_fts_updates connects as nagios (or provided username).
It should not use the configured zulip username, but should instead
pull from the login user (likely `nagios`), or an explicit alternate
provided PostgreSQL username.  Failure to do so results in Nagios
failures because the `nagios` login does not have permissions to
authenticated the `zulip` PostgreSQL user.

This requires CI changes, as the install tests install as the `zulip`
login username, which allowed Nagios tests to pass previously; with
the custom database and username, however, they must be passed to
process_fts_updates explicitly when validating the install.
2021-12-14 14:48:53 -08:00
Mateusz Mandera 323f5dbba2 lint: Enforce consistent style of using transaction.atomic decorator.
When decorating a function, @transaction.atomic and
@transaction.atomic() are equivalent. We can add a linting rule to
enforce consistency.
2021-12-12 11:15:33 -08:00
Steve Howell 1db74fcf2f test-all: Call documentation tests again.
I accidentally checked in a change where I commented
out the test.
2021-12-11 12:46:44 -08:00
Steve Howell b2d83a8300 tests: Split out SingleUserExportTest.
This is mostly moving code, plus I now just
call shutil.rmtree directly.
2021-12-11 13:06:41 -05:00
Alya Abbott 0836b51738 policies: Update Kandra Labs policies pages. 2021-12-10 17:56:12 -08:00
Eeshan Garg 5aaeb1a432 use_cases: Rename /for/companies to /for/business. 2021-12-09 17:16:52 -08:00
Tim Abbott f8e35b5b36 tools: Document indentation policy. 2021-12-06 15:01:29 -08:00
Steve Howell 1509e7f355 templates: Always indent "each" blocks. 2021-12-06 14:58:29 -08:00
Steve Howell cbcfbf83ae check-tempates: Improve error message for missing end tag. 2021-12-04 15:35:24 -08:00
Steve Howell fb574431cb check-templates: Rewrite pretty_print (again).
It now does everything based on the tokens, rather
than walking the lines and trying to match up tokens
to lines.
2021-12-03 14:35:34 -08:00
Steve Howell 7e7b628054 check-templates: Parse code blocks properly. 2021-12-03 14:35:34 -08:00
Steve Howell ddfb4def4e check-templates: Improve error output.
I have looked at maybe ~100 errors in the last week as part
of fixing the tooling, and it's quite common to want to just
see what the improved file would look like.  Now I show the
desired output with line numbers.

I also try to encourage devs to scroll up, since newbies
often don't do that for some reason when confronted with
error output.

Finally, I add some color. I try to repeat myself without
color for certain things in case colors on certain
backgrounds are hard to read.

A fast way to test this is to just break up a long tag
into two lines.
2021-12-03 14:35:34 -08:00
Anders Kaseorg c5a8894dfb install-shfmt: Upgrade shfmt from 3.4.0 to 3.4.1.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-12-03 14:33:53 -08:00
Anders Kaseorg befb56f905 install-shellcheck: Upgrade ShellCheck from 0.7.2 to 0.8.0.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-12-03 14:33:53 -08:00
Aman Agrawal 25ee6a795e compose: Never hide send button.
`Press Enter to send` used to hide `Send` button, we remove that
behaviour.

We show the current state of `Enter` hotkey action via text below
`Send` button which can toggle behaviour on click.
2021-12-02 15:11:15 -08:00
Mateusz Mandera 72e5664031 semgrep: Enforce use of get_object_from_key for Confirmation fetching.
get_object_from_key should be used when trying to fetch a Confirmation
object. There are some places that need to make
Confirmation.objects.filter(...) queries, so we can't completely ban the
pattern, but we can ban .get(...) and
.filter(..., confirmation_key=..., ...).
2021-12-02 12:05:08 -08:00
Steve Howell 42061359a9 dead code: Kill off most of html_branches.
The code that was removed here was intended
for tooling that either never got built or
became obsolete.
2021-12-02 09:49:10 -08:00
Steve Howell 5a5dcd6962 minor: Streamline IGNORE_IDS in check-templates. 2021-12-02 09:49:10 -08:00
Steve Howell c0d72ba236 check-templates: Avoid duplicate tokenizing step.
Now we only tokenize the file once, and we pass
**validated** tokens to the pretty printer.

There are a few reasons for this:

    * It obviously saves a lot of extra computation
      just in terms of tokenization.

    * It allows our validator to add fields
      to the Token objects that help the pretty
      printer.

I also removed/tweaked a lot of legacy tests for
pretty_print.py that were exercising bizarrely
formatted HTML that we now simply ban during the
validation phase.
2021-12-02 09:49:10 -08:00
Steve Howell 0decfa8da0 check-templates: Remove prevent_dangling_tags.
This was a misfeature--it occasionally flagged
code that was ugly but necessarily ugly to
ensure proper rendering.
2021-12-02 09:49:10 -08:00
Steve Howell 6152bed378 check-templates: Prevent excessive newlines. 2021-12-02 09:49:10 -08:00
Steve Howell 6fc445ddd9 check-templates: Defer some indentation checks.
This accomplishes a few things:

    * lighten the load for the main validation loop
    * defer indentation checks until we are sure the author
      even knows how to match up tags
    * add some info to the Token objects that we may soon
      consume in our pretty-printer
2021-12-02 09:49:10 -08:00
Steve Howell f1fa9bff83 refactor: Extract tag_flavor(). 2021-12-02 09:49:10 -08:00
Steve Howell c6799c0903 refactor: Move logic for HTML_VOID_TAGS.
This sets up for the next commit.
2021-12-02 09:49:10 -08:00
Steve Howell c4b181a169 check-templates: Insist on 4-space indents.
We now complain about programmers who don't use
4-space indents in template files, rather than
letting the pretty printer fix them.

This is partly just to simplify the pretty printer
code (in future commits), but it also makes the
symptom more obvious to newbie developers. They
are probably just as able to react to the direct
error messages as they are able to figure out how
to read diffs from the pretty printer and grok
the --fix syntax. And once they learn the convention
and configure their editor, it should then be a
one time problem.
2021-12-02 09:49:10 -08:00
Steve Howell 2f0f27b841 templates: Fix minor whitespace errors.
(we also add validation)
2021-12-02 09:49:10 -08:00
Steve Howell a744e38e67 check-templates: Make parser more thorough (and faster).
We now create tokens for whitespace and text, such that you
could rebuild the template file with "".join(token.s for
token in tokens).

I also fixed a few bugs related to not parsing
whitespace-control tokens.

We no longer ignore template variables, although we could do
a lot better at validating them.

The most immediate use case for the more thorough parser is
to simplify the pretty printer, but it should also make it
less likely for us to skip over new template constructs
(i.e. the tool will fail hard rather than acting strange).

Note that this speeds up the tool by almost 3x, which may be
slightly surprising considering we are building more tokens.
The reason is that we are now munching efficiently through
big chunks of whitespace and text at a time, rather than
checking each individual character to see if it starts one
of the N other token types.

The changes to the pretty_print module here are a bit ugly,
but they should mostly be made irrelevant in subsequent
commits.
2021-12-02 09:49:10 -08:00
Steve Howell 2eac0560b2 check-templates: Explicitly skip comment tags. 2021-12-02 09:49:10 -08:00
Steve Howell 7f651b6aa2 refactor: Remove pseudo singleton end tags.
These aren't necessary any more.
2021-12-02 09:49:10 -08:00
Steve Howell 2f5ae1a54b check-templates: Be more flexible with inline tags.
We now only complain about inline tags having mismatched
indents if the author clearly tried to format them as
blocks.
2021-12-02 09:49:10 -08:00
Steve Howell d197813b88 check-templates: Extract/improve report_problem.
We extract the function for modularity and to
allow early-return.

We also add checks for "else" and improve a few
error messages.
2021-12-02 09:49:10 -08:00
Steve Howell 00e80b8c91 tools: Remove pretty-print-html.
This tool has been unmaintained since our initial code
sweep to fix templates, and it has possibly bit-rotted
during unrelated code sweeps like introducing mypy, etc.

It's not documented anywhere.

The preferred method now is to run:

    ./tools/check-templates --fix
2021-12-02 09:49:10 -08:00
Anders Kaseorg 5344e86c57 requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-12-02 08:16:20 -08:00
Palash 67f35d913a capitalization: Add 'I understand' to the IGNORED_PHRASES list.
String 'Here are a few messages I understand:'(next commit) was failing
./tools/check-capitalization check because of the capital I. I added
'I understand' to the IGNORED_PHRASES list in tools/lib/capitalization.py.

Adding "I" was working as well but didn't seem to me as a very great fix.
Strangely enough, adding " I " to the list made the test fail again
(With a lot of failed strings this time) as mentioned in the following
CZO thread.

Relevent CZO chat -
https://chat.zulip.org/#narrow/stream/49-development-help/topic/capitalization.20confusion.2E
2021-12-01 15:47:24 -08:00
Anders Kaseorg 513848a9d2 requirements: Install Gitlint from PyPI again.
In https://github.com/jorisroovers/gitlint/pull/246 I split the
gitlint package into gitlint and gitlint-core, where the latter avoids
pinning exact versions of its requirements so we can use it again.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-11-29 13:35:58 -08:00
Sahil Batra 0a949274e9 js: Rename night_mode.ts to dark_theme.ts. 2021-11-26 22:03:29 -08:00
Alex Vandiver 0ae375e0f9 ci: Test upgrades from the latest minor release. 2021-11-25 08:00:34 -08:00
Steve Howell 733cad1f5c check-templates: Tweak message for fixes. 2021-11-24 14:29:07 +00:00
Steve Howell d565387657 check-templates: Make --fix fix all files.
The 0/1 convention confused the person who implemented
the --fix option.  Now we use bools.
2021-11-24 14:15:01 +00:00
Steve Howell 2134a26873 templates: Clean up zephyr-mirror.html.
This allows us to eliminate conditionals related
to ignoring files and checking indentation.
2021-11-24 13:56:41 +00:00
Steve Howell 981a8d0189 templates: Clean up recipient_row.hbs. 2021-11-24 13:56:39 +00:00
Alya Abbott d09997f32e dev docs: Improve new contributor guide.
The main focus is on improving the instructions around claiming issues
to try to create less issue claiming spam.

Additionally, we haven't done a detailed update in a few years, and
some of the content is stale/irrelevant.
2021-11-23 17:01:53 -08:00
Steve Howell fdd63546b2 linters: Rewrite check-templates.
I rewrote most of tools/lib/pretty-printer.py, which
was fairly easy due to being able to crib some
important details from the previous implementation.

The main motivation for the rewrite was that we weren't
handling else/elif blocks correctly, and it was difficult
to modify the previous code. The else/elif shortcomings
were somewhat historical in nature--the original parser
didn't recognize them (since they weren't in any Zulip
templates at the time), and then the pretty printer was
mostly able to hack around that due to the "nudge"
strategy. Eventually the nudge strategy became too
brittle.

The "nudge" strategy was that we would mostly trust
the existing templates, and we would just nudge over
some lines in cases of obviously faulty indentation.

Now we are bit more opinionated and rigorous, and
we basically set the indentation explicitly for any
line that is not in a code/script block. This leads
to this diff touching several templates for mostly
minor fix-ups.

We aren't completely opinionated, as we respect the
author's line wrapping decisions in many cases, and
we also allow authors not to indent blocks within
the template language's block constructs.
2021-11-23 15:55:54 -08:00
Steve Howell 4792af5682 templates: Prevent dangling end tags.
In cases where an opening tag is so long that we stretch
it to 2+ lines of code, we should try to use block-style
formatting in the template code.

Unfortunately, we have lots of legacy code that violates
this concept, so this is a timid fix.

There are also legit use cases like textarea where we
probably need to keep the ugly template syntax for things
to render properly.
2021-11-23 15:55:54 -08:00
Steve Howell a6ee54d99d check-templates: Complain about stray text.
We disallow this HTML:

    junk-text-before-open-tag<p>
        This is a paragraph.
    </p>

We rarely see the above mistake, but we want to eliminate
the possibility to be somewhat rigorous, and so that we
can eliminate a pretty-printer mis-feature.
2021-11-23 15:55:54 -08:00
Ganesh Pawar f5fbf5f0e0 change_password: Migrate modal to dialog_widget. 2021-11-23 15:41:54 -08:00
Alex Vandiver 73055255bb ci: Test pgroonga installation. 2021-11-20 07:13:50 -08:00
Sahil Batra 16d35f5b4e streams: Hide fake emails in stream creation form.
We hide fake emails in "People to add" section of
stream creation form and instead show user id
along with specifying that email is hidden.
2021-11-18 12:49:10 -08:00
YashRE42 b0be6a0f92 test_script: Reword provision warning not to assume running tests.
Previously, running `./tools/run-dev.py` when provision was required
would lead to a warning along the lines of:
```
Before we run tests, we make sure your provisioning version
is correct by looking at var/provision_version, which is at
version 165.1, and we compare it to the version in source
control (version.py), which is 165.2.

It looks like you checked out a branch that has added
dependencies beyond what you last provisioned. Your command
is likely to fail until you add dependencies by provisioning.

Do this: `./tools/provision`

If you really know what you are doing, use --skip-provision-check to
run anyway.
```

The assumption that we're trying to run tests might cause some
confusion, especially if its the first time you're seeing the
provision warning. Hence, we reword the first paragraph to avoid
making that assumption.

The second paragraph has also been slightly altered, since (1) it's
possible that we didn't checkout a different branch, but eg just
rebased with upstream and (2) we might not be on a VM.

The warning you'd get after this commit would be along the lines of:
```
Provisioning state check failed! This check compares
`var/provision_version` (currently 165.2) to the version in
source control (`version.py`), which is 164.6, to see if you
likely need to provision before this command can run
properly.

The branch you are currently on expects an older version of
dependencies than the version you provisioned last. This may
be ok, but it's likely that you either want to rebase your
branch on top of upstream/main or re-provision your machine.

Do this: `./tools/provision`

If you really know what you are doing, use --skip-provision-check to
run anyway.
```
or along the lines of:
```
Provisioning state check failed! This check compares
`var/provision_version` (currently 165.2) to the version in
source control (`version.py`), which is 167.2, to see if you
likely need to provision before this command can run
properly.

The branch you are currently on has added dependencies beyond
what you last provisioned. Your command is likely to fail
until you add dependencies by provisioning.

Do this: `./tools/provision`

If you really know what you are doing, use --skip-provision-check to
run anyway.
```
2021-11-16 16:29:34 -08:00
Alex Vandiver 64268f47e8 queue_processors: Drop unused current_queue_size, which was local size.
The `current_queue_size` key in the queue monitoring stats file was
the local queue size, not the global queue size -- d5a6b0f99a
renamed the function, but did not adjust the queue monitoring JSON,
despite the last use of it having been removed in cd9b194d88.

The function is still used to mark "we emptied our queue," and it
remains a reasonable metric for that.
2021-11-16 11:48:50 -08:00
Mateusz Mandera 3688ff38bd custom_check: Improve check for exit(...) call in management commands.
It's better for this to catch all exit(...) calls with non-zero exit
code, given the purpose is to catch all exits with failure, as opposed
to only exit(1).
2021-11-16 11:21:05 -08:00
Mateusz Mandera 7cc345d7b1 process_queue: Improve handling of exceptions in process_queue.
Unhandled exceptions propagating to process_queue were not caught there,
causing improper logging - errors didn't land in errors.log as expected.
Exceptions should be caught and explicitly logged by the process_queue
logger. Exceptions occurring during consuming events are caught and
handled inside the worker's logic - however those that happen while
setting up the worker were not addressed at all, and that's the core bug
we mean to address here.

Furthermore, in multi-threaded mode we want the autoreload mechanism to
be working - which it doesn't without catching the exceptions. The
correct approach is to - again - catch the exception, log it and then
send SIGUSR1 signal to trigger exit and autoreload.
2021-11-16 11:21:05 -08:00
Anders Kaseorg 7a431a690d dev-vagrant-docker: Upgrade docker-systemctl-replacement to 1.5.4505.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-11-10 09:02:56 -08:00
Alex Vandiver 2b025608d0 provision: Remove unsupported Ubuntu versions, add Impish. 2021-11-08 18:21:46 -08:00
Anders Kaseorg b54007d36c clean-repo: Remove.
Unless this was working around a specific bug that has somehow
persisted for eight years, this was just pointlessly defeating
Python’s bytecode cache every time you started the dev server.

This reverts commits deffda072f,
59228f7458, and
ae45217671.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-11-05 17:44:24 -07:00
Anders Kaseorg a3445dac95 install-shfmt: Upgrade shfmt from 3.3.1 to 3.4.0.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-11-05 17:34:13 -07:00
Vishnu KS 6c06858e02 billing: Migrate to Stripe hosted checkout page. 2021-11-05 17:23:10 -07:00
Alex Vandiver c9ebccdf70 ci: Update outdated comments to generated Dockerfiles.
These were missed in the doc updates in d78723b6e8, which were for
behavior changes in 16067bc4fc.
2021-11-05 15:45:46 -07:00
Kevin Scott 64f099d2f5 compose: Add compose box button to insert global times.
Fixes #20045.
2021-11-01 16:20:05 -07:00
Anders Kaseorg 8bb540fa31 run-codespell: Fix invocation with multiple arguments.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-10-28 09:59:27 -07:00
rht a3a71487b0 CI: Add Codespell linter.
This tool helps catch common typos in code and documentation, which is
particularly useful for our many contributors who are not native
English speakers.

The config is based on the codespell that I ran in
https://github.com/zulip/zulip/pull/18535.
2021-10-27 16:49:30 -07:00
Kartik Soneji 6705085e25 provision: Silently unset PIP_USER if set. 2021-10-22 13:43:40 -07:00
Anders Kaseorg 2c27a87bcd provision: Refuse to run outside an existing Vagrant environment.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-10-20 10:35:46 -07:00
Anders Kaseorg 51971a48f3 debug-require-webpack-plugin: Handle missing chunkGraph.
This seems to be triggered by mini-css-extract-plugin@2.4.0.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-10-17 07:15:09 -07:00
Anders Kaseorg 9cf5a03f2a ci: Migrate to new Codecov uploader.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-10-14 17:56:52 -07:00
Anders Kaseorg 9f2ac49fb3 ci: Change github UID from 3434 to 1001.
CircleCI expected 3434; GitHub Actions expects 1001.  This is the
reason zulip-ci.yml currently needs to mess with the permissions in
/__w.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-10-14 13:01:47 -07:00