Commit Graph

4717 Commits

Author SHA1 Message Date
Alex Vandiver fb338f22d7 ci: Stop rabbitmq when making CI images.
The production CI image starts `rabbitmq-server` but does not stop it,
which leaves a stale `/var/run/rabbitmq/pid` file in the image.

`rabbitmqctl wait --timeout 600 /var/run/rabbitmq/pid`, which is run
after starting the rabbitmq node, reads the PID file and waits for the
PID to be running, and for rabbitmq's port to be responding to pings.

If it reads an old PID file before the new PID is written, it
aborts (all but the first and last lines are output from `rabbitmqctl
wait` that is hidden by `/etc/init.d/rabbitmq-server`):

```
 * Starting RabbitMQ Messaging Server rabbitmq-server
Waiting for pid file '/var/run/rabbitmq/pid' to appear
pid is 341
Waiting for erlang distribution on node 'rabbit@fc8f64d6acdb' while OS process '341' is running
Error:
process_not_running
 * FAILED - check /var/log/rabbitmq/startup_\{log, _err\}
 ```

If it failed, the `production-upgrade` script tried to start
`rabbitmq` again -- despite it already still starting in the
background.  These two attempts conflicted, and often one or both
failed.

Stop `rabbitmq-server` when building the image, which removes the
stale PID file.
2022-05-19 13:30:59 -07:00
Adam Sah 04ccd8c6d8 testing: 100% code coverage for zerver/lib/outgoing_webhook.py. 2022-05-17 13:51:00 -07:00
Alex Vandiver c93024cd5b oneclick: Fail if the fab command fails. 2022-05-17 13:41:12 -07:00
Alex Vandiver ff647dff03 oneclick: Do not use a stale Zulip client.
Initializing the Zulip client opens a long-lived TCP connection due to
connection pooling in urllib3.  In Github Actions, the network kills
such requests after ~270s, making the later `send_message` call fail.

Use a singular call to `zulip.Client()` early on to verify the
credentials, and do not cache the resulting client object.  Instead,
re-create it during the final step when it is needed, so we do not run
afoul of bad TCP connection state.

This would ideally be fixed via connection keepalive or retry at the
level of the Zulip module.
2022-05-17 13:41:12 -07:00
Adam Sah 534754442a test-backend: Improve performance by disabling XML report.
We don't use the XML report ourselves. We add options to make this
easy to control if specific circumstances indicate doing so.
2022-05-16 16:38:02 -07:00
Adam Sah 492272d597 test-backend: Display test function in HTML coverage reports. 2022-05-16 16:38:02 -07:00
Alex Vandiver 9e42cf435b tail-ses: Add a tool to tail SES delivery/bounce/complaint queues. 2022-05-06 17:49:00 -07:00
Ganesh Pawar 59e810790f user_groups: Convert inline form to modal.
Fixes part of #21298.
2022-05-04 17:46:30 -07:00
Raghav Luthra fb89c14f34 lint: Improve data-tippy-allowHTML error message.
The error message a user gets from the linter when using the
data-tippy-allowHTML attribute now conveys the fact that the
<template> tag is supposed to hold the tooltip content. This
might make understanding the correct workflow easier for
someone who encounters this error.
2022-05-04 17:38:30 -07:00
Adam Sah a9ce02348b testing: Improve backend code coverage for semaphore webhook. 2022-05-04 16:13:24 -07:00
Alex Vandiver bf562f8fff ci: Update upgrade test to 5.2, from 5.1. 2022-05-04 11:37:15 -07:00
Tim Abbott 25999b7056 lint: Ban Help Center relative links without leading /.
Such links would likely end up broken if we change our URL scheme to
no longer be based around URL fragments.
2022-05-04 11:18:27 -07:00
patricklauer d86b7c4ef7
provision: Add missing ENDC to avoid colouring all output. 2022-05-03 12:41:43 -07:00
Anders Kaseorg e952641013 install: Resupport Ubuntu 22.04.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-05-03 09:41:08 -07:00
Anders Kaseorg 7acb642fa5 requirements: Upgrade to Tornado 6.
Fixes #8913.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-05-02 17:41:49 -07:00
Anders Kaseorg f23bfe91c0 run-dev: Fix types.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-05-02 17:41:49 -07:00
Anders Kaseorg c9faefd50e cache: Instantiate only one BMemcached cache backend.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-05-02 17:41:49 -07:00
Anders Kaseorg 0ef9309e92 run-dev: Switch to asyncio event loop.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-05-02 17:41:49 -07:00
Anders Kaseorg bded7180f7 tornado: Unfork tornado.autoreload.
We previously forked tornado.autoreload to work around a problem where
it would crash if you introduce a syntax error and not recover if you
fix it (https://github.com/tornadoweb/tornado/issues/2398).

A much more maintainable workaround for that issue, at least in
current Tornado, is to use tornado.autoreload as the main module.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-05-02 17:41:49 -07:00
Anders Kaseorg 1c7954b452 run-dev: Avoid deprecated tornado.gen.engine.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-05-02 17:41:49 -07:00
Anders Kaseorg 080a806d60 build-pgroonga: Update PGroonga to 2.3.6.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-29 16:02:45 -07:00
Tim Abbott cd011b1a94 ci: Verify production servers using /api/v1/server_settings.
The previous commit changed our response for the root domain on a
self-hosted Zulip server containing no realms to be a 404.

This made the HTTP headers verification done here less robust than
would be ideal, in that one could imagine a bug that makes every URL
404 passing that test.

So we instead change the check to verify the /server_settings API,
which doesn't have that problem.
2022-04-28 12:34:59 -07:00
Aman Agrawal 4e08c737ca home: For web public realms, skip login for spectators.
To provide a smoother experience of accessing a web public stream,
we don't ask user to login unless user directly requests a
`/login` URL.

Fixes #21690.
2022-04-28 12:34:29 -07:00
Alex Vandiver 4325bf0fc9 droplets: Add a tool to clean up old hostnames in DNS. 2022-04-28 11:30:47 -07:00
Alex Vandiver b5bcc5f1b4 droplets: Switch to Ubuntu 20.04 base image. 2022-04-28 11:30:47 -07:00
Alex Vandiver 702807829f droplets: Add IPv6 addresses to droplets. 2022-04-28 11:30:47 -07:00
Alex Vandiver 738d21f24b droplets: Allow specifying the region for the droplet. 2022-04-28 11:30:47 -07:00
Alex Vandiver b18b9d25e1 droplets: Add a script to do the provisioning. 2022-04-28 11:30:47 -07:00
Aman Agrawal 44efc8eb37 web_public_view: Redirect to current narrow after login.
Add current hash as `next` URL to all login buttons
a spectator can access while in app.
2022-04-28 10:47:49 -07:00
Anders Kaseorg a2825e5984 python: Use Python 3.8 typing.{Protocol,TypedDict}.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-27 12:57:49 -07:00
Anders Kaseorg 098a514599 python: Use Python 3.8 shlex.join function.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-27 12:57:49 -07:00
Anders Kaseorg bd072d79a4 test-backend: Respect --parallel=N when running specific tests.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-26 18:34:14 -07:00
Anders Kaseorg a543dcc8e3 Remove Debian 10 support.
As a consequence:

• Bump minimum supported Python version to 3.8.
• Move Vagrant environment to Ubuntu 20.04, which has Python 3.8.
• Move CI frontend tests to Ubuntu 20.04.
• Move production build test to Ubuntu 20.04.
• Move 3.4 upgrade test to Ubuntu 20.04.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-26 16:32:02 -07:00
Anders Kaseorg 63a1ef0e91 configure-rabbitmq: Remove use of sudo.
It already runs as root everywhere except in provision_inner, so move
the sudo there.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-19 12:36:31 -07:00
Anders Kaseorg cc30ed8ec7 actions: Delete zerver.lib.actions.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:38 -07:00
Anders Kaseorg 59f6b090c7 actions: Split out zerver.actions.realm_settings.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:37 -07:00
Anders Kaseorg ec174dfb47 actions: Split out zerver.actions.bots.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:36 -07:00
Anders Kaseorg cbad5739ab actions: Split out zerver.actions.create_user.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:35 -07:00
Anders Kaseorg 5fcbc412cf actions: Split out zerver.actions.streams.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:35 -07:00
Anders Kaseorg 975066e3f0 actions: Split out zerver.actions.message_send.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:34 -07:00
Anders Kaseorg ec6355389a actions: Split out zerver.actions.user_settings.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:34 -07:00
Anders Kaseorg d7981dad62 actions: Split out zerver.actions.users.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:34 -07:00
Anders Kaseorg a29f1b39da actions: Move part into zerver.lib.streams.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 17:14:32 -07:00
Anders Kaseorg dd8b1aaba6 actions: Add zerver/actions directory.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-14 13:48:35 -07:00
Anders Kaseorg a682530fd4 node_tests: Enforce 100% coverage for test files.
This makes it easier to find obsolete parts of the tests that should
be cleaned up.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-10 19:31:56 -04:00
Alex Vandiver e2a3fe0930 ci: Test upgrades from 3.x, 4.x and 5.x. 2022-04-08 17:10:03 -07:00
Alex Vandiver e6e4b7b3ef timeout: Add test coverage. 2022-04-07 17:26:01 -07:00
Alex Vandiver 35e27aef4a migrations: Remove the possibly-duplicated emoji re-uploading.
In 85e531e377, we duplicated this block
of migration code to fix a bug, but moving it (aka deleting the
original copy) is a cleaner solution.
2022-04-01 17:51:00 -07:00
Tim Abbott 85e531e377 migrations: Repeat part of migration 0376.
The blockquote explains the motivation for this change in detail.

Fixes #21608.
2022-04-01 15:20:43 -07:00
Tim Abbott d9bf8baca1 tools: Add per-repository commit counts in contributions tool.
This makes the output nice enough to include in the blog post.
2022-03-29 14:13:17 -07:00
Tim Abbott 13da2c2fb7 total-contributions: Clone repositories if already available.
This makes it more convenient to manage.
2022-03-28 23:01:58 -07:00
Tim Abbott a7f13ba723 total-contributions: Support all repositories and improve output. 2022-03-28 23:01:58 -07:00
Tim Abbott 7cbe9665b6 total-contributions: Exclude dependabot commits. 2022-03-28 23:01:58 -07:00
Tim Abbott a234fe9c4c total-contributions: Use commits by date, not by release. 2022-03-28 23:01:58 -07:00
Tim Abbott 6b5ce3579c total-contributions: Do a fetch for zulip/zulip as well.
This avoids producing stale output by having a consistent algorithm
for all repositories.
2022-03-28 23:01:58 -07:00
Anders Kaseorg 2762121162 python: Convert last type comments to annotations.
We had skipped these in #14693 so we could keep generating a friendly
error on Python 3.5, but we gave that up in #19801.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-03-24 20:32:39 -07:00
Abhishek Reddypalle cab37b4aca tools: Add tool to count contributions across all major repositories.
This will allow Zulip release announcements to credit contributions
made to Zulip projects beyond the server in our release announcements.

Fixes #19044.
2022-03-23 14:20:30 -07:00
Tim Abbott 9d460a513e tools: Fix typo in previous commit. 2022-03-18 16:57:51 -07:00
Tim Abbott a9d7a15ce2 tools: Add missing exclude to check-capitalization.
This tool needs improvement, but this fixes main failing CI.
2022-03-18 16:41:48 -07:00
Anders Kaseorg e2e645a183 browserslist: Stop transpiling for pre-ES2019 browsers.
Specifically, this desupports:

android 4.4.3-4.4.4
baidu 7.12
ie 11
kaios 2.5
op_mini all

although we’ve already been blocking IE 11 since 3.0 (#14662).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-03-17 15:24:46 -07:00
Alex Vandiver 6b0876063f bootstrap-awscli: Upgrade awscli version. 2022-03-17 11:44:05 -07:00
Alex Vandiver d150236217 ci: Test upgrades from 4.11. 2022-03-15 16:00:02 -07:00
Alex Vandiver 7650b5a972 session: Enforce that changes cannot happen in a transaction. 2022-03-15 13:52:15 -07:00
Anders Kaseorg 29ecf415fc validator: Add WildValue class for enforcing JSON type checking.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-03-15 13:02:02 -07:00
Steve Howell 8dcb7c571f node tests: Test get_convos directly.
In order to simplify our node tests, we are willing
to sacrifice coverage on UI-oriented glue modules.
2022-03-15 11:33:16 -07:00
Alex Vandiver 2fc7054a09 droplet: Fix printed instructions to have the right username/hostname. 2022-03-10 12:25:05 -08:00
Alex Vandiver 72b10937fc droplet: Factor out common droplet_domain_name. 2022-03-10 12:25:05 -08:00
Alex Vandiver 5086241361 droplet: Set a secure erlang cookie at startup. 2022-03-10 12:25:05 -08:00
Alex Vandiver aa9039d83e droplet: Switch to a new Debian 10 template snapshot. 2022-03-10 12:25:05 -08:00
Alex Vandiver d8c77eafb4 droplet: Always create with the "dev" tag.
This allows the firewall to be enforced on new hosts.
2022-03-10 12:25:05 -08:00
Alex Vandiver 25d753889b droplet: Allow overriding the subdomain. 2022-03-10 12:25:05 -08:00
Steve Howell c43d48b22f stream create: Overhaul create-stream add-subscribers UI.
The most notable change here is that when you are adding
subscribers to a stream as part of creating the stream,
you can now use the same essential pill-based UI for
adding users as we do when you edit subscribers for an
existing stream.

We don't try to exactly mimic the edit-stream UI or
implementation, since when you are adding subscribers
during create-stream, we are just updating a list in
memory, whereas in the edit-stream UI, we immediately
send info to the server.

Fixes #20499
2022-03-07 16:58:58 -08:00
Tim Abbott 92cc771392 test-all: Pass --skip-external-links to documentation tests.
This is what we already do in CI.  The external links often cause
these tests to fail, and it's not helpful for test-all to not match
CI.
2022-03-07 11:54:01 -08:00
Lauryn Menard 58f21fc748 edit_history: Remove `LEGACY_PREV_TOPIC` constant from code base.
Removes `LEGACY_PREV_TOPIC` which is no longer needed due to the
message edit history migration.

Also remove additions to the linter exclude list that were added
earlier in this commit series.
2022-03-04 10:25:48 -08:00
Tim Abbott 5e819cdebc edit_history: Support prev_subject not being present in database.
This commit serves as a checkpoint that can be deployed to production
before running the big Message format migration in the next commit.
2022-03-04 10:25:48 -08:00
Tim Abbott 85222b790d edit_history: Refactor to use FormattedEditHistoryEvent type.
We fix the mutation of caller and other bad patterns, as well as
adding explicit typing to make the code readable.

We also update the OpenAPI documentation for previously
undocumented `prev_strem` field in the `/get-message-history`
endpoint for API validation testing.

Co-authored-by: Lauryn Menard <lauryn.menard@gmail.com>
2022-03-04 10:25:48 -08:00
Tim Abbott f1e5ed91a1 types: Add EditHistoryEvent and APIEditHistoryEvent types.
These types will help make iteration on this code easier.

Note that `user_id` can be null due to the fact that
edit history entries before March 2017 did not log
the user that made the edit, which was years after
supporting topic edits (discovered in test deployment
of migration on chat.zulip.org).

Co-authored-by: Lauryn Menard <lauryn.menard@gmail.com>
2022-03-04 10:25:48 -08:00
Dinesh 74989d8171 tooling: test-js-with-puppeteer: Accept full relative file path.
Changes in a529dc8 to raise exception for invalid file name
has removed support for passing full file paths.
This commit fixes it.

Thanks to Steve Howell (showell) for reporting this.
2022-03-04 04:48:03 -05:00
Anders Kaseorg 44767dd653 tippyjs: Avoid unsafe allowHTML API in favor of <template> elements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-03-03 10:13:24 -08:00
Dinesh 36cc322525 tooling: Add --loop option to test-js-with-puppeteer. 2022-03-02 12:57:09 -05:00
Dinesh 02cf8e831f refactor: Extract code of running a single puppeteer test into a function. 2022-03-02 12:57:09 -05:00
Dinesh a529dc8c76 tooling: Make test-js-with-puppeteer fail fast on passing invalid file names.
fixes: #20970
2022-03-02 12:57:09 -05:00
Anders Kaseorg bff2741499 install-shfmt: Upgrade shfmt from 3.4.2 to 3.4.3.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-03-01 23:09:46 -08:00
Anders Kaseorg 503127c2e9 docs: Fix Git capitalization.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-28 16:04:35 -08:00
Alex Vandiver 62f4f3435f ci: Test upgrades from 4.10. 2022-02-25 16:28:33 -08:00
Anders Kaseorg 894a50b5c9 install: Support Ubuntu 22.04.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-25 14:49:07 -08:00
Anders Kaseorg 21cd1c10b3 docs: Add missing space in “time zone”.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-24 14:05:12 -08:00
Anders Kaseorg 076b0f06a2 capitalization: Prune unused phrases from IGNORED_PHRASES.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-23 12:08:20 -08:00
Anders Kaseorg cb9630e0db capitalization: Escape regex metacharacters in IGNORED_PHRASES.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-23 12:08:20 -08:00
Anders Kaseorg b3260bd610 docs: Use Debian and Ubuntu version numbers over development codenames.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-23 12:04:24 -08:00
Steve Howell ca17a452fd subscribers: Rename file to stream_edit_subscribers.js.
We are going to move to this code organization for
managing streams:

    stream_create.js
        stream_create_subscribers.js
    stream_edit.js
        stream_edit_subscribers.js

The modules stream_create.js and stream_edit.js historically
manage the entire process of creating and editing stream
data (respectively).

Going forward both will delegate most of the subscriber-specific
pieces to either stream_create_subscribers or stream_edit_subscribers.

The stream_*_subscribers modules will be somewhat similar in
nature, but the way that we manage subscribers at creation time
is a bit different than how we manage subscribers at edit time.
2022-02-22 16:29:36 -08:00
Steve Howell ebda1dfc8b manage streams: Extract add_subscribers_pill.js.
This is mostly a pure code move.  A few small tweaks:

    * The create() function is new.
    * The new module doesn't assume a `pill_widget`
      global.

This module represents the truly re-usable code
that can be shared during these two user actions:

    * edit-stream subscribers (now)
    * create-stream subscribers (future)

In both situations the input pill has (or will have)
essentially the same behavior, and the next commit
will tighten up the abstraction.

(The two processes will both also use fairly similar
ListWidgets, but the mechanics of managing the list
are going to be different, so we do not intend
to keep around stream_subscribers_ui in its current
name. More on that later.)
2022-02-22 16:29:36 -08:00
Steve Howell 65b51ae3bd subscribers: Extract subscriber_api.
This simplifies some of our dependencies.

As an example, we really don't want compose.js
to depend on stream_subscribers_ui.js, since
the former doesn't use any actual UI code from
the latter.

We also rename the two functions here:

    invite_user_to_stream -> add_user_ids_to_stream
    remove_user_from_stream -> remove_user_id_from_stream

(The notion of "inviting" somebody to a stream is
somewhat misleading, since there is really no invitation
mechanism; you just add them.)

Apart from naming changes this is a verbatim code move.

Finally, we eliminate a little bit of test cruft--the
`override` helper already ensures that a function gets
called at least once during a test.
2022-02-22 16:29:36 -08:00
Steve Howell 57398c9933 node tests: Remove stream_edit tests.
These tests have been historically difficult to maintain.

We have pretty good direct test coverage on the
components used by stream_edit.

The code tested here was mostly glue code and jQuery
code, which the node tests are particularly poorly
suited for testing.

Note that we lose 100% line coverage on
stream_settings_containers.js, but that module
is literally a single-line function to describe
a jQuery container, and the node tests for that
would be more convoluted than helpful.
2022-02-22 16:29:36 -08:00
Anders Kaseorg 1629d6bfb3 python: Reformat with Black 22 (stable).
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-18 18:03:13 -08:00
Aman Agrawal 9b622b7d25 gear_menu: Allow user to configure preferred theme.
We save the preferred theme in localstorage so that user doesn't
have to re-select the theme on every reload. Users on slow
computers might see flash of a theme change, if it happens.
2022-02-18 11:29:48 -08:00
Anders Kaseorg 924df5aaf5 provision: Binary-patch OpenSSL ARM64 assembly bug.
Fixes #20910.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-18 10:54:02 -08:00
Anders Kaseorg f6a701090c setup-apt-repos: Don’t install lsb_release.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-14 16:38:53 -08:00
Anders Kaseorg 7213116dd3 provision: Use apt-get --allow-downgrades.
Needed for commit 9c8d2b7be3 (#21115).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-14 16:02:35 -08:00
Anders Kaseorg 961633fcec timezone: List only canonical timezone identifiers.
For aliases that will no longer be listed, see the third column of

grep '^L ' zulip-py3-venv/lib/python3.*/site-packages/pytz/zoneinfo/tzdata.zi

Time zones previously set to an alias will be canonicalized on demand.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-11 17:38:57 -08:00