Commit Graph

1453 Commits

Author SHA1 Message Date
Tim Abbott 0a756c652c push_notifications: Shard mobile push notifications. 2024-05-02 14:25:10 -07:00
Alex Vandiver 9dfaa83aa8 invites: Remove invites worker, make confirmation object in-process.
The "invites" worker exists to do two things -- make a Confirmation
object, and send the outgoing email.  Making the Confirmation object
in a background process from where the PreregistrationUser is created
temporarily leaves the PreregistrationUser in invalid state, and
results in 500's, and the user not immediately seeing the sent
invitation.  That the "invites" worker also wants to create the
Confirmation object means that "resending" an invite invalidates the
URL in the previous email, which can be confusing to the user.

Moving the Confirmation creation to the same transaction solves both
of these issues, and leaves the "invites" worker with nothing to do
but send the email; as such, we remove it entirely, and use the
existing "email_senders" worker to send the invites.  The volume of
invites is small enough that this will not affect other uses of that
worker.

Fixes: #21306
Fixes: #24275
2024-05-02 14:23:04 -07:00
Anders Kaseorg 0c7fd45c83 install-node: Upgrade Node.js from 20.12.0 to 20.12.2.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-04-25 16:46:03 -07:00
Alex Vandiver 7f4bc05f8e install: Check for a supported version of the OS in shell.
Testing for it in Python means that we have to worry about keeping the
`upgrade-zulip-stage-2` backwards-compatible with all versions of
Python which we could ever be upgrading from -- which is all of them.

Factor out the "supported operating systems" check, and share it
between upgrade and install codepaths.
2024-04-16 11:13:33 -07:00
Alex Vandiver a853b8849f install: Actually add --puppet-classes argument.
7187146422 only added it to the internal parsing and the help, not
the getopt invocation which would make the command accept the
argument.
2024-04-11 15:34:00 -07:00
Anders Kaseorg 36aa0177bd Revert "apt-repos: Disallow libmagic1 1:5.45-2 (Ubuntu 24.04) again."
This reverts commit 13e28fc3ac.
2024-04-10 16:07:25 -07:00
Alex Vandiver 7187146422 install: Move PUPPET_CLASSES env var to --puppet-classes argument.
`--no-init-db` is used to silence the need for `--hostname` and
`--email` arguments; it is a proxy for "this is not a frontend host."
We would ideally like to use `has_class` to know if the user's
provided puppet classes are include an `app_frontend`, and thus
`--hostname` and `--email` are required -- but doing that requires
several other steps, and we would like this feedback to be immediate.

We make the presence of `--puppet-classes` equivalent to
`--no-init-db`, since nearly every configuration with
`--puppet-classes` does not install both a database and a frontend,
which is what is required to initialize a database.
2024-04-04 16:49:43 -07:00
Alex Vandiver 89484fbbe6 install: Remove the hidden VIRTUALENV_NEEDED parameter.
We can do this behaviour better by using `has_roles`.
2024-04-04 16:49:42 -07:00
Alex Vandiver efdfaaf5a2 install: Only install node on frontend hosts. 2024-04-04 16:49:22 -07:00
Alex Vandiver acc94a5f32 create-database: Skip stop-server and flush-memcached on standalone Pg.
If running on a stand-alone PostgreSQL server, then supervisor does
exist -- but `stop-server` is useless, and in fact cannot run because
the Zulip directory may not be readable by the `zulip` user.

Detect if this is an application front-end server by looking for
`/home/zulip/deployments`, and use the stop-server and flush-memcached
from there if it exists.  The `create-db.sql` and
`terminate-psql-sessions` files are still read from the local
directory, but those already have precautions from being from a
non-world-readable directory, and are more obviously important to keep
in sync with the `create-database` script.
2024-04-04 16:48:58 -07:00
Anders Kaseorg 0bae651e23 upgrade-zulip-stage-2: Add Ubuntu 20.04, Debian 11 to UNSUPPORTED_DISTROS.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-04-04 16:28:12 -07:00
Anders Kaseorg ad5abbe54f apt-repos: Remove Ubuntu 24.04 gnupg workaround.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-04-04 10:48:22 -07:00
Anders Kaseorg 13e28fc3ac apt-repos: Disallow libmagic1 1:5.45-2 (Ubuntu 24.04) again.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-04-03 11:28:49 -07:00
Anders Kaseorg 6e871e7731 ruff: Fix UP036 Version block is outdated for minimum Python version.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-04-01 18:32:52 -07:00
Anders Kaseorg 70914b0475 Remove support for Ubuntu 20.04 and Debian 11.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2024-04-01 13:27:39 -07:00
Anders Kaseorg 4ebc734632 apt-repos: Temporarily work around Ubuntu 24.04 non-installable gnupg.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-03-31 13:51:31 -07:00
Anders Kaseorg 5d6e616808 Revert "apt-repos: Disallow libmagic1 1:5.45-2 (Ubuntu 24.04)."
This reverts commit 44e38e8ea7.
2024-03-30 11:29:07 -07:00
Anders Kaseorg da984d0ab3 install-node: Upgrade Node.js from 20.11.1 to 20.12.0.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-03-29 09:37:38 -07:00
Anders Kaseorg 708c3e7182 node_cache: Invoke pnpm via corepack.
This avoids prompting the user that corepack is about to download
pnpm.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-03-29 09:37:38 -07:00
Anders Kaseorg 44e38e8ea7 apt-repos: Disallow libmagic1 1:5.45-2 (Ubuntu 24.04).
This package is replaced by libmagic1t64 1:5.45-3 for the Ubuntu
64-bit time_t transition, but hasn’t been deleted from the archive
yet.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-03-28 16:47:30 -07:00
Tim Abbott 8f0f41e236 pythonrc: Add various frequently handy imports. 2024-03-27 12:49:24 -07:00
Anders Kaseorg 9015cd7da3 apt-repos: Use Ubuntu 22.04 Teleport repo on Ubuntu 22.04.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2024-03-25 17:57:24 -04:00
Tim Abbott 2980d14cb4 scripts: Use Apache Arrow from Groonga repository.
This lets us avoid CI failures related to that repository's signing
key having expired. See https://github.com/apache/arrow/issues/40744.
2024-03-24 20:46:24 -07:00
Anders Kaseorg a1a341f0ae ruff: Fix UP032 Use f-string instead of `format` call.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-03-21 08:59:17 -07:00
Alex Vandiver c13e3dee24 katex: Replace subprocess call with minimal external service.
Replace a separate call to subprocess, starting `node` from scratch,
with an optional standalone node Express service which performs the
rendering.  In benchmarking, this reduces the overhead of a KaTeX call
from 120ms to 2.8ms.  This is notable because enough calls to KaTeX in
a single message would previously time out the whole message
rendering.

The service is optional because he majority of deployments do not use
enough LaTeX to merit the additional memory usage (60Mb).

Fixes: #17425.
2024-03-15 15:34:12 -07:00
Alex Vandiver 0f70ab7679 config: Use logic from zulip_tools, to support bools. 2024-03-15 15:34:12 -07:00
Alex Vandiver b36f389e9e zulip_tools: Make default default_value be None, not "". 2024-03-15 15:34:12 -07:00
Alex Vandiver f9f9392b93 zulip_tools: Unify get_config and get_config_bool using overloads. 2024-03-15 15:34:12 -07:00
Alex Vandiver 0ff1fb2e21 zulip_tools: Support non-lowercase true values.
This matches the ruby implementation.
2024-03-15 15:34:12 -07:00
Alex Vandiver 3905416a0e zulip_tools: Fix path to zulipconf.rb.
This was moved in 16aa7c0923.
2024-03-15 15:34:12 -07:00
Alex Vandiver e0cf82a526 email-mirror-postfix: Use get_config_bool, instead of checking "true". 2024-03-15 15:34:12 -07:00
Alex Vandiver 5123477d55 upgrade-zulip-from-git: Provide a flag to use a local branch.
While this could be done previously by calling
`upgrade-zulip-from-git --remote-url /srv/zulip.git`, the explicit
argument makes this more straightforward, and avoids churning the
`refs/remotes/origin/` namespace.
2024-03-08 12:15:25 -08:00
Alex Vandiver b0283c42f9 create-database: Treat "zulip" db without "zerver_messages" as empty.
A user who somehow got an empty `zulip` database, but without a
`zerver_messages` table in it, would get stuck in the installer at:

```
++ su postgres -c 'cd / && psql -v ON_ERROR_STOP=1 -Atc '\''SELECT COUNT(*) FROM zulip.zerver_message;'\'' zulip'
ERROR:  relation "zulip.zerver_message" does not exist
LINE 1: SELECT COUNT(*) FROM zulip.zerver_message;
                             ^
+ records=
```

Treat a failure to select from `zerver_messages` as having 0 messages,
and continue with the `DROP DATABASE IF EXISTS` / `CREATE DATABASE`
that `create-db.sql` usually does.

Fixes: #29110.
2024-03-08 10:10:19 -08:00
Anders Kaseorg 066ea3ebf9 install: Support Ubuntu 24.04.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-03-01 17:38:08 -08:00
Anders Kaseorg 55954c9819 build-pgroonga: Upgrade PGroonga from 3.1.0 to 3.1.8.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-03-01 17:38:08 -08:00
Alex Vandiver 3480d204de reload-clients: Log how many reload events were actually sent. 2024-03-01 09:31:20 -08:00
Alex Vandiver 2d59e07e58 reload-clients: Only sleep between batches.
This avoids an unnecessary sleep at the end of all of the Tornado
shards.
2024-03-01 09:31:20 -08:00
Anders Kaseorg 82a9fd927b ruff: Fix E226 Missing whitespace around arithmetic operator.
This is a preview rule, not yet enabled by default.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-03-01 09:30:04 -08:00
Anders Kaseorg 553f268b04 ruff: Fix RUF027 Possible f-string without an `f` prefix.
This is a preview rule, not yet enabled by default.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-03-01 09:30:04 -08:00
Anders Kaseorg 865febb307 ruff: Fix RUF021 Parenthesize when chaining `and` and `or` together.
This is a preview rule, not yet enabled by default.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-03-01 09:30:04 -08:00
Anders Kaseorg 570f3dd447 python: Reformat with Ruff formatter.
https://docs.astral.sh/ruff/formatter/

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-02-29 17:07:16 -08:00
Alex Vandiver 175e61486e log-search: Auto-limit to the correct logfile when searching datetimes. 2024-02-27 17:04:03 -08:00
Alex Vandiver 9eaaacba52 log-search: Return the filter terms, rather than changing them.
This leaves `args.filter_terms` as the raw values the user specified,
before they may have been transformed to the shape that we use for
substring matching.
2024-02-27 17:04:03 -08:00
Alex Vandiver 39a7c5d106 log-search: Add datetime prefix limit. 2024-02-27 17:04:03 -08:00
Alex Vandiver 281b74a264 log-search: Factor out nginx date conversion. 2024-02-27 17:04:03 -08:00
Alex Vandiver bdc210702c log-search: Document HTTP method limits. 2024-02-27 17:04:03 -08:00
Alex Vandiver 2122780d41 log-search: Fix attempts to limit to events requests in Django logs. 2024-02-26 08:19:57 -08:00
Alex Vandiver b7e42ca17a log-search: Support user-agent / client limit. 2024-02-26 08:19:57 -08:00
Alex Vandiver 9ca0cd9bbd log-search: Remove old /report/ endpoints. 2024-02-26 08:19:57 -08:00
Anders Kaseorg 94cf99bcf2 install-node: Upgrade Node.js from 20.9.0 to 20.11.1.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-02-22 12:46:40 -08:00