Commit Graph

1731 Commits

Author SHA1 Message Date
Tim Abbott 54c2c02011 thumbnail: Add support for multiple queue workers.
There's no need for sharding, but this allows one to spend a bit of
extra memory to reduce image-processing latency when bursts of images
are uploaded at once.
2024-07-21 19:15:43 -07:00
Anders Kaseorg f3c67103cf memcached_exporter: Explicitly close memcached connection.
Fixes warnings like “ResourceWarning: unclosed <socket.socket fd=5,
family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0,
laddr=('127.0.0.1', 39702), raddr=('127.0.0.1', 11211)>” with warnings
enabled.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-16 14:50:09 -07:00
Alex Vandiver 2e38f426f4 upload: Generate thumbnails when images are uploaded.
A new table is created to track which path_id attachments are images,
and for those their metadata, and which thumbnails have been created.
Using path_id as the effective primary key lets us ignore if the
attachment is archived or not, saving some foreign key messes.

A new worker is added to observe events when rows are added to this
table, and to generate and store thumbnails for those images in
differing sizes and formats.
2024-07-16 13:22:15 -07:00
Alex Vandiver bf9539386a katex: Default the server on.
The memory costs are low (~60MB), and likely worth the stability.
2024-07-15 13:16:31 -07:00
Anders Kaseorg a73b8a34d9 supervisor: Fix katex_server.js path for tarball deployments.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-15 13:16:31 -07:00
Anders Kaseorg 0fa5e7f629 ruff: Fix UP035 Import from `collections.abc`, `typing` instead.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-13 22:28:22 -07:00
Anders Kaseorg 531b34cb4c ruff: Fix UP007 Use `X | Y` for type annotations.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-13 22:28:22 -07:00
Anders Kaseorg e08a24e47f ruff: Fix UP006 Use `list` instead of `List` for type annotation.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-13 22:28:22 -07:00
Lauryn Menard 80b86c7b16 documentation: Fix "you you" typos in docs/comments/logs. 2024-07-02 11:57:44 -07:00
Alex Vandiver 6a811cb306 process_exporter: Use -recheck-with-time-limit to catch process renames. 2024-05-30 22:24:53 -07:00
Alex Vandiver 2e6504cab7 puppet: Upgrade dependencies. 2024-05-30 22:24:53 -07:00
Alex Vandiver a6d2112453 kandra: Run rabbitmq cron jobs as root.
This is a regression introduced in
f246b82f67.
2024-05-28 15:15:53 -07:00
Alex Vandiver d2516607bb puppet: Silence "needrestart" nags about kernel upgrades.
Ksplice keeps the kernel updated without restarts.
2024-05-28 14:34:26 -07:00
Laura Hausmann 5edcc209c3 nginx: Send SNI for proxied S3 requests.
Some S3 backends (e.g. garage or minio behind caddy) are unable to
respond to TLS requests that only have the Host header set. This makes
sure those configurations are supported going forward.
2024-05-28 16:30:44 -04:00
Alex Vandiver 549f4fe00b nginx: Strip off request headers which might affect S3's behaviour.
Clients making requests to Zulip with a `Authorization: Basic ...` for
an upload in S3 pass along all of their request headers to the S3
backend -- causing errors of the form:

```xml
<?xml version="1.0" encoding="UTF-8"?>
<Error>
  <Code>InvalidArgument</Code>
  <Message>Only one auth mechanism allowed; only the X-Amz-Algorithm
  query parameter, Signature query string parameter or the
  Authorization header should be specified</Message>
  <ArgumentName>Authorization</ArgumentName>
  <ArgumentValue>Basic ...</ArgumentValue>
  <RequestId>...</RequestId>
  <HostId>...</HostId>
</Error>
```

Strip off all request headers which AWS reports that S3 may read[^1].

Fixes: #30180.

[^1]: https://docs.aws.amazon.com/AmazonS3/latest/API/RESTCommonRequestHeaders.html
2024-05-28 12:04:19 -07:00
Alex Vandiver c98bf184bb postfix: Increase the max message size to 25MB.
The default is 10MB[^1]; increasing this to 25MB aligns it with the
email mirror's internal limit.

[^1]: https://www.postfix.org/postconf.5.html#message_size_limit
2024-05-24 14:54:29 -07:00
Alex Vandiver c7f42de8ba cron: Monitor "command"s with sentry, now that they are one command.
These no longer involve any shell, so are safe to prepend with the
sentry wrapper.
2024-05-24 11:31:25 -07:00
Alex Vandiver f246b82f67 puppet: Factor out pattern of writing a nagios state file atomically. 2024-05-24 11:31:25 -07:00
Alex Vandiver 230040caa9 puppet: Remove check_postgresql_backup.
We have replaced this monitoring with the black-box wal-g monitoring,
which is more accurate.
2024-05-24 11:27:59 -07:00
Alex Vandiver 2218de0399 management: Disable Sentry for management commands run interactively.
This adds `--automated` and `--no-automated` flags to all Zulip
management commands, whose default is based on if STDIN is a TTY.
This enables cron jobs and supervisor commands to continue to report
to Sentry, and manually-run commands (when reporting to Sentry does
not provide value, since the user can see them) to not.

Note that this only applies to Zulip commands -- core Django
commands (e.g. `./manage.py`) do not grow support for `--automated`
and will always report exceptions to Sentry.

`manage.py` subcommands in the `upgrade` and `restart-server` paths
are marked as `--automated`, since those may be run semi-unattended,
and they are useful to log to Sentry.
2024-05-24 10:30:16 -07:00
Alex Vandiver cf24d2c25e check_send_receive_time: Use time.perf_counter() for duration timing. 2024-05-08 15:51:20 -07:00
Alex Vandiver f42153f670 check_send_receive_time: Use machine.deploy_type to check "staging".
This brings it in line with other locations (e.g. Sentry) and is less
likely to have accidental false-positives.
2024-05-08 15:51:20 -07:00
Alex Vandiver 8bdf1e4a10 check_send_receive_time: Move "states" to inside where they are used. 2024-05-08 15:51:20 -07:00
Alex Vandiver 04e21044b9 check_send_receive_time: Default --site usefully.
This saves us the time of shelling out to a new python process,
loading all of Django, and printing one value we could just have read
in-process.  It is unclear why we ever did it this way.
2024-05-08 15:51:20 -07:00
Alex Vandiver cbc9065ed2 check_send_receive_time: Remove no-longer-used "config" option.
This become unused in 927660a7b6.
2024-05-08 15:51:20 -07:00
Alex Vandiver 2bd60e8562 check_send_receive_time: Print no output on success. 2024-05-08 15:51:20 -07:00
Alex Vandiver 294fd914e1 kandra: Fix cron specification, to run once, not every minute from 7-8. 2024-05-08 12:16:36 -07:00
Alex Vandiver 908a805d3e kandra: Change the remaining check_send_receive_time to use zulip::cron.
This was mistakenly left off of 6e981c18d5.
2024-05-08 09:00:45 -07:00
Alex Vandiver e7511d43b8 kandra: Put the use_proxy override on the job it belongs on. 2024-05-08 09:00:45 -07:00
Alex Vandiver f960f5d9fd puppet: Fix a typo in zulip::cron.
This went undetected because the clause is only used on kandra hosts.
2024-05-08 09:00:45 -07:00
Tim Abbott 0a756c652c push_notifications: Shard mobile push notifications. 2024-05-02 14:25:10 -07:00
Alex Vandiver 9dfaa83aa8 invites: Remove invites worker, make confirmation object in-process.
The "invites" worker exists to do two things -- make a Confirmation
object, and send the outgoing email.  Making the Confirmation object
in a background process from where the PreregistrationUser is created
temporarily leaves the PreregistrationUser in invalid state, and
results in 500's, and the user not immediately seeing the sent
invitation.  That the "invites" worker also wants to create the
Confirmation object means that "resending" an invite invalidates the
URL in the previous email, which can be confusing to the user.

Moving the Confirmation creation to the same transaction solves both
of these issues, and leaves the "invites" worker with nothing to do
but send the email; as such, we remove it entirely, and use the
existing "email_senders" worker to send the invites.  The volume of
invites is small enough that this will not affect other uses of that
worker.

Fixes: #21306
Fixes: #24275
2024-05-02 14:23:04 -07:00
Alex Vandiver 11dd6791c4 management: Provide a common lockfile dir, and a decorator for it.
Factor out the repeated pattern of taking a lock, or immediately
aborting with a message if it cannot be acquired.  The exit code in
that situation is changed to be exit code 1, rather than the successful
0; we are likely missing new work since that process started.

We move the lockfiles to a common directory under `/srv/zulip-locks`
rather than muddy up `/home/zulip/deployments`.
2024-04-24 14:40:28 -07:00
Alex Vandiver 572fafd6b9 cron: Set environment variables to use Smokescreen, if configured. 2024-04-24 14:40:28 -07:00
Alex Vandiver f355a08111 sentry: Configure cron job watching, if sentry.project_dsn is set. 2024-04-24 14:40:28 -07:00
Alex Vandiver 528d475053 puppet: Factor out sentry-cli installation. 2024-04-24 14:40:28 -07:00
Alex Vandiver 2df91c70ef puppet: Move rabbitmq monitoring into kandra/, where it is used from. 2024-04-24 14:40:28 -07:00
Alex Vandiver 6e981c18d5 puppet: Factor out cron job creation. 2024-04-24 14:40:28 -07:00
Alex Vandiver c607594d11 puppet: Re-indent. 2024-04-24 14:40:28 -07:00
Alex Vandiver 503bddab9c cron: Remove stale comment.
122d0bca83 removed the hour-specific exception.
2024-04-24 14:40:28 -07:00
Alex Vandiver b298d08fda wal-g: Support parameterizing the bucket which is fetched. 2024-04-24 09:04:16 -07:00
Alex Vandiver 49422c05c8 kandra: Add teleport services to "needsrestart" skip list.
These are often how one is connected to the node, and restarting them
would drop the connection one us actively using.
2024-04-19 09:55:17 -07:00
Alex Vandiver a4e6037dc4 kandra: Automate the second step of configuring database replicas.
If there is a replication primary configured, and no current database,
then we check all of the required secrets are in place, then pull down
the latest backup and trigger a PostgreSQL restart, which will pick up
downloading the remaining WAL logs to catch up, then start streaming
from the configured primary.
2024-04-17 17:31:49 -07:00
Alex Vandiver a8dbdd6d92 kandra: Ensure that pg_hba.conf is in place before starting PostgreSQL. 2024-04-17 17:31:49 -07:00
Alex Vandiver 311dfdaa38 puppet: Only restart PostgreSQL if there is a datadir to read.
This is specifically to support Kandra's `setup_disks`, which stops
PostgreSQL and moves the data directory out of the way while mounting
a new disk; restarting PostgreSQL would fail in this state.  We
install secrets and re-run puppet to finish bootstrapping the
database, all of which expects the PostgreSQL server to be stopped
anyways.
2024-04-17 17:31:49 -07:00
Alex Vandiver 4be2abc55b puppet: If needed, ensure wal-g is installed before we start PostgreSQL.
PostgreSQL will need to use wal-g to pull needed WAL files.  We do not
express this as a direct dependency because it is possible to have
wal-g without PostgreSQL, as well as PostgreSQL without wal-g.
2024-04-17 17:31:49 -07:00
Alex Vandiver b0e3191434 puppet: Stop relying on "tidy" ordering, which ignores metaparams.
The `tidy` parameter is buggy, and ignores all ordering
metaparameters.  This is fixed in Puppet 7[^1], but it's helpful to
resolve it now.  Specifically, this fixes bugs with tidy running too
early, and deleting the old version of a package before its new
version is installed or symlinked, leaving a race condition if
anything tries to run the binary in this window.

This is mostly not a problem for Supervisor-managed processes, since
the binary is already running, and can continue to run if it is tidied
out from under the running process.  For stand-alone tools like wal-g,
which are run frequently by PostgreSQL, this may cause issues if
PostgreSQL tries to call them during a puppet run.

Remove all complicated uses of tidy, and replace them with an `exec`
which does the equivalent.  We also generate `file` resources for
binaries, making them easier (and clearer) to specify as dependencies.

[^1]: https://puppet.atlassian.net/browse/PUP-10688
2024-04-15 14:30:24 -07:00
Alex Vandiver f7bc881ca3 puppet: Update dependencies. 2024-04-12 15:06:42 -07:00
Alex Vandiver 8d3120bf17 kandra: Remove zulip::static_asset_compiler include.
This was removed in 263212decf.
2024-04-12 15:00:33 -07:00
Alex Vandiver 30f71639f0 pg_backup_and_purge: Properly preserve needed base backups.
Without `FIND_FULL`, `wal_g delete before ...` will fail, rather than
delete a base backup which is needed by the delta backups after it.
By passing `FIND_FULL`[^1], we tell it explicitly that we're OK
preserving files before the specified one, as long as they are
necessary for the delta chain.

[^1]: https://github.com/wal-g/wal-g/blob/master/docs/README.md#delete
2024-04-12 11:39:54 -07:00