Commit Graph

1699 Commits

Author SHA1 Message Date
Alex Vandiver 802477ca1a memcached-exporter: Add metrics for max item size and max connections. 2024-02-27 10:16:00 -08:00
Alex Vandiver 11e0c448da memcached-exporter: Fix descriptions on read/written_bytes_total. 2024-02-27 10:16:00 -08:00
Alex Vandiver be59ef9c02 runtornado: Swap deferred reload events to the default.
This makes no immediate reloads the default for runtornado, matching
the production configuration, and changes the development incantation
to be the one to specify the departure from the norm, with
--immediate-reloads.
2024-02-26 22:29:19 -08:00
Alex Vandiver 8673f87907 puppet: Allow enabling "stats sizes" in memcached. 2024-02-23 13:32:04 -08:00
Alex Vandiver dfdaddea38 kandra: Add a memcached exporter using bmemcached. 2024-02-23 13:32:04 -08:00
David Rosa e891c57d0e docs: Update links to new "Reverse proxies" page. 2024-02-20 21:03:32 -08:00
Alex Vandiver ec6f64f7b0 tornado: Perform rolling client restarts after servers are restarted.
Decouple the sending of client restart events from the restarting of
the servers.  Restarts use the new Tornado restart-clients endpoint to
inject "restart" events into queues of clients which were loaded from
the previous Tornado process.  The rate is controlled by the
`application_server.client_restart_rate`, in clients per minute, or a
flag to `restart-clients` which overrides it.  Note that a web client
will also spread its restart over 5 minutes, so artificially-slow
client restarts are generally not very necessary.

Restarts of clients are deferred to until after post-deploy hooks are
run, such that the pre- and post- deploy hooks are around the actual
server restarts, even if pushing restart events to clients takes
significant time.
2024-02-15 15:42:50 -08:00
Alex Vandiver fa6f4cc039 kandra: Skip hostnames which are undef.
If zulip.conf is not fully configured, do not error out if one of
the $hostname values is undef.
2024-02-07 10:42:12 -08:00
Alex Vandiver 3d63a87384 kandra: Puppet github.com keys to both root and zulip users.
We update to add the ecdsa-sha2-nistp256 key as well.
2024-02-07 10:42:12 -08:00
Alex Vandiver b1f899512a pack-local-script: Match mode of file outside of packing. 2024-02-07 10:42:12 -08:00
Alex Vandiver 927660a7b6 kandra: Remove munin. 2024-02-06 21:34:56 -08:00
Alex Vandiver 96d237d57e kandra: Pull in known_hosts to the nagios user.
This prevents failures when new hosts are initially added to the
configuration.
2024-02-07 00:02:40 -05:00
Alex Vandiver 2cbd9c2a47 kandra: Update default chrony configuration for 22.04. 2024-02-07 00:02:32 -05:00
Alex Vandiver b23d90ed62 puppet: Rename puppet/zulip_ops to puppet/kandra.
This makes for easier tab-completion, and also is a bit more explicit
about the expected consumer.
2024-02-06 17:56:27 -08:00
Alex Vandiver ccc106dca3 puppet: Add redis host to ~redistunnel/.ssh/known_hosts. 2024-02-06 11:34:09 -08:00
Alex Vandiver aaf3eb514d puppet: Allow controlling known_hosts.
Note that this uses `ssh-keyscan` to write in the currently-observed
host fingerprint; if DNS or network is untrusted during initial puppet
apply, this can allow attackers to write their own host key, obviating
the utility of known_hosts.

We do not view this as a likely attack mechanism, since in our
deployment the network and DNS is almost certainly trusted, and if
not, the timing attack to catch only initial configuration is likely
impossible.
2024-02-06 11:34:09 -08:00
Alex Vandiver fe1fd30a58 puppet: Create redistunnel group.
This was left off of 1fe5e7c7da.
2024-02-05 12:15:15 -08:00
Alex Vandiver dc950235e5 puppet: Raise net.core.somaxconn on redis. 2024-02-02 17:54:02 -08:00
Alex Vandiver ec59b4fd78 puppet: Enact redis' kernel config suggestions. 2024-02-02 17:54:02 -08:00
Alex Vandiver 069f262e7b puppet: Inline all sysctl settings, and always check for containers. 2024-02-02 17:54:02 -08:00
Alex Vandiver fd69f5f8e2 puppet: Call systemd reload after adding or changing a teleport service. 2024-02-02 17:54:02 -08:00
Alex Vandiver 1ac6e24eaa puppet: Rename exec resource of systemd_daemon_reload.
This makes it easier to put as a "notify" resource.
2024-02-02 17:54:02 -08:00
Alex Vandiver 4947753bfd puppet: Remove unused systemd_daemon_reload.
572443edc6 removed the callsite that triggered the exec in
`zulip::systemd_daemon_reload`, making its inclusion and ordering via
`require` moot.

Remove the call.
2024-02-02 17:54:02 -08:00
Alex Vandiver 6e9b25d993 puppet: Add key to allow prod to write ccache on zmirrorp. 2024-02-02 17:24:12 -08:00
Alex Vandiver 96b65cbeab install-ssh-authorized-keys: Merge multiple authorized_keys secrets. 2024-02-02 17:24:12 -08:00
Alex Vandiver 1567275959 install-ssh-authorized-keys: Use --check rather than another argument. 2024-02-02 17:24:12 -08:00
Alex Vandiver afa24923e5 install-ssh-authorized-keys: Enable -o pipefail to catch AWS errors. 2024-02-02 17:24:12 -08:00
Alex Vandiver a0347e76cb install-ssh-keys: Enable -o pipefail to catch AWS errors. 2024-02-02 17:24:12 -08:00
Alex Vandiver f8a9edc382 install-ssh-*: Pass -a to rsync to preserve rights, owners, and times. 2024-02-02 17:24:12 -08:00
Alex Vandiver 0bd1e2b434 puppet: Rename and limit production key distribution. 2024-02-02 17:24:12 -08:00
Alex Vandiver d910ea27fe puppet: Allow profiles to override zulip_ops::profile::base. 2024-02-02 17:24:12 -08:00
Tim Abbott a0e7f1296f puppet: Increase minimum memory for multiprocess queue workers.
This should give some more room for systems that are still below 4GB
of RAM to use the lower-memory multithreaded mode, which is less
likely to have OOM kills (a very bad experience).

There should be little cost, as few systems are likely allocated with
memory in this range.
2024-02-02 13:45:25 -08:00
Tim Abbott 43c0c77610 puppet: Update rules for number of uwsgi processes.
The defaults for how many uwsgi processes to run no longer depend on
the queue processor mode, but instead the total memory on the system.
2024-02-02 13:45:25 -08:00
Alex Vandiver 0e6f013e03 puppet: Drop mosh package -- teleport is now the supported login. 2024-01-31 16:41:04 -08:00
Alex Vandiver 16117e6139 puppet: Drop now-unnecessary common-session change. 2024-01-31 16:41:04 -08:00
Alex Vandiver 9810200d78 puppet: Stop writing custom sshd_config.
The only relevant changes are `PasswordAuthentication no` (which
is now the default) and `MaxStartups 40:50:60` (which is now
unneccesary due to autossh tunnels.
2024-01-31 16:41:04 -08:00
Alex Vandiver 1fe5e7c7da puppet: Use a dedicated user for redis tunneling. 2024-01-31 16:41:04 -08:00
Alex Vandiver 795621771f puppet: Pull authorized_keys from AWS secretsmanager. 2024-01-31 16:41:04 -08:00
Alex Vandiver ff00c01538 bootstrap-aws-installer: Pull all keys from secretsmanager. 2024-01-31 16:41:04 -08:00
Alex Vandiver 65d2e855a0 puppet: Switch removal of MOTD directories for .hushlogin.
Rather than have to keep resolving apt conflicts with the lack of
`/etc/update-motd.d`, have each user disable the motds with a
`.hushlogin` file.
2024-01-31 16:41:04 -08:00
Alex Vandiver 1bddf41731 puppet: Factor out creation of basic user dotfiles. 2024-01-31 16:41:04 -08:00
Alex Vandiver 69ef808d7b puppet: Use IAM Roles Anywhere to get AWS credentials outside EC2. 2024-01-31 16:41:04 -08:00
Alex Vandiver 16305761ac puppet: Use IAM join method, when possible. 2024-01-31 16:41:04 -08:00
Alex Vandiver dbb60dbeb9 puppet: Factor out $is_ec2, clarify comments. 2024-01-31 16:41:04 -08:00
Alex Vandiver 6902d5db47 install-aws-cli: Also install and keep up to date using Puppet.
We previously only did this install on the developer machine and on
initial boot.  Also run it from puppet to make sure we keep the binary
up-to-date.
2024-01-31 16:41:04 -08:00
Alex Vandiver e49fc8873b install-aws-cli: Check version before no-op'ing.
We previously just ensured that _a_ version was installed.  Since we
were only using this at initial boot, that was all that was required.
2024-01-31 16:41:04 -08:00
Alex Vandiver bd87f53c86 install-aws-server: Build a tool to smuggle scripts inline in the bootdata. 2024-01-31 16:41:04 -08:00
Alex Vandiver 1a84a20ca0 install-aws-cli: Add a shebang so it is callable directly. 2024-01-31 16:41:04 -08:00
Alex Vandiver 38bf1c5d22 install-aws-cli: Move into puppet files. 2024-01-31 16:41:04 -08:00
Alex Vandiver 7eeec09425 puppet: Include epmd on the list of do-not-auto-restart services.
This brings it into parity with the list of unattended-upgrade
packages.
2024-01-31 16:41:04 -08:00
Alex Vandiver d02354be6c puppet: statuspage-pusher uses zulip.conf for page_id.
This was changed midway through the implementation, from reading it
from `zulip-secrets.conf`, and a couple locations still reference the
secrets path.
2024-01-25 15:37:03 -08:00
Alex Vandiver cd565058cd puppet: Add vector pipelines for other Akamai SQS queues. 2024-01-25 15:36:40 -08:00
Alex Vandiver 39046afbaf nagios: Remove locks alert.
Using locks is not itself a bad sign; there is no need to alert on it,
as there is no value which can be chosen as being "too many."
2024-01-25 13:14:08 -08:00
Alex Vandiver 147fe19c1f puppet: Fix grafana tarball path.
Grafana 10.2.1 and up package their tarball with a `grafana-v10.2.1`
and not `grafana-10.2.1` as previously.
2024-01-25 13:03:05 -08:00
Tim Abbott 004563b380 puppet: Fix bugs in sysctl configuration. 2024-01-23 09:32:15 -08:00
Alex Vandiver 3bf047beb8 iptables: Skip conntrack for DNS queries.
Under heavy request load, it is possible for the conntrack kernel
table to fill up (by default, 256k connections).  This leads to DNS
requests failing because they cannot make a new conntrack entry.

Allow all port-53 UDP traffic in and out without connection tracking.
This means that unbound port-53 traffic is no longer filtered out by
the on-host firewall -- but it is already filtered out at the border
firewall, so this does not change the external network posture.
`systemd-resolve` also only binds to 127.0.0.53 on the loopback
interface, so there is no server to attack on inbound port 53.
2024-01-10 09:07:00 -08:00
Alex Vandiver d18de3e0a4 puppet: Add a knob to adjust conntrack max size. 2024-01-10 09:07:00 -08:00
Alex Vandiver 588aec96f9 puppet: Factor out a sysctl operator. 2024-01-10 09:07:00 -08:00
Alex Vandiver 4da87524ff
nagios: Remove provisioning of zulip contact alias.
fcf096c52e removed the callsite which would have notified this
contact. Note that the source config file was presumably installed via the
python-zulip-api package.
2024-01-09 16:01:07 -08:00
Alex Vandiver b000328ba5 puppet: Adjust uptrack permissions and ownership to match package's.
This reverts a759d26a327cd4337d68eaa1d45d6a69edc9161c; apparently the
package has switched back.
2024-01-09 12:31:02 -08:00
Alex Vandiver 7a6acc9bbf logrotate: smokescreen has its own config file.
149bea8309 added a separate config file
for smokescreen (which is necessary because it can be installed
separately) but failed ot notice that `zulip.template.erb` already had
a config line for it.  This leads to failures starting the logrotate
service:

```
logrotate[4158688]: error: zulip:1 duplicate log entry for /var/log/zulip/smokescreen.log
logrotate[4158688]: error: found error in file zulip, skipping
```

Remove the duplicate line.
2024-01-08 14:05:01 -08:00
Alex Vandiver 1ba2f39854 install: Support PostgreSQL 16. 2023-12-23 14:57:12 -08:00
Anders Kaseorg bac027962f models: Extract zerver.models.clients.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-12-16 22:08:44 -08:00
Anders Kaseorg cd96193768 models: Extract zerver.models.realms.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-12-16 22:08:44 -08:00
Anders Kaseorg 45bb8d2580 models: Extract zerver.models.users.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-12-16 22:08:44 -08:00
Alex Vandiver 4989221b9e nginx: Limit the methods that we proxy to Tornado.
While the Tornado server supports POST requests, those are only used
by internal endpoints.  We only support OPTIONS, GET, and DELETE
methods from clients, so filter everything else out at the nginx
level.

We set `Accepts` header on both `OPTIONS` requests and 405 responses,
and the CORS headers on `OPTIONS` requests.
2023-12-08 09:23:30 -08:00
Alex Vandiver ca57d360e6 puppet: Update dependencies. 2023-12-07 18:45:10 -08:00
Anders Kaseorg 3853fa875a python: Consistently use from…import for urllib.parse.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-12-05 13:03:07 -08:00
Anders Kaseorg 8a7916f21a python: Consistently use from…import for datetime.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-12-05 12:01:18 -08:00
Alex Vandiver 9b1bdfefcd nagios: Use a better index on UserActivity for zephyr alerting.
Limiting only by client_name and query leads to a very poorly-indexed
lookup on `query` which throws out nearly all of its rows:

```
Nested Loop  (cost=50885.64..60522.96 rows=821 width=8)
  ->  Index Scan using zerver_client_name_key on zerver_client  (cost=0.28..2.49 rows=1 width=4)
        Index Cond: ((name)::text = 'zephyr_mirror'::text)
  ->  Bitmap Heap Scan on zerver_useractivity  (cost=50885.37..60429.95 rows=9052 width=12)
        Recheck Cond: ((client_id = zerver_client.id) AND ((query)::text = ANY ('{get_events,/api/v1/events}'::text[])))
        ->  BitmapAnd  (cost=50885.37..50885.37 rows=9052 width=0)
              ->  Bitmap Index Scan on zerver_useractivity_2bfe9d72  (cost=0.00..16631.82 rows=..large.. width=0)
                    Index Cond: (client_id = zerver_client.id)
              ->  Bitmap Index Scan on zerver_useractivity_1b1cc7f0  (cost=0.00..34103.95 rows=..large.. width=0)
                    Index Cond: ((query)::text = ANY ('{get_events,/api/v1/events}'::text[]))
```

A partial index on the client and query list is extremely effective
here in reducing PostgreSQL's workload; however, we cannot easily
write it as a migration, since it depends on the value of the ID of
the `zephyr_mirror` client.

Since this is only relevant for Zulip Cloud, we manually create the
index:

```sql
CREATE INDEX CONCURRENTLY zerver_useractivity_zehpyr_liveness
    ON zerver_useractivity(last_visit)
 WHERE client_id = 1005
   AND query IN ('get_events', '/api/v1/events');
```

We rewrite the query to do the time limit, distinct, and count in SQL,
instead of Python, and make use of this index.  This turns a 20-second
query into two 10ms queries.
2023-11-30 16:01:55 -08:00
Alex Vandiver c4b619af15 puppet: Change /etc/rabbitmq to be owned by rabbitmq.
The Ubuntu and Debian package installation scripts for
`rabbitmq-server` install `/etc/rabbitmq` (and its contents) owned by
the `rabbitmq` user -- not `root` as Puppet does.  This means that
Puppet and `rabbitmq-server` unnecessarily fight over the ownership.

Create the `rabbitmq` user and group, to the same specifications that
the Debian package install scripts do, so that we can properly declare
the ownership of `/etc/rabbitmq`.
2023-11-29 21:45:35 -08:00
Alex Vandiver c47ee4a296 zulip_ops: Configure stats to be pushed to status.zulip.com. 2023-11-16 16:21:12 -05:00
Alex Vandiver 5e49804004 puppet_ops: Include Akamai log parser on prometheus server. 2023-11-13 14:35:39 -05:00
Alex Vandiver 5591d6f65c zulip_ops: Add configuration for Vector Akamai stats.
Akamai writes access logs to S3; we use an SQS events queue, combined
with Vector, to transform those into Prometheus statistics.
2023-11-13 09:53:20 -08:00
Tim Abbott b59e90d100 puppet: Fix buggy media-src Content-Security-Policy.
The colon is invalid syntax. Verified the updated policy using an
online CSP checker.
2023-11-06 14:45:05 -05:00
Alex Vandiver 803b7b4b93 puppet: Fix SHA256sum of sentry-cli binary. 2023-10-31 10:24:49 -07:00
Alex Vandiver 37b261ef0f puppet: Update dependencies. 2023-10-30 16:10:25 -07:00
Aman Agrawal f3ab45a152 uploads-internal: Mark `self` as a valid source of loading media.
Without this, browser refused to play the video. To reproduce press `open`
on an uploaded video on CZO. Chrome gives us the following error
in console:

Refused to load media from '<source>' because it violates the
following Content Security Policy directive: "default-src 'none'".
Note that 'media-src' was not explicitly set, so 'default-src' is
used as a fallback.
2023-10-12 09:57:21 -07:00
Anders Kaseorg 835ee69c80 docs: Fix grammar errors found by mwic.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-10-09 13:24:09 -07:00
Anders Kaseorg 4cb2eded68 typos: Fix typos caught by typos.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-10-09 11:55:16 -07:00
Alex Vandiver 528d0ebcf0 puppet: Serve /etc/zulip/well-known/ in nginx as /.well-known/. 2023-10-04 15:56:42 -07:00
Aman Agrawal 8ef52d55d3 markdown: Add support for inline video thumbnails. 2023-10-02 22:39:02 -07:00
Alex Vandiver 5308fbdeac puppet: Add postgresql-client depenencies to monitoring.
The `unless` step errors out if /usr/bin/psql does not exist at
first evaluation time -- protect that with a `test -f` check, and
protect the actual `createuser` with a dependency on `postgresql-client`.
To work around `Zulip::Safepackage` not actually being safe to
instantiate more than once, we move the instantiation of
`Package[postgresql-client]` into a class which can be safely
included one or more times.
2023-09-22 11:45:00 -07:00
Alex Vandiver 5ee4b642ad views: Add a /health healthcheck endpoint.
This endpoint verifies that the services that Zulip needs to function
are running, and Django can talk to them.  It is designed to be used
as a readiness probe[^1] for Zulip, either by Kubernetes, or some other
reverse-proxy load-balancer in front of Zulip.  Because of this, it
limits access to only localhost and the IP addresses of configured
reverse proxies.

Tests are limited because we cannot stop running services (which would
impact other concurrent tests) and there would be extremely limited
utility to mocking the very specific methods we're calling to raising
the exceptions that we're looking for.

[^1]: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/
2023-09-20 09:53:59 -07:00
Alex Vandiver f778316b5a uwsgi: Ensure that the master process cannot load the application.
The rolling restart configuration of uwsgi attempted to re-chdir the
CWD to the new `/home/zulip/deployments/current` before `lazy-apps`
loaded the application in the forked child.  It successfully did so --
however, the "main" process was still running in the original
`/home/zulip/deployments/current`, which somehow (?) tainted the
search path of the children processes.

Set the parent uwsgi process to start in `/`, so that the old deploy
directory cannot taint the load order of later children processes.
2023-09-18 13:13:34 -07:00
Alex Vandiver a6d5d7740e uwsgi: Always enable lazy-apps.
Enabling `lazy-apps` defers loading of the uwsgi application until
after the fork, instead of happening prior to forking workers[^1].  The
nominal reason to not enable this is that it increases the memory
footprint of the server (since no memory is shared across processes),
and may slow down worker initialization, since each worker has to load
the files from disk.

However, Django defers loading the majority of the code until the
first request[^2].  As such, our current non-`lazy-apps` gains nothing
over `lazy-apps`.  For consistency, switch to using `lazy-apps` for
all deployments, rolling restart or no.

[^1]: https://uwsgi-docs.readthedocs.io/en/latest/articles/TheArtOfGracefulReloading.html#preforking-vs-lazy-apps-vs-lazy
[^2]: https://uwsgi-docs.readthedocs.io/en/latest/articles/TheArtOfGracefulReloading.html#preforking-vs-lazy-apps-vs-lazy
2023-09-18 13:13:34 -07:00
Alex Vandiver f95c8b894a nagios: Remove load monitoring.
Load monitoring alerts are extremely noisy, and do not reliably
indicate an issue which is affecting users.
2023-09-14 09:29:29 -07:00
Anders Kaseorg 2665a3ce2b python: Elide unnecessary list wrappers.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-09-13 12:41:23 -07:00
Alex Vandiver 135acfea93 nginx: Suppress proxy warnings when the proxy itself sent the request.
This is common in cases where the reverse proxy itself is making
health-check requests to the Zulip server; these requests have no
X-Forwarded-* headers, so would normally hit the error case of
"request through the proxy, but no X-Forwarded-Proto header".

Add an additional special-case for when the request's originating IP
address is resolved to be the reverse proxy itself; in these cases,
HTTP requests with no X-Forwarded-Proto are acceptable.
2023-09-12 10:10:58 -07:00
Alex Vandiver ccbd834a86 postgres_exporter: Rebase the per-index stats branch.
The branch from the PR is somewhat stale, and is missing important bugfixes.
2023-09-11 17:59:54 -07:00
Alex Vandiver 0c88cfca63 postgres_exporter: Build from source for per-index stats.
This builds prometheus-community/postgres_exporter#843 to track
per-index statistics.
2023-09-11 11:59:39 -07:00
Alex Vandiver fdd811bec1 postgres_exporter: Explicitly specify the zulip database.
Some of the collectors (e.g. `pg_stat_user_tables`) don't appear to
work with `--auto-discover-databases`, which is deprecated since
version 0.13.0[^1].

Explicitly set the database name.

[^1]: https://github.com/prometheus-community/postgres_exporter/releases/tag/v0.13.0
2023-09-06 09:20:57 -07:00
Alex Vandiver 5d3ce8b2d4 puppet: Update dependencies. 2023-09-06 09:20:06 -07:00
Alex Vandiver f8636e7d2b iptables: Stop logging on dropped packets.
We never examine these logs, and it fills dmesg.  We have flow logging at the AWS stack layer.
2023-08-30 15:29:01 -07:00
Alex Vandiver e8c8544028 nginx: Do not forward X-amz-cf-id header to S3.
All `X-amz-*` headers must be included in the signed request to S3;
since Django did not take those headers into account (it constructed a
request from scratch, while nginx's request inherits them from the
end-user's request), the proxied request fails to be signed correctly.

Strip off the `X-amz-cf-id` header added by CloudFront.  While we
would ideally strip off all `X-amz-*` headers, this requires a
third-party module[^1].

[^1]: https://github.com/openresty/headers-more-nginx-module#more_clear_input_headers
2023-08-28 12:30:14 -07:00
Anders Kaseorg c43629a222 ruff: Fix PLW1510 `subprocess.run` without explicit `check` argument.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-08-17 17:05:34 -07:00
Alex Vandiver c5cace3600 puppet: Fix includes for new name of zulip_ops::prometheus::tornado.
This fixes the `include` name for the file renamed in 740a494ba4.
2023-08-09 02:32:28 +00:00
Anders Kaseorg 0b95d83f09 ruff: Fix PERF402 Use `list` or `list.copy` to create a copy of a list.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-08-07 17:23:55 -07:00
Alex Vandiver 740a494ba4 puppet: Rename and generalize Tornado process exporter.
Exporting stats about all of the various Zulip processes is useful for
tracking memory leaks, etc.
2023-08-06 13:41:10 -07:00
Anders Kaseorg 211934a9d9 nginx: Remove gzip_disable "msie6".
We don’t support IE 6.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-07-20 13:09:53 -07:00