Commit Graph

838 Commits

Author SHA1 Message Date
Anders Kaseorg 263d71bf2b nginx: Add CORS headers to /user_uploads.
Fixes: #12980.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-12 15:35:35 -07:00
Anders Kaseorg 2e57f3ffae puppet: “Resolve” puppet-lint warnings.
Introduced by #12966.

puppet/zulip/manifests/base.pp - WARNING: double quoted string containing no variables on line 93
puppet/zulip/manifests/base.pp - WARNING: string containing only a variable on line 93

scanf doesn’t accept a number as input, so uh, add a dummy space
character.

What.  You can’t give me a bad language and then complain when I write
bad programs in it.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-12 15:29:23 -07:00
Anders Kaseorg 820165e4da Merge pull request #12968 from andersk/ffdhe2048
nginx: Use fixed ffdhe2048 DH parameter (RFC 7919)
2019-08-09 16:29:10 -07:00
Anders Kaseorg 4e9fb05c4f puppet: Use built-in memorysize_mb fact.
Fixes this warning:

    Warning: The string '8167976' was automatically coerced to the numerical value 8167976 (file: /root/zulip/puppet/zulip/manifests/base.pp, line: 93, column: 19)

Fixes #9682.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 16:28:09 -07:00
Tim Abbott de0a41bc9c provision: Fix missing dependency on unzip.
Because this is often installed by default, we hadn't noticed that our
Slack importer doesn't run without it.

Thanks to Ray Kraesig for the report.
2019-08-08 10:49:20 -07:00
Anders Kaseorg 0962393933 cleanup: Delete trailing newlines.
Delete trailing newlines from all files, except
tools/ci/success-http-headers.txt and tools/setup/dev-motd, where they
are significant, and static/third, where we want to stay close to
upstream.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-06 23:29:11 -07:00
Anders Kaseorg becef760bf cleanup: Delete leading newlines.
Previous cleanups (mostly the removals of Python __future__ imports)
were done in a way that introduced leading newlines.  Delete leading
newlines from all files, except static/assets/zulip-emoji/NOTICE,
which is a verbatim copy of the Apache 2.0 license.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-06 23:29:11 -07:00
Anders Kaseorg 68dd8e4ec8 mypy: Migrate from mypy_extensions to typing_extensions.
This gives us access to typing_extensions.Deque, which was not added
to typing until 3.5.4.

(PROVISION_VERSION is not bumped because the transitive dependency set
in dev.txt hasn’t changed.)

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-05 17:24:09 -07:00
Wyatt Hoodes a109508e34 typing: Remove now-unnecessary conditional import.
As a result of dropping support for trusty, we can remove our old
pattern of putting `if False` before importing the typing module,
which was essential for Python 3.4 support, but not required and maybe
harmful on newer versions.

cron_file_helper
check_rabbitmq_consumers
hash_reqs
check_zephyr_mirror
check_personal_zephyr_mirrors
check_cron_file
zulip_tools
check_postgres_replication_lag
api_test_helpers
purge-old-deployments
setup_venv
node_cache
clean_venv_cache
clean_node_cache
clean_emoji_cache
pg_backup_and_purge
restore-backup
generate_secrets
zulip-ec2-configure-interfaces
diagnose
check_user_zephyr_mirror_liveness
2019-07-29 15:18:22 -07:00
Anders Kaseorg b758ed5ac1 nginx: Remove invalid extra headers for OPTIONS /api/v1/events.
Since 204 responses don’t contain a payload body, Content-Type is
neither required nor encouraged (RFC 7231 §3.1.1.5), and ours was
missing a semicolon to boot; Content-Length is expressly
forbidden (RFC 7230 §3.3.2).

Furthermore, these add_header directives were silencing the CORS
headers set in api_headers, because add_header inheritance doesn’t
work the way you think it does, as was known before commit
5614d51afc.

Fixes: #12902.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-29 14:58:35 -07:00
Anders Kaseorg 6d5a20ac62 requirements: Remove django-pipeline.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-24 17:40:31 -07:00
Tim Abbott 2eb855b302 puppet: Include . separator in email mirror rules.
This is required for the postfix-localmail integration to use the new
`.` format email addresses.
2019-07-22 11:13:36 -07:00
Wyatt Hoodes e331a758c3 python: Migrate open statements to use with.
This is low priority, but it's nice to be consistently using the best
practice pattern.

Fixes: #12419.
2019-07-20 15:48:52 -07:00
Anders Kaseorg c97ca677c9 nginx: Update TLS settings based on Mozilla recommendations 5.0.
Disable TLS 1.0 and TLS 1.1.  (We no longer need to support IE8 on
Windows XP.)

Prefer client-selected cipher order.  (Now that all enabled ciphers
provide good security, this allows mobile clients lacking AES hardware
acceleration to pick ChaCha20 for better performance.)

Disable session tickets.  (Mozilla discourages them based on
https://www.imperialviolet.org/2013/06/27/botchingpfs.html.)

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-08 15:51:02 -07:00
Anders Kaseorg 079ddae4c8 minify-js: Remove; everything has been migrated to Webpack.
min/sockjs-0.3.4.min.js is not used.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-03 13:58:21 -07:00
Tim Abbott aea1279e8c puppet: Remove trusty configuration for static_asset_compiler.
Trusty is desupported.
2019-06-26 11:32:06 -07:00
Tim Abbott 8fbd965ab5 puppet: Remove legacy pgtune related configuration for trusty.
Since we no longer support Ubuntu Trusty, we no longer need this
backwards-compatibility cruft (which we only kept around to avoid
randomizing configuration for existing systems).
2019-06-26 11:32:06 -07:00
Anders Kaseorg 33c941407b puppet: Remove legacy unauthenticated local uploads backend.
This was only used in Ubuntu 14.04 Trusty.

Removing this also finally lets us simplify our security model
discussion of uploaded files.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-06-26 11:31:46 -07:00
Tim Abbott 271319fb13 puppet: Fix hacky release test for whether we're in EC2.
The result is still a bit hacky, but guaranteed to be correct if we
adjust the OS version of our systems, which we of course will do over
time.
2019-06-25 22:19:04 -07:00
Tim Abbott 8d8cfb314b puppet: Remove zulip_ops configuration for trusty.
There are no longer any zulip_ops systems using trusty.
2019-06-25 22:09:06 -07:00
Tim Abbott 88b77af54f puppet: Add support for changing the nginx port directly.
This provides a clean process for changing Zulip's nginx port.
2019-06-17 12:24:22 -07:00
Fabian Stanke 51ba9ddd89 postfix: Inserted compulsory setting for postfix ≥ 2.10.
One of smtpd_relay_restrictions or smtpd_recipient_restrictions is
required by postfix ≥ 2.10 (see
http://www.postfix.org/SMTPD_ACCESS_README.html).

This is important for using the email mirror on Ubuntu Bionic.
2019-06-16 18:48:39 -07:00
Tim Abbott b41c2d93d1 puppet: Exclude squashfs filesystems from nagios disk checks.
These generally aren't being written to.
2019-06-16 16:22:23 -07:00
Tim Abbott 0ec1b4e82c puppet: Move check_send_receive_time to the _once ruleset.
We don't actually want to run this bundle of message-sending Nagios
checks to run on every single server.
2019-06-16 15:48:35 -07:00
Tim Abbott df83979c76 zulip_ops: Extract a prod_app_frontend_once ruleset. 2019-06-16 15:48:35 -07:00
Tim Abbott 738cfe54c3 puppet: Move app_frontend_once out of prod configuration.
That logic made it inconvenient to run multiple prod servers with the
same top-level puppet configuration.
2019-06-16 15:24:20 -07:00
Tim Abbott e85250941d puppet: Fix quoting of commented-out python3-boto.
This will avoid a linter error if/when we uncomment it.
2019-06-13 14:39:24 -07:00
Tim Abbott 337efe0fb7 puppet: Remove puppet-el, which no longer exists.
This package was only every available on Ubuntu Xenial.
2019-06-13 14:39:24 -07:00
Tim Abbott afb0d1ccce Revert "puppet: Use nice to deprioritize various processes."
This reverts commit d959de7a89.

This broken our Travis CI, so I'm pulling it off while we investigate.
2019-06-05 12:55:56 -07:00
Tim Abbott d959de7a89 puppet: Use nice to deprioritize various processes.
Our priority hierarchy is:
(1) Tornado and base services like memcached, redis, etc.
(2) Django and message sender queue workers.
(3) Everything else.

Ideally, we'd have something a bit more fine-grained (e.g. some queue
workers are potentially in the sending path, while others aren't), but
this should have a big impact on ensuring Tornado gets the resources
it needs during load spikes.

I think this has a good chance of causing some load spikes that would
previously have resulted in a user-facing delivery delays no longer
having any significant user-facing impact.
2019-06-05 11:56:48 -07:00
Tim Abbott cd1ec37404 puppet: Make uwsgi listen backlog limit configurable.
This can be useful for busy servers to limit the risk of bursts of
traffic causing them to reject requests.
2019-05-17 12:38:56 -07:00
Tim Abbott ca48b4ec9f puppet: Set postgres max_connections to 1000.
There isn't much legitimate reason to have a limit as low as 100,
given how few resources a connection consumes.
2019-05-13 17:19:31 -07:00
Tim Abbott b7d50190b7 process_fts_updates: Batch updates when catching up.
Previously, if process_fts_updates ended up very far behind
(e.g. 100,000s of messages), it was unable to recover without doing
some very expensive databsae operations to fetch and then delete the
list of message IDs needing updates.  This change fixes that issue by
doing the catch-up work in batches.
2019-05-09 22:44:07 -07:00
Vishnu Ks ecdd3bea43 billing: Add cron job to run invoice_plans once a day.
Fixes #11960
2019-04-29 11:23:17 -07:00
Anders Kaseorg 643bd18b9f lint: Fix code that evaded our lint checks for string % non-tuple.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-04-23 15:21:37 -07:00
Anders Kaseorg 5290519a62 scripts: Always use ON_ERROR_STOP=1 when running psql.
Also use psql -e (--echo-queries) in scripts that use ‘set -x’, so
errors can be traced to a specific query from the output.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-04-22 14:54:19 -07:00
Puneeth Chaganti 9876f1b14e check_rabbitmq_queue: Fix the time period when we ignore long queues.
The commit 87d1809657 changed the time when
digests are sent by 3 hours to account for moving from the US East Coast to the
West Coast, but didn't change the time period exception in the
`check-rabbitmq-queue` script.

Closes #5415
2019-04-13 20:43:07 -07:00
Anders Kaseorg 9f7c0b7e65 postgres_master.pp: Fix wacky su command line.
The construction `su postgres -c -- bash -c 'psql …'` didn’t behave the
way it reads, and only worked by accident:

1. `-c --` sets the command to `--`.
2. `bash` sets the first argument to `bash`.
3. `-c 'psql …'` replaces the command with `psql …`.

Thus, `su` ended up executing `<shell> -c 'psql …' bash`, where
`<shell>` is the `postgres` user’s login shell, usually also `bash`,
which then executed 'psql …' and ignored the extra `bash`.

Unconfuse this construction.

Note from tabbott: The old code didn't even work by accident, it was
just broken.  The right fix is to move the quoting around properly.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-04-12 17:27:23 -07:00
Tim Abbott b1da797955 puppet: Make uwsgi buffer size configurable. 2019-03-18 22:43:59 -07:00
Anders Kaseorg fd6f18f7cf nginx: Improve TLS settings based on Mozilla config generator.
Lengthen the session timeout and enlarge the session cache.  Upgrade
Diffie-Hellman parameters from fixed 1024-bit to custom 2048-bit.
Enable OCSP stapling.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-03-11 23:40:34 -07:00
Tim Abbott e0c6136ce1 puppet: Fix nginx configuration logic for S3 backend.
Apparently, our testing environment for this configuration was broken
and didn't test the code we thought it did; as a result, a variable
redefinition bug slipped through.

Fixes #11786.
2019-03-06 13:17:11 -08:00
Tim Abbott 5614d51afc nginx: Restructure how we manage uploaded file routes.
The overall goal of this change is to fix an issue where on Ubuntu
Trusty, we were accidentally overriding the configuration to serve
uploads from disk with the regular expressions for adding access
control headers.

However, while investigating this, it became clear that we could
considerably simplify the mental energy required to understand this
system by making the uploads-route file be unconditionally available
and included from `zulip-include/app` (which means the zulip_ops code
can share behavior here).

We also move the Access-Control-Allow-* headers to a separate include
file, to avoid duplicating it in 5 places.  Fixing this duplication
discovered a potential bug in the settings used for Tornado, where
DELETE was not allowed on a route that definitely expects DELETE.

Fixes #11758.
2019-03-02 12:14:28 -08:00
Anders Kaseorg 649235cfec python: Remove unused imports.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-02-22 16:54:36 -08:00
Tim Abbott a0add8f651 puppet: Add IPv6 support to standard nginx listen directives.
This should save some setup work for anyone wanting to setup nginx on
their Zulip server.
2019-02-13 15:00:21 -08:00
Tim Abbott ab18dbfde5 uwsgi: Increase buffer-size to 8192.
For users putting Zulip behind certain proxies (and potentially some
third-party API clients), buffer sizes can exceed the uwsgi default of
4096.  Since we aren't doing such high-throughput APIs that a small
buffer size is valuable, we should just raise this for everyone.
2019-02-13 11:17:55 -08:00
Anders Kaseorg c109690cf8 puppet: Remove unused Python imports.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-02-02 17:02:12 -08:00
Tim Abbott 68552c31cb Revert "puppet: Increase process listening count for uwsgi."
This reverts commit ccce83d0f0.

This needs sysctl changes as well.
2019-01-23 11:02:14 -08:00
Tim Abbott ccce83d0f0 puppet: Increase process listening count for uwsgi.
The default limit is too low for situations right around a server
restart when there might be a large burst of connections.
2019-01-23 10:34:01 -08:00
Harshit Bansal 50ef91bb08 scripts: Add argparse option to `restart-zerver` for `--fill-cache`.
Nowm unless you specify `--fill-cache`, memcached caches will not be
pre-filled after a server restart. This will be helpful when someone
is in a hurry (e.g. if the server is down right now, or if he/she
testing a configuration change in a newly setup server), it's best to
just restart without pre-filling the cache.

Fixes: #10900.
2019-01-14 15:20:01 -08:00
Tom Daff fbffbf8ef0 puppet/nginx: Update to recommended SSL ciphers.
Update the list of ciphers that nginx will use to the current
Mozilla recommended ones.

These are Intermediate compatibility ones suitable for clients
running anything newer than Firefox 1, Chrome 1, IE 7, Opera 5
and Safari 1. Modern compatibility is not suitable as it excludes
Andriod 4 which is still seen on ~1% of traffic.

More info: https://wiki.mozilla.org/Security/Server_Side_TLS
2019-01-08 14:19:49 -08:00