Commit Graph

1123 Commits

Author SHA1 Message Date
Alex Vandiver 2c73fbdcb6 puppet: Remove munin monitoring for no-longer-used "postgres_other".
The `wiki` and `trac` products are no longer used.
2020-05-28 17:24:35 -07:00
Tim Abbott 0b93e09e72 puppet: Add nginx configuration for blog.zulip.org move. 2020-05-26 14:47:05 -07:00
Anders Kaseorg f5b33f9398 python: Further pyupgrade changes.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-26 11:43:40 -07:00
Anders Kaseorg 333f7d16c9 logging: Pass more format arguments to logging.
Commit bdc365d0fe (#14852) missed this
because of https://github.com/returntocorp/semgrep/issues/831.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-26 11:42:23 -07:00
Anders Kaseorg 824d97987b process_fts_updates: Use cursor.execute correctly.
Commit b501d04f6a (#14841) missed this
because of https://github.com/returntocorp/semgrep/issues/831.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-26 11:42:23 -07:00
arpit551 439f0d3004 install: Ad production support for Zulip on Ubuntu Focal.
Install script now runs on Focal.  Python 2 is now installed via the
`python2` package in Focal.
2020-05-25 16:58:42 -07:00
Tim Abbott 220620e7cf sharding: Add basic sharding configuration for Tornado.
This allows straight-forward configuration of realm-based Tornado
sharding through simply editing /etc/zulip/zulip.conf to configure
shards and running scripts/refresh-sharding-and-restart.

Co-Author-By: Mateusz Mandera <mateusz.mandera@zulip.com>
2020-05-20 13:47:20 -07:00
Tim Abbott cdd3b7efbc tornado: Configure upstreams for TORNADO_PROCESSES. 2020-05-20 13:43:48 -07:00
Tim Abbott c3d3324295 puppet: Add link to the sources for Zephyr patches. 2020-05-19 20:54:11 -07:00
Tim Abbott a35e71ebbc puppet: Update package name for boto-on-python3.
The python3-boto3 package is the maintained fork that supports Python
3; it was renamed in Ubuntu Bionic from the original Ubuntu Xenial name.
2020-05-19 20:25:11 -07:00
Tim Abbott 1c28770810 puppet: Fix apt_repo_debathena setup_file path.
There was a typo introduced here when scripts_path was added.
2020-05-19 20:21:30 -07:00
Tim Abbott c43b3d95e2 puppet: Switch env-wal-e to use wal-g rather than wal-e.
wal-g is the modern reimplementation of wal-e that supports current
postgres.  It requires a bit of extra configuration to specify the AWS
region.
2020-05-15 16:45:36 -07:00
Anders Kaseorg fcca4a38b6 puppet: Work around memcached SASL configuration path bug.
memcached 1.5.22 in Ubuntu 20.04 has a bug where it looks for its SASL
configuration at /etc/sasl2/memcached.conf/memcached.conf instead of
/etc/sasl2/memcached.conf.

https://bugs.launchpad.net/ubuntu/+source/memcached/+bug/1878721

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-14 23:25:24 -07:00
Tim Abbott b3c5f2c13e puppet: Remove check_postgres_replication_lag hostname hardcoding.
Since this runs on the Nagios server, which already has the relevant
hostnames defined in zulip.conf, we can just read it from there.
2020-05-11 23:42:36 -07:00
Tim Abbott 225bbf3633 puppet: Update check_postgres_replication_lag for postgres 10.
These functions were renamed in postgres 10.
2020-05-11 15:59:23 -07:00
Tim Abbott d8ea649869 puppet: Cast tornado_processes to Integer.
This is the latest mechanism in puppet for turning a string into an
integer.

We update an adjacent comment while we're at it.
2020-05-11 00:54:48 -07:00
Tim Abbott 6319c181eb puppet: Use actual name for the bind9-host package.
Using the `host` virtual package confused Puppet into reporting it was
doing work every time one did a puppet run, resulting in unnecessarily
spammy output.
2020-05-11 00:51:53 -07:00
Mateusz Mandera dd40649e04 queue_processors: Remove the slow_queries queue.
While this functionality to post slow queries to a Zulip stream was
very useful in the early days of Zulip, when there were only a few
hundred accounts, it's long since been useless since (1) the total
request volume on larger Zulip servers run by Zulip developers, and
(2) other server operators don't want real-time notifications of slow
backend queries.  The right structure for this is just a log file.

We get rid of the queue and replace it with a "zulip.slow_queries"
logger, which will still log to /var/log/zulip/slow_queries.log for
ease of access to this information and propagate to the other logging
handlers.  Reducing the amount of queues is good for lowering zulip's
memory footprint and restart performance, since we run at least one
dedicated queue worker process for each one in most configurations.
2020-05-11 00:45:13 -07:00
Tim Abbott 21a04e2dbc puppet: Use nice to deprioritize various processes.
Our priority hierarchy is:
(1) Tornado and base services like memcached, redis, etc.
(2) Django and message sender queue workers.
(3) Everything else.

Ideally, we'd have something a bit more fine-grained (e.g. some queue
workers are potentially in the sending path, while others aren't), but
this should have a big impact on ensuring Tornado gets the resources
it needs during load spikes.

I think this has a good chance of causing some load spikes that would
previously have resulted in a user-facing delivery delays no longer
having any significant user-facing impact.
2020-05-10 23:28:25 -07:00
shubhamgupta2956 9cd8644c7c uploads: Add support for ".jpe" file extension.
Currently when the user uploads files with ".jpe" file extension, the
markdown is converted to link but the image is not embedded.

This commit adds the support for ".jpe" file extension.

Fixes #14863
2020-05-10 22:55:52 -07:00
Anders Kaseorg 8cdf2801f7 python: Convert more variable type annotations to Python 3.6 style.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-08 16:42:43 -07:00
Anders Kaseorg 708c6f4f11 puppet: Finally vanquish the cursed integer conversion conditional.
We no longer support Puppet 3.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-08 16:42:43 -07:00
Tim Abbott 50d8d61d3c puppet: Remove unnecssary/broken ;.
This breaks the Xenial build, which we're removing soon, but it's
unnecessary in any case.
2020-05-07 16:23:37 -07:00
Tim Abbott 03991d098a puppet: Add optional postgres version override.
This makes it convenient to run an alternative postgres version.
2020-05-07 09:33:24 -07:00
Mateusz Mandera 4643e48f60 retention: Add a daily cron job.
This will run archive_messages management command at 6am every day, 1
hour after soft_deactivate_users (which runs at 5am).
2020-05-05 10:11:38 -07:00
Tim Abbott 4034f6f99e nagios: Fix check_postgres_replication_lag.
This expects to be run outside a virtualenv and thus without
typing_extensions available.
2020-05-03 00:14:54 -07:00
Tim Abbott 4f3976b917 process_fts_updates: Clean up logging output.
This saves a couple lines of spammy output in the run-dev.py startup
experience, and will be better output in production as well.
2020-05-01 11:51:20 -07:00
Anders Kaseorg c0ffa71fa9 nginx: Replace unanchored regexes in location directives.
We could anchor the regexes, but there’s no need for the power (and
responsibility) of regexes here.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-24 16:58:19 -07:00
Anders Kaseorg 5e01a0ae8b zulip-ec2-configure-interfaces: Convert function type annotations.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-24 13:06:54 -07:00
Anders Kaseorg f8339f019d python: Convert assignment type annotations to Python 3.6 style.
Commit split by tabbott; this has changes to scripts/, tools/, and
puppet/.

scripts/lib/hash_reqs.py, scripts/lib/setup_venv.py,
scripts/lib/zulip_tools.py, and tools/lib/provision.py are excluded so
tools/provision still gives the right error message on Ubuntu 16.04
with Python 3.5.

Generated by com2ann, with whitespace fixes and various manual fixes
for runtime issues:

-shebang_rules: List[Rule] = [
+shebang_rules: List["Rule"] = [

-trailing_whitespace_rule: Rule = {
+trailing_whitespace_rule: "Rule" = {

-whitespace_rules: List[Rule] = [
+whitespace_rules: List["Rule"] = [

-comma_whitespace_rule: List[Rule] = [
+comma_whitespace_rule: List["Rule"] = [

-prose_style_rules: List[Rule] = [
+prose_style_rules: List["Rule"] = [

-html_rules: List[Rule] = whitespace_rules + prose_style_rules + [
+html_rules: List["Rule"] = whitespace_rules + prose_style_rules + [

-    target_port: int = None
+    target_port: int

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-24 13:06:54 -07:00
Anders Kaseorg 09ea778db1 nginx: Listen for ACME challenges on port 80 too.
This should make Certbot renewals more reliable.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-23 16:22:04 -07:00
Aman Agrawal 2dc6d09c2a python3-upgrade: Move python2 scripts to run on python3. 2020-04-22 16:13:15 -07:00
Anders Kaseorg 5901e7ba7e python: Convert function type annotations to Python 3 style.
Generated by com2ann (slightly patched to avoid also converting
assignment type annotations, which require Python 3.6), followed by
some manual whitespace adjustment, and six fixes for runtime issues:

-    def __init__(self, token: Token, parent: Optional[Node]) -> None:
+    def __init__(self, token: Token, parent: "Optional[Node]") -> None:

-def main(options: argparse.Namespace) -> NoReturn:
+def main(options: argparse.Namespace) -> "NoReturn":

-def fetch_request(url: str, callback: Any, **kwargs: Any) -> Generator[Callable[..., Any], Any, None]:
+def fetch_request(url: str, callback: Any, **kwargs: Any) -> "Generator[Callable[..., Any], Any, None]":

-def assert_server_running(server: subprocess.Popen[bytes], log_file: Optional[str]) -> None:
+def assert_server_running(server: "subprocess.Popen[bytes]", log_file: Optional[str]) -> None:

-def server_is_up(server: subprocess.Popen[bytes], log_file: Optional[str]) -> bool:
+def server_is_up(server: "subprocess.Popen[bytes]", log_file: Optional[str]) -> bool:

-    method_kwarg_pairs: List[FuncKwargPair],
+    method_kwarg_pairs: "List[FuncKwargPair]",

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-18 20:42:48 -07:00
pemontto fd34bc5161
puppet: Allow /etc/zulip to be a symlink.
This PR updates the puppet manifest to allow /etc/zulip to be a symlink. The current behaviour overwrites /etc/zulip if it is link to another directory, which is problematic with docker-zulip and 
in particular the `LINK_SETTINGS_TO_DATA` setting.
2020-04-17 12:45:05 -07:00
Tim Abbott 777a3b6c18 puppet: Fix nagios check to not require typing_extensions. 2020-04-16 17:56:05 -07:00
Tim Abbott e1ce53ac46 puppet: Update nagios checks for disk to exclude kernel filesystems.
The fact that we have to explicitly list these is almost certainly a
bug in check_disk, but at least this works.
2020-04-16 17:49:29 -07:00
Tim Abbott cfbb617f5c puppet: Update nagios configuration for checking local disk. 2020-04-16 17:48:36 -07:00
Tim Abbott 9821dfa9fc puppet: The letsencrypt package is debian is now certbot.
It was an alias starting with Ubuntu Xenial, and will eventually be
removed.
2020-04-16 17:30:01 -07:00
Tim Abbott 8e5a866122 puppet: Update tuning for load average monitoring. 2020-04-16 16:47:05 -07:00
Tim Abbott b1ff823798 puppet: Remove old zulipbot configuration.
We haven't used zulipbot hosted here for years.
2020-04-16 16:18:48 -07:00
Anders Kaseorg 99242138a7 static: Serve webpack bundles from the root domain.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-10 00:48:02 -07:00
Anders Kaseorg c734bbd95d python: Modernize legacy Python 2 syntax with pyupgrade.
Generated by `pyupgrade --py3-plus --keep-percent-format` on all our
Python code except `zthumbor` and `zulip-ec2-configure-interfaces`,
followed by manual indentation fixes.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-09 16:43:22 -07:00
Vishnu KS 449f7e2d4b team: Generate team page data using cron job.
This eliminates the contributors data as a possible source of
flakiness when installing Zulip from Git.

Fixes #14351.
2020-04-08 12:52:31 -07:00
Anders Kaseorg 15d68c40dd nginx: Set X-XSS-Protection: 1; mode=block.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-05 16:13:53 -07:00
Anders Kaseorg 79c215626e nginx: Set X-Content-Type-Options: nosniff globally.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-05 16:13:53 -07:00
Anders Kaseorg 06e7d4ec19 nginx: Don’t override HSTS, X-Frame-Options with other ‘add_header’s.
The nginx ‘add_header’ directive doesn’t inherit the way you’d
want (https://trac.nginx.org/nginx/ticket/854), so we need to manually
simulate inheritance using ‘include’, like we previously did with
api_headers.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-05 16:13:53 -07:00
Mateusz Mandera 5252b081bd queue_processors: Gather statistics on queue worker operations. 2020-04-01 16:44:06 -07:00
Stefan Weil d2fa058cc1
text: Fix some typos (most of them found and fixed by codespell).
Signed-off-by: Stefan Weil <sw@weilnetz.de>
2020-03-27 17:25:56 -07:00
Anders Kaseorg 7ff9b22500 docs: Convert many http URLs to https.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-03-26 21:35:32 -07:00
Anders Kaseorg 687553a661 setup_path_on_import: Replace with setup_path function.
isort 5 knows not to reorder imports across function calls, so this
will stop isort from breaking our code.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-02-25 15:40:21 -08:00
Anders Kaseorg 9d598d95a6 puppet: Fix puppet-lint warning.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-02-20 19:51:48 -08:00
Anders Kaseorg 91edb7dc43 puppet: Fix regeneration of memcached-sasldb2 on password changes.
Puppet doesn’t re-run an exec blocks that’s declared as creating an
existing file, even if it’s notified.  Remove the creates declaration.

Fixes #13730.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-02-19 14:20:43 -08:00
Mateusz Mandera 4c5a8e6f0c queue: Remove missedmessage_email_senders. 2020-01-31 12:13:51 -08:00
Tim Abbott dd969b5339 install: Remove references to "Zulip Voyager".
"Zulip Voyager" was a name invented during the Hack Week to open
source Zulip for what a single-system Zulip server might be called, as
a Star Trek pun on the code it was based on, "Zulip Enterprise".

At the time, we just needed a name quickly, but it was never a good
name, just a placeholder.  This removes that placeholder name from
much of the codebase.  A bit more work will be required to transition
the `zulip::voyager` Puppet class, as that has some migration work
involved.
2020-01-30 12:40:41 -08:00
Tim Abbott d70e799466 bots: Remove FEEDBACK_BOT implementation.
This legacy cross-realm bot hasn't been used in several years, as far
as I know.  If we wanted to re-introduce it, I'd want to implement it
as an embedded bot using those common APIs, rather than the totally
custom hacky code used for it that involves unnecessary queue workers
and similar details.

Fixes #13533.
2020-01-25 22:41:39 -08:00
Anders Kaseorg 3360df7ad1 generate_secrets: Enable memcached authentication in production.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-01-15 17:35:15 -08:00
Anders Kaseorg cdda983e90 settings: Support optional memcached authentication.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-01-15 17:35:15 -08:00
Anders Kaseorg ea6934c26d dependencies: Remove WebSockets system for sending messages.
Zulip has had a small use of WebSockets (specifically, for the code
path of sending messages, via the webapp only) since ~2013.  We
originally added this use of WebSockets in the hope that the latency
benefits of doing so would allow us to avoid implementing a markdown
local echo; they were not.  Further, HTTP/2 may have eliminated the
latency difference we hoped to exploit by using WebSockets in any
case.

While we’d originally imagined using WebSockets for other endpoints,
there was never a good justification for moving more components to the
WebSockets system.

This WebSockets code path had a lot of downsides/complexity,
including:

* The messy hack involving constructing an emulated request object to
  hook into doing Django requests.
* The `message_senders` queue processor system, which increases RAM
  needs and must be provisioned independently from the rest of the
  server).
* A duplicate check_send_receive_time Nagios test specific to
  WebSockets.
* The requirement for users to have their firewalls/NATs allow
  WebSocket connections, and a setting to disable them for networks
  where WebSockets don’t work.
* Dependencies on the SockJS family of libraries, which has at times
  been poorly maintained, and periodically throws random JavaScript
  exceptions in our production environments without a deep enough
  traceback to effectively investigate.
* A total of about 1600 lines of our code related to the feature.
* Increased load on the Tornado system, especially around a Zulip
  server restart, and especially for large installations like
  zulipchat.com, resulting in extra delay before messages can be sent
  again.

As detailed in
https://github.com/zulip/zulip/pull/12862#issuecomment-536152397, it
appears that removing WebSockets moderately increases the time it
takes for the `send_message` API query to return from the server, but
does not significantly change the time between when a message is sent
and when it is received by clients.  We don’t understand the reason
for that change (suggesting the possibility of a measurement error),
and even if it is a real change, we consider that potential small
latency regression to be acceptable.

If we later want WebSockets, we’ll likely want to just use Django
Channels.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-01-14 22:34:00 -08:00
Anders Kaseorg 6749810c2e puppet: Fix zuli-redis.conf path typo.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-01-13 17:37:09 -08:00
Anders Kaseorg 79cae1e7e0 puppet: Delete legacy rediscleanup code.
It was added in commit 9afb1c7a71 from
before 1.4.0.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-01-13 17:37:09 -08:00
Anders Kaseorg 5526af32f3 puppet: Switch double quoted strings to single quoted.
Resolves these warnings from puppet-lint.

puppet-lint| puppet/zulip/manifests/app_frontend_base.pp - WARNING: double quoted string containing no variables on line 14
puppet-lint| puppet/zulip/manifests/app_frontend_base.pp - WARNING: double quoted string containing no variables on line 19

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-01-07 15:10:17 -08:00
rht 70dfb423e4 puppet: Specify CentOS specific path to ca certificates for nginx. 2020-01-07 13:25:25 -08:00
rht d5284b177e puppet: Convert memorysize_mb to integer depending on Puppet version. 2020-01-07 13:25:25 -08:00
rht dccfb0ebe9 puppet: Remove duplicate postgresql-client safepackage check on CentOS. 2020-01-07 13:25:25 -08:00
Anders Kaseorg a78f8647d8 install: Run generate_secrets.py before zulip-puppet-apply.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-01-05 22:48:08 -08:00
Vishnu KS 8b57e39c7e settings: Add option to set remote postgres port. 2019-12-12 12:17:11 -08:00
Anders Kaseorg 0d20145b93 mypy: Upgrade from 0.730 to 0.740.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-11-13 12:38:45 -08:00
Anders Kaseorg 0ae2c5c96e nginx: Enable TLS 1.3 if supported.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-10-30 13:09:57 -07:00
Anders Kaseorg ee9a6071fd 5xx.html: Build with webpack.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-10-28 15:53:15 -07:00
Tim Abbott f84c037225 puppet: Tune check_postgres_locks parameters.
This has been a spurious alert for a long time.

It's unclear that this check is useful at all, but if it spikes
dramatically above what's normal, there's perhaps still utility in
being alerted.
2019-10-23 15:04:38 -07:00
Tim Abbott e4dee9532c nagios: Update configuration for user_activity worker change.
Since LoopQueueProcessingWorker jobs cannot be monitored by checking
for connected consumers (since they poll, rather than consuming as
events arrive), they can't be monitored with check_consumers.  It's
OK, because that monitoring was redundant with monitoring for
potential growth in their queue that we have as well.

Also clean up the block comments for the two other similar queue
procesors.
2019-09-23 11:49:46 -07:00
Anders Kaseorg b72bb8171b nginx: Add CORS, HSTS, and X-Frame-Options headers to error responses.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-19 12:10:18 -07:00
Anders Kaseorg 6701c4463c search: Remove now unnecessary tsearch_extra dependency.
Now that we're implemented tsearch_extras in pure postgres, we no
longer need a custom extension.  This should help us considerably, as
it means we no longer need to ship custom apt packages at all.

Fixes #467.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-29 12:49:26 -07:00
Anders Kaseorg b2e1af90fc process_fts_updates: Reconnect on OperationalError.
This allows process_fts_updates to recover if Postgres is restarted.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-21 11:00:58 -07:00
Anders Kaseorg fb42cd3af9 process_fts_updates: Fix log message.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-21 11:00:58 -07:00
Anders Kaseorg 473c4abca5 process_fts_updates: Use psycopg2.connect kwargs.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-21 11:00:58 -07:00
Anders Kaseorg fa11b2d806 nginx: Don’t gzip files that are already compressed.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-21 10:51:37 -07:00
Anders Kaseorg 4e620ed43c nginx: Enable http2 in on-premise configuration.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-21 10:51:37 -07:00
Hemanth V. Alluri dac068df31 production: Finish adding production support for Zulip on Debian Buster.
This commit finishes adding end-to-end support for the install script
on Debian Buster (making it production ready). Some support for this
was already added in prior commits such as
99414e2d96.

We plan to revert the postgres hunks of this once we've built
tsearch_extras for our packagecloud archive.

Fixes #9828.
2019-08-17 12:22:32 -07:00
Hemanth V. Alluri 083723b6a9 puppet: Add Zulip specific postgres configuration for 11.
Based on the work done in a03e4784c7.
2019-08-17 11:41:11 -07:00
Hemanth V. Alluri 792283c441 puppet: Commit an upstream version of postgres 11 config.
In preparation for adding production support for Debian Buster. Based
on the work done in commit 964a1ac8a7.
2019-08-17 11:41:11 -07:00
Hemanth V. Alluri 5dd45b4b2e puppet: Fix the release detection regex patterns in base.pp.
The issue here was that the '.' character was unescaped and the
regex was not anchored with a terminal '$'. This was detected by
Anders Kaseorg.

Co-authored-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-17 11:41:11 -07:00
Anders Kaseorg 66649d84cb puppet: Reload postfix on /etc/postfix/virtual changes.
`/etc/postfix/virtual` is of `regexp:` type, not `hash:` type, so
running `postmap` on it has no effect; we need to reload Postfix when
it changes.

http://www.postfix.org/DATABASE_README.html#detect

In the interest of forcing a reload now, optimize the regexes by
eliding the unanchored `.*`s at the beginnings and ends.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-15 22:45:26 -07:00
rht 61be9fb4bd puppet: Add Zulip-specific postgres configuration for 10 on Centos. 2019-08-14 14:31:16 -07:00
rht 03fb4b5f90 puppet: Commit an upstream CentOS version of postgres 10 sample config. 2019-08-14 14:31:16 -07:00
Anders Kaseorg 263d71bf2b nginx: Add CORS headers to /user_uploads.
Fixes: #12980.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-12 15:35:35 -07:00
Anders Kaseorg 2e57f3ffae puppet: “Resolve” puppet-lint warnings.
Introduced by #12966.

puppet/zulip/manifests/base.pp - WARNING: double quoted string containing no variables on line 93
puppet/zulip/manifests/base.pp - WARNING: string containing only a variable on line 93

scanf doesn’t accept a number as input, so uh, add a dummy space
character.

What.  You can’t give me a bad language and then complain when I write
bad programs in it.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-12 15:29:23 -07:00
Anders Kaseorg 820165e4da Merge pull request #12968 from andersk/ffdhe2048
nginx: Use fixed ffdhe2048 DH parameter (RFC 7919)
2019-08-09 16:29:10 -07:00
Anders Kaseorg 4e9fb05c4f puppet: Use built-in memorysize_mb fact.
Fixes this warning:

    Warning: The string '8167976' was automatically coerced to the numerical value 8167976 (file: /root/zulip/puppet/zulip/manifests/base.pp, line: 93, column: 19)

Fixes #9682.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 16:28:09 -07:00
Tim Abbott de0a41bc9c provision: Fix missing dependency on unzip.
Because this is often installed by default, we hadn't noticed that our
Slack importer doesn't run without it.

Thanks to Ray Kraesig for the report.
2019-08-08 10:49:20 -07:00
Anders Kaseorg 0962393933 cleanup: Delete trailing newlines.
Delete trailing newlines from all files, except
tools/ci/success-http-headers.txt and tools/setup/dev-motd, where they
are significant, and static/third, where we want to stay close to
upstream.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-06 23:29:11 -07:00
Anders Kaseorg becef760bf cleanup: Delete leading newlines.
Previous cleanups (mostly the removals of Python __future__ imports)
were done in a way that introduced leading newlines.  Delete leading
newlines from all files, except static/assets/zulip-emoji/NOTICE,
which is a verbatim copy of the Apache 2.0 license.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-06 23:29:11 -07:00
Anders Kaseorg 68dd8e4ec8 mypy: Migrate from mypy_extensions to typing_extensions.
This gives us access to typing_extensions.Deque, which was not added
to typing until 3.5.4.

(PROVISION_VERSION is not bumped because the transitive dependency set
in dev.txt hasn’t changed.)

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-05 17:24:09 -07:00
Wyatt Hoodes a109508e34 typing: Remove now-unnecessary conditional import.
As a result of dropping support for trusty, we can remove our old
pattern of putting `if False` before importing the typing module,
which was essential for Python 3.4 support, but not required and maybe
harmful on newer versions.

cron_file_helper
check_rabbitmq_consumers
hash_reqs
check_zephyr_mirror
check_personal_zephyr_mirrors
check_cron_file
zulip_tools
check_postgres_replication_lag
api_test_helpers
purge-old-deployments
setup_venv
node_cache
clean_venv_cache
clean_node_cache
clean_emoji_cache
pg_backup_and_purge
restore-backup
generate_secrets
zulip-ec2-configure-interfaces
diagnose
check_user_zephyr_mirror_liveness
2019-07-29 15:18:22 -07:00
Anders Kaseorg b758ed5ac1 nginx: Remove invalid extra headers for OPTIONS /api/v1/events.
Since 204 responses don’t contain a payload body, Content-Type is
neither required nor encouraged (RFC 7231 §3.1.1.5), and ours was
missing a semicolon to boot; Content-Length is expressly
forbidden (RFC 7230 §3.3.2).

Furthermore, these add_header directives were silencing the CORS
headers set in api_headers, because add_header inheritance doesn’t
work the way you think it does, as was known before commit
5614d51afc.

Fixes: #12902.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-29 14:58:35 -07:00
Anders Kaseorg 6d5a20ac62 requirements: Remove django-pipeline.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-24 17:40:31 -07:00
Tim Abbott 2eb855b302 puppet: Include . separator in email mirror rules.
This is required for the postfix-localmail integration to use the new
`.` format email addresses.
2019-07-22 11:13:36 -07:00
Wyatt Hoodes e331a758c3 python: Migrate open statements to use with.
This is low priority, but it's nice to be consistently using the best
practice pattern.

Fixes: #12419.
2019-07-20 15:48:52 -07:00
Anders Kaseorg c97ca677c9 nginx: Update TLS settings based on Mozilla recommendations 5.0.
Disable TLS 1.0 and TLS 1.1.  (We no longer need to support IE8 on
Windows XP.)

Prefer client-selected cipher order.  (Now that all enabled ciphers
provide good security, this allows mobile clients lacking AES hardware
acceleration to pick ChaCha20 for better performance.)

Disable session tickets.  (Mozilla discourages them based on
https://www.imperialviolet.org/2013/06/27/botchingpfs.html.)

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-08 15:51:02 -07:00
Anders Kaseorg 079ddae4c8 minify-js: Remove; everything has been migrated to Webpack.
min/sockjs-0.3.4.min.js is not used.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-03 13:58:21 -07:00
Tim Abbott aea1279e8c puppet: Remove trusty configuration for static_asset_compiler.
Trusty is desupported.
2019-06-26 11:32:06 -07:00
Tim Abbott 8fbd965ab5 puppet: Remove legacy pgtune related configuration for trusty.
Since we no longer support Ubuntu Trusty, we no longer need this
backwards-compatibility cruft (which we only kept around to avoid
randomizing configuration for existing systems).
2019-06-26 11:32:06 -07:00
Anders Kaseorg 33c941407b puppet: Remove legacy unauthenticated local uploads backend.
This was only used in Ubuntu 14.04 Trusty.

Removing this also finally lets us simplify our security model
discussion of uploaded files.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-06-26 11:31:46 -07:00
Tim Abbott 271319fb13 puppet: Fix hacky release test for whether we're in EC2.
The result is still a bit hacky, but guaranteed to be correct if we
adjust the OS version of our systems, which we of course will do over
time.
2019-06-25 22:19:04 -07:00
Tim Abbott 8d8cfb314b puppet: Remove zulip_ops configuration for trusty.
There are no longer any zulip_ops systems using trusty.
2019-06-25 22:09:06 -07:00
Tim Abbott 88b77af54f puppet: Add support for changing the nginx port directly.
This provides a clean process for changing Zulip's nginx port.
2019-06-17 12:24:22 -07:00
Fabian Stanke 51ba9ddd89 postfix: Inserted compulsory setting for postfix ≥ 2.10.
One of smtpd_relay_restrictions or smtpd_recipient_restrictions is
required by postfix ≥ 2.10 (see
http://www.postfix.org/SMTPD_ACCESS_README.html).

This is important for using the email mirror on Ubuntu Bionic.
2019-06-16 18:48:39 -07:00
Tim Abbott b41c2d93d1 puppet: Exclude squashfs filesystems from nagios disk checks.
These generally aren't being written to.
2019-06-16 16:22:23 -07:00
Tim Abbott 0ec1b4e82c puppet: Move check_send_receive_time to the _once ruleset.
We don't actually want to run this bundle of message-sending Nagios
checks to run on every single server.
2019-06-16 15:48:35 -07:00
Tim Abbott df83979c76 zulip_ops: Extract a prod_app_frontend_once ruleset. 2019-06-16 15:48:35 -07:00
Tim Abbott 738cfe54c3 puppet: Move app_frontend_once out of prod configuration.
That logic made it inconvenient to run multiple prod servers with the
same top-level puppet configuration.
2019-06-16 15:24:20 -07:00
Tim Abbott e85250941d puppet: Fix quoting of commented-out python3-boto.
This will avoid a linter error if/when we uncomment it.
2019-06-13 14:39:24 -07:00
Tim Abbott 337efe0fb7 puppet: Remove puppet-el, which no longer exists.
This package was only every available on Ubuntu Xenial.
2019-06-13 14:39:24 -07:00
Tim Abbott afb0d1ccce Revert "puppet: Use nice to deprioritize various processes."
This reverts commit d959de7a89.

This broken our Travis CI, so I'm pulling it off while we investigate.
2019-06-05 12:55:56 -07:00
Tim Abbott d959de7a89 puppet: Use nice to deprioritize various processes.
Our priority hierarchy is:
(1) Tornado and base services like memcached, redis, etc.
(2) Django and message sender queue workers.
(3) Everything else.

Ideally, we'd have something a bit more fine-grained (e.g. some queue
workers are potentially in the sending path, while others aren't), but
this should have a big impact on ensuring Tornado gets the resources
it needs during load spikes.

I think this has a good chance of causing some load spikes that would
previously have resulted in a user-facing delivery delays no longer
having any significant user-facing impact.
2019-06-05 11:56:48 -07:00
Tim Abbott cd1ec37404 puppet: Make uwsgi listen backlog limit configurable.
This can be useful for busy servers to limit the risk of bursts of
traffic causing them to reject requests.
2019-05-17 12:38:56 -07:00
Tim Abbott ca48b4ec9f puppet: Set postgres max_connections to 1000.
There isn't much legitimate reason to have a limit as low as 100,
given how few resources a connection consumes.
2019-05-13 17:19:31 -07:00
Tim Abbott b7d50190b7 process_fts_updates: Batch updates when catching up.
Previously, if process_fts_updates ended up very far behind
(e.g. 100,000s of messages), it was unable to recover without doing
some very expensive databsae operations to fetch and then delete the
list of message IDs needing updates.  This change fixes that issue by
doing the catch-up work in batches.
2019-05-09 22:44:07 -07:00
Vishnu Ks ecdd3bea43 billing: Add cron job to run invoice_plans once a day.
Fixes #11960
2019-04-29 11:23:17 -07:00
Anders Kaseorg 643bd18b9f lint: Fix code that evaded our lint checks for string % non-tuple.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-04-23 15:21:37 -07:00
Anders Kaseorg 5290519a62 scripts: Always use ON_ERROR_STOP=1 when running psql.
Also use psql -e (--echo-queries) in scripts that use ‘set -x’, so
errors can be traced to a specific query from the output.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-04-22 14:54:19 -07:00
Puneeth Chaganti 9876f1b14e check_rabbitmq_queue: Fix the time period when we ignore long queues.
The commit 87d1809657 changed the time when
digests are sent by 3 hours to account for moving from the US East Coast to the
West Coast, but didn't change the time period exception in the
`check-rabbitmq-queue` script.

Closes #5415
2019-04-13 20:43:07 -07:00
Anders Kaseorg 9f7c0b7e65 postgres_master.pp: Fix wacky su command line.
The construction `su postgres -c -- bash -c 'psql …'` didn’t behave the
way it reads, and only worked by accident:

1. `-c --` sets the command to `--`.
2. `bash` sets the first argument to `bash`.
3. `-c 'psql …'` replaces the command with `psql …`.

Thus, `su` ended up executing `<shell> -c 'psql …' bash`, where
`<shell>` is the `postgres` user’s login shell, usually also `bash`,
which then executed 'psql …' and ignored the extra `bash`.

Unconfuse this construction.

Note from tabbott: The old code didn't even work by accident, it was
just broken.  The right fix is to move the quoting around properly.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-04-12 17:27:23 -07:00
Tim Abbott b1da797955 puppet: Make uwsgi buffer size configurable. 2019-03-18 22:43:59 -07:00
Anders Kaseorg fd6f18f7cf nginx: Improve TLS settings based on Mozilla config generator.
Lengthen the session timeout and enlarge the session cache.  Upgrade
Diffie-Hellman parameters from fixed 1024-bit to custom 2048-bit.
Enable OCSP stapling.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-03-11 23:40:34 -07:00
Tim Abbott e0c6136ce1 puppet: Fix nginx configuration logic for S3 backend.
Apparently, our testing environment for this configuration was broken
and didn't test the code we thought it did; as a result, a variable
redefinition bug slipped through.

Fixes #11786.
2019-03-06 13:17:11 -08:00
Tim Abbott 5614d51afc nginx: Restructure how we manage uploaded file routes.
The overall goal of this change is to fix an issue where on Ubuntu
Trusty, we were accidentally overriding the configuration to serve
uploads from disk with the regular expressions for adding access
control headers.

However, while investigating this, it became clear that we could
considerably simplify the mental energy required to understand this
system by making the uploads-route file be unconditionally available
and included from `zulip-include/app` (which means the zulip_ops code
can share behavior here).

We also move the Access-Control-Allow-* headers to a separate include
file, to avoid duplicating it in 5 places.  Fixing this duplication
discovered a potential bug in the settings used for Tornado, where
DELETE was not allowed on a route that definitely expects DELETE.

Fixes #11758.
2019-03-02 12:14:28 -08:00
Anders Kaseorg 649235cfec python: Remove unused imports.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-02-22 16:54:36 -08:00
Tim Abbott a0add8f651 puppet: Add IPv6 support to standard nginx listen directives.
This should save some setup work for anyone wanting to setup nginx on
their Zulip server.
2019-02-13 15:00:21 -08:00
Tim Abbott ab18dbfde5 uwsgi: Increase buffer-size to 8192.
For users putting Zulip behind certain proxies (and potentially some
third-party API clients), buffer sizes can exceed the uwsgi default of
4096.  Since we aren't doing such high-throughput APIs that a small
buffer size is valuable, we should just raise this for everyone.
2019-02-13 11:17:55 -08:00
Anders Kaseorg c109690cf8 puppet: Remove unused Python imports.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-02-02 17:02:12 -08:00
Tim Abbott 68552c31cb Revert "puppet: Increase process listening count for uwsgi."
This reverts commit ccce83d0f0.

This needs sysctl changes as well.
2019-01-23 11:02:14 -08:00
Tim Abbott ccce83d0f0 puppet: Increase process listening count for uwsgi.
The default limit is too low for situations right around a server
restart when there might be a large burst of connections.
2019-01-23 10:34:01 -08:00
Harshit Bansal 50ef91bb08 scripts: Add argparse option to `restart-zerver` for `--fill-cache`.
Nowm unless you specify `--fill-cache`, memcached caches will not be
pre-filled after a server restart. This will be helpful when someone
is in a hurry (e.g. if the server is down right now, or if he/she
testing a configuration change in a newly setup server), it's best to
just restart without pre-filling the cache.

Fixes: #10900.
2019-01-14 15:20:01 -08:00
Tom Daff fbffbf8ef0 puppet/nginx: Update to recommended SSL ciphers.
Update the list of ciphers that nginx will use to the current
Mozilla recommended ones.

These are Intermediate compatibility ones suitable for clients
running anything newer than Firefox 1, Chrome 1, IE 7, Opera 5
and Safari 1. Modern compatibility is not suitable as it excludes
Andriod 4 which is still seen on ~1% of traffic.

More info: https://wiki.mozilla.org/Security/Server_Side_TLS
2019-01-08 14:19:49 -08:00
rht 3f0bae8c38 puppet: Disable camo when not on Debian. 2019-01-07 18:52:45 -08:00
rht bf65f86a0b puppet: Abstract out ssl certs and private keys dirs. 2019-01-07 18:52:45 -08:00
rht d9ef3fd505 puppet: Manually create ssl-cert group on CentOS to acess ssl private key. 2019-01-07 18:51:39 -08:00
rht 6c3bb507b0 puppet: Ensure nginx sites-available & sites-enabled dirs exist on CentOS.
These are automatically created on Debian.
2019-01-07 17:09:42 -08:00
rht f2b6a2c68a puppet: Add CentOS version of the command to start supervisor. 2019-01-05 15:57:53 -08:00
rht 39f28a0d0f puppet: Abstract out supervisor service name. 2019-01-05 15:57:53 -08:00
rht d2069f7720 puppet: Include yum repository for CentOS voyager. 2019-01-05 15:57:45 -08:00
rht 1da17be52a puppet: Ensure supervisord conf.d directory is created on CentOS. 2019-01-05 15:55:43 -08:00
rht 902bb7a80c puppet: Add CentOS version of supervisor conf.d path. 2019-01-05 15:54:21 -08:00
rht 6b0bf828f7 puppet: Add CentOS version of supervisord.conf path. 2019-01-05 15:49:03 -08:00
rht 9ee2ee046a puppet: Use systemctl instead of pg_ctlcluster on CentOS. 2019-01-05 15:49:03 -08:00
rht 2bcf83d940 puppet: Add CentOS packages to static_asset_compiler.pp. 2019-01-05 15:49:03 -08:00
rht 071e32985c puppet: Generalize redis.conf path to CentOS. 2019-01-05 15:49:03 -08:00
rht acaf001cdd puppet: Group commonly reused variables into zulip::common. 2019-01-05 15:49:03 -08:00
rht 766ff38586 puppet: Abstract out nagios plugins directory. 2019-01-05 15:49:03 -08:00
rht b22f6c6a99 puppet: Abstract out postgresql package. 2019-01-05 15:49:03 -08:00
rht 43fdb00fc7 puppet: Abstract out nginx package. 2019-01-05 15:49:03 -08:00
rht 5424fca168 puppet: Add CentOS packages to postgres_appdb_base.pp. 2019-01-05 15:49:03 -08:00
rht 21c71a0c52 puppet: Use generic erlang package variable for all dependencies. 2019-01-05 15:49:02 -08:00
Tim Abbott 047817b6b0 puppet: Disable log2zulip cron job.
It hasn't been working for years, but more importantly, it spams up
root's mail queue so that one can't find important things in there
(e.g. the fact that the long-term-idle cron job was failing).
2019-01-05 10:56:44 -08:00
rht 801b04c057 puppet: Abstract out nagios-plugins package. 2019-01-04 15:27:03 -08:00
rht 04372e3300 puppet: Add CentOS packages to postgres_common.pp. 2019-01-04 15:24:42 -08:00
rht bdf36bdc3d puppet: Use pip to install python dependencies on CentOS. 2019-01-04 15:23:45 -08:00
rht 008879eb22 puppet: Add postgresql.conf path for CentOS. 2019-01-03 14:36:43 -08:00
rht dce43e1a0e puppet: Add CentOS-version of pg data path at pg_backup_and_purge. 2019-01-03 14:36:43 -08:00
rht 59993aea80 puppet: Abstract out path to postgresql.conf. 2019-01-03 14:36:43 -08:00
rht c189409ffd puppet: Initialize yum_repository.pp to wrap setup-yum-repo. 2019-01-03 14:36:43 -08:00
rht 1b02fb6d6d puppet: Add CentOS packages to rabbit.pp. 2019-01-03 14:36:43 -08:00
rht a3d67e52fe puppet: Add CentOS packages to redis.pp. 2019-01-03 14:36:43 -08:00
rht 788128f05c puppet: Add CentOS packages to nginx.pp. 2019-01-03 14:36:43 -08:00
rht 1965cc1491 puppet: Add CentOS packages to base.pp. 2019-01-03 14:36:42 -08:00
rht 0caaed4e1f puppet: Add CentOS packages to apache_sso.pp. 2019-01-03 14:36:41 -08:00
rht 9fd18a2c7f puppet: Detect CentOS for release_name. 2019-01-03 14:35:09 -08:00
Fabian Tribrunner 013b169469 check_email_deliverer_backlog: Switch down from root if required.
This now checks if the user is zulip, and if not, switches to the
zulip user, making it possible to run it as root.

Significantly modified by tabbott to not break existing behavior.
2019-01-02 10:16:56 -08:00
Fabian Tribrunner 4612a84a67 check_email_deliverer_process: Fix typo in process name.
This nagios check has never worked, since it had the wrong process
name from the beginning.
2018-12-30 09:41:42 -08:00
Anders Kaseorg 392175d6e8 Use #!/usr/bin/env for bash shebangs.
/bin/sh and /usr/bin/env are the only two binaries that NixOS provides
at a fixed path (outside a buildFHSUserEnv sandbox).

This discussion was split from #11004.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-12-17 17:21:08 -08:00
Tim Abbott 2558f101af docs: Add documentation for `if False` mypy pattern in scripts.
This should help make it clear what's going on with these scripts.
2018-12-17 11:12:53 -08:00
Tim Abbott bce90a3340 lint: Add lint rule for scripts importing typing improperly.
This is a common bug that users might be tempated to introduce.

And also fix two instances of this bug that were present in our
codebase, including an important one in our upgrade code path.
2018-12-17 10:46:37 -08:00
rht 43bedc0909 provision: Use vendored pg_hba.conf on CentOS. 2018-12-16 13:21:54 -08:00
rht c9d54f7854 puppet: Remove vendored puppetlabs apt and stdlibs dependencies.
This commit works by vendoring the couple functions we still use from
puppetlabs stdlib (join and range), but removing the rest of the
puppetlabs codebase, and of course cleaning up our linter rules in the
process.

Fixes #7423.
2018-12-11 13:03:26 -08:00
rht d2aa81858c puppet/zulip_ops: Replace apt::source with setup-apt-repo-debathena.
Tweaked by tabbott to use a clearer name.
2018-12-11 13:02:56 -08:00
rht 97766102df puppet/zulip: Replace apt::source and apt::ppa with setup-apt-repo. 2018-12-11 13:01:26 -08:00
Tim Abbott b218c2a70e loadbalancer: Use same certbot cert for zulipstaging.com.
This is a simple configuration improvement.
2018-12-07 13:43:21 -08:00
Tim Abbott 467694c1fa nginx: Enable http2 in external nginx configuration.
This should be a nice performance improvement for browsers that
support it.

We can't yet enabled this in the Zulip on-premise nginx configuration,
because that still has to support Trusty.
2018-12-07 13:43:02 -08:00
Tim Abbott e609e10229 puppet: Fix missing dependency of tsearch-extras on apt repository.
This isn't super required, in that we add these repositories via
`setup-apt-repo` in any case, but the previous code was wrong and
worth fixing in any case.
2018-11-30 10:45:04 -08:00
Tim Abbott ededdc512b nginx: Fix missing API authentication configuration.
This fixes a bug where our API routes for uploaded files (where we
need to use a consistent URL between session auth and API auth) were
not properly configured to pass through the API authentication headers
(and otherwise provide REST endpoint settings).

In particular, this prevented the Zulip mobile apps from being able to
access authenticated image files using these URLs.
2018-11-16 11:25:54 -08:00
Tim Abbott f62050212b tornado: Fix supervisord configuration for multiple processes.
Apparently, we can use the process group naming style of having dashes
in the names without using the explicit nun_procs feature of
supervisord configuration.

The new configuration is perfectly satisfactory, so there's no real
reason to prefer the old approach.
2018-11-06 17:56:06 -08:00
Tim Abbott 5abf4dee92 nagios: Add new host groups for Tornado processes.
We also move all the existing Tornado monitoring rules to the
singletornado_frontends rule.
2018-11-06 16:33:18 -08:00
Tim Abbott 5f3b79c9e7 nagios: Fix tab-based whitespace. 2018-11-06 16:30:29 -08:00
Tim Abbott 5e7aa27c29 puppet: Add supervisord support for multiple tornado processes. 2018-11-02 16:55:33 -07:00
Tim Abbott a5acbd51c3 settings: Add new zulip.conf setting for number of Tornado processes.
This will eventually be used to support Tornado sharding; for now,
it's just used to contain the code intended to support that feature.
2018-11-02 16:47:26 -07:00
Tim Abbott dc7d44a245 puppet: Don't run calculate-first-visible-message-id on most systems.
This should only be run on systems that are running zilencer, because
the cron job is part of the zilencer project.
2018-10-30 11:40:24 -07:00
Tim Abbott a4df001cef check_queue_worker_errors: Add support for running unprivileged.
Previously, this script needed access to Django settings, which in
turn required access to /etc/zulip/zulip-secrets.conf.  Since that
isn't world-readable, this meant that this couldn't run as an
unprivileged `nagios` user.

Fix that by just hardcoding the appropriate path under /var/log/.
2018-10-18 15:03:17 -07:00
Tim Abbott 98d89b676d pg_backup_and_purge: Fix incorrect conversion to use python3 types.
When using the Python 3 typing style, Python scripts can't import from
typing inside an `if False` (in contrast, one needs to import inside
an `if False` to support the Python 3 syntax without needing
python-typing installed).  So this was just incorrectly half-converted
from the Python 2 style to the Python 3 style.
2018-10-16 11:12:52 -07:00
Tim Abbott 2c7f9ce0fc puppet: Fix puppet-lint warnings in various manifests.
Apparently, `puppet-lint` on Ubuntu trusty throws warnings for certain
quoting patterns that are OK in modern `puppet-lint`.  I believe the
old Zulip code was actually correct (i.e. the old `puppet-lint`
implementation was the problem), but it seems worth changing anyway to
suppress the warnings.

We also exclude more of puppet-apt from linting, since it's
third-party code.
2018-08-28 13:46:31 -07:00
Tim Abbott b53a712856 nginx: Update configuration for using certbot certs everywhere. 2018-08-22 11:59:15 -07:00
Abhilash Verma 0e2322a322 logging: Show timestamp in UTC in non-django production scripts.
Done in pair programming with @aero31aero.

Fixes #9678.
2018-08-20 12:52:40 -07:00
Tim Abbott 5021f7b76f puppet: Fix accidental conflict on apache2 package.
Apparently, the work to force installation of the Python 3 version of
mod_wsgi was buggy and tried to force uninstall apache2.

Fixes #10318.
2018-08-16 14:15:35 -07:00
Tim Abbott 90828297e4 puppet-lint: Enforce double_quoted_strings check.
This makes our puppet codebase more consistent by using single-quoted
strings consistently.
2018-08-13 12:31:19 -07:00
Tim Abbott d0b51b70f4 puppet-lint: Enforce 2sp_soft_tables puppet-lint check.
This cleans up the puppet codebase's whitespace formatting to be more
consistent.
2018-08-13 12:31:16 -07:00
Tim Abbott b26e0a957d puppet-lint: Enforce arrow_alignment check.
This fixes all exceptions in our puppet codebase to this lint rule.
2018-08-13 12:30:57 -07:00
Tim Abbott 054c07b585 mypy: Fix run types in pg_backup_and_purge. 2018-08-09 12:57:53 -07:00
Tim Abbott db1f706d09 pg_backup_and_purge: Fix buggy recovery status parsing.
This was converted to Python 3 incorrectly, in a way that actually
completely broke the script (the .decode() that this adds is critical,
since 'f' != b'f').

We fix this, and also add an assert that makes the parsing code
safer against future refactors.
2018-08-09 11:48:48 -07:00
Aditya Bansal 5bfe24beef puppet-lint: Fix an error with defined type safepackage in base.pp.
We fix "ERROR: safepackage not in autoload module layout" error
which was caused by a defined type "safepackage" definitation
lying in the wrong place. We refactor to create the defined type
according to puppet guidelines. Link below:
https://docs.puppet.com/puppet/2.7/lang_defined_types.html
2018-08-07 10:03:40 -07:00
Aditya Bansal 710d4507de puppet-lint: Fix lines longer than 140 characters lint warnings.
We fix these by adding ignore statements in a bunch of files
where this error popped up. We target only specific lines using
the ignore statements and not the entire files.
2018-08-07 10:03:40 -07:00
Anders Kaseorg edfd5ef992 setup_disks.sh: Fix shellcheck warnings.
In puppet/zulip_ops/files/postgresql/setup_disks.sh line 15:
array_name=$(mdadm --examine --scan | sed 's/.*name=//')
^-- SC2034: array_name appears unused. Verify use (or export if used externally).

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-08-03 09:15:26 -07:00
Anders Kaseorg 5a0fecc2d5 munin_plugins: Fix shellcheck warnings.
In puppet/zulip_ops/files/munin-plugins/rabbitmq_connections line 66:
echo "connections.value $(HOME=$HOME rabbitmqctl list_connections | grep -v "^Listing" | grep -v "done.$" | wc -l)"
                                                                                         ^-- SC2126: Consider using grep -c instead of grep|wc -l.

In puppet/zulip_ops/files/munin-plugins/rabbitmq_consumers line 32:
VHOST=${vhost:-"/"}
^-- SC2034: VHOST appears unused. Verify use (or export if used externally).

In puppet/zulip_ops/files/munin-plugins/rabbitmq_messages line 32:
VHOST=${vhost:-"/"}
^-- SC2034: VHOST appears unused. Verify use (or export if used externally).

In puppet/zulip_ops/files/munin-plugins/rabbitmq_messages_unacknowledged line 32:
VHOST=${vhost:-"/"}
^-- SC2034: VHOST appears unused. Verify use (or export if used externally).

In puppet/zulip_ops/files/munin-plugins/rabbitmq_messages_uncommitted line 32:
VHOST=${vhost:-"/"}
^-- SC2034: VHOST appears unused. Verify use (or export if used externally).

In puppet/zulip_ops/files/munin-plugins/rabbitmq_queue_memory line 32:
VHOST=${vhost:-"/"}
^-- SC2034: VHOST appears unused. Verify use (or export if used externally).

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-08-03 09:15:08 -07:00
Anders Kaseorg 16ed5d5e79 env-wal-e: Fix shellcheck warnings.
In puppet/zulip/files/postgresql/env-wal-e line 6:
export AWS_ACCESS_KEY_ID=$(crudini --get "$ZULIP_SECRETS_CONF" secrets s3_backups_key)
       ^-- SC2155: Declare and assign separately to avoid masking return values.

In puppet/zulip/files/postgresql/env-wal-e line 7:
export AWS_SECRET_ACCESS_KEY=$(crudini --get "$ZULIP_SECRETS_CONF" secrets s3_backups_secret_key)
       ^-- SC2155: Declare and assign separately to avoid masking return values.

In puppet/zulip/files/postgresql/env-wal-e line 9:
if [ $? -ne 0 ]; then
     ^-- SC2181: Check exit code directly with e.g. 'if mycmd;', not indirectly with $?.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-08-03 09:13:07 -07:00
Anders Kaseorg d5bf4eed9a check_worker_memory: Fix shellcheck warnings.
In puppet/zulip/files/nagios_plugins/zulip_app_frontend/check_worker_memory line 12:
ps -o vsize,size,pid,user,command --sort -vsize $processes > "$datafile"
                                                ^-- SC2086: Double quote to prevent globbing and word splitting.

In puppet/zulip/files/nagios_plugins/zulip_app_frontend/check_worker_memory line 14:
top_worker=$(cat "$datafile" | head -n2 | tail -n1)
                 ^-- SC2002: Useless cat. Consider 'cmd < file | ..' or 'cmd file | ..' instead.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-08-03 08:42:08 -07:00
Anders Kaseorg eb4855b77b check_email_deliverer_process: Fix shellcheck warnings.
In puppet/zulip/files/nagios_plugins/zulip_app_frontend/check_email_deliverer_process line 16:
elif [ "$(echo "$STATUS" | egrep '(STOPPED)|(STARTING)|(BACKOFF)|(STOPPING)|(EXITED)|(FATAL)|(UNKNOWN)$')" ]
       ^-- SC2143: Use egrep -q instead of comparing output with [ -n .. ].
                           ^-- SC2196: egrep is non-standard and deprecated. Use grep -E instead.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-08-03 08:42:08 -07:00
Anders Kaseorg e3253a7a1b check_email_deliverer_backlog: Fix shellcheck warnings.
In puppet/zulip/files/nagios_plugins/zulip_app_frontend/check_email_deliverer_backlog line 8:
cd /home/zulip/deployments/current
^-- SC2164: Use 'cd ... || exit' or 'cd ... || return' in case cd fails.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-08-03 08:42:08 -07:00
Anders Kaseorg 510c97d861 scripts: Use shell quoting when displaying commands to be run.
This way, commands with arguments containing whitespace or
metacharacters are unambiguously readable.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-07-30 22:39:08 -07:00
Tim Abbott 7853254df7 puppet: Run thumbor by default on voyager systems.
With this change, all that one needs to do to start using thumbor in
production is to set the `THUMBOR_URL` setting.

Since without THUMBOR_URL enabled, the thumbor service doesn't
actually do anything, this is pretty safe.
2018-07-30 16:16:52 -07:00
Tim Abbott 02ae71f27f api: Stop using API keys for Django->Tornado authentication.
As part of our effort to change the data model away from each user
having a single API key, we're eliminating the couple requests that
were made from Django to Tornado (as part of a /register or home
request) where we used the user's API key grabbed from the database
for authentication.

Instead, we use the (already existing) internal_notify_view
authentication mechanism, which uses the SHARED_SECRET setting for
security, for these requests, and just fetch the user object using
get_user_profile_by_id directly.

Tweaked by Yago to include the new /api/v1/events/internal endpoint in
the exempt_patterns list in test_helpers, since it's an endpoint we call
through Tornado. Also added a couple missing return type annotations.
2018-07-30 12:28:31 -07:00
Tim Abbott 07af59d4cc tornado: Split get_events_backend into two functions.
The lower-layer function, now called get_events_backend, is intended
to be called by multiple code paths (including the upcoming
get_events_internal).
2018-07-30 12:28:31 -07:00
Anders Kaseorg dbe65231fc puppet/zulip/files/nagios_plugins/zulip_app_frontend/check_send_receive_time: Avoid shelling out for mv.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-07-19 10:43:37 -07:00
Joshua Schmidlkofer b1a57d144f thumbor: Add production installer/puppet support.
This commits adds the necessary puppet configuration and
installer/upgrade code for installing and managing the thumbor service
in production.  This configuration is gated by the 'thumbor.pp'
manifest being enabled (which is not yet the default), and so this
commit should have no effect in a default Zulip production environment
(or in the long term, in any Zulip production server that isn't using
thumbor).

Credit for this effort is shared by @TigorC (who initiated the work on
this project), @joshland (who did a great deal of work on this and got
it working during PyCon 2017) and @adnrs96, who completed the work.
2018-07-12 20:37:34 +05:30
Tim Abbott afdfdf775c nginx: Set X-Frame-Options header to DENY.
While there are legitimate use cases for embedded Zulip in an iFrame,
they're rare, and it's more important to prevent this category of
attack by default.

Sysadmins can switch this to a whitelist when they want to use frames.
2018-05-30 09:24:17 -07:00
Sampriti Panda 250015a5d5 pgroonga: Fix issues with HTML escaping in queries. 2018-05-28 16:53:30 -07:00
Vishnu Ks 54a002c2e2 requirements: Upgrade pyflakes to 2.0.0.
We fix a few errors that only the new version finds.
2018-05-24 11:31:36 -07:00
Tim Abbott 42da4522a9 puppet-apt: Fix buggy access to caller_module_name.
New versions of Puppet on Ubuntu bionic don't like this.
2018-05-24 09:52:16 -07:00
Tim Abbott b83ba85100 puppet: Switch memcached to using common total_memory_mb value.
This just cuts a bit of unnecessary code duplication.
2018-05-24 09:49:43 -07:00
Tim Abbott 9b4b15cd0a static_asset_compiler: Remove dependency on node packages.
We no longer need or use these, since Zulip installs a pinned version
of node directly with the scripts/setup/install-node tool.

Noticed because in the effort of adding Ubuntu bionic support, we
noticed the package names changed again.
2018-05-24 09:43:45 -07:00
Tim Abbott c843276196 nginx: Fix accidental load-balancing between IPv4 and IPv6.
Apparently, our nginx configuration's use of "localhost", combined
with the default in modern Linux of having localhost resolve to both
the IPv4 and IPv6 addresses on a given machine, resulted in `nginx`
load-balancing requests to a given Zulip server between the IPv4 and
IPv6 addresses.  This, in turn, resulted in irrelevant 502 errors
problems every few minutes on the /events endpoints for some clients.

Disabling IPv6 on the server resolved the problem, as does simply
spelling localhost as 127.0.0.1 for the `nginx` upstreams that we
declare for proxying to non-Django services on localhost.
2018-05-22 11:56:59 -07:00
Tim Abbott 12dcabcdbd docker: Remove need for static_asset_compiler.
Now that the way we're installing from Git involving building a
release tarball with a 2-stage build, we no longer need to do this.
2018-05-20 13:15:21 -07:00
Tim Abbott 61d6965634 puppet: Add option for controlling file upload nginx config.
Now, one can just set `no_serve_uploads` in `zulip.conf` to prevent
`nginx` from serving locally uploaded files.

This should help simplify the S3 integration setup process.
2018-05-17 07:02:30 -07:00
Tim Abbott 6d74ba8271 puppet: Add zulip.conf option for HTTP only.
This option is intended to support situations like a quick Docker
setup where doing HTTPS adds more setup overhead than it's worth.
It's not intended to be used in actual production environments.
2018-05-17 06:58:35 -07:00
Jason Michalski 3d8e424d84 puppet: Add cron package dependency
The Zulip puppet installs various cron jobs and will fail if cron is not
installed. This was found when installing Zulip in a minimal docker
image.
2018-05-16 15:04:31 -07:00
Tim Abbott f2efa122a6 puppet: Include static_asset_compiler in dockervoyager.
This is required to build static assets from Git.
2018-05-15 18:27:01 -07:00
Tim Abbott 9498260516 puppet: Include process_fts_updates in dockervoyager manifest.
This is preferred, since we don't currently have a way to run Django
logic on the postgres hosts with the Docker implementation.

This is a necessary part of removing the need for the docker-zulip
package to patch this file to make Zulip work with Docker.
2018-05-15 15:37:12 -07:00
Tim Abbott ee3cd95bd1 puppet: Remove python 2 psycopg2 package.
We no longer need this, since we're a Python 3 project now.
2018-05-15 15:37:12 -07:00
Tim Abbott bd5e2ddc74 puppet: Extract zulip::process_fts_updates.
In theory, one might want to run this either on the postgres server or
on an application server.
2018-05-15 15:37:12 -07:00
Tim Abbott b656388ee2 check_send_receive_time: Fix mypy Optional issues.
These were not correctly interacting with the Zulip API, and improved
annotations in the API bindings are now actually being type-checked.
2018-05-15 09:06:56 -07:00
Tim Abbott dc0696af74 puppet: Ensure zulip user owns key /etc/zulip files.
The main purpose of this change is to make it guaranteed that
`manage.py register_server --rotate-key` can edit the
/etc/zulip/zulip-secrets.conf configuration via crudini.

But it also adds value by ensuring zulip-secrets.conf is not readable
by other users.
2018-05-06 21:54:02 -07:00
Tim Abbott 3ac7f01e4b puppet: Replace dockervoyager supervisord hack with a better one.
This hack is still pretty bad, but at least it still works on puppet 4.
2018-05-06 21:28:03 -07:00
Tim Abbott eab1d1d9e7 puppet: Fix puppet-apt bug with modern apt-key.
With modern apt-key, the fingerprints are displayed in the more fully
written-out format with spaces, and so `apt-key add` was being run
every time.

This fixes some unnecessary work being done on each puppet run on
Debian stretch.

I would have preferred to not need to do this by upgrading to
upstream, but see #7423 for notes on why that isn't going to work
(basically they broke support for puppet older than 4).
2018-05-06 21:10:24 -07:00
Tim Abbott 7fe19ef8e7 puppet: Remove </% example characters from sample postgres config.
Apparently, these confused the puppet template parser, since they are
somewhat similar to its syntax, resulting in errors trying to use
these templates.  It's easy enough to just remove the example
content from the base postgres config file.
2018-05-06 19:50:38 -07:00
Tim Abbott 63fe39e381 zulip_ops: Disable Ubuntu's built-in update-motd.d files.
We can't really do this in the zulip manifests (since it's sorta a
sysadmin policy decision), but these scripts can cause significant
load when Nagios logs into a server (because many of them take 50ms or
more of work to run).  So we just get rid of them.
2018-05-06 18:47:40 -07:00
Tim Abbott 427b404b9b puppet: Fix detection of Debian/Ubuntu version.
The previous configuration, missing a ^, had the 8.x for Debian Jessie
match Ubuntu 18.04 by accident.
2018-05-05 11:52:45 -07:00
Tim Abbott cf90b9cec0 puppet: Make memory computations work with Puppet 4.
The actual approach for achieving this goal is to take our manual
parsing and move it to the central base.pp.
2018-05-05 11:25:48 -07:00
Tim Abbott 0c1a0a35ec static_asset_compiler: Assume non-trusty is newer.
It seems unlikely we're going to add support for additional older
Debian-based distributions, so it makes sense to just use an else
statement.  This should save a bit of busywork every time we add a new
distro.
2018-05-05 10:50:17 -07:00
Tim Abbott 8ea8bfe285 puppet: Add basic configuration for Ubuntu bionic. 2018-05-05 10:49:09 -07:00
Tim Abbott a03e4784c7 puppet: Add Zulip specific postgres configuration for 10.
Mostly, this involves adding the big block at the bottom and making
10 a variable so that it's easier to compare different versions of
these.

I did an audit of the configuration changes between 9.6 and 10, so
this should be fine, but it hasn't been tested yet.
2018-05-05 10:48:46 -07:00
Tim Abbott 964a1ac8a7 puppet: Commit an upstream version of postgres 10 config. 2018-05-05 10:48:37 -07:00
Tim Abbott 4ee762a52c apt: Add packagecloud repository for Debian. 2018-05-05 10:03:03 -07:00
Tim Abbott 06cfc591fe setup-apt-repo: Require apt-transport-https be installed.
Doing our apt operations over HTTPS has better security properties.
2018-05-05 10:02:50 -07:00
Tim Abbott 4e8487c886 nagios: Bump maximum processes limits.
These seemed to be flapping for no good reason.
2018-05-02 11:12:47 -07:00
Tim Abbott 2f937d81e2 puppet: Add Zulip specific postgres configuration for 9.6.
Mostly, this involves adding the big block at the bottom and making
9.6 a variable so that it's easier to compare different versions of
these.
2018-04-23 18:33:49 -07:00
Tim Abbott 9930e3de09 puppet: Add a stock Postgres 9.6 configuration file from Debian.
This will make it easier to see what we customize.
2018-04-23 18:29:02 -07:00
Tim Abbott 718492638b puppet: Fix name for dhcpcd5 package.
Apparently the name dhcpcd isn't installable.
2018-04-23 11:32:07 -07:00
Tim Abbott 56b0479656 puppet: Clean up indentation in various manifests.
These are inspired by puppet-lint, though we didn't take all of their
changes, since some seem to be bugs in the tool.
2018-04-23 00:15:54 -07:00
Tim Abbott b493748ddb puppet: Use single quotes where valid.
This brings our puppet codebase more in line with the standard puppet
style guide.  Changes done via `puppet-lint --fix`.
2018-04-23 00:15:54 -07:00
Tim Abbott 871078db30 puppet: Fix alignment of arrow operators.
This was done using puppet-lint --fix.
2018-04-23 00:15:54 -07:00
Tim Abbott 19cee30bf8 puppet: Fix use of under-scoped variables. 2018-04-22 23:53:34 -07:00
Tim Abbott 6e55aa2ce6 puppet: Fix mispelled variable name.
Apparently, we weren't uninstalling the old WSGI module properly.
2018-04-22 23:53:34 -07:00