Commit Graph

535 Commits

Author SHA1 Message Date
Tim Abbott 17c8527856 nagios: Clean up check_send_receive_time arguments. 2017-03-15 12:52:27 -07:00
Tim Abbott 0ff1f3d663 nagios: Error on heartbeat events in check_send_receive_time.
It was probably going to fail anyway if those show up, but this
produces a clearer failure mode.
2017-03-15 12:51:19 -07:00
Tim Abbott 6a5e98b77e puppet: Increase MaxStartups SSH configuration. 2017-03-08 22:28:16 -08:00
K.Kanakhin 1e441b8d7c uwsgi: Add master mode to the main uwsgi process.
- Enable `master` parameter for `uswgi` configuration.
  It allows cleaning leaked processes if the parent
  process is closed unexpectedly or with SIGKILL command.
  Child processes follow to the master and kill themselves
  after the main process.

Fixes #3855
2017-03-07 21:35:51 -08:00
K.Kanakhin 6a801db1c2 missed-emails-sending: Move email sending to separate queue worker.
- Add new 'missedmessage_email_senders' queue for sending missed messages emails.
- Add the new worker to process 'missedmessage_email_senders' queue.
- Split aggregation missed messages and sending missed messages email
  to separate queue workers.
- Adapt tests for sending missed emails to the new logic.

Fixes #2607
2017-03-07 20:08:40 -08:00
Tim Abbott 67219cf660 puppet: Use restart-server for weekly server restarts.
Using `supervisorctl restart all` carried longer downtime (since it
just restarts everything at the same moment) and was less under our
control; I'm not sure it had any advantages.
2017-03-05 11:36:10 -08:00
Tim Abbott ab2ce9d5e9 lint: Fix whitespace error with recent mypy annotations. 2017-03-04 15:41:11 -08:00
Tim Abbott 75e81253f2 mypy: Work around several new mypy bugs in 0.501. 2017-03-04 15:33:39 -08:00
Raghav Jajodia a3a03bd6a5 mypy: Added Dict, List and Set imports.
Fixed mypy errors associated with the upgrade.
2017-03-04 14:33:44 -08:00
Rishi Gupta 3d07ac0c49 Change timezone-naive datetimes to use timezone.now() where safe to do so.
Change timezone-naive datetimes to use timezone.now() in cases where there
is no change in behavior.
2017-03-01 22:54:28 -08:00
Rishi Gupta 0218422e96 Use time.time() instead of datetime.now() to measure elapsed time.
Both because it is more idiomatic and because we will soon start enforcing
that all datetimes in Zulip are timezone aware.
2017-03-01 22:54:28 -08:00
Tim Abbott aed3632cbb puppet: Convert remaining queue workers to _ style. 2017-02-19 16:18:37 -08:00
Tim Abbott 4b54307d94 puppet: Generate most of main supervisor config with template. 2017-02-19 16:18:37 -08:00
Tim Abbott ae09d55e46 puppet: Move zulip.conf to be a template file. 2017-02-19 16:18:37 -08:00
Kouhei Sutou a2d935a2ee puppet: Fix PostgreSQL user to create PGroonga extension
"root" user isn't a PostgreSQL administrator. "postgres" is a PostgreSQL
administrator.
2017-02-08 12:57:56 -08:00
Tim Abbott fa8045a484 puppet: Add websockets Nagios test to configuration.
Since browser clients send messages via websockets and not the API,
this is an important element in making sure mission-critical Zulip
functionality is working.
2017-02-08 11:13:19 -08:00
Tim Abbott 8db13d0bb9 check_send_receive_time: Use a different state file for websockets.
Otherwise, the two Nagios checks will fight over the same state file
if both are in use.
2017-02-08 11:13:19 -08:00
Tim Abbott ba5f454be5 puppet: Extract zulip::analytics.
I'm not altogether happy with this (a better solution would be
database-level locking), but I think it solves the immediate problem
of folks with 2 servers being very likely to run analytics on both of
them.
2017-02-07 12:29:15 -08:00
Tim Abbott 70388b17d2 puppet: Add missing dependency on ssl-cert. 2017-02-06 15:51:38 -08:00
umkay 76f3d02590 analytics: Add cron job to run analytics jobs.
This adds a cron job to update the Zulip analytics counts, complete
with locking etc.

Substantially tweaked by tabbott.
2017-02-01 17:02:46 -08:00
Tim Abbott 2fb51ff876 puppet: Use SIGINT to restart uwsgi.
This results in a brief service interruption (not a graceful restart),
but fixes a bug where on a `supervisorctl restart zulip-django`, we'd
end up leaking a bunch of uwsgi processes.

The mechanism was that sending SIGHUP to uwsgi was a command for it to
gracefully restart, so it'd start doing that (whereas supervisor
expected it to be dying)... and then supervisor would start up the new
uwsgi process group, resulting in 2 uwsgi process groups running.

This, in turn, led to a memory leak that could eventually result in
OOM kills.
2017-01-28 22:26:12 -08:00
Tim Abbott 36d54cf5ff Replace references to zulip.com/dist with zulip.org/dist.
Now that zulip.org has all the files to distribute, there's no reason
to still point to the soon-to-be-decommissioned zulip.com/dist.
2017-01-28 17:56:25 -08:00
Eitan Adler 0ce29d7ad6 Remove some some duplicate words in copy. 2017-01-23 23:15:04 -08:00
Tim Abbott 4e171ce787 lint: Clean up E126 PEP-8 rule. 2017-01-23 22:06:13 -08:00
Tim Abbott d6e38e2a5c lint: Clean up E123 PEP-8 rule. 2017-01-23 21:34:26 -08:00
Tim Abbott bde2da7dfd lint: clean up PEP-8 W391 rule. 2017-01-23 20:39:02 -08:00
Tim Abbott bbd853e208 puppet: Add redirect to https to zulip.org. 2017-01-22 21:52:50 -08:00
Tim Abbott 44776c43a1 puppet: Add configuration for zulip.org website.
This puppet configuration, plus cloning the zulip.github.io repo and
letsencrypt key setup, is all we need to run a zulip.org server.
2017-01-22 21:48:48 -08:00
JefftheBest1 9de75f5167 Fixed typos with separate 2017-01-12 04:52:05 -08:00
JefftheBest1 ff8639f9db Fixed typos with threshold. 2017-01-12 04:50:20 -08:00
JefftheBest1 5008f45112 Fixed typo in munin.conf.erb 2017-01-12 04:49:19 -08:00
Tim Abbott 3e32102016 nagios: Fix various critical issues not tagged as pageable. 2017-01-06 21:49:20 -08:00
Tim Abbott edebf7619b puppet: Add PAM common_session disabling systemd-login.
This fixes a weird problem with systemd where logging into a server
via ssh frequently has a 15s+ lag.
2017-01-06 21:49:15 -08:00
Tim Abbott 93c2c19775 nagios: Increase process count limits. 2017-01-06 21:49:15 -08:00
Tim Abbott 2c6cb37385 munin: Add default munin configuration template. 2017-01-06 21:44:57 -08:00
Tim Abbott 9ab8e7ba34 nagios: Disable swap checks for servers with no swap. 2017-01-06 21:39:07 -08:00
Tim Abbott 3e01ed1f73 nagios: Increase NTP max_check_attempts.
NTP often suffers from brief interruptions of service that lead to
spurious Nagios alerts; it makes sense to suppress these.
2017-01-06 21:32:43 -08:00
Tim Abbott e4420b08d2 zulip_ops: Disable unattended upgrades of security packages.
Since Zulip does not handle e.g. postgres server restarts gracefully,
it's best for a system administrator to manually trigger security
updates.
2017-01-06 21:30:56 -08:00
Tim Abbott 6f9c73d0e5 zmirror: Update Debathena release in configuration.
The zulip_ops configuration is now for xenial, not obsolete wheezy.
2017-01-06 21:30:41 -08:00
Tim Abbott bd9176d1d9 nagios: Remove some default files.
Nagios ships with a bunch of default configuration files that one
needs to delete in order to configure it.
2017-01-06 21:25:12 -08:00
Tim Abbott 7083899e77 zulip_ops: Add postgres config for enabling Nagios.
The old zulip_ops Nagios configuration depended on Nagios having the
ability to login as the zulip user (with essentially full write
access); this configuration is helpful for limiting nagios to special
"nagios" user with more limited credentials.
2017-01-06 21:24:24 -08:00
Tim Abbott 204edb0f85 zulip_ops: Cleanup pg_hba.conf configuration. 2017-01-06 21:23:51 -08:00
Tim Abbott 30c57eb2ae zulip_ops: Add basic .emacs for production. 2017-01-06 21:20:21 -08:00
Tim Abbott eb87d04168 puppet: Remove xxxxx password hardcoding in recovery.conf. 2017-01-06 21:20:21 -08:00
Tim Abbott 6404a1a5ff zulip_ops: Add nagios-plugins-contrib.
This has a number of useful nagios plugins.
2017-01-06 21:19:59 -08:00
Tim Abbott f7b77008ef zulip_ops: Add aptitude dependency.
This is useful for `aptitude why`.
2017-01-06 21:19:50 -08:00
Tim Abbott 2510a51a8a zulip_ops: Add letsencrypt dependency. 2017-01-06 21:19:31 -08:00
Tim Abbott 65774e1c4f zulip_ops: use check_postgres package from apt. 2017-01-06 21:18:55 -08:00
Tim Abbott 165b4d3126 nagios: Fix check_send_receive_time threshholds.
Previously, the CRITICAL state would never fire (because x > 6 =>
x > 3).  Additionally, 6s is not so unusually high as to deserve being
immediately pageable.
2017-01-06 21:16:37 -08:00
K.Kanakhin 0d8c18a6dd nagios-plugins: Add websocket checking to nagios message sending test.
- Add websocket client to create connection with SockJS websocket server.
  It contains callback method to launch after connection setup.
- Add '--websocket' parameter to 'check_send_receive_time' script to
  check websocket connection.
- Add testing  websocket connection to production installation checking.
- Add cronjob to launch websocket connection nagios test.

This makes it possible for Zulip Nagios monitoring to check for
problems impacting the websockets sending code path, which is what all
web users use.
2016-12-30 15:36:37 -08:00