4cd3fd234c
puppet: Add supervisord configuration for feedback-bot.
...
(imported from commit c7deece3e48d59de856393a4a6b7929757bc1c7c)
2013-02-05 14:27:56 -05:00
f5b44cf349
nagios: Add monitoring for zmirror subscriptions syncing.
...
(imported from commit 2e4ae2c35d589f14b57758cd68a58f8b49b7ecf3)
2013-02-05 14:27:56 -05:00
a7281f7e5a
Add notes to puppet config about manually deployed symlinks.
...
(imported from commit 219f3b407bd83e0728f049820ad06092d6eed12a)
2013-02-01 16:04:11 -05:00
3c6dc21b05
Add pagerduty_nagios.cfg to git.
...
(imported from commit 2f7110d5ab65893afcb83e6f38944bf065abedff)
2013-02-01 14:50:28 -05:00
26aece90b8
nagios: Enable the Nagios commands feature.
...
This allows us to in particular reschedule a Nagios check to run
immediately, which I've in the past found super useful when trying to
figure out whether we actually fixed a problem.
Unfortunately, Nagios config sucks and there's no easy way to create a
group containing all of us as people able to issue commands; you have
to list them in like 8 different places.
(imported from commit 2c1e53330eff1e47e09d0b1917136f101d64e86a)
2013-02-01 14:50:28 -05:00
1fe6045288
nagios: add check that process_user_activity is running.
...
This fixes trac #670 , and also adds the "-u humbug" parameter on the
other check_procs run, since that is a good practice move to help
avoid the check counting its parent process as one of the matches.
(imported from commit 43ae9b4863ba67579a21c86a910b73019f85a538)
2013-02-01 14:50:28 -05:00
2dd2bc8759
nagios: Make default contact_groups not page.
...
This will help us avoid making things accidentally pageable.
Also, explicitly set contact_groups for all our services, to help
encourage making explicit decisions about which new items are
pageable.
(imported from commit 740c6550d4a7091e58681435eeb7aaabf98df75c)
2013-02-01 14:50:28 -05:00
6990260b59
[manual] Minify JavaScript and CSS in production
...
Manual deployment steps: The same Nginx reload as for "Get rid of the
static-access-control mechanism". If deploying both commits at once,
just do it once.
(imported from commit dd8dbbf14b95fce0a4b6f66f462fa0a6b50bfb8c)
2013-01-31 15:41:01 -05:00
ee6f668c4d
puppet: Install django-pipeline on app servers
...
This is in Debian unstable but unfortunately not older versions.
(imported from commit b82654edef270ef06fcf5015dd5d20db1c5c92f5)
2013-01-31 15:34:13 -05:00
f57126d169
puppet: Install yui-compressor on app servers
...
We have lots of choices for JS / CSS minifier. This one works fine
and is in Debian.
(imported from commit bb5a05a8e59d0821e746116af0ef7e3c8ad59aaa)
2013-01-31 15:34:12 -05:00
5e9b0ba79d
[manual] Get rid of the static-access-control mechanism
...
We will minify our code, rather than trying to restrict who can see the
un-minified code. Removing access control first simplifies things.
Manual deployment steps:
scp servers/puppet/files/nginx/humbug-include/app root@staging.humbughq.com:/etc/nginx/humbug-include/
ssh root@staging.humbughq.com service nginx reload
and then the same for app.humbughq.com once deployed to prod.
(imported from commit 63788aa3fa7ba5fd97fcf85b05760abb5e7cae4b)
2013-01-31 15:34:12 -05:00
6e9b8d895c
Add munin plugin for send-receive timing
...
(imported from commit e2ae0775379ce59ab43213e68ade4d3f88b578e6)
2013-01-31 13:02:57 -05:00
14d0ec1096
nagios: add several postgres checks.
...
(imported from commit 5440b2b14d5db11fa9794fe4bcb86a1d6fe90b5d)
2013-01-30 10:55:35 -05:00
a5337033b7
nagios: add a send-receive delay check.
...
(imported from commit ed58f49440fc1e8175ea02eb5d1b0ae8b53472f0)
2013-01-30 10:55:35 -05:00
726ba8dad9
Make Postgres have a log prefix more like what pgFouine requires
...
We'll still need a conversion script, but it should be easy.
pgFouine requires a log prefix of '%t [%p]: [%l-1] '. We instead use
'%m [%c]: [%l-1] ' which contains strictly more data. Specifically,
"%m" is "%t" (time) but with milliseconds and "%c" is "%p" (pid) but
with the process start time.
(imported from commit a0bb583b563bdea0ca19b8b21677df0b9a18092a)
2013-01-28 16:21:42 -05:00
767bf16c1c
Hack up paths to be able to import both the API and Django model.
...
(imported from commit ca89d6bf6208455db4b636198737698ffe575698)
2013-01-24 13:36:11 -05:00
61843b8645
nagios: Add plugin to watch the latency for a message roundtrip.
...
(imported from commit 75888fa4f7ceedb4a95e9b6c4012c32e106ee1ad)
2013-01-24 13:36:11 -05:00
2be39640d3
Add postgres config for new frontend.
...
(imported from commit 0b67ec1cb2c4b06d85d875c14154dd3e453f05c2)
2013-01-17 22:08:39 -05:00
54748643d5
puppet: Install South on our servers.
...
(imported from commit 18b25c6329508a235229992da48a9888b2cc244c)
2013-01-17 22:08:39 -05:00
c9a555b605
Nginx: Drop caching directive for /static
...
This might fix problems where users were running old code even after reloading.
(imported from commit dedc4d513f884aa2bafa0c7cc7a817d6715b48a0)
2013-01-16 15:03:40 -05:00
d0a5d7f7e2
Serve static content in /dist on app
...
(imported from commit b5850ee1f6c6663a27fee14f430f1fae7b690725)
2013-01-15 19:10:09 -05:00
e592e71515
[manual] Use rabbitmq queue to process UserActivity.
...
Before this is deployed, we need to install rabbitmq and pika on the
target server (see the puppet part of this commit for how).
When this is deployed, we need to start the new user activity bot:
./manage.py process_user_activity
in the screen session on the relevant server, or user_activity logs
won't be processed (which will eventually result in all users getting
notifications about how their mirrors are out of date).
(imported from commit 44d605aca0290bef2c94fb99267e15e26b21673b)
2013-01-14 13:28:23 -05:00
6d7ef69cda
nginx: Add config for plant.humbughq.com
...
(imported from commit e90b8e350014b49de53bfd5640442060672e691d)
2013-01-11 17:41:11 -05:00
56660f30f8
nginx: Factor out shared parts of app / staging config
...
(imported from commit e00d5eec1bc58754db6e97935bc803fe3a4fe291)
2013-01-11 17:39:51 -05:00
ef6a5220c8
nginx: Remove unused config humbug-dev
...
(imported from commit 178a320bf56076c61f4010bf6cb89ba04798b4a4)
2013-01-11 17:39:48 -05:00
9730a65f59
nagios: revamp check_user_zephyr_mirror_liveness to monitor sudden drops in mirror use.
...
(imported from commit e92df66c40065584e84c049cfab8d82f71d6dddd)
2013-01-08 10:53:33 -05:00
0655397536
Give the NTP check the default number of retries.
...
It had a max_check_attempts of 1, which makes it susceptible to
network blips.
(imported from commit 20e51878d75bef36d02c5afaab78b8cdd701077f)
2013-01-08 10:53:33 -05:00
8d0a17cbc3
puppet: fix installed packages typo in humbug_apache_base.
...
(imported from commit f503c767cadd9ce5f501233859faafd652f2c4e8)
2013-01-08 10:53:33 -05:00
c186e8ad96
puppet: Add a humbug_bots class with supervisord dependencies.
...
(imported from commit f5bdf6bccf10c7c7f21cc96c415014a26d04c019)
2013-01-08 10:53:32 -05:00
62284f39f4
nagios: monitor feedback bot liveness.
...
(imported from commit 64a97e74b8a44bf0a6faf97398f843d8209b8e36)
2013-01-08 10:53:32 -05:00
5d7b64993b
nagios: Add monitoring for clock skew.
...
(imported from commit 1db47e7c6b28c9dd119e4c50309867d52d3c294b)
2013-01-03 10:21:16 -05:00
ee0b01b8a3
puppet: munin: Document the manual SSH tunnel setup required.
...
The full documentation, referenced in the config file, is at
https://wiki.humbughq.com/Deployment%20process/components#munin .
(imported from commit b7f989accb2ee8c5f400e68bf7a7491115a7d0b3)
2013-01-02 17:41:50 -05:00
7c7263ebfe
puppet: Add munin packages.
...
(imported from commit 4cefc2505b03df7de42c8e7e2adcff9490753476)
2013-01-02 17:41:50 -05:00
9083b0f184
puppet: Add munin and munin-node config files.
...
(imported from commit fa9d7b191fe89894f61f4fd15cb7382663e34837)
2013-01-02 17:41:50 -05:00
d8cd78ec85
nagios: Add and make the default contact a PagerDuty group.
...
(imported from commit 6ab1fd777f3ec7804e6b4f31eaa5efad51993f1a)
2013-01-02 17:41:50 -05:00
cfad014596
nagios: Do check_user_zephyr_mirror_liveness as user humbug.
...
That user has the necessary database certs.
(imported from commit 2f0778a1c5ca5259143b8e7ab25b557a6ddd76df)
2013-01-02 17:41:49 -05:00
a40b5da432
puppet: Use PostgreSQL's internal logging system
...
This also requires disabling logrotate for postgres log files.
(imported from commit eeedb87a4f488829c59eddecc041654e762d6d0e)
2013-01-02 16:56:57 -05:00
f7237ac2aa
puppet: Install postgres config files
...
(imported from commit a01bd1d0b14b1436e50605fc14c6267fe77989fc)
2013-01-02 16:56:57 -05:00
779191b30e
puppet: Add postgres server configuration files
...
(imported from commit bbfe6e9246a9a172a48c4cf8257d32936de009f9)
2013-01-02 16:56:57 -05:00
45e6550270
Add starting point for zmirror puppet configuration.
...
I expect this will be fleshed out more later.
(imported from commit c05e994e737ca2ab4ca24d4a4ac03ce46c52336a)
2013-01-02 15:03:42 -05:00
a2f26f1106
Nagios: Fix retry interval of zephyr_mirror_forwarding check.
...
(imported from commit eae984669dad0a2dd6779092e9759909fbbd1da7)
2012-12-19 11:21:47 -05:00
1aa825e6d0
puppet: Add generic nagios monitoring for postgres.humbughq.com
...
(imported from commit 9e732b69580bc3da8507a5fe6fdd81f044fb4443)
2012-12-13 11:30:02 -05:00
cd73e13601
puppet: Add python postgres module (python-psycopg2) to humbug_app_frontend packages
...
(imported from commit 3f41629d6f1e2c26458e223bc2135a53ac3bdd14)
2012-12-13 11:30:02 -05:00
dc6d48611d
puppet: Accept traffic on port 5432 (postgresql)
...
(imported from commit bf30d0af2377209f3d5c10add3a526a1fee28dd8)
2012-12-13 11:30:02 -05:00
4d2899b5f8
puppet: Add postgres config
...
(imported from commit ca932a1a1af7e7236ff1f47785acf4b412b16650)
2012-12-13 11:30:02 -05:00
375f8e3540
nagios: disable flap detection.
...
This will ensure that we always get state change alerts, even when the
service is changing states frequently.
(imported from commit 57fa5a941dd1a6042eb782dbac2fed0e4cb934ba)
2012-12-11 10:22:52 -05:00
5212a48d3b
puppet: Only install ipython on the app servers
...
I'd like to have this everywhere, but it has a bunch of X dependencies.
(imported from commit c0c4089909ab7b3a5b6f9620c19eb0435b72762c)
2012-12-05 14:12:36 -05:00
01b070a122
puppet: Install emacs without X support
...
(imported from commit b15e63613c6b6cf1815a8f5bb660bd8c8e80604c)
2012-12-05 14:12:36 -05:00
c34d39caf8
puppet: Separate out some packages needed only by the app servers
...
(imported from commit 447837f1d5f68d0bf160dec2a9a37fc1cb7e62d5)
2012-12-05 14:12:36 -05:00
d8b4cefccb
nagios: Remove AllowOverride AuthConfig
...
We don't use it.
(imported from commit 875148e24e0de2815737b6bc03eeb7f1cb8d770d)
2012-12-03 17:54:16 -05:00
Tim Abbott
Keegan McAllister
Leo Franchi
Jessica McKellar
Zev Benjamin
Luke O'Malley
Luke Faraone