9730a65f59
nagios: revamp check_user_zephyr_mirror_liveness to monitor sudden drops in mirror use.
...
(imported from commit e92df66c40065584e84c049cfab8d82f71d6dddd)
2013-01-08 10:53:33 -05:00
0655397536
Give the NTP check the default number of retries.
...
It had a max_check_attempts of 1, which makes it susceptible to
network blips.
(imported from commit 20e51878d75bef36d02c5afaab78b8cdd701077f)
2013-01-08 10:53:33 -05:00
8d0a17cbc3
puppet: fix installed packages typo in humbug_apache_base.
...
(imported from commit f503c767cadd9ce5f501233859faafd652f2c4e8)
2013-01-08 10:53:33 -05:00
c186e8ad96
puppet: Add a humbug_bots class with supervisord dependencies.
...
(imported from commit f5bdf6bccf10c7c7f21cc96c415014a26d04c019)
2013-01-08 10:53:32 -05:00
62284f39f4
nagios: monitor feedback bot liveness.
...
(imported from commit 64a97e74b8a44bf0a6faf97398f843d8209b8e36)
2013-01-08 10:53:32 -05:00
5d7b64993b
nagios: Add monitoring for clock skew.
...
(imported from commit 1db47e7c6b28c9dd119e4c50309867d52d3c294b)
2013-01-03 10:21:16 -05:00
ee0b01b8a3
puppet: munin: Document the manual SSH tunnel setup required.
...
The full documentation, referenced in the config file, is at
https://wiki.humbughq.com/Deployment%20process/components#munin .
(imported from commit b7f989accb2ee8c5f400e68bf7a7491115a7d0b3)
2013-01-02 17:41:50 -05:00
7c7263ebfe
puppet: Add munin packages.
...
(imported from commit 4cefc2505b03df7de42c8e7e2adcff9490753476)
2013-01-02 17:41:50 -05:00
9083b0f184
puppet: Add munin and munin-node config files.
...
(imported from commit fa9d7b191fe89894f61f4fd15cb7382663e34837)
2013-01-02 17:41:50 -05:00
d8cd78ec85
nagios: Add and make the default contact a PagerDuty group.
...
(imported from commit 6ab1fd777f3ec7804e6b4f31eaa5efad51993f1a)
2013-01-02 17:41:50 -05:00
cfad014596
nagios: Do check_user_zephyr_mirror_liveness as user humbug.
...
That user has the necessary database certs.
(imported from commit 2f0778a1c5ca5259143b8e7ab25b557a6ddd76df)
2013-01-02 17:41:49 -05:00
a40b5da432
puppet: Use PostgreSQL's internal logging system
...
This also requires disabling logrotate for postgres log files.
(imported from commit eeedb87a4f488829c59eddecc041654e762d6d0e)
2013-01-02 16:56:57 -05:00
f7237ac2aa
puppet: Install postgres config files
...
(imported from commit a01bd1d0b14b1436e50605fc14c6267fe77989fc)
2013-01-02 16:56:57 -05:00
779191b30e
puppet: Add postgres server configuration files
...
(imported from commit bbfe6e9246a9a172a48c4cf8257d32936de009f9)
2013-01-02 16:56:57 -05:00
45e6550270
Add starting point for zmirror puppet configuration.
...
I expect this will be fleshed out more later.
(imported from commit c05e994e737ca2ab4ca24d4a4ac03ce46c52336a)
2013-01-02 15:03:42 -05:00
a2f26f1106
Nagios: Fix retry interval of zephyr_mirror_forwarding check.
...
(imported from commit eae984669dad0a2dd6779092e9759909fbbd1da7)
2012-12-19 11:21:47 -05:00
1aa825e6d0
puppet: Add generic nagios monitoring for postgres.humbughq.com
...
(imported from commit 9e732b69580bc3da8507a5fe6fdd81f044fb4443)
2012-12-13 11:30:02 -05:00
cd73e13601
puppet: Add python postgres module (python-psycopg2) to humbug_app_frontend packages
...
(imported from commit 3f41629d6f1e2c26458e223bc2135a53ac3bdd14)
2012-12-13 11:30:02 -05:00
dc6d48611d
puppet: Accept traffic on port 5432 (postgresql)
...
(imported from commit bf30d0af2377209f3d5c10add3a526a1fee28dd8)
2012-12-13 11:30:02 -05:00
4d2899b5f8
puppet: Add postgres config
...
(imported from commit ca932a1a1af7e7236ff1f47785acf4b412b16650)
2012-12-13 11:30:02 -05:00
155e2c4943
install-server: Allow users to use an alternate humbug root
...
(imported from commit 1b5e57c1ec8c175733c8fb15343b096c46e6b6b2)
2012-12-13 11:30:02 -05:00
ab373d6457
install-server: Use "apt-get -y" instead of "yes '' | apt-get"
...
(imported from commit 0157f179928d69a5f0ff574a2d003187f28c1772)
2012-12-13 11:21:25 -05:00
11f8dc644d
install-server: Quote filenames in existance checks
...
(imported from commit ed17b65d875b5321c57fff4e16263282cccf4dff)
2012-12-13 11:21:20 -05:00
16a5af0b8d
install-server: Accept apt-get prompts
...
(imported from commit 2f69d047488d3d82689a4fc71777e3c4667b36d5)
2012-12-13 11:21:16 -05:00
b6b0ab80cb
install-server: Check for humbug-self-signed.key before running
...
(imported from commit 0c5ab50fbb278db740690522e2354f33f1958cc7)
2012-12-13 11:21:11 -05:00
d90fb5d00f
install-server: Use named constants for file paths
...
(imported from commit 6178f8110c6f79c642dd3c8cde149be6e4d72e16)
2012-12-13 11:21:05 -05:00
375f8e3540
nagios: disable flap detection.
...
This will ensure that we always get state change alerts, even when the
service is changing states frequently.
(imported from commit 57fa5a941dd1a6042eb782dbac2fed0e4cb934ba)
2012-12-11 10:22:52 -05:00
5212a48d3b
puppet: Only install ipython on the app servers
...
I'd like to have this everywhere, but it has a bunch of X dependencies.
(imported from commit c0c4089909ab7b3a5b6f9620c19eb0435b72762c)
2012-12-05 14:12:36 -05:00
01b070a122
puppet: Install emacs without X support
...
(imported from commit b15e63613c6b6cf1815a8f5bb660bd8c8e80604c)
2012-12-05 14:12:36 -05:00
c34d39caf8
puppet: Separate out some packages needed only by the app servers
...
(imported from commit 447837f1d5f68d0bf160dec2a9a37fc1cb7e62d5)
2012-12-05 14:12:36 -05:00
d8b4cefccb
nagios: Remove AllowOverride AuthConfig
...
We don't use it.
(imported from commit 875148e24e0de2815737b6bc03eeb7f1cb8d770d)
2012-12-03 17:54:16 -05:00
2cf49c4ff2
nagios: Go straight to the service detail page
...
This bypasses the side navigation frame, but I think said frame currently
provides negative value.
(imported from commit b067d546e4a7fb95e7de2a35be7e7f947c7a0da1)
2012-12-03 17:54:16 -05:00
d435f29308
Add X-Frame-Options header on nagios, trac, wiki
...
Prevents clickjacking attacks.
(imported from commit 8b3872e607d8a4e714c280a3226465fde0d5a6ed)
2012-12-03 17:54:16 -05:00
7c495d7232
Move the nagios Apache authentication directives to a <Location> block
...
Following the trac Apache config.
(imported from commit 01e773f2361d85f45f190f6ade2510b84a2f88ee)
2012-12-03 17:54:15 -05:00
41319fe820
Rework the nagios Apache config as a proper vhost
...
This also adds HSTS. Based on the trac Apache config.
Fixes #435 .
Suggested viewing: git show -w
(imported from commit e7e9fe74687b88497ddb21f74febfc7fdf9b1979)
2012-12-03 17:54:15 -05:00
a9c16b38ce
Fix up whitespace in Apache configs
...
(imported from commit 605253abf9b029e18774f80979d23c60ffca034b)
2012-12-03 17:54:15 -05:00
922b44a1da
Add iptables config for zmirror.humbughq.com
...
For now we allow all UDP traffic. I'll look into doing something clever.
This isn't puppetized, either.
(imported from commit bdf53df87a5f6c8af6d950b25946b5ec8a4f910b)
2012-12-03 17:43:04 -05:00
ed0cb0a5f8
Puppetize nginx.conf
...
Fixes #201 .
(imported from commit 0feaff372d94009fa51dabf2bda55062826e2ed5)
2012-12-03 15:58:16 -05:00
4aa7615234
Nginx: Use $host instead of $server_name
...
The latter is just the first name in the 'server_name' directive.
The former uses the HTTP Host header, if provided.
This fixes the redirect
from http://zephyr.humbughq.com
to https://zephyr.humbughq.com
(imported from commit be47b05f4f055bb2d1d82aebbe155579f49c538d)
2012-11-30 17:12:42 -05:00
500a5e29c3
Nginx: Redirect unknown hostnames to https://humbughq.com
...
(imported from commit f6dd65c1db033d09f1df8f0a5972f067f3aeb80a)
2012-11-30 15:32:32 -05:00
ac18c533c8
Nginx: Serve the cert for zephyr.humbughq.com rather than app.humbughq.com
...
This will cause SSL errors for anyone still using the deprecated
app.humbughq.com name, which we concluded is (almost?) nobody.
(imported from commit 7f3c149a4064e7bdae8ec944f2bb8a482df6f90d)
2012-11-30 15:32:32 -05:00
2fcb9cfd49
Nginx: Make zephyr.humbughq.com an alias for humbughq.com
...
(imported from commit d23ef5aeed990a04f294b7dffe322b8d174c1f07)
2012-11-30 15:32:32 -05:00
0f20150a81
Nagios: move /var/lib/nagios/humbug-api to /usr/local/lib/humbug
...
(imported from commit ff3ff1e3cc54a4c556479e62e058002229143627)
2012-11-26 16:58:51 -05:00
d7b3afef6b
Send Nagios alerts to Humbug
...
Fixes #385 .
(imported from commit 7dac013debd6ccff031fc4da0dd7185e198b4498)
2012-11-26 14:42:55 -05:00
b609840e82
puppet: Install memcached on app servers
...
We use the default Debian configuration, which listens on localhost only.
(imported from commit efa8333c7fa423e71a99ec06b2b420cae36fddfb)
2012-11-26 11:59:48 -05:00
be27ec1ad4
nagios: Change zephyr mirror liveness check to only care about aggregate statistics.
...
Too many individual users occasionally don't update their mirrors,
causing us to be permanently alerting; we have sufficient user
notification at this point (plus Waseem keeping an eye on /activity)
that we don't need to alert on individual users.
We do, however, still care if something happens (say, Linerva going
down) that causes many users' mirrors to go down.
(imported from commit 392952c95739e183d4a711120e3a963671cec289)
2012-11-26 10:31:29 -05:00
75526a2c67
nagios: Drop ssh -o StrictHostKeyChecking=no
...
This is bad for security.
I've checked that all currently known hosts for nagios@nagios.humbughq.com
match one of our existing servers. When adding servers to nagios in the
future, it will be necessary to do an initial manual ssh from nagios@ and check
the host key fingerprint.
(imported from commit adfd1d29f03343d4be04e87c5e26a018f31e5194)
2012-11-26 00:25:15 -05:00
043f0d8e15
nagios: Use lowercase host aliases
...
(imported from commit c653d5948894e651a5040339e8cd6af50af712b3)
2012-11-26 00:25:15 -05:00
1939b55b5a
nagios: Remove wiki.humbughq.com
...
This is the same machine as git.humbughq.com.
(imported from commit 8aa9306668d672052aa38a2f4453cb0127ab5cc5)
2012-11-26 00:25:15 -05:00
f761643724
nagios: Monitor bots.humbughq.com
...
(imported from commit 83cb5cc3c3c4bf54e1339d877bd60cd05586783b)
2012-11-26 00:25:15 -05:00
f8a065ed2c
nagios: Remove dev.humbughq.com
...
This is an alias for staging.humbughq.com.
(imported from commit b2d2777e57773052dd59b3f5c067e23eafb60681)
2012-11-26 00:25:15 -05:00
b9c8f4a770
nagios: Enable ssh-based checks on all machines
...
(imported from commit 3905ad03cc2ed5dec6f8eb6b20d4b4f0896f164c)
2012-11-26 00:25:14 -05:00
25916d9ce5
nagios: Check Debian update availability
...
(imported from commit 59da03c409281b6b2f5cf3612e5f7bd0caa76226)
2012-11-26 00:25:14 -05:00
4b1a2e8a8a
Install nagios-plugins-basic on every machine
...
We need this for check_debian_packages.
(imported from commit 588dfbe7d5b69acfd1db7fcf4060b64ec5151c2d)
2012-11-26 00:25:14 -05:00
5836462cb4
Import check_debian_packages Nagios plugin
...
From http://exchange.nagios.org/directory/Plugins/Operating-Systems/Linux/check-debian-packages/details
(imported from commit 6304a2aa315a91fd48e9ad79fcdb584ba8a2ccb4)
2012-11-26 00:25:14 -05:00
685deba16b
Puppetize APT::Periodic config
...
(imported from commit 2ccdeb4f9c8173a83c7014987977304187651f67)
2012-11-26 00:25:14 -05:00
8dd1f1efc0
Puppetize iptables config
...
(imported from commit aa58d06255aaf5a2979a7fcc4e0746c1ac2d91a7)
2012-11-19 11:06:33 -05:00
905f2d3235
nagios: add monitoring for the liveness of our users' zephyr mirrors.
...
Using the check_user_zephyr_mirror_liveness plugin.
(imported from commit c17e112fe8696fab583a0dbc228ea9fb6e6988b0)
2012-11-16 11:36:33 -05:00
5498557b4b
nagios: add monitoring for Zephyr mirroring.
...
Using the check_zephyr_mirror plugin.
(imported from commit 8ef5d4870c3a2ec547729c191de838504cea1d3d)
2012-11-16 11:36:16 -05:00
b86ddf4ddc
nagios: add a check_user_zephyr_mirror_liveness plugin.
...
It will alert when our users' mirrors don't appear to be running, as
assessed by having recently made a get_message API request.
(imported from commit 4b8c5f51b007568a90a92f7b095c51f3566d5117)
2012-11-16 11:28:37 -05:00
fcf5eb8f1f
nagios: add a check_zephyr_mirror plugin.
...
It checks the output of api/bots/check-mirroring and alerts if we
aren't able to send and receive mirrored Zephyrs.
(imported from commit 6c9abc380fca955d00462f829fa7dcadfef24221)
2012-11-16 11:28:28 -05:00
a0aa1b31c8
nagios: Add remote disk and load checks.
...
(imported from commit 1f0a1f5540212357ac2ed0c8d50fb2291a1812ed)
2012-11-16 11:25:22 -05:00
5ec66f467b
nagios: add basic monitoring for new servers, and a hostgroup for SSH-based checks.
...
(imported from commit 7e5ad2bb024eb935bf6640a894cad762e45c0ab0)
2012-11-16 11:25:14 -05:00
010f15c66e
nagios: add a test contact and contact group for testing new alerts.
...
(imported from commit 9cc1ef2b7af6c84bfd87dc38c6a558ea3b36d267)
2012-11-16 11:24:46 -05:00
609ff161a1
nagios: send the full multi-line alert data in Nagios e-mails.
...
(imported from commit c906bd2b6a2a1e0f009e4743a0f7b1968f371919)
2012-11-16 11:24:45 -05:00
a4289e6553
Install the 'host' command on our servers.
...
(imported from commit 21171e553cf6974cd19170c47a79e3e7389b5534)
2012-11-14 16:57:21 -05:00
35171b9d3e
Tweak gitit config
...
The "signup code" was left over from a very early era. We now use HTTP auth
and there's no way to register an account within Gitit at all.
(imported from commit 20f1e10de1fd978d0045c2fed2254e37ab6f7b6c)
2012-11-07 17:46:46 -05:00
b9452b5644
apache/ports.conf: Document which part is custom
...
(imported from commit b25c4ce8847509ce07d98e1caee402aa33369c4d)
2012-11-07 17:46:46 -05:00
adab0c1880
Add staging/dev certs to puppet configuration.
...
(imported from commit 1415d909a4619adecc3b43ad0b7817f473bc2a73)
2012-11-06 16:59:28 -05:00
0c25a091d2
Install ntp on all our servers.
...
(imported from commit 59cb9ef4350a8ec9a528623fb3247e7ba6c15405)
2012-11-06 14:19:14 -05:00
4a3bf99fa0
Add staging server nginx configuration.
...
(imported from commit 560621e48098925d526c7a29681dc03c4508a878)
2012-11-06 14:12:18 -05:00
4aa91336b5
install-server: Set the hostname automatically.
...
(imported from commit c8081845eac0cf8d21711eb7c836f2ce39a6d9f6)
2012-11-02 10:50:59 -04:00
44bee33c41
Setup servers id_rsa for the humbug user as well.
...
(imported from commit 19a36fc829882eff7bfdab3a22765c5df6bdbd0d)
2012-11-02 10:50:59 -04:00
7d35c3135b
Puppet: Update default classes for building new server.
...
(imported from commit 7283498779108992456c98d3d18b01751ccbb5b6)
2012-11-02 10:50:59 -04:00
c0b75ed93a
Add Nagios config files to git.
...
(imported from commit 5d6ba166cf35afdd76ca4f2cfc8a13988cfdeaea)
2012-11-01 10:47:50 -04:00
2c577d70b3
Restart apache2, ssh, and nginx after running puppet.
...
(imported from commit 0e0b7fd10d1742efe86ff6bdab8d3ac4e4d291a9)
2012-10-30 14:21:31 -04:00
545476c6a4
Puppet: Don't install sudo
...
We're no longer using it, and fewer setuid programs is better.
Fixes #225 .
(imported from commit 68b06bb8afedc0854d96ad072b5de718832932ed)
2012-10-30 12:30:18 -04:00
f0c2421f00
Customize the 404 error page Nginx serves for missing static content
...
(imported from commit 70fc821f9ae29b8a902c48ce57e39273c90f57ff)
2012-10-30 11:00:10 -04:00
233cba1380
Remove old copy of my.cnf
...
This is now found at servers/puppet/files/mysql/my.cnf . It's identical to the
one we remove here.
(imported from commit fe70529a716842a57058386eaf697a23a6d6251c)
2012-10-30 00:03:31 -04:00
9de0d3c7b8
install-server: Add note about hostname domain
...
(imported from commit 1fb9fd79cfc88d7ec191b1116b45e4b7b7bb9c2f)
2012-10-29 23:21:00 -04:00
e9a8d3e447
install-server: Restart sshd after configuring it
...
Needed for disabling password authentication to take effect. Fixes #202 .
(imported from commit f082b08076ecb1f0facb3f66d5b416ea545dfa4f)
2012-10-29 23:21:00 -04:00
85ead77956
Splitting SSL out didn't fix the dependency problems.
...
(imported from commit 14515ea1abecb6212842a4a5cba90eb705f65755)
2012-10-29 13:19:41 -04:00
e6f196f372
Move certs .gitignore to where the certs are now.
...
(imported from commit c1a92978a7d835cb3d3eec5647ef7aa4f4f73b35)
2012-10-29 13:19:41 -04:00
0f211673ac
Puppet: Add missing packages python-flup
...
Needed for Django runfcgi.
(imported from commit cfd1f20a2f7a08c21e8ab3b321c2928a28319a54)
2012-10-27 12:11:22 -04:00
d71b9594fe
Configure nginx for the new server
...
(imported from commit d073276912ea844e75fd710689f152fd7a2213c7)
2012-10-27 11:38:15 -04:00
92b10e3bc2
settings.py: Change deployed check
...
This is security-critical so we have two checks.
(imported from commit adaa1cefe2d08526cdaac2fb0d8cc02773390224)
2012-10-27 11:18:51 -04:00
cbdbc12ab0
Remove config for Apache as app frontend
...
(imported from commit ae4072bdf59cdfccec76eeee7fd3b99a899eaa19)
2012-10-27 10:56:14 -04:00
fdba3addc8
Update puppet configuration to be slightly more accurate.
...
This still needs a lot of work though.
(imported from commit 4472488c399f7c5b96bcf900b1a5e957625cb450)
2012-10-26 14:58:05 -04:00
8c20bafb98
Add Nagios configuration to puppet.
...
(imported from commit 34c09661e63b31bd177b9704b69a0d8d0d644de7)
2012-10-26 14:15:27 -04:00
f8540dcdae
Wrap some other extremely long lines
...
(imported from commit e7d55f318c8865ca953bf4520d1b07f7e84a4aeb)
2012-10-25 15:22:18 -04:00
2f5e7ba1ad
Properly format wiki code highlighting CSS
...
(imported from commit 216ff55aa55847fa61a32a1d87737de273c21ae4)
2012-10-25 15:22:18 -04:00
811604021e
puppet: Install requests module on servers.
...
(imported from commit 2aa2e5740065e9e46020cca0d22854e57e81a407)
2012-10-24 16:24:02 -04:00
7be626ff2f
Run trac on top of Apache rather than the standalone server.
...
(imported from commit 2e9ee69a6f36b4c145d83abdf975bbe5d7ec1c7d)
2012-10-23 15:58:02 -04:00
c5f262987f
Add apache2 site configuration for trac.
...
(imported from commit 81d173070a449168d6d8e08a557134dbda66f2bb)
2012-10-23 14:52:04 -04:00
7079efba8d
puppet: Make humbug's authorized_keys file owned by humbug.
...
(imported from commit 4237203722d7782cb5c479e408966494bc703149)
2012-10-23 14:52:04 -04:00
7a56448a35
Move humbug-self-signed.key to new certs directory.
...
(imported from commit ed8ff32739e21cca3d6173e19bb425da8a3a18ea)
2012-10-23 14:52:04 -04:00
9ed02e220c
puppet: Disable creating new servers as wiki/frontend for now.
...
(imported from commit 20250365a3d82479d5dd33ecb0ee9b5c3db68029)
2012-10-23 13:32:34 -04:00
9ff633ccfe
Add nginx config
...
This is not yet hooked up to Puppet.
(imported from commit 1d0368285a5a9ef5a4af0651001db4930cf77578)
2012-10-22 18:00:37 -04:00
0dcc7c2914
puppet: Remove humbug from sudoers
...
(imported from commit cdc8aafdcbfafe5fe97a18dfd32b5a7f15a77102)
2012-10-15 14:56:12 -04:00
45273643ec
Move humbug-self-signed.crt into a common directory
...
(imported from commit 0c914b87c06be7cd7b370d8d0f38efc9f1aaf57f)
2012-10-15 13:29:47 -04:00
Jessica McKellar
Zev Benjamin
Tim Abbott
Keegan McAllister