9d8f847fed
[manual] Run server using supervisord
...
This change will make it so that processes related to the app.humbughq.com
server are run under supervisord, which uses a state machine model to ensure
that programs are running. It also ensure process startup order.
We will need to manually switch the old way of running server (in screen) into
this new way of doing things, on both staging and prod (app_frontend.pp has been
updated appropriately). This means:
1) cp servers/puppet/modules/humbug/files/supervisord/conf.d/humbug.conf /etc/supervisord/conf.d
2) installing the supervisor package.
3) killing those while loops in that screen session
4) mkdir /var/log/humbug (as root)
5) /etc/init.d/supervisord start
6) check that nothing broke
(imported from commit 055269a70973db89acd69049e01b185fabdc8f90)
2013-05-20 23:42:28 -04:00
25b915fa6a
Enable rabbitmq consumser checks on app
...
(imported from commit e3df8bc849dc0e1ae2e7782c0c9be5c08d4818c2)
2013-05-20 23:29:54 -04:00
3d4e239247
Check rabbitmq consumers for all important queues
...
(imported from commit 1279d33e3e1c36ee8da01859875d24b54e14e2e6)
2013-05-17 01:02:35 -04:00
8c2bbf7d45
puppet: Add missing frontend dependency
...
(imported from commit 465c7b1b1e8d6f0e5ef555374187414fc762cf9c)
2013-05-17 01:02:33 -04:00
b940693df4
puppet: Fix conflicting puppet declaration
...
(imported from commit da0ca23a9b8a0a7a6de7d12ef4d47f468075aeec)
2013-05-17 01:02:33 -04:00
c3421b31b9
Include certificate configuration for www.humbughq.com via Comodo
...
This expires in on Aug 11 23:59:59 2013 GMT.
I've set a calendar event for this :)
(imported from commit fb426b703c88dd255536e10285375dc997e47b01)
2013-05-17 01:02:32 -04:00
0a36340216
check_user_zephyr_mirror_liveness: Fix query for new API.
...
(imported from commit f6c477a1d5f0237109be339d099c41c7db5186cc)
2013-05-10 10:46:49 -04:00
d0540efa6a
nagios check_disk: check inode disk usage too.
...
(imported from commit e920c4a11c2797904f0ca397ebdcd8b0a9fef8cf)
2013-05-09 10:35:47 -04:00
5a5ed28ab0
Create aggregate all-active-users data
...
(imported from commit 4009a4eb15a3efb1c05e1e80151db7d1074f0617)
2013-05-01 17:24:38 -04:00
52f6c720d9
Add new stats server to logging
...
(imported from commit b3647ab039c902d09a92082c3e98b5b066e6a5c8)
2013-04-29 16:44:41 -04:00
2aadf6fc6e
[schema] [manual] Create a Postgres text search configuration for use with Humbug
...
Text search was not that great partially because Postgres wasn't
using a ispell dictionary (Postgres term) before. We now pull in
Hunspell and use its dictionary and affix rules.
It is Ok to run with this new configuration before updating our full
text column and index that will be coming in the next few commits.
Manual steps for deploy:
1) On both postgres0 and postgres1 (both before moving on to step 2),
install the hunspell-en-us package
2) On staging, run migration 0022
3) On both postgres0 and postgres1, copy the appropriate postgresql.conf
file over
4) On both postgres0 and postgres1, run `pg_ctlcluster 9.1 main reload`
(imported from commit 706bf0f6ecc46c712cea10b73c34fd9d1dfd4767)
2013-04-27 20:06:26 -04:00
5c0cfc44e7
Add iptables rule for statsd
...
(imported from commit 5311be29fd63151fb9d5a5c0f80ed34f8e8b76f5)
2013-04-26 17:47:00 -04:00
af3ef8636c
puppet: Add Postgres recovery.conf
...
Note that this file needs to be copied over manually as part of the
process of starting up a new replica.
(imported from commit a9f14b695ef2b6b4d48b6180d187c3babf5a667c)
2013-04-22 16:36:09 -04:00
986ca06c44
puppet: Add wal-e to Postgres config
...
(imported from commit 55727a95cc51afb69f14c27df89a6ae287ec0f3f)
2013-04-22 16:36:09 -04:00
f280e7cdfa
puppet: Use deadline scheduler for disks on Postgres master
...
(imported from commit 41061cb4535b94b4afea8c3a2228120073bf06ee)
2013-04-22 16:36:09 -04:00
092cdff061
puppet: Log Postgres checkpoint information
...
(imported from commit 41603ad1c3cf8419d315b44d5679e0817062ced0)
2013-04-22 16:36:09 -04:00
387f63deaa
puppet: Add vm sysctl settings to Postgres configs
...
(imported from commit e557815f490a603da635fb60d39569346a72aa85)
2013-04-22 16:36:09 -04:00
a13b929d1f
puppet: Add script to configure Postgres master disks
...
(imported from commit 61004aa839df8f3fa82ba0c4ea9e2a01ae43464c)
2013-04-22 16:36:09 -04:00
e7cdea1c43
puppet: Tweak Postgres master tunables for its hardware
...
(imported from commit 8644e82d00944203728a3214b2141f778e1c54ed)
2013-04-22 16:36:09 -04:00
336db5c709
puppet: Split Postgres puppet config into master and slave versions
...
(imported from commit adb02cc1904875eb8f56fe272b44dd51bb7d939d)
2013-04-22 16:36:09 -04:00
55449fb724
Fix carbon aggregation by sending to aggregator daemon not cache
...
(imported from commit 1f96a6edd019d8be2844b33588fcdc2ebd61fff6)
2013-04-22 11:07:41 -04:00
b3a3054f64
Slightly raise thresholds for load on nagios
...
(imported from commit 2dbc06c8ba204c10f6d6b590bc4858e07692540b)
2013-04-22 10:22:35 -04:00
499ef75c26
Add configuration files for graphite and statsd
...
(imported from commit bb2c14d816f9ead54bed9da1f227c5e35c9a36bb)
2013-04-18 18:05:51 -04:00
e9f6d9ceff
puppet: Fix location of stats directory
...
(imported from commit b482d6c22e5c1844a65cbee41d1e39378500a9c7)
2013-04-18 17:14:32 -04:00
350cf79ba0
Add a nagios check for a notify_tornado consumer
...
(imported from commit 050536bb4ac7384d5b98d5cf6cb7430b2b00dbd5)
2013-04-17 09:24:28 -04:00
99ce1ce9ac
munin: run the humbug_send_receive plugin against the current site.
...
(imported from commit 594e77dd32b9ab0db0002e7dc357ebe93b3ca9cd)
2013-04-16 12:02:42 -04:00
5b1b2257bd
nagios: Commit Luke's testing contact.
...
(imported from commit d88951f42ad7753777b8e0ab2d47b9bb61ff3f76)
2013-04-16 12:02:42 -04:00
bb3b63206a
nagios: Comment out the postgres time checks (they're too noisy).
...
(imported from commit c9569cdbd2909ea7fb8c8c14a681201ee033c62b)
2013-04-16 12:02:42 -04:00
b73ac39a25
nagios: Run check_send_receive_time check against both staging and prod.
...
(imported from commit 749c5f04fba4832debe8a4e702914fa47d1fbeaa)
2013-04-16 12:02:42 -04:00
73886a95fd
nagios: Update app.humbughq.com to use its primary hostname.
...
(imported from commit 39d291e06b0fa223ae4bb76022b26464b969a505)
2013-04-16 12:02:42 -04:00
1b8cf16988
[manual] Update deployment process to run atomically.
...
This requires manual steps on deploy to each of staging and prod:
(1) Run the new update-deployment code to setup the initial deployment directory.
(2) Restart all the programs running in screen sessions.
(3) Deploy the nginx changes and restart nginx.
(imported from commit 1ffe27933ee79274dc0a93d35c9938712de0ef36)
2013-04-12 11:54:50 -04:00
c784457d36
nagios: update feedback bot check to reflect API directory reorg.
...
(imported from commit 01389b0f3f8bf68249cf91b4986e44763fb9a4a0)
2013-04-10 17:40:48 -04:00
fe7fedd252
nagios: add check for send_invite_emails process.
...
(imported from commit b30e55241249a02ee61fac2d3f7abecc4d8318bd)
2013-04-10 16:58:17 -04:00
d89f5670bb
Add nagios check to verify mailchimp is running on staging/app.
...
(imported from commit 2aa79cc6252aadaa0a212b5c60eff9c5c55b7781)
2013-04-05 14:44:18 -07:00
00900b99fe
Bump nginx max file upload limit to 25M.
...
(imported from commit f8f82e718aeab52a0ea228737d144487c49fa706)
2013-04-05 13:07:17 -07:00
64b0106e47
puppet: Log slow autovacuum actions
...
(imported from commit 8f311261cfe41acd1fbda0db5c52f2816e234157)
2013-04-05 13:12:46 -04:00
1443edce00
Access the UserProfile's new email field rather than using User.
...
This is preparatory for stopping using the User model.
(imported from commit a1b0808c8cc2ddd19a25163f91c4f18620c9ce90)
2013-04-02 12:07:08 -04:00
3ff7e885a3
puppet: Add munin graphs for event queues
...
(imported from commit 6ef730551e2ce1e953f116b3797206098bc23455)
2013-04-01 17:22:12 -04:00
b5c2002f62
Add a script to build and install Node.js on one of our servers
...
This should be part of the Puppet config eventually, but I'm not sure how to do
it.
(imported from commit 23063e1e492f6e1cad0afbcf4adffbb6e25aedf7)
2013-04-01 15:21:05 -04:00
29f759e74f
puppet: Install build dependencies of Node.js on app servers
...
I've already installed these packages on staging, and will install them on prod
when the commit hits master. So there is no manual prod deployment step here.
(imported from commit 1b77e771f938305dfbeb797c3ea2a7e3897e78a7)
2013-04-01 15:21:05 -04:00
401fa6063e
[manual] Add get_events URL routing
...
The new nginx configuration file needs to be copied to
/etc/nginx/humbug-include and nginx needs to be restarted when this
commit is deployed.
(imported from commit 6c43f3c2c7a6acee6a852c672c96a38bda01dd0d)
2013-03-28 16:48:52 -04:00
0d51e59fd5
Implement URLs for API redesign.
...
(imported from commit 2020491a737ec4c1e99a63f84eb6cfc594a2dd56)
2013-03-28 07:57:38 -07:00
dc64345c8c
puppet: Add TODO item for Django 1.5 being installed on frontends.
...
(imported from commit b4a884036d8050e2ecb780ce5779997ffce3a7b8)
2013-03-28 07:36:10 -04:00
7103c4df59
puppet: Ensure Python 2.5 is not installed.
...
Django 1.5 won't install if Python 2.5 is installed.
(imported from commit 949b81b6456b0dff4f5d11f0a9d8c7018605c492)
2013-03-28 07:36:10 -04:00
f5a19348e5
[puppet] Increase memcached memory limit to 256MB.
...
Because we're storing 25,000+ message content objects in memcached on
server restart, we were sometimes exceeding the 64MB cap even during
the restart process. 256MB should be safely large enough to not have
the issue but not so large as to seriously consume resources on our
app frontends (which currently have 7GB of RAM).
(imported from commit 1a64319e50c9dadf0bc65e2e4dbf08f4cc1b9c38)
2013-03-27 12:59:24 -04:00
6d41fa4118
[puppet] Add memcached.conf to puppet.
...
(imported from commit 67fcf112ca876e89f69fe58f905cf4bf3195797a)
2013-03-27 12:59:23 -04:00
4fd1281f3c
puppet: Log when Postgres uses temporary files
...
(imported from commit fee3df561c865d5238d4409f0607dba9b1fce4d3)
2013-03-27 11:36:00 -04:00
6a6cfaf56a
puppet: Add more postgresql.conf settings for streaming replication
...
(imported from commit 6cc6180053e038777b65c6bbdac1944d30bcc93b)
2013-03-27 11:36:00 -04:00
2faaf50514
puppet: Add postgres replication user
...
(imported from commit 92c3a28584f6d83cc187314f5e7b17b4def18321)
2013-03-27 11:36:00 -04:00
1658d31004
puppet: Go back to using IP addresses in pg_hba.conf
...
Postgres requires DNS reverse resolution in order to use hostnames.
(imported from commit 57304a6703c06e3860a87d021b7f99561ca4bac6)
2013-03-27 11:36:00 -04:00
7bd21857c8
puppet: When using hostnames in pg_hba.conf, don't specify a CIDR mask
...
(imported from commit c60dd315886e13693de4603c3965db22a3caccfd)
2013-03-27 11:36:00 -04:00
2a334a6328
Tighten rabbitmq thresholds and page_admins
...
(imported from commit 373014bf75346286b55b0ea7d370b21de49ffa33)
2013-03-22 15:55:49 -04:00
fdbf1cff6f
puppet: Fix postgresql.conf typo
...
The name of the archive directory doesn't actually matter, but this
is what we're running in prod, so our config in git should match.
(imported from commit c3fbba4f0c988811b11f2c21cf4a2a32327575aa)
2013-03-19 12:51:46 -04:00
cfe691dc48
puppet: Use hostnames in pg_hba.conf
...
We can do this now that we can use our servers' DNS names internally.
(imported from commit a8e672843e99c1b549a44b3528a5f6698ea9df54)
2013-03-19 12:16:39 -04:00
700b0eebbe
puppet: Add config variables for Postgres master streaming replication
...
(imported from commit 6ffdb08c335452c60223c3129cf13c70a6ef439f)
2013-03-18 12:57:33 -04:00
cb7b80ac28
[manual] Add django-bitfield to the puppet configuration.
...
This will require manual installation on existing servers.
(imported from commit 87bb6fdf1af4f9281e18150e2ae7326da2487eb2)
2013-03-12 12:51:12 -04:00
0fe0cf0ffb
[manual] Implement backend support for authenticating a user via Google.
...
This code adds a dependency on python-django-auth-openid, installable as
django-openid-auth from PyPI.
On prod, one needs to run a syncdb in order to create the required
tables. A database *migration* is not required, as these are new tables
only.
(imported from commit c902a0df8d589d93743b27e480154a04402b2c41)
2013-02-27 10:16:54 -05:00
72d7adce93
nagios: Lower default check intervals and default counts.
...
The defaults are quite large for a small site like ours where on
server down means an outage (e.g. only check every 5 minutes and then
require 4 failures before we alert the admins).
(imported from commit 3b2f436bbb716262f4ee939434749be535ffd6d3)
2013-02-20 16:47:55 -05:00
f547bdce9e
nagios: Add swap check.
...
(imported from commit 37ffdb8dfc117e728acc6c3fe4bae671c66ce4c9)
2013-02-20 11:10:45 -05:00
63b899fc09
puppet: Add note about pagerduty config to nagios.pp.
...
(imported from commit de4642739b4c41468bbb61540482f1ad1b0d37ef)
2013-02-19 15:40:18 -05:00
be834815aa
nagios: Rename paging_admins to page_admins.
...
I think the name is a little clearer.
(imported from commit cd707b76339cb85365f007701c6313aa6d65b4a3)
2013-02-19 15:40:18 -05:00
02ff5bc38d
Nagios: Change new services to paging mode.
...
(imported from commit 4406485179224287f4b7dfbaaa8ed4f97e6debbc)
2013-02-19 15:40:18 -05:00
a89b129781
Fix path to rabbitmqctl and clarify comment
...
(imported from commit 3a9a9c718db3e6221b4073b122f301fb85be6be3)
2013-02-19 15:36:57 -05:00
97b6c1f9ca
Fix path to bots after folder refactor
...
(imported from commit c43e8794e2de5a7feb8b1ae713de2aa63c3d3dd3)
2013-02-19 15:22:55 -05:00
9bb699f917
Add a nagios plugin for checking rabbitmq queue sizes
...
(imported from commit 32bd03bcfe4c4a4221ace17f83adb175f591c8ea)
2013-02-19 15:22:55 -05:00
e819995454
Set better munin thresholds for rabbitmq
...
(imported from commit 032fec63873a05ea7c03fc738d0f82d74322ca3e)
2013-02-15 15:54:12 -05:00
63827c2301
Make the Nagios integration configurable, available, and documented.
...
(imported from commit 1208fc08ed366a892763c3b29b9aeafa90b29981)
2013-02-14 17:50:00 -05:00
67a41143f5
[manual] Add rabbitmq monitoring to munin.
...
When deploying to prod, this will require a manual install of the
rabbitmq_* files to /etc/munin/plugins/ and an edit of
/etc/munin/plugin-conf.d/munin-node.conf
(imported from commit 4c10e634b04200dda1c4f4989e37fe232143240f)
2013-02-12 10:55:01 -05:00
0a0c4bb9a0
[manual] Use rabbitmq for asynchronous presence updating
...
Note: When deploying, restarting the process-user-activity-commandline script is needed
(imported from commit 63ee795c9c7a7db4a40170cff5636dc1dd0b46a8)
2013-02-11 18:05:57 -05:00
c80a8a50f4
puppet: fix app_frontend pip command
...
(imported from commit 58e2e57892d6af68c386f4838d0de26c7d2e861e)
2013-02-11 13:19:42 -05:00
98add6efa2
puppet: Store the machine's type in /etc and add script to reapply the appropriate puppet config
...
Fixes #815
(imported from commit 0c90b74b9b11e4abcc875fbd7a2f0c48356678b2)
2013-02-11 13:19:41 -05:00
801d2c40e2
puppet: Restart iptables-persistent from puppet
...
(imported from commit d46271046b4a341ab5a5f636908583323e107c69)
2013-02-11 12:55:01 -05:00
b1e04f8feb
Fix path to send-receive script post api relayout
...
(imported from commit 69cf0334b7bd363a862f468ea9d5e73b777bdd73)
2013-02-08 16:56:52 -05:00
8132cf2ea2
puppet: Have install-server use its 'type' argument to install the correct puppet manifest
...
This also removes the "run puppet twice" hack, which I believe is no
longer needed for Apache.
(imported from commit 20016c3e8f0d267e04aeef585f19f5910bf01ddc)
2013-02-08 16:06:34 -05:00
0e66607910
puppet: Fix wiki dependencies
...
(imported from commit a16675d203363b4cd535c2714ad77fd4d0c239d3)
2013-02-08 16:06:34 -05:00
61466c0637
puppet: Fix trac copy and paste error
...
(imported from commit e201ea56a3111423f65e7a386685a4d90bc1080b)
2013-02-08 16:06:34 -05:00
b91d510dac
puppet: Fix apache dependencies
...
(imported from commit 340dde4d045da17dcc7dab492a6fc12e0df3539c)
2013-02-08 16:06:34 -05:00
a4f6a3e83f
puppet: Fix iptables dependency
...
(imported from commit 8c3275f4b875b318931a6f6917e6293b111b247f)
2013-02-08 16:06:34 -05:00
153777bd01
puppet: Fix pip dependencies
...
(imported from commit 34d2ef0fbf39734c66d7ed777c225045e1b8619d)
2013-02-08 16:06:34 -05:00
c76ae76d51
puppet: Ensure a particular 30-postgresql-shm.conf file instead of appending lines to its contents
...
(imported from commit 884e1f2663763d7286146583a097f4548253e347)
2013-02-08 16:06:34 -05:00
fb5e5519d9
puppet: Ensure a particular sshd_config instead of appending a line to its contents
...
(imported from commit 4e745e23afe0cf8e6dd117cdeb6d6ec3a14ef24b)
2013-02-08 16:06:34 -05:00
da95bb2988
puppet: Move all puppetized config files to the humbug module and reference them with puppet URLs
...
(imported from commit f0f325bbad381b87c12c6f7888f4dd5d6989f09f)
2013-02-08 16:06:34 -05:00
beb2ecf5c9
puppet: Ensure that sshd is restarted after turning off password auth
...
(imported from commit 46fef98df9ea1d9ee4038f400cc7c8689d80a0ec)
2013-02-08 16:05:51 -05:00
10f3853abd
puppet: Make common::line use absolute paths
...
(imported from commit c9a3b184433709361243d5c3af13d290cc710ecf)
2013-02-08 16:05:51 -05:00
3e7a6619bd
puppet: Use exec's 'creates' parameter instead of onlyif
...
(imported from commit 94b5e76ca5119443f143e4af5c86e3c16c99dc1e)
2013-02-08 16:05:51 -05:00
a7ca48e2ff
puppet: Use common::line instead of our own common::append
...
(imported from commit 230efb3409eaa451fa28d1655b27a2f5e5f9d382)
2013-02-08 16:05:51 -05:00
eba1008b1c
puppet: Use absolute paths in execs
...
(imported from commit 426c572a4653ad44aa315b43d49c0d6ce001a58d)
2013-02-08 16:05:50 -05:00
5c6a3f3e66
puppet: move all our puppet classes into a 'humbug' module
...
(imported from commit 69b42598c003bbe85dfa4266c56dd019304ea7fb)
2013-02-08 16:05:50 -05:00
bcec450c49
[manual] Remove hunt server from Postgres config
...
Deployment steps: TBD. Will do this when the commit hits master.
(imported from commit 1eebbe873d520cb44c6605845afc9421448c6fe4)
2013-02-08 13:33:28 -05:00
a306c28aa2
puppet: Add documentation on setting up the API distribution site.
...
(imported from commit ea298e8123bb5fee079cf969802fcb8201ed3111)
2013-02-07 14:28:06 -05:00
4cd3fd234c
puppet: Add supervisord configuration for feedback-bot.
...
(imported from commit c7deece3e48d59de856393a4a6b7929757bc1c7c)
2013-02-05 14:27:56 -05:00
f5b44cf349
nagios: Add monitoring for zmirror subscriptions syncing.
...
(imported from commit 2e4ae2c35d589f14b57758cd68a58f8b49b7ecf3)
2013-02-05 14:27:56 -05:00
a7281f7e5a
Add notes to puppet config about manually deployed symlinks.
...
(imported from commit 219f3b407bd83e0728f049820ad06092d6eed12a)
2013-02-01 16:04:11 -05:00
3c6dc21b05
Add pagerduty_nagios.cfg to git.
...
(imported from commit 2f7110d5ab65893afcb83e6f38944bf065abedff)
2013-02-01 14:50:28 -05:00
26aece90b8
nagios: Enable the Nagios commands feature.
...
This allows us to in particular reschedule a Nagios check to run
immediately, which I've in the past found super useful when trying to
figure out whether we actually fixed a problem.
Unfortunately, Nagios config sucks and there's no easy way to create a
group containing all of us as people able to issue commands; you have
to list them in like 8 different places.
(imported from commit 2c1e53330eff1e47e09d0b1917136f101d64e86a)
2013-02-01 14:50:28 -05:00
1fe6045288
nagios: add check that process_user_activity is running.
...
This fixes trac #670 , and also adds the "-u humbug" parameter on the
other check_procs run, since that is a good practice move to help
avoid the check counting its parent process as one of the matches.
(imported from commit 43ae9b4863ba67579a21c86a910b73019f85a538)
2013-02-01 14:50:28 -05:00
2dd2bc8759
nagios: Make default contact_groups not page.
...
This will help us avoid making things accidentally pageable.
Also, explicitly set contact_groups for all our services, to help
encourage making explicit decisions about which new items are
pageable.
(imported from commit 740c6550d4a7091e58681435eeb7aaabf98df75c)
2013-02-01 14:50:28 -05:00
6990260b59
[manual] Minify JavaScript and CSS in production
...
Manual deployment steps: The same Nginx reload as for "Get rid of the
static-access-control mechanism". If deploying both commits at once,
just do it once.
(imported from commit dd8dbbf14b95fce0a4b6f66f462fa0a6b50bfb8c)
2013-01-31 15:41:01 -05:00
ee6f668c4d
puppet: Install django-pipeline on app servers
...
This is in Debian unstable but unfortunately not older versions.
(imported from commit b82654edef270ef06fcf5015dd5d20db1c5c92f5)
2013-01-31 15:34:13 -05:00
f57126d169
puppet: Install yui-compressor on app servers
...
We have lots of choices for JS / CSS minifier. This one works fine
and is in Debian.
(imported from commit bb5a05a8e59d0821e746116af0ef7e3c8ad59aaa)
2013-01-31 15:34:12 -05:00
5e9b0ba79d
[manual] Get rid of the static-access-control mechanism
...
We will minify our code, rather than trying to restrict who can see the
un-minified code. Removing access control first simplifies things.
Manual deployment steps:
scp servers/puppet/files/nginx/humbug-include/app root@staging.humbughq.com:/etc/nginx/humbug-include/
ssh root@staging.humbughq.com service nginx reload
and then the same for app.humbughq.com once deployed to prod.
(imported from commit 63788aa3fa7ba5fd97fcf85b05760abb5e7cae4b)
2013-01-31 15:34:12 -05:00
6e9b8d895c
Add munin plugin for send-receive timing
...
(imported from commit e2ae0775379ce59ab43213e68ade4d3f88b578e6)
2013-01-31 13:02:57 -05:00
14d0ec1096
nagios: add several postgres checks.
...
(imported from commit 5440b2b14d5db11fa9794fe4bcb86a1d6fe90b5d)
2013-01-30 10:55:35 -05:00
a5337033b7
nagios: add a send-receive delay check.
...
(imported from commit ed58f49440fc1e8175ea02eb5d1b0ae8b53472f0)
2013-01-30 10:55:35 -05:00
726ba8dad9
Make Postgres have a log prefix more like what pgFouine requires
...
We'll still need a conversion script, but it should be easy.
pgFouine requires a log prefix of '%t [%p]: [%l-1] '. We instead use
'%m [%c]: [%l-1] ' which contains strictly more data. Specifically,
"%m" is "%t" (time) but with milliseconds and "%c" is "%p" (pid) but
with the process start time.
(imported from commit a0bb583b563bdea0ca19b8b21677df0b9a18092a)
2013-01-28 16:21:42 -05:00
767bf16c1c
Hack up paths to be able to import both the API and Django model.
...
(imported from commit ca89d6bf6208455db4b636198737698ffe575698)
2013-01-24 13:36:11 -05:00
61843b8645
nagios: Add plugin to watch the latency for a message roundtrip.
...
(imported from commit 75888fa4f7ceedb4a95e9b6c4012c32e106ee1ad)
2013-01-24 13:36:11 -05:00
2be39640d3
Add postgres config for new frontend.
...
(imported from commit 0b67ec1cb2c4b06d85d875c14154dd3e453f05c2)
2013-01-17 22:08:39 -05:00
54748643d5
puppet: Install South on our servers.
...
(imported from commit 18b25c6329508a235229992da48a9888b2cc244c)
2013-01-17 22:08:39 -05:00
c9a555b605
Nginx: Drop caching directive for /static
...
This might fix problems where users were running old code even after reloading.
(imported from commit dedc4d513f884aa2bafa0c7cc7a817d6715b48a0)
2013-01-16 15:03:40 -05:00
d0a5d7f7e2
Serve static content in /dist on app
...
(imported from commit b5850ee1f6c6663a27fee14f430f1fae7b690725)
2013-01-15 19:10:09 -05:00
e592e71515
[manual] Use rabbitmq queue to process UserActivity.
...
Before this is deployed, we need to install rabbitmq and pika on the
target server (see the puppet part of this commit for how).
When this is deployed, we need to start the new user activity bot:
./manage.py process_user_activity
in the screen session on the relevant server, or user_activity logs
won't be processed (which will eventually result in all users getting
notifications about how their mirrors are out of date).
(imported from commit 44d605aca0290bef2c94fb99267e15e26b21673b)
2013-01-14 13:28:23 -05:00
6d7ef69cda
nginx: Add config for plant.humbughq.com
...
(imported from commit e90b8e350014b49de53bfd5640442060672e691d)
2013-01-11 17:41:11 -05:00
56660f30f8
nginx: Factor out shared parts of app / staging config
...
(imported from commit e00d5eec1bc58754db6e97935bc803fe3a4fe291)
2013-01-11 17:39:51 -05:00
ef6a5220c8
nginx: Remove unused config humbug-dev
...
(imported from commit 178a320bf56076c61f4010bf6cb89ba04798b4a4)
2013-01-11 17:39:48 -05:00
9730a65f59
nagios: revamp check_user_zephyr_mirror_liveness to monitor sudden drops in mirror use.
...
(imported from commit e92df66c40065584e84c049cfab8d82f71d6dddd)
2013-01-08 10:53:33 -05:00
0655397536
Give the NTP check the default number of retries.
...
It had a max_check_attempts of 1, which makes it susceptible to
network blips.
(imported from commit 20e51878d75bef36d02c5afaab78b8cdd701077f)
2013-01-08 10:53:33 -05:00
8d0a17cbc3
puppet: fix installed packages typo in humbug_apache_base.
...
(imported from commit f503c767cadd9ce5f501233859faafd652f2c4e8)
2013-01-08 10:53:33 -05:00
c186e8ad96
puppet: Add a humbug_bots class with supervisord dependencies.
...
(imported from commit f5bdf6bccf10c7c7f21cc96c415014a26d04c019)
2013-01-08 10:53:32 -05:00
62284f39f4
nagios: monitor feedback bot liveness.
...
(imported from commit 64a97e74b8a44bf0a6faf97398f843d8209b8e36)
2013-01-08 10:53:32 -05:00
5d7b64993b
nagios: Add monitoring for clock skew.
...
(imported from commit 1db47e7c6b28c9dd119e4c50309867d52d3c294b)
2013-01-03 10:21:16 -05:00
ee0b01b8a3
puppet: munin: Document the manual SSH tunnel setup required.
...
The full documentation, referenced in the config file, is at
https://wiki.humbughq.com/Deployment%20process/components#munin .
(imported from commit b7f989accb2ee8c5f400e68bf7a7491115a7d0b3)
2013-01-02 17:41:50 -05:00
7c7263ebfe
puppet: Add munin packages.
...
(imported from commit 4cefc2505b03df7de42c8e7e2adcff9490753476)
2013-01-02 17:41:50 -05:00
9083b0f184
puppet: Add munin and munin-node config files.
...
(imported from commit fa9d7b191fe89894f61f4fd15cb7382663e34837)
2013-01-02 17:41:50 -05:00
d8cd78ec85
nagios: Add and make the default contact a PagerDuty group.
...
(imported from commit 6ab1fd777f3ec7804e6b4f31eaa5efad51993f1a)
2013-01-02 17:41:50 -05:00
cfad014596
nagios: Do check_user_zephyr_mirror_liveness as user humbug.
...
That user has the necessary database certs.
(imported from commit 2f0778a1c5ca5259143b8e7ab25b557a6ddd76df)
2013-01-02 17:41:49 -05:00
a40b5da432
puppet: Use PostgreSQL's internal logging system
...
This also requires disabling logrotate for postgres log files.
(imported from commit eeedb87a4f488829c59eddecc041654e762d6d0e)
2013-01-02 16:56:57 -05:00
f7237ac2aa
puppet: Install postgres config files
...
(imported from commit a01bd1d0b14b1436e50605fc14c6267fe77989fc)
2013-01-02 16:56:57 -05:00
779191b30e
puppet: Add postgres server configuration files
...
(imported from commit bbfe6e9246a9a172a48c4cf8257d32936de009f9)
2013-01-02 16:56:57 -05:00
45e6550270
Add starting point for zmirror puppet configuration.
...
I expect this will be fleshed out more later.
(imported from commit c05e994e737ca2ab4ca24d4a4ac03ce46c52336a)
2013-01-02 15:03:42 -05:00
a2f26f1106
Nagios: Fix retry interval of zephyr_mirror_forwarding check.
...
(imported from commit eae984669dad0a2dd6779092e9759909fbbd1da7)
2012-12-19 11:21:47 -05:00
1aa825e6d0
puppet: Add generic nagios monitoring for postgres.humbughq.com
...
(imported from commit 9e732b69580bc3da8507a5fe6fdd81f044fb4443)
2012-12-13 11:30:02 -05:00
cd73e13601
puppet: Add python postgres module (python-psycopg2) to humbug_app_frontend packages
...
(imported from commit 3f41629d6f1e2c26458e223bc2135a53ac3bdd14)
2012-12-13 11:30:02 -05:00
dc6d48611d
puppet: Accept traffic on port 5432 (postgresql)
...
(imported from commit bf30d0af2377209f3d5c10add3a526a1fee28dd8)
2012-12-13 11:30:02 -05:00
4d2899b5f8
puppet: Add postgres config
...
(imported from commit ca932a1a1af7e7236ff1f47785acf4b412b16650)
2012-12-13 11:30:02 -05:00
155e2c4943
install-server: Allow users to use an alternate humbug root
...
(imported from commit 1b5e57c1ec8c175733c8fb15343b096c46e6b6b2)
2012-12-13 11:30:02 -05:00
ab373d6457
install-server: Use "apt-get -y" instead of "yes '' | apt-get"
...
(imported from commit 0157f179928d69a5f0ff574a2d003187f28c1772)
2012-12-13 11:21:25 -05:00
11f8dc644d
install-server: Quote filenames in existance checks
...
(imported from commit ed17b65d875b5321c57fff4e16263282cccf4dff)
2012-12-13 11:21:20 -05:00
16a5af0b8d
install-server: Accept apt-get prompts
...
(imported from commit 2f69d047488d3d82689a4fc71777e3c4667b36d5)
2012-12-13 11:21:16 -05:00
b6b0ab80cb
install-server: Check for humbug-self-signed.key before running
...
(imported from commit 0c5ab50fbb278db740690522e2354f33f1958cc7)
2012-12-13 11:21:11 -05:00
d90fb5d00f
install-server: Use named constants for file paths
...
(imported from commit 6178f8110c6f79c642dd3c8cde149be6e4d72e16)
2012-12-13 11:21:05 -05:00
375f8e3540
nagios: disable flap detection.
...
This will ensure that we always get state change alerts, even when the
service is changing states frequently.
(imported from commit 57fa5a941dd1a6042eb782dbac2fed0e4cb934ba)
2012-12-11 10:22:52 -05:00
5212a48d3b
puppet: Only install ipython on the app servers
...
I'd like to have this everywhere, but it has a bunch of X dependencies.
(imported from commit c0c4089909ab7b3a5b6f9620c19eb0435b72762c)
2012-12-05 14:12:36 -05:00
01b070a122
puppet: Install emacs without X support
...
(imported from commit b15e63613c6b6cf1815a8f5bb660bd8c8e80604c)
2012-12-05 14:12:36 -05:00
c34d39caf8
puppet: Separate out some packages needed only by the app servers
...
(imported from commit 447837f1d5f68d0bf160dec2a9a37fc1cb7e62d5)
2012-12-05 14:12:36 -05:00
d8b4cefccb
nagios: Remove AllowOverride AuthConfig
...
We don't use it.
(imported from commit 875148e24e0de2815737b6bc03eeb7f1cb8d770d)
2012-12-03 17:54:16 -05:00
2cf49c4ff2
nagios: Go straight to the service detail page
...
This bypasses the side navigation frame, but I think said frame currently
provides negative value.
(imported from commit b067d546e4a7fb95e7de2a35be7e7f947c7a0da1)
2012-12-03 17:54:16 -05:00
d435f29308
Add X-Frame-Options header on nagios, trac, wiki
...
Prevents clickjacking attacks.
(imported from commit 8b3872e607d8a4e714c280a3226465fde0d5a6ed)
2012-12-03 17:54:16 -05:00
7c495d7232
Move the nagios Apache authentication directives to a <Location> block
...
Following the trac Apache config.
(imported from commit 01e773f2361d85f45f190f6ade2510b84a2f88ee)
2012-12-03 17:54:15 -05:00
41319fe820
Rework the nagios Apache config as a proper vhost
...
This also adds HSTS. Based on the trac Apache config.
Fixes #435 .
Suggested viewing: git show -w
(imported from commit e7e9fe74687b88497ddb21f74febfc7fdf9b1979)
2012-12-03 17:54:15 -05:00
acrefoot
Leo Franchi
Zev Benjamin
Luke Faraone
Tim Abbott
Jessica McKellar
Luke Faraone
Keegan McAllister
Luke O'Malley