350cf79ba0
Add a nagios check for a notify_tornado consumer
...
(imported from commit 050536bb4ac7384d5b98d5cf6cb7430b2b00dbd5)
2013-04-17 09:24:28 -04:00
99ce1ce9ac
munin: run the humbug_send_receive plugin against the current site.
...
(imported from commit 594e77dd32b9ab0db0002e7dc357ebe93b3ca9cd)
2013-04-16 12:02:42 -04:00
5b1b2257bd
nagios: Commit Luke's testing contact.
...
(imported from commit d88951f42ad7753777b8e0ab2d47b9bb61ff3f76)
2013-04-16 12:02:42 -04:00
bb3b63206a
nagios: Comment out the postgres time checks (they're too noisy).
...
(imported from commit c9569cdbd2909ea7fb8c8c14a681201ee033c62b)
2013-04-16 12:02:42 -04:00
b73ac39a25
nagios: Run check_send_receive_time check against both staging and prod.
...
(imported from commit 749c5f04fba4832debe8a4e702914fa47d1fbeaa)
2013-04-16 12:02:42 -04:00
73886a95fd
nagios: Update app.humbughq.com to use its primary hostname.
...
(imported from commit 39d291e06b0fa223ae4bb76022b26464b969a505)
2013-04-16 12:02:42 -04:00
1b8cf16988
[manual] Update deployment process to run atomically.
...
This requires manual steps on deploy to each of staging and prod:
(1) Run the new update-deployment code to setup the initial deployment directory.
(2) Restart all the programs running in screen sessions.
(3) Deploy the nginx changes and restart nginx.
(imported from commit 1ffe27933ee79274dc0a93d35c9938712de0ef36)
2013-04-12 11:54:50 -04:00
c784457d36
nagios: update feedback bot check to reflect API directory reorg.
...
(imported from commit 01389b0f3f8bf68249cf91b4986e44763fb9a4a0)
2013-04-10 17:40:48 -04:00
fe7fedd252
nagios: add check for send_invite_emails process.
...
(imported from commit b30e55241249a02ee61fac2d3f7abecc4d8318bd)
2013-04-10 16:58:17 -04:00
d89f5670bb
Add nagios check to verify mailchimp is running on staging/app.
...
(imported from commit 2aa79cc6252aadaa0a212b5c60eff9c5c55b7781)
2013-04-05 14:44:18 -07:00
00900b99fe
Bump nginx max file upload limit to 25M.
...
(imported from commit f8f82e718aeab52a0ea228737d144487c49fa706)
2013-04-05 13:07:17 -07:00
64b0106e47
puppet: Log slow autovacuum actions
...
(imported from commit 8f311261cfe41acd1fbda0db5c52f2816e234157)
2013-04-05 13:12:46 -04:00
1443edce00
Access the UserProfile's new email field rather than using User.
...
This is preparatory for stopping using the User model.
(imported from commit a1b0808c8cc2ddd19a25163f91c4f18620c9ce90)
2013-04-02 12:07:08 -04:00
3ff7e885a3
puppet: Add munin graphs for event queues
...
(imported from commit 6ef730551e2ce1e953f116b3797206098bc23455)
2013-04-01 17:22:12 -04:00
b5c2002f62
Add a script to build and install Node.js on one of our servers
...
This should be part of the Puppet config eventually, but I'm not sure how to do
it.
(imported from commit 23063e1e492f6e1cad0afbcf4adffbb6e25aedf7)
2013-04-01 15:21:05 -04:00
29f759e74f
puppet: Install build dependencies of Node.js on app servers
...
I've already installed these packages on staging, and will install them on prod
when the commit hits master. So there is no manual prod deployment step here.
(imported from commit 1b77e771f938305dfbeb797c3ea2a7e3897e78a7)
2013-04-01 15:21:05 -04:00
401fa6063e
[manual] Add get_events URL routing
...
The new nginx configuration file needs to be copied to
/etc/nginx/humbug-include and nginx needs to be restarted when this
commit is deployed.
(imported from commit 6c43f3c2c7a6acee6a852c672c96a38bda01dd0d)
2013-03-28 16:48:52 -04:00
0d51e59fd5
Implement URLs for API redesign.
...
(imported from commit 2020491a737ec4c1e99a63f84eb6cfc594a2dd56)
2013-03-28 07:57:38 -07:00
dc64345c8c
puppet: Add TODO item for Django 1.5 being installed on frontends.
...
(imported from commit b4a884036d8050e2ecb780ce5779997ffce3a7b8)
2013-03-28 07:36:10 -04:00
7103c4df59
puppet: Ensure Python 2.5 is not installed.
...
Django 1.5 won't install if Python 2.5 is installed.
(imported from commit 949b81b6456b0dff4f5d11f0a9d8c7018605c492)
2013-03-28 07:36:10 -04:00
f5a19348e5
[puppet] Increase memcached memory limit to 256MB.
...
Because we're storing 25,000+ message content objects in memcached on
server restart, we were sometimes exceeding the 64MB cap even during
the restart process. 256MB should be safely large enough to not have
the issue but not so large as to seriously consume resources on our
app frontends (which currently have 7GB of RAM).
(imported from commit 1a64319e50c9dadf0bc65e2e4dbf08f4cc1b9c38)
2013-03-27 12:59:24 -04:00
6d41fa4118
[puppet] Add memcached.conf to puppet.
...
(imported from commit 67fcf112ca876e89f69fe58f905cf4bf3195797a)
2013-03-27 12:59:23 -04:00
4fd1281f3c
puppet: Log when Postgres uses temporary files
...
(imported from commit fee3df561c865d5238d4409f0607dba9b1fce4d3)
2013-03-27 11:36:00 -04:00
6a6cfaf56a
puppet: Add more postgresql.conf settings for streaming replication
...
(imported from commit 6cc6180053e038777b65c6bbdac1944d30bcc93b)
2013-03-27 11:36:00 -04:00
2faaf50514
puppet: Add postgres replication user
...
(imported from commit 92c3a28584f6d83cc187314f5e7b17b4def18321)
2013-03-27 11:36:00 -04:00
1658d31004
puppet: Go back to using IP addresses in pg_hba.conf
...
Postgres requires DNS reverse resolution in order to use hostnames.
(imported from commit 57304a6703c06e3860a87d021b7f99561ca4bac6)
2013-03-27 11:36:00 -04:00
7bd21857c8
puppet: When using hostnames in pg_hba.conf, don't specify a CIDR mask
...
(imported from commit c60dd315886e13693de4603c3965db22a3caccfd)
2013-03-27 11:36:00 -04:00
2a334a6328
Tighten rabbitmq thresholds and page_admins
...
(imported from commit 373014bf75346286b55b0ea7d370b21de49ffa33)
2013-03-22 15:55:49 -04:00
fdbf1cff6f
puppet: Fix postgresql.conf typo
...
The name of the archive directory doesn't actually matter, but this
is what we're running in prod, so our config in git should match.
(imported from commit c3fbba4f0c988811b11f2c21cf4a2a32327575aa)
2013-03-19 12:51:46 -04:00
cfe691dc48
puppet: Use hostnames in pg_hba.conf
...
We can do this now that we can use our servers' DNS names internally.
(imported from commit a8e672843e99c1b549a44b3528a5f6698ea9df54)
2013-03-19 12:16:39 -04:00
700b0eebbe
puppet: Add config variables for Postgres master streaming replication
...
(imported from commit 6ffdb08c335452c60223c3129cf13c70a6ef439f)
2013-03-18 12:57:33 -04:00
cb7b80ac28
[manual] Add django-bitfield to the puppet configuration.
...
This will require manual installation on existing servers.
(imported from commit 87bb6fdf1af4f9281e18150e2ae7326da2487eb2)
2013-03-12 12:51:12 -04:00
0fe0cf0ffb
[manual] Implement backend support for authenticating a user via Google.
...
This code adds a dependency on python-django-auth-openid, installable as
django-openid-auth from PyPI.
On prod, one needs to run a syncdb in order to create the required
tables. A database *migration* is not required, as these are new tables
only.
(imported from commit c902a0df8d589d93743b27e480154a04402b2c41)
2013-02-27 10:16:54 -05:00
72d7adce93
nagios: Lower default check intervals and default counts.
...
The defaults are quite large for a small site like ours where on
server down means an outage (e.g. only check every 5 minutes and then
require 4 failures before we alert the admins).
(imported from commit 3b2f436bbb716262f4ee939434749be535ffd6d3)
2013-02-20 16:47:55 -05:00
f547bdce9e
nagios: Add swap check.
...
(imported from commit 37ffdb8dfc117e728acc6c3fe4bae671c66ce4c9)
2013-02-20 11:10:45 -05:00
63b899fc09
puppet: Add note about pagerduty config to nagios.pp.
...
(imported from commit de4642739b4c41468bbb61540482f1ad1b0d37ef)
2013-02-19 15:40:18 -05:00
be834815aa
nagios: Rename paging_admins to page_admins.
...
I think the name is a little clearer.
(imported from commit cd707b76339cb85365f007701c6313aa6d65b4a3)
2013-02-19 15:40:18 -05:00
02ff5bc38d
Nagios: Change new services to paging mode.
...
(imported from commit 4406485179224287f4b7dfbaaa8ed4f97e6debbc)
2013-02-19 15:40:18 -05:00
a89b129781
Fix path to rabbitmqctl and clarify comment
...
(imported from commit 3a9a9c718db3e6221b4073b122f301fb85be6be3)
2013-02-19 15:36:57 -05:00
97b6c1f9ca
Fix path to bots after folder refactor
...
(imported from commit c43e8794e2de5a7feb8b1ae713de2aa63c3d3dd3)
2013-02-19 15:22:55 -05:00
9bb699f917
Add a nagios plugin for checking rabbitmq queue sizes
...
(imported from commit 32bd03bcfe4c4a4221ace17f83adb175f591c8ea)
2013-02-19 15:22:55 -05:00
e819995454
Set better munin thresholds for rabbitmq
...
(imported from commit 032fec63873a05ea7c03fc738d0f82d74322ca3e)
2013-02-15 15:54:12 -05:00
63827c2301
Make the Nagios integration configurable, available, and documented.
...
(imported from commit 1208fc08ed366a892763c3b29b9aeafa90b29981)
2013-02-14 17:50:00 -05:00
67a41143f5
[manual] Add rabbitmq monitoring to munin.
...
When deploying to prod, this will require a manual install of the
rabbitmq_* files to /etc/munin/plugins/ and an edit of
/etc/munin/plugin-conf.d/munin-node.conf
(imported from commit 4c10e634b04200dda1c4f4989e37fe232143240f)
2013-02-12 10:55:01 -05:00
0a0c4bb9a0
[manual] Use rabbitmq for asynchronous presence updating
...
Note: When deploying, restarting the process-user-activity-commandline script is needed
(imported from commit 63ee795c9c7a7db4a40170cff5636dc1dd0b46a8)
2013-02-11 18:05:57 -05:00
c80a8a50f4
puppet: fix app_frontend pip command
...
(imported from commit 58e2e57892d6af68c386f4838d0de26c7d2e861e)
2013-02-11 13:19:42 -05:00
98add6efa2
puppet: Store the machine's type in /etc and add script to reapply the appropriate puppet config
...
Fixes #815
(imported from commit 0c90b74b9b11e4abcc875fbd7a2f0c48356678b2)
2013-02-11 13:19:41 -05:00
801d2c40e2
puppet: Restart iptables-persistent from puppet
...
(imported from commit d46271046b4a341ab5a5f636908583323e107c69)
2013-02-11 12:55:01 -05:00
b1e04f8feb
Fix path to send-receive script post api relayout
...
(imported from commit 69cf0334b7bd363a862f468ea9d5e73b777bdd73)
2013-02-08 16:56:52 -05:00
8132cf2ea2
puppet: Have install-server use its 'type' argument to install the correct puppet manifest
...
This also removes the "run puppet twice" hack, which I believe is no
longer needed for Apache.
(imported from commit 20016c3e8f0d267e04aeef585f19f5910bf01ddc)
2013-02-08 16:06:34 -05:00
Leo Franchi
Tim Abbott
Jessica McKellar
Luke Faraone
Zev Benjamin
Keegan McAllister