Commit Graph

348 Commits

Author SHA1 Message Date
Zev Benjamin 8884e81bb3 Move the current deployment symlink in restart-server
This will help minimize downtime.

(imported from commit 47fb66f0d2e21fc12f62c69b7c59ca6828553309)
2013-06-05 16:33:43 -04:00
Zev Benjamin 300e57fcbd Fill memcached caches synchronously before restarting the server
(imported from commit a45fa845e94a1fc6e96a1aafca31e9a6fc2b7526)
2013-06-05 16:33:43 -04:00
acrefoot 8447174325 Make testing message appear on its own line in code reviews
(imported from commit 5d37c7dc51723d08aac7d12fa71e9d7da0f3f62c)
2013-06-05 15:38:16 -04:00
Tim Abbott 74f70f3915 generate-activity-metrics: Display by-week usage numbers changes.
(imported from commit 22bc1592c35c5f86ab89f7f4fc14ba7dbd2375db)
2013-06-05 10:24:23 -04:00
Tim Abbott 42ddabf91a Do only one bucket, as specified on command line.
(imported from commit 4ca4f9609a01a6cfedc55efdbe2423673bbe4904)
2013-06-05 10:24:23 -04:00
Tim Abbott 6f38feca05 Change display format for metrics
(imported from commit eb87f1958fd52098d8bdcf205efc95cdaad56787)
2013-06-05 10:24:23 -04:00
Leo Franchi b6a3446b62 Refactor to use wildcard/exclusion for all
(imported from commit 2874361aef2cbc14a3047f964eb584f27a29323c)
2013-06-05 10:24:23 -04:00
Tim Abbott 5f86ed92f3 generate-activity-metrics: Add support for specifying realm.
(imported from commit 09e2adb6ae8d9fb0a140fbbe9a0f1272e2aa81a7)
2013-06-05 10:24:23 -04:00
Tim Abbott 6425a894f5 generate-activity-metrics: Fix import issues.
(imported from commit a8dd3690200ea42616301adc29d136e7554f45df)
2013-06-05 10:24:23 -04:00
Tim Abbott efcf88a707 puppet: Fix paths in feedback-bot configuration.
(imported from commit e9407af884dc75490de5168e067453e77aa612d7)
2013-06-04 19:48:13 -04:00
Tim Abbott 0c78bb87fa puppet: Add tool to call puppet apply for the current machine type.
(imported from commit 995f32d3daed69f8fc01e950f7c40fc71cdad684)
2013-06-04 19:48:13 -04:00
Tim Abbott 0378c43dc6 update-deployment: Move Schema change detected to post-receive.
Since update-deployment is run on the host being deployed to and only
has access to a recent clone of the git repository, it doesn't
necessarily have the old refs available for reverts.

(imported from commit 3652f58a7b165c805822bf6d8a4f0792c629e28e)
2013-06-04 18:42:00 -04:00
Tim Abbott 795173ac86 update-deployment: Move color codes to humbug_tools.py.
(imported from commit 96a79a699ea032714b24aa6572fd188433b979a8)
2013-06-04 18:42:00 -04:00
Tim Abbott b753eb0c1e [manual] Include the events API in the API tarballs.
Previously our receive API bindings were broken in our API tarballs
because we weren't including the receive API bindings which they used.

This requires our deploying the built API tarball to the prod server
when we deploy it so that the link on /api isn't broken.

(imported from commit 14ecaab34556f4e29c72f4f567d8af73c89d6297)
2013-06-04 15:53:36 -04:00
Leo Franchi f9a99192df Add supervisor conf file for stats
(imported from commit e9104676e714dc36050fef50cabe8386b6c52e4d)
2013-06-03 16:16:22 -04:00
Tim Abbott 8d40864c8b Include the commit that was deployed in deployment finished notices.
With our currently rather slow deploys, this helps with the confusing
case where 2 deploys happen in quick succession and the second is
blocked by the lock.

(imported from commit 5a8bd449ecbe0825838faf72099b67f94db7797a)
2013-05-31 17:01:34 -04:00
Steve Howell 322128a771 Check for whitespace in check-all.
Exit with a failing status if any of your
non-excluded JS/Python files have trailing whitespace.

(imported from commit 0aa184fa85a4a1ddeef6ea40dff57ce43c10e459)
2013-05-31 15:25:35 -04:00
Tim Abbott 05f943395b post-receive: Send a notification when deployment process completes.
This is a little hackish in that I'm using humbug-send rather than
importing humbug_git_config.py to get the API keys and all, but this
was quick to do and I think it's valuable for us to have this output
on our deploys.

(imported from commit 24d7954d04aadeff77ea9cdb76fdc8be055f3ac5)
2013-05-30 16:42:14 -04:00
Zev Benjamin 6e0a5e0293 Add notice of inclusion of third-party code
(imported from commit 62d141e6ebd3368bd1cb3154d6903f9408f2b77c)
2013-05-30 13:51:47 -04:00
Tim Abbott 1ab3918338 Add pre-commit and post-commit hooks for use on dev systems.
As well as a little script to automatically configure them.

(imported from commit c2e7f990f5f2091b5bebdd1b3dad402457943c95)
2013-05-30 10:29:44 -04:00
Steve Howell 35ac129496 Remove unnecessary code in check-all.js.
(imported from commit 3b35f8f951692955402524100efc49cc1bffef69)
2013-05-29 17:52:14 -04:00
Tim Abbott 1960c8e8f1 Show data for every day we have, for now.
(imported from commit d1f26176c1efaf8ba5fa1a74bcd53d38627757d5)
2013-05-29 15:39:53 -04:00
Leo Franchi 444bec285b Add a rough tool to get day over day and week over week activity metrics
(imported from commit fd6e6771fd29a31d553620c0f29ed663e2d680aa)
2013-05-29 15:39:53 -04:00
Zev Benjamin ad2e59b506 update-deployment: Log the fact that we're about to clone the repo
(imported from commit 5dd09ed84c9aec228c700dfff651aea0e74a4a5b)
2013-05-29 15:29:52 -04:00
Tim Abbott 7927edb374 Ensure your dev environment API key matches your ~/.humbugrc.
(imported from commit d4d0699ce912685ede445ed677c46dfeefa68b42)
2013-05-29 15:11:24 -04:00
Zev Benjamin 4cd9abdf10 Fix post-receive to work with new update-deployment
update-deployment now expects the ref name to already be relative to
origin

(imported from commit 0a1792119dd2565c2413acbca25e0fe370f09bf8)
2013-05-29 13:53:41 -04:00
acrefoot c58ac9cc0c Re-focus on the compose box after a send
Re-focuses on the compose box after a send, but only if
the compose-box was opened by responding to the message
at the cursor (by hitting "r", enter, or clicking on the message)

(imported from commit 8e7560c8ea31397b57b2bc3e2e7d9dd996226a6f)
2013-05-29 13:07:45 -04:00
Zev Benjamin 6824c94b7e [manual] Remove dependence on /home/humbug/humbug git checkout on app frontends
Modified files need to be copied into the right place.  The checkout
on git.humbughq.com also needs to be updated.

(imported from commit dbe9e05a0512e1f59c7819dd8d44c2c4e9c83bcf)
2013-05-29 12:00:03 -04:00
Zev Benjamin de3b7253c2 update-deployment: Do a fresh git checkout instead of a copy
(imported from commit 13de8b0daf7725b8aa6ea46a7ee26649ceb4c0b0)
2013-05-29 12:00:03 -04:00
Zev Benjamin cce8dfab84 [manual] Use the same socket across server restarts
We let supervisor create the socket for us by making humbug-django a
fcig-program.  Unfortunately, supevisor doesn't support putting
fcgi-programs in groups (see
https://github.com/Supervisor/supervisor/issues/148), so we have to
restart tornado and django separately.

To deploy, copy the config files over and restart nginx and
supervisor (via stopping and then starting it because restart is
broken).  I believe the automated restart as part of
update-deployment will fail because of the way supervisor treats
programs in groups.  If so, after restarting supervisor, you will
also need to run restart-server manually to fill the caches and then
delete the lock directory in humbug-deployments.

(imported from commit bfb5db7dd42dcbc4bfefa2944355b3cbb2ef9104)
2013-05-23 00:19:17 -04:00
Tim Abbott 00a6cc902e postgres-init-db: Auto-detect whether we need to use sudo.
(imported from commit 4b01672835139c57486811e51399c3f03bc4182b)
2013-05-22 14:58:43 -04:00
Zev Benjamin a4bda36ab4 Add a wrapper to mixpanel so that we can selectively enable metrics collection
(imported from commit 1d72f2e98c4c756a8a988d08731282e4195b8e1a)
2013-05-21 17:56:49 -04:00
Zev Benjamin 8fd72a09bc Restart Django and Tornado separately from the other worker processes
The amount of process downtime during a supervisord-mediated restart
appears to be linear in the number of processes that are being
restarted.  Therefore, restarting just Django and Tornado causes less
downtime than doing them at the same time as the other worker
processes.

(imported from commit 1fa9ef547bcd88caeec49800664e37d5f2fcb7a8)
2013-05-21 16:13:39 -04:00
acrefoot 94c61f5507 [manual] Humbug testing databases now backed by postgres
Currently our test database is backed by sqlite; this commit moves
us to using postgres for our all database needs. This, in conjunction
with the patched django on github, allow us to have fewer hacks and
more true-to-life tests. It also sets the stage for testing the bulk_create
and schema search_path patches made to django.

Developers will need to run:
./tools/postgres-init-test-db
./tools/do-destroy-rebuild-test-database

this is assuming that they have already run:
./tools/postgres-init-db
./tools/do-destroy-rebuild-database

at some point on this pg_cluster. (The ordering is important; it will other-
wise complain about the south_migration table).

(imported from commit c56c6f27e13df7ae10b2e643e65d669dde61af3d)
2013-05-20 23:55:03 -04:00
acrefoot 9d8f847fed [manual] Run server using supervisord
This change will make it so that processes related to the app.humbughq.com
server are run under supervisord, which uses a state machine model to ensure
that programs are running. It also ensure process startup order.

We will need to manually switch the old way of running server (in screen) into
this new way of doing things, on both staging and prod (app_frontend.pp has been
updated appropriately). This means:
1) cp servers/puppet/modules/humbug/files/supervisord/conf.d/humbug.conf /etc/supervisord/conf.d
2) installing the supervisor package.
3) killing those while loops in that screen session
4) mkdir /var/log/humbug (as root)
5) /etc/init.d/supervisord start
6) check that nothing broke

(imported from commit 055269a70973db89acd69049e01b185fabdc8f90)
2013-05-20 23:42:28 -04:00
Steve Howell 62d6360a48 Extracted model code out to unread.js.
Most of the model logic pertaining to unread counts had been in
zephyr.js, along with a couple global variables.  Now the code
is encapsulated in unread.js.  It was a pretty straightforward
extraction with some minor method name changes.  Also, a small
bit of the logic had also been in stream_list.js.

Conflicts:
	tools/jslint/check-all.js

(imported from commit f0abdd48f26ab20c5beaef203479eb5a70dacfff)
2013-05-20 23:42:11 -04:00
Tim Abbott 2bdecd130a Add user interface frontend for editing messages.
The only known outstanding bug with this is that it doesn't properly
handle the updating of a message's highlighting/presence in a narrowed
view (e.g. in theory, a message should disappear if it is edited such
that its subject doesn't match your narrow or it no longer matches
your search).  I think I'll just open a trac ticket about that once
this is merged, since it's a little hairy to deal with and kinda a
marginal use case.

Also it's not pretty, but that should be easy to tweak once we get the
framework merged.

Conflicts:
	tools/jslint/check-all.js

(imported from commit 2d0e3a440bcd885546bd8e28aff97bf379649950)
2013-05-20 23:40:57 -04:00
Leo Franchi a6486a1fa0 Add a tab bar at the top of the Message list to aid navigation
The intent is that this will also make it easier to tell when you are narrowed.

(imported from commit 5e37721fbad304d30fc725aacf2cc9e6560e9da2)
2013-05-20 23:37:57 -04:00
Tim Abbott fe987fb1c3 [manual] Excise South migrations prior to the User merge.
South doesn't properly deal with removing the Django User model, so
this commit redoes our South history to instead start after that
migration has already been applied.  This allows us to get rid of some
annoying hacks.

Note that developers and staging will need to run

./manage.py migrate --delete-ghost-migrations zephyr

in order to clear out the old versions of the migrations.

(imported from commit 7f45ea601b809dde33720f76e7dfb0ab348b0e65)
2013-05-20 23:29:54 -04:00
Zev Benjamin 4bfda4c49e update-deployment: Purge old deployments after deploy
(imported from commit 9a320617e22a1d71506fbb76443a00c178ae5367)
2013-05-17 01:02:34 -04:00
Zev Benjamin 30d417a692 Add tool to purge old deployments
(imported from commit d214e1a843ea5e5902b14551730d004ff04e903d)
2013-05-17 01:02:34 -04:00
Zev Benjamin 4cdecdea21 update-deployment: Move constants to humbug_tools
(imported from commit d3699d684b74b15a10f0a464f58623a376f7d486)
2013-05-17 01:02:34 -04:00
Zev Benjamin 6b3b418010 update-deployment: Make the deployment directory a constant
(imported from commit fcbd5f3e98881f33c5b50b28a90b4627593abe83)
2013-05-17 01:02:34 -04:00
Zev Benjamin 578eead441 update-deployment: Create the logs directory in addition to the humbug-deployments directory
The logs directory needs to be there the first time we start a frontend.

(imported from commit c37825f5bd41cbfa3ac22f61d05552e3d4f3ccbb)
2013-05-17 01:02:34 -04:00
Zev Benjamin e4e9613a59 Add a deployment lock to prevent concurrent deployments
The lock file is a directory in case we want to add future metadata
information.

(imported from commit 505a684870baff94f28bf689b0c5ca2e8265a3c0)
2013-05-17 01:02:34 -04:00
Steve Howell f024a7ab14 Call process_loaded_for_unread() more directly.
This is a prefactoring to eventually eliminate the home_unread_messages
global variable.  More commits to follow.

In order to set up process_loaded_for_unread() not to modify
global variable to get its job done, we want to pull it out of
add_messages(), so that add_messages() doesn't have to pass back
state to the 9 different places in the codebase where it's called.
There are only 2 places where process_loaded_for_unread() get
called after this commit.

In order to facilitate pulling up process_loaded_for_unread(), I
made it so that the contract for add_messages() was to accept
already-hydrated messages.  This way I could hydrate the messages
before calling process_loaded_for_unread() without have to
worry about double-caching them in add_messages.  This will
slightly improve performance, but it was mostly done for code
clarity.

(imported from commit ad5aaad5b1f22c31647370f4c9dcb5f89d7d99a7)
2013-05-14 15:21:27 -04:00
Jessica McKellar 76a57e78ed Enumerate onboarding steps client-side, and give a way to update them.
(imported from commit 0a993976e898cdb20ebb814a19d520572ef8b7e6)
2013-05-14 10:27:39 -04:00
Zev Benjamin 028d3adabf [manual] Lower the max number of child processes that runfcgi can spawn
The default is 50, which we believe was causing too much CPU
contention in prod during restarts.  This commit lowers the max to 20.

The django-command-line script needs to be restarted manually to pick
up this change.

(imported from commit c2d097d2141be50b6222efb4a34142108c241cce)
2013-05-10 17:39:49 -04:00
Leo Franchi 77fd5f1113 Also kill/restart send_missedmessage_reminders on a deploy
(imported from commit 560f6aabb0daec63d55806f8caab76d3fb7d260a)
2013-05-10 09:41:49 -04:00
Leo Franchi 1267ffe542 Show unread counts next to expanded subject names
(imported from commit 6de1145cdec8d7e68395c1f9a7a95cd59136d805)
2013-05-09 10:41:32 -04:00