Commit Graph

42 Commits

Author SHA1 Message Date
Tim Abbott 26e9d55e16 deployments: Refactor locking libraries into zulip_tools.py.
The code in update-deployment and upgrade-zulip for managing the
deployment lock was nearly identical.
2016-01-11 21:36:42 -08:00
Tim Abbott 186f563176 Fix deployment locks being leaked when a deployment fails.
The point of the lock is to prevent two deployments happening at the
same time and racing with each other, not to prevent doing any future
deployments after an error happens (which is what the current
implementation does in practice).

Addresses part of #208.
2016-01-11 21:36:41 -08:00
Tim Abbott 7395003e6a Fix buggy #! lines using "/usr/bin/env python2.7 -u".
The #! line processing interpreted the argument to pass to `env` as
"python2.7 -u", which obviously isn't a real program.

We fix this by setting the PYTHONUNBUFFERED environment variable
inside the program, which has the same effect.

Thanks to Dan Fedele for the bug report and suggested solution!
2015-11-16 08:46:48 -08:00
Steven Oud d5435fad1d Consistently use /usr/bin/env python2.7 in shebangs and commands. 2015-10-21 22:58:21 +00:00
Tim Abbott d7618ff8dc Don't include zulip-dropbox.png in open source project.
(imported from commit 3b0068521818534ce34f039a3e16c418c448ed9c)
2015-09-24 22:40:59 -07:00
Zev Benjamin 031dfbcc46 zulip_tools.py: Add a make_deploy_path function and make it invokable from the commandline
(imported from commit 94578d117864fba76f9353784734c712c89c4bf2)
2013-11-13 16:26:04 -05:00
Tim Abbott 5f92ccb422 [manual] Move update-prod-static and update-deployments back to tools/.
update-deployment has been replaced by upgrade-zulip for local server
instances, since it won't be running off a git repository, and
update-prod-static won't be needed since we plan on shipping minified
javascript.

When we deploy this, the deployment will fail, and then we'll need to
update the git checkout from which post-receive runs on git.zulip.net.

(imported from commit 86aaedbab09c60ae86ac1d0ae492d0d1bc45569f)
2013-11-04 13:22:41 -05:00
Tim Abbott 56e9ad230e [manual] Move our deployment scripts to scripts/.
This will require updating the post-receive code on git.zulip.net to
work.

(imported from commit 2e51fa2d7b891c1138d3f22ae534cfb8a6cf174c)
2013-10-28 10:54:48 -04:00
Tim Abbott e11ae77ba6 [manual] Rename /home/humbug to /home/zulip.
This may require just doing an mv on the home directory, plus changing
the home directory in /etc/passwd.  It should of course be done carefully.

(imported from commit 660997d897ee6d33563af74f0fc5d4267a911755)
2013-10-08 08:57:29 -04:00
Luke Faraone 80c2eb0367 Repoint to new repository
(imported from commit bc90453bf9776b1e3d05c222f78cc66383278c32)
2013-10-07 13:43:23 -04:00
Tim Abbott 130d381302 Rename humbug_tools to zulip_tools.
(imported from commit 7f21fdc2c2d6ad0bdbd99eb616ffc75c347d8dcb)
2013-08-07 10:00:08 -04:00
Tim Abbott 3fcd6eb9e6 Update humbughq.com => zulip.net in tools.
This shouldn't be pushed until we get graphite.zulip.net working.

(imported from commit c0dd208222385a9ee92ad1ca8fbf141894f7c7b6)
2013-07-25 17:27:45 -04:00
Scott Feeney 3942907eec Fix pathname in an update-deployment error message
(imported from commit 1f21593056d4a272c0ad9f7f16ed5664f9d393f1)
2013-07-12 11:59:11 -04:00
Scott Feeney 2c33320746 Reuse minified JS from previous deploys
This is a big change affecting lots of areas:

* Pipeline no longer deals with JS (though it still minifies CSS)
* A new script, tools/minify-js (called from update-prod-static),
  minifies JavaScripts
* A command-line argument --prev-deploy, if passed to minify-js or
  update-prod-static, is used to copy minified JS from a previous
  deploy (i.e., a previous git checkout), if the source files have
  not changed
* update-deployment passes --prev-deploy
* Scripts are now included with the minified_js template tag, rather
  than Pipeline's compressed_js

Also, as a side benefit of this commit, our Handlebars templates will
no longer be copied into prod-static/ and accessible in production.

Unminification is probably broken, but, per Zev and Trac ticket #1377,
it wasn't working perfectly before this change either.

(Based on code review, this commit has been revised to:
 * Warn if git returns an error in minify-js
 * Add missing output redirects in update-prod-static
 * Use DEPLOY_ROOT instead of manually constructing that directory
 * Use old style formatting)

(imported from commit e67722ea252756db8519d5c0bd6a421d59374185)
2013-07-12 11:59:04 -04:00
Tim Abbott 493d8276e5 update-deployment: Fix buffering argument.
(imported from commit 7a62af4727384d4c1c0f87513aa62c25ab075569)
2013-07-09 14:41:06 -04:00
Steve Howell ca3f2f4c4d Do not buffer deployment output.
(imported from commit a4ffde1096cc8eb5ca64b0feafd33063e6a159d1)
2013-07-09 12:39:21 -04:00
Tim Abbott 8dcdf74af8 update-deployment: Wait up to 5 minutes for lock, rather than just failing.
(imported from commit 3677354841d6084f0917acde4baa79f9c8c3d26b)
2013-07-02 11:36:40 -04:00
Tim Abbott 152ba9ccc7 update-deployment: Remove unnecessary arguments.
(imported from commit b28336b1e45de85a7bc24ee89327d19e106cf3cb)
2013-07-02 11:36:40 -04:00
Scott Feeney 83cd963c49 Remove unused imports
(imported from commit 9e3050c72a2d1137b9096c6cfa1c3945341b9a56)
2013-06-27 16:22:39 -04:00
Zev Benjamin 8884e81bb3 Move the current deployment symlink in restart-server
This will help minimize downtime.

(imported from commit 47fb66f0d2e21fc12f62c69b7c59ca6828553309)
2013-06-05 16:33:43 -04:00
Tim Abbott 0378c43dc6 update-deployment: Move Schema change detected to post-receive.
Since update-deployment is run on the host being deployed to and only
has access to a recent clone of the git repository, it doesn't
necessarily have the old refs available for reverts.

(imported from commit 3652f58a7b165c805822bf6d8a4f0792c629e28e)
2013-06-04 18:42:00 -04:00
Tim Abbott 795173ac86 update-deployment: Move color codes to humbug_tools.py.
(imported from commit 96a79a699ea032714b24aa6572fd188433b979a8)
2013-06-04 18:42:00 -04:00
Zev Benjamin ad2e59b506 update-deployment: Log the fact that we're about to clone the repo
(imported from commit 5dd09ed84c9aec228c700dfff651aea0e74a4a5b)
2013-05-29 15:29:52 -04:00
Zev Benjamin de3b7253c2 update-deployment: Do a fresh git checkout instead of a copy
(imported from commit 13de8b0daf7725b8aa6ea46a7ee26649ceb4c0b0)
2013-05-29 12:00:03 -04:00
Zev Benjamin 4bfda4c49e update-deployment: Purge old deployments after deploy
(imported from commit 9a320617e22a1d71506fbb76443a00c178ae5367)
2013-05-17 01:02:34 -04:00
Zev Benjamin 4cdecdea21 update-deployment: Move constants to humbug_tools
(imported from commit d3699d684b74b15a10f0a464f58623a376f7d486)
2013-05-17 01:02:34 -04:00
Zev Benjamin 6b3b418010 update-deployment: Make the deployment directory a constant
(imported from commit fcbd5f3e98881f33c5b50b28a90b4627593abe83)
2013-05-17 01:02:34 -04:00
Zev Benjamin 578eead441 update-deployment: Create the logs directory in addition to the humbug-deployments directory
The logs directory needs to be there the first time we start a frontend.

(imported from commit c37825f5bd41cbfa3ac22f61d05552e3d4f3ccbb)
2013-05-17 01:02:34 -04:00
Zev Benjamin e4e9613a59 Add a deployment lock to prevent concurrent deployments
The lock file is a directory in case we want to add future metadata
information.

(imported from commit 505a684870baff94f28bf689b0c5ca2e8265a3c0)
2013-05-17 01:02:34 -04:00
Tim Abbott 1b8cf16988 [manual] Update deployment process to run atomically.
This requires manual steps on deploy to each of staging and prod:
(1) Run the new update-deployment code to setup the initial deployment directory.
(2) Restart all the programs running in screen sessions.
(3) Deploy the nginx changes and restart nginx.

(imported from commit 1ffe27933ee79274dc0a93d35c9938712de0ef36)
2013-04-12 11:54:50 -04:00
Tim Abbott 88e2949818 Move update-prod-static etc. out of restart-server.
(imported from commit 763af1c8b1937cda860acc635d9d87c12d66a938)
2013-04-12 11:54:49 -04:00
Keegan McAllister 5f33298bf4 [manual] tools: Move check_output into a shared file
We leave the stuff under api/ alone for now, since we need to be able to ship
it as a standalone thing.

tools/post-receive wasn't using the function anyway.

For push to master: Push this commit, update post-receive per instructions at
the top of that file, then push the rest of the branch to confirm that the hook
still works.

No manual instructions for prod.

(imported from commit 9bcbe14c08d15eda47d82f0b702bad33e217a074)
2013-02-20 16:02:30 -05:00
Keegan McAllister 75704ad1e8 Consistently use #!/usr/bin/env python
At Ksplice we used /usr/bin/python because we shipped dependencies as Debian /
Red Hat packages, which would be installed against the system Python.  We were
also very careful to use only Python 2.3 features so that even old system
Python would still work.

None of that is true at Humbug.  We expect users to install dependencies
themselves, so it's more likely that the Python in $PATH is correct.  On OS X
in particular, it's common to have five broken Python installs and there's no
expectation that /usr/bin/python is the right one.

The files which aren't marked executable are not interesting to run as scripts,
so we just remove the line there.  (In general it's common to have libraries
that can also be executed, to run test cases or whatever, but that's not the
case here.)

(imported from commit 437d4aee2c6e66601ad3334eefd50749cce2eca6)
2013-02-20 16:02:30 -05:00
Tim Abbott 9cc3e29de5 Split restart-server code out of update-deployment.
(imported from commit 3ae913b950be0a0c94fbaf0173012ea315f36d62)
2013-01-31 12:09:16 -05:00
Tim Abbott d121c51584 update-deployment: Delete pyc files on deploy.
(imported from commit 852258b1fd1b837807d11e5f4693012cd438c395)
2013-01-14 11:52:39 -05:00
Keegan McAllister 41e43cb707 Flush memcached on every deploy
This uses pylibmc, the same memcached client library we use from Django.  If
that becomes a problem for whatever reason, all we really need to do is send
"flush_all\n" to TCP port 11211 on localhost.

(imported from commit 0b9736bd31b0549b5dabd4b735706351635a9cf2)
2012-11-26 16:59:37 -05:00
Tim Abbott 33b9a9e8b4 post-receive: Add support for deploying to staging server.
After this commit, we're tracking two branches: "master" and "prod".
Pushed to "prod" deploy to production, while pushes to "master" deploy
to the staging server.

(imported from commit 6ce429a1d6f606fb6136341dc393d93fd1228a21)
2012-11-06 14:12:18 -05:00
Keegan McAllister d9195cd189 update-deployment: Fix undefined name
(imported from commit 3b9ffea14b1e643dd2dacf1212f9f2fbd9e2f33e)
2012-10-19 20:51:56 -04:00
Keegan McAllister f109d23bba Restart runtornado on deploy, as well
We need this for client auto-reload on static file changes, because the server
generation comes from get_updates.

(imported from commit c5e97421883383c12891fdcb177d0f6fda4b4857)
2012-10-18 16:50:01 -04:00
Keegan McAllister 2c9cc7d393 Restart runfcgi on deploy
(imported from commit b6316135b5ab9e7fcedf404e81dc74b2cf9495c5)
2012-10-17 23:47:10 -04:00
Tim Abbott 71dd5aed0a Update automated deployment code.
Will warn about schema changes based on '[schema]' appearing anywhere
in the commit message.

(imported from commit 0092f12c1a2dad3f909ec1934c162776d72263b4)
2012-08-31 14:44:36 -04:00
Tim Abbott 260e646abc Add continuous integration automation scripts.
(imported from commit 61fc5fa1540732b0d1ad02faa95724cc6e2699c0)
2012-08-31 13:53:56 -04:00