Commit Graph

49 Commits

Author SHA1 Message Date
Leo Franchi 65b247edec Add a event_queue gc hook for missed messages
(imported from commit a799abb35a9622e6aa59e9499cab3281ccb6859f)
2013-05-24 21:23:59 -04:00
Tim Abbott 5e22778843 Use a function for stopping/restarting time logging for longpolling.
(imported from commit 11b772deaa126fcc7e7605d467022b22d9e98cb0)
2013-04-24 10:44:56 -04:00
Luke Faraone 71a91197fa Enable absolute imports.
See PEP 328[1] for details. This feature was introduced in Python 2.5 and
will become mandatory in Python 3.

[1]: http://www.python.org/dev/peps/pep-0328

(imported from commit 7444eeba8a08d5f91b94c7921848f2274979bd76)
2013-04-23 09:51:17 -07:00
Leo Franchi d7a33485ad Register a tornado atexit handler to disconnect from rabbitmq
(imported from commit b70650070f1df548794a9e3ff2948d134fd0c5de)
2013-04-16 11:49:03 -04:00
Zev Benjamin c6df5af84d Import tornado ioloop directly instead of passing it from runtornado
(imported from commit ce901d7daca81bb1682827175bdcd83d3e5c2344)
2013-03-28 16:57:48 -04:00
Zev Benjamin e4fba59538 Make event queues persistent across Tornado restarts using cPickle
(imported from commit 1434d1e9d394d725827b1740c0c07249d5e716ed)
2013-03-28 16:57:48 -04:00
Zev Benjamin 401fa6063e [manual] Add get_events URL routing
The new nginx configuration file needs to be copied to
/etc/nginx/humbug-include and nginx needs to be restarted when this
commit is deployed.

(imported from commit 6c43f3c2c7a6acee6a852c672c96a38bda01dd0d)
2013-03-28 16:48:52 -04:00
Zev Benjamin 8f4eaa63ad Initial event system implementation
This version has several limitations that are addressed in later
commits in this series.

(imported from commit 5d452b312d4204935059c4d602af0b9a8be1a009)
2013-03-28 16:48:14 -04:00
Tim Abbott 7685e47f07 Fix Tornado idle time logging.
When we added rabbitmq usage within Tornado, we inadvertently caused
the Tornado ioloop to be initialized in runtornado.py's imports,
before we overwrote the _poll method.  The end result was that we
weren't running the our instrumented Tornado poll function.

Fix this by moving that code to its own file which we import at the
top of runtornado.py, and adding comments documenting the situation so
we don't break this in some future import reorganization.

(imported from commit 016717476f10566fef4ed2b656f29f865d2084db)
2013-03-28 14:40:49 -04:00
Luke Faraone 0d51e59fd5 Implement URLs for API redesign.
(imported from commit 2020491a737ec4c1e99a63f84eb6cfc594a2dd56)
2013-03-28 07:57:38 -07:00
Luke Faraone b63e5e3235 Refactor runtorando AsyncDjangoHandler URLs into a list.
(imported from commit f5c65f69dd2ccb5a0ba6339b28b8c486cde2c7ff)
2013-03-28 07:53:36 -07:00
Tim Abbott 144987e581 logs: Provide total time passed for longpolled requests.
Also improve display of times passed -- we now use display short times
in milliseconds for easier reading.

(imported from commit 08e1e7e6acbef48453080864946f7602a3395e7c)
2013-03-27 08:19:27 -04:00
Tim Abbott f3ad2d7a6b Centralize dispatch logic for which queue processor to use.
Previous we had around 4 copies of the logic for deciding whether we
should publish data via a SimpleQueueClient queue, a
TornadoQueueClient queue, or to directly handle the operation, which
resulted in their getting out of sync and buggy (see e.g. the previous
commit).

We need to add a lock around adding things to the queue to work around
a bug with pika's BlockingConnection.

I should note that the previous logic in some places had a bunch of
tests of the form "elif settings.TEST_SUITE" for doing the work that
would have been done by the queue processor directly; these should
have just been "else" clauses -- since we generally want that code to
run on development environments whether or not the test suite is
currently running.

(imported from commit 16bdbed4fff04b1bda6fde3b16bee7359917720b)
2013-03-27 08:19:27 -04:00
Leo Franchi f64bc59dfb Automatically register consumer once asynchronous connection is completed
(imported from commit 3d860a9b79a42beffbd3b73f49aa2c25986dd3c6)
2013-03-22 15:55:49 -04:00
Leo Franchi c346abe6e2 Send Tornado callback notifications via RabbitMQ
(imported from commit 83814d8c6b78fe725aedea9d98fb588ed31123e7)
2013-03-22 15:55:40 -04:00
Keegan McAllister 23b8833ec5 Handle Tornado callback notify in a more generic, less HTTP-centric way
(imported from commit 85a74d1b40461236c4c95ad688e9796ab50f0bbf)
2013-03-22 15:51:23 -04:00
Keegan McAllister 591816d80e Use TornadoQueueClient in runtornado
(imported from commit cc4d5505987dc2826d177ff93d037b17f78072b8)
2013-03-22 15:39:07 -04:00
Tim Abbott 722918aa61 Use humbug_finish() wrapper for finish() with required arguments.
I think this is a little cleaner than the previous approach.

(imported from commit fa6233721f43c94003de643d7801a88a156affd2)
2013-03-15 12:53:07 -04:00
Tim Abbott 419e153422 runtornado: Mark block of get_response modified from the original Django.
(imported from commit 383d6960f0043175c026956e1724ec1d1157767b)
2013-03-15 12:53:06 -04:00
Tim Abbott ade7146cd9 Disable tornado request logging in favor of our Django logging.
(imported from commit 2144b2ac1cdb6062bb32fca4cc69ea3f4570cf69)
2013-03-15 12:53:06 -04:00
Tim Abbott 4df6736225 Call the Django response middleware from Tornado views.
But discard any changes the Django response middleware may have made
to anything other than the content.

This allows us to, for example, output our nice database query logging
for get_updates requests.

(imported from commit e1d2fd38ceb4d73ff50bdfaad7c72ddb24d0fe16)
2013-03-15 12:53:06 -04:00
Tim Abbott 549577a928 Pass the original request object to Tornado's finish().
This is preparatory for running the Django response middleware on
our Tornado responses.

(imported from commit 05da8ea9cb663a928b2f98a928f3992aae4f067c)
2013-03-15 12:53:06 -04:00
Zev Benjamin 22457277f4 Move send_with_safety_check functionality into AsyncDjangoHandler
This will mainly be useful in the event system branch, where we want
to actually send a response from a file other than tornadoviews.py.

(imported from commit b7ae9bb9b062215ab44eb5f0a3a72d6baeee1d07)
2013-03-14 18:48:15 -04:00
Tim Abbott acab720502 Add support for making Tornado dump its stack via SIGUSR1/2.
With this change,

pkill -SIGUSR1 -f runtornado

will dump the stack and SIGUSR2 will enable an interactive debugging
session.

This fixes #613 for Tornado which was the original motive for that
ticket; I'm not sure whether we want to do this for our Django
processes as well, but it would be easy to do so if we did.

(imported from commit a7de7c6070f4bf0404bed6f434e6a6b291d66a26)
2013-03-06 14:19:32 -05:00
Tim Abbott 3b7d61e45f tornado: Get User and UserProfile objects from a memcached.
This commit has the effect of eliminating all of the non-UserActivity
database queries from the Tornado process -- at least in the uncached
case.

This is safe to do, if a bit fragile, since our Tornado code only
accesses these objects (as opposed to their IDs) in a few places that
are all fine with old data, and I don't expect us to add any new ones
soon:

* UserActivity logging, which I plan to move out of Tornado entirely

* Checking whether we're authenticated in our decorators (which could
  be simplified -- the actual security check is just whether the
  Django session object has a particular field)

* Checking the user realm for whether we should sync to the client
  notices about their Zephyr mirror being up to date, which is quite
  static and I think we can move out of this code path.

But implementation constraints around mapping the user_ids to
user_profile_ids mean that it makes sense to get the actual objects
for now.

This code is not what I want to do long-term.  I expect we'll be able
to clean up the dual User/UserProfile nonsense once we integrate the
upcoming Django 1.5 release, with its support for pluggable User
models, and after that I change, I expect it'll be fairly easy to make
the Tornado code only work with the user ID, not the actual objects.

(imported from commit 82e25b62fd0e3af7c86040600c63a4deec7bec06)
2013-01-11 16:11:07 -05:00
Keegan McAllister 0fdc3b772e runtornado: Print busy percentage at most once every 5s
(imported from commit 017bd01f96689f06c6d166bea47b5112fad05f8c)
2013-01-10 17:21:24 -05:00
Keegan McAllister 1d9bcf8da9 runtornado: Report the percentage of time spent outside the event loop
Note that on local dev servers, this will print out every half second, as
Tornado polls for file changes for autoreloading.  In production it will only
print out when network events occur.

(imported from commit adfe88879e4e446b7dfa6ee69e0a9ad013e9c4d4)
2013-01-10 17:07:46 -05:00
Keegan McAllister aa1df57163 runtornado: Drop call to tornado.autoreload
tornado.web already does this, based on the setting of the 'debug' kwarg.

Dropping this in production saves us waking up twice a second to stat()
a bunch of files.

We already explicitly restart the server on deploys.

(imported from commit 283bb0da609acb2699a04111a74c13224fe5124c)
2013-01-09 15:50:56 -05:00
Keegan McAllister b5dccbfa87 Return a special value to initiate longpolling
Fixes #261.

(imported from commit dd0312c3574c9ca924ab76edcb2fb6c03085be32)
2012-11-28 15:28:14 -05:00
Keegan McAllister e9c81a4a8b runtornado: Don't apply response middleware twice
This was causing us to log some requests twice, and might have more serious
consequences as well.

(imported from commit 0bb2d7207ee3e4e04679215a7f5ae637cd26aa19)
2012-11-06 13:15:49 -05:00
Keegan McAllister 8259059119 runtornado: Remove ability to run without specifying exactly one port
The zero-port case never actually worked, because addrport wasn't an optional
parameter in run_one.  And multiple ports was implemented using the
multiprocessing library, which is just bad news.  Since we have no need for
this, remove it before it can cause trouble.

(imported from commit 9d913924701f30d23ebe878b76c8f1f0da2800e2)
2012-11-06 13:15:49 -05:00
Keegan McAllister da252e8499 runtornado: Abort view processing if request or view middleware returns a response
This is the behavior specified by Django.  Since this was broken before,
our CSRF protection had no effect on Tornado views other than printing
a warning message :(

(imported from commit 7975d3c9b6c18915f917ac2da4592a55f6b6a658)
2012-10-27 17:28:07 -04:00
Zev Benjamin 732ca19729 Synchronize the pointer across sessions
The client may now optionally send its current pointer during
get_updates and the server will return the latest pointer if it
differs and was updated more recently by a different session.

(imported from commit e43b377d7dfb52f83cefb0b1003863d5407caf80)
2012-10-22 16:44:57 -04:00
Keegan McAllister a545876d56 Rename notify_waiting_clients -> notify_new_message
We might have other URLs for other notifications.

(imported from commit 4c1c5fe2f039816fef4c268f34692ca4f19d81e8)
2012-10-17 18:23:01 -04:00
Keegan McAllister 5e70b5a291 Split off the Tornado code into a separate process
(imported from commit 95dbd0f438cdba06d6e6c6c539a2a3d49c577cfd)
2012-10-17 18:23:01 -04:00
Tim Abbott 3e994c16b7 Rename/reorganize our urls to be more consistent.
(imported from commit ca3cc7ccd5d7da83a9c60968527378ee1118648e)
2012-10-16 15:56:06 -04:00
Tim Abbott 83ceba0a2d Remove old /api/get_updates view.
(imported from commit 5a33a61bfb4b3aa40e28c356f5b4048db775b901)
2012-10-03 16:58:31 -04:00
Tim Abbott 5a5b5de650 Rename /api/v1/get_updates to /api/v1/get_messages.
(imported from commit f856637158cf9fda64ba333b532a5941de8fcbab)
2012-10-02 16:29:34 -04:00
Tim Abbott 18a3888373 [schema] Add an API for sending/receiving messages.
(imported from commit 209d525dc5892fc4c392a8ced1588c838cbb17c4)
2012-10-02 15:49:25 -04:00
Tim Abbott b43dd303b6 Use tornado for new /api/get_updates query.
(imported from commit 84b85fc5bb0de4144fde00ea13d14f74b1dd9bd1)
2012-09-27 17:37:05 -04:00
Keegan McAllister 20240d869c Rename get_updates_longpoll to get_updates
(imported from commit 15635904584b324bdfa1414caea12c3bc4ce54c8)
2012-09-27 15:46:36 -04:00
Keegan McAllister c887a5f977 Debug Tornado when Django DEBUG = True
(imported from commit ebd81e5d9344f5fb09c59a2afcaf83e9b1fc2a1d)
2012-09-18 13:45:52 -04:00
Keegan McAllister 7d51dfc0a7 Remove dead function in runtornado.py
The code is buggy; the variable 'escape' is undefined.  Right now we aren't
using this code because we bypass Tornado for login.

(imported from commit 91cf6e8fc9477039c8847c7e9501d645993278b8)
2012-09-07 13:57:32 -04:00
Keegan McAllister f837ec66e8 Remove more dead code/variables in runtornado.py
(imported from commit 0d754ca1f9ce8bc53db878d6165354149016bea5)
2012-09-07 13:57:32 -04:00
Keegan McAllister ffb69ad1c8 Fix (?) logging in runtornado.py
(imported from commit d5158005abebe496959989acaa69db269126e502)
2012-09-07 13:57:32 -04:00
Keegan McAllister 80c6d09197 Remove unused imports
(imported from commit 9f5c226708e44935e8d32c18ea7dd1ad2615d8d5)
2012-09-07 13:57:32 -04:00
Keegan McAllister 2332fa2c72 Log server events with timestamps
(imported from commit b74274ccafcca716871058b8f9bde8dfe9e081ee)
2012-09-05 13:47:25 -04:00
Tim Abbott 20215eef86 Remove a bunch more trailing whitespace.
(imported from commit 5d1cc8dcb0b26eaf95ddca26574b361b6948cdb8)
2012-09-05 11:23:58 -04:00
Tim Abbott 9afd63692f Implement long polling using Tornado.
(imported from commit 4385304b27d7fe55a57a23133cd214fe8fc33482)
2012-08-30 14:29:34 -04:00