83dd51dcd6
Remove now-obsolete get_message_by_id_dbwarn transition code.
2016-01-26 21:05:11 -08:00
eecd1513b3
Don't access/store full Client objects in Tornado queue servers.
...
At present, we only do a few simple checks on the client type inside
the event system, and this saves database/memcached queries.
Note that this preserves the structure of the marshalled name in
to_dict/from_dict as client_type to avoid an unnecessary migration.
2016-01-26 21:04:32 -08:00
e3b6bfa3ca
Remove code for pickle event queue store.
...
Since we've had the JSON store in all environments for some time, we
no longer need this legacy code.
2016-01-26 20:59:34 -08:00
f6073d1708
Move HTTPResponse construction out of event_queue.py.
2016-01-26 20:59:19 -08:00
a9bf4b4cc7
Store the client name for the current handler in ClientDescriptor.
2016-01-26 20:58:01 -08:00
c7e3c3ce38
Look up client descriptors by handler_id.
...
Previously, client descriptors were referenced directly from the
handler object. Once we split the Tornado process into separate queue
and connection servers, these will no longer be in the same process,
so we need to reference them by ID instead.
2016-01-26 20:57:25 -08:00
ae760a351e
Move most of ClientDescriptor.finish_current_handler into library.
2016-01-26 20:56:08 -08:00
7df61fccbd
Add user_profile_email field to ClientDescriptor.
2016-01-26 20:55:59 -08:00
2ea0daab19
Track Tornado handlers by uniquely assigned IDs rather than objects.
...
This is early preparation for splitting apart Tornado into a queue
server and a frontend server.
2016-01-26 20:55:41 -08:00
8b42fdd0d7
Move get_events logic into a backend function in event_queue.py.
...
This commit is somewhat ugly, but its purpose is to be early
preparation for splitting Tornado into a queue server and a frontend
server, and this code belongs, by and large, in the queue server
component.
2016-01-26 20:51:54 -08:00
2ea0663a4a
Apply Python 3 futurize transform libmodernize.fixes.fix_imports_six.
2015-11-01 09:26:16 -08:00
b3ac668779
Apply Python 3 futurize transform libmodernize.fixes.fix_filter.
2015-11-01 09:26:16 -08:00
0494e40c39
Merge zerver/tornado_callbacks.py into zerver/lib/event_queue.py.
...
It had stopped being a coherently distinct component a while ago.
(imported from commit 0617957bcfe8dcaf69143c88a96ddd51ecb31a98)
2014-04-23 17:22:31 -07:00
fb602da502
Don't log exceptions that we know aren't bugs when adding events to event queues
...
(imported from commit 55074e68603119bd11cc5218453a4e1eab710feb)
2014-01-03 14:40:55 -05:00
2276c6e524
Add support for loading a narrows-only embedded Zulip window.
...
Features:
* Only shows messages in the narrow
* New messages in the narrow will arrive as they are sent
* Works even for streams you're not subscribed to
* Automatically subscribes you to a stream on send
* Doesn't update your pointer
* All searches etc. automatically have the narrow added
(imported from commit 2e12b76849f6ca0f53dda5985dad477a04f7bbac)
2013-12-16 17:46:02 -05:00
7c829c4024
Extract adding clients to the various dicts to a function.
...
(imported from commit 667e5736710c23becdf4ebe5688a31421ac632f5)
2013-12-16 17:46:02 -05:00
af347acf99
Fix narrow stream event queues with all event types.
...
(imported from commit 69c2f6f1be3dd864b5da85104b83235f0dd4df50)
2013-12-16 17:46:02 -05:00
a3fe72642b
Add save/reload of narrows of server restart.
...
(imported from commit 2ad77845ce7d8fe365c893ab5cad06e3c725b421)
2013-12-12 17:36:39 -05:00
71ad82b6aa
Add support for event queues with an included narrow.
...
This will allow us to substantially decrease the server-side work that
we do to support our Mirroring systems (since the personal mirrors can
request only messages that user sent) and also is what we need to
support a single-stream Zulip widget that we embed in webpages.
(imported from commit 055f2e9a523920719815181f8fdb44d3384e4a34)
2013-12-12 17:36:39 -05:00
48963c1cdf
Replace accepts_event_type with accepts_event.
...
This is in preparation for allowing queues to only accept some
messages.
(imported from commit 6bca11a9084aeff52931762085c4c771edbce56e)
2013-12-12 17:36:38 -05:00
b8579c6848
Disconnect handlers if the client closes a connection.
...
Tested using the following procedure (run-dev.py won't pass through
the client connect closing)
tabbott@monastery:~/zulip$ curl http://localhost:9991/api/v1/register -u email:key -d 'event_types=["message"]'
{"msg":"","max_message_id":2259,"last_event_id":-1,"result":"success","queue_id":"1386884005:0"}
tabbott@monastery:~/zulip$ curl -G http://localhost:9993/api/v1/events -u email:key -d "last_event_id=0" -d "queue_id=1386884005:0"
(then hit ctrl-C)
(imported from commit 3c4f3d5caac97b3de53da994ff9cd9ef67b2b9ea)
2013-12-12 16:52:25 -05:00
cc0ce6b437
Finish event handlers when disconnecting from an event queue.
...
This should help prevent timeouts from clients whose requests have
been supplanted.
(imported from commit fdb3a89c4ec02bb23d0fba50ea558d48cb786916)
2013-12-12 16:52:25 -05:00
291506a2b2
Add logging for when we disconnected an active handler.
...
(imported from commit 18c21a93bdf14485e5656bd7a38f2069eb8f2aad)
2013-12-11 17:23:56 -05:00
fcf6ec16aa
Fix pruning of event queues that only had virtual events.
...
(imported from commit 7b43d2164df6e95a6feb073b8f90c0862245d629)
2013-12-11 17:20:34 -05:00
b10278c8e0
events: Fix incorrectly empty event queues with virtual events.
...
I'm not sure this had much impact, but it's definitely a subtle bug.
(imported from commit 0260d4c5c35ce8eac683e84f1d939a6704ce4e2b)
2013-12-11 16:59:48 -05:00
874d4282c0
Disconnect handlers from event queues via a finally block
...
(imported from commit ac63a962f978b05a5d8d6bf81861b908b5fd54b3)
2013-12-10 17:44:35 -05:00
b566c0f201
Catch a wider range of exceptions when adding events to event queues
...
We have observed additional exceptions being thrown from zulip_finish and we
need to make sure that the handler is disconnected from the queue, or else the
event queue will keep throwing exceptions due to the handler being closed.
(imported from commit 59273aa14495216430b9eb1525b2cce230d8913d)
2013-12-10 17:40:30 -05:00
fed3902fc5
Clean up debugging logs from tracking down slow receive.
...
(imported from commit e4b1f763522406255ce8be3c8d51b5d855a770cd)
2013-12-05 14:36:29 -05:00
abded5f886
Upgrade event flag collapsing to support all flag changes.
...
(imported from commit a2ca74b779d67148aee41e92e4a4cb716620e566)
2013-12-03 12:21:04 -05:00
d2486670a8
event: Add deduplication of server restart events.
...
(imported from commit e464aa8b68958fb9f9fe8c199b9e73df510d22bc)
2013-12-03 12:20:36 -05:00
54b06d046b
events: Add support for not storing redundant events.
...
This should dramatically improve the speed of the dump/load part of
our restart process, especially with large long-lived event queues.
(imported from commit ae4ae20ba2ca4433e25a5e7beeb4fa4882c53972)
2013-12-03 12:19:42 -05:00
ad9659ea54
events: Fix zulip_finish to not make assumptions about how queues work.
...
Previously, we had an issue with the ACKing protocol, where if a
virtualizable event (like a "read" flag) was dispatched to a queue
client immediately, we would not properly ACK the change because it
had been made a virtual event.
(imported from commit ea09812f8a5ba1d5aad65f536022e3dbc77b0f9e)
2013-12-03 12:19:42 -05:00
559f0896a1
Revert "events: Add support for not storing redundant events."
...
This reverts commit cc493fa50b4c339257e060b3f0c0956c682e449d.
(imported from commit 9e8edfc11d415d61581da482d66bc9fa30cbefdf)
2013-12-02 11:03:04 -05:00
5b601d8492
Revert "event: Add deduplication of server restart events."
...
This reverts commit 05467929ba376a0ce173119c5d90d83671fc544b.
(imported from commit 4f24a0f1fe72e71fa368e1d818d9406a76c131be)
2013-12-02 11:03:04 -05:00
2c2f404cb6
Revert "Upgrade event flag collapsing to support all flag changes."
...
This reverts commit dadf3852f2ca7aaa85fb97b3d265c990b4e5a38a.
(imported from commit de5a95572aa955b4a9419b80a99d2446c51730e7)
2013-12-02 11:03:04 -05:00
10bce7ffe8
Revert "Fix loading event queues that don't have a virtual_events attribute"
...
This reverts commit 99c5ae332d253313ed728883a388c61191ed3021.
(imported from commit d7b601a2897bac6e14b675e689ee63ccc6e8d67d)
2013-12-02 11:03:03 -05:00
5bc826067e
Revert "Add comment in ClientDescriptor.to_dict and EventQueue.to_dict to avoid future bugs"
...
This reverts commit 45560539d09e9e1922130349be729005b786e632.
(imported from commit aa266941f03cdbe12f4315e65ec6c472cc207204)
2013-12-02 11:03:03 -05:00
9d0f801ad6
Exempt a particular client name from heartbeats for testing
...
(imported from commit 3914130384a7a9854b006405ffd2fe379bbd090d)
2013-11-27 10:39:31 -05:00
8899c62567
Add comment in ClientDescriptor.to_dict and EventQueue.to_dict to avoid future bugs
...
(imported from commit 45560539d09e9e1922130349be729005b786e632)
2013-11-26 18:39:31 -05:00
ce00fe0cd5
Fix loading event queues that don't have a virtual_events attribute
...
(imported from commit 99c5ae332d253313ed728883a388c61191ed3021)
2013-11-26 18:32:58 -05:00
e9f4612281
Upgrade event flag collapsing to support all flag changes.
...
(imported from commit dadf3852f2ca7aaa85fb97b3d265c990b4e5a38a)
2013-11-26 18:12:52 -05:00
fb5c42707e
event: Add deduplication of server restart events.
...
(imported from commit 05467929ba376a0ce173119c5d90d83671fc544b)
2013-11-26 18:12:52 -05:00
1db8fba5aa
events: Add support for not storing redundant events.
...
This should dramatically improve the speed of the dump/load part of
our restart process, especially with large long-lived event queues.
(imported from commit cc493fa50b4c339257e060b3f0c0956c682e449d)
2013-11-26 18:12:52 -05:00
272e1c2d92
Revert "Don't send heartbeat events to clients that don't request them."
...
This reverts commit 1147814b22fb9737a807057ddbdbe0e9554086e0.
This seems to with some probability screw up our Zephyr mirroring
script.
(imported from commit 4f82452f1b0ca98e6b895db020e071d2daa325e4)
2013-11-21 12:53:26 -05:00
ca8225cf47
[manual] Add endpoint to cleanup a finished events queue.
...
This requires a puppet apply on each of staging and prod0 to update
the nginx configuration to support the new URL when it is deployed.
(imported from commit a35a71a563fd1daca0d3ea4ec6874c5719a8564f)
2013-11-20 18:34:15 -05:00
25c63f08de
Use ujson instead of pickle for serializing event queues
...
(imported from commit 2d90574ae97cc973c9686c2bd557cc199d4b0850)
2013-11-20 14:25:36 -05:00
69e5e2c3d2
Remove old migration code
...
(imported from commit 5366411706ef296dc14d0d0dd9e2566cb8abb8b7)
2013-11-20 14:25:36 -05:00
06ee83b531
Construct EventQueues externally from ClientDescriptors
...
(imported from commit 7ef882074ed1c3663939da199d54e4959df8f127)
2013-11-20 14:25:36 -05:00
db29e60249
Move the last event queue pickle file to /var/tmp instead of deleting it
...
(imported from commit b8edaa686e2dadfdb2111ee1da93456591e44a50)
2013-11-19 15:05:19 -05:00
74c304ae40
Don't send heartbeat events to clients that don't request them.
...
(imported from commit 1147814b22fb9737a807057ddbdbe0e9554086e0)
2013-11-18 13:57:01 -05:00
Tim Abbott
Tim Abbott
Tim Abbott
Zev Benjamin