Commit Graph

95 Commits

Author SHA1 Message Date
Tim Abbott 50356cc325 event_queue: Eliminate buggy need_timeout heartbeat code.
As best I can tell, this option was completely confused in two ways:
* The name is confusing; it actually controls whether we _clear_ the
  timeout associated with the current handler
* It's not clear why one would not want it to be unconditionally true.

From reading the history, I'm pretty sure I had just misread the code
when I created this.

And this caused a real bug; a later refactoring caused us to basically
never cancel the timeouts, which in turn resulted in 90% of all events
traffic being hearbeats with a much lower frequency (~5s) than the
intended 45s.  Removing this code fixes that nasty bug.
2016-10-19 14:32:49 -07:00
Tim Abbott a6e779a56c events: Add event types to logging.
This should make it a lot easier to analyze our events traffic and
debug potential issues with it.
2016-10-19 14:32:16 -07:00
Tim Abbott 63a49be09e event_queue: Import tornado.ioloop correctly.
We actually use tornado.ioloop, not the root tornado module, and while
the current code does work, it causes mypy errors with the incremental
cache.
2016-10-17 17:57:46 -07:00
Umair Khan 6a12ad7c33 Don't trust env while accessing Tornado.
This fixes a problem where the requests to Tornado would attempt to
use a configured outgoing HTTP proxy, when really we want to connect
directly to localhost.

Fixes: #468.
2016-09-06 14:17:50 -07:00
Umair Khan a4466c93a3 Use connection pooling for Django -> Tornado reqs.
Fixes: #391.
2016-09-06 14:15:35 -07:00
Tim Abbott 72e948d19a Remove now-unused message_cache_key message cache.
Originally this cache was used to transmit data from Django to Tornado
(and also for general message caching purposes), but now nothing
actually reads from this cache, so we can eliminate it.
2016-07-08 17:58:56 -07:00
Conrad Dean bbf7a9c801 Annotate zerver/lib/event_queue.py. 2016-07-04 03:01:33 +05:30
Conrad Dean fe2c352ac0 ClientDescriptor: Pass inline sets, not lists, to do_gc_event_queues.
This allows us to more precisely type do_gc_event_queues.
2016-06-07 13:28:45 -07:00
Conrad Dean d77c70220c send_event: Remove useless return value and annotate.
Detected by mypy.

[tweaked by tabbott to pass mypy check and remove annotations]
2016-06-07 13:27:40 -07:00
Conrad Dean a4704ba8b2 event_queue: Fix deque values type annotation.
Event IDs in here are ints, as shown by the prune operation.
2016-06-07 13:05:46 -07:00
Umair Khan 08fbd57245 [i18n] Make error messages translatable.
Make all strings passing through `json_error` and `JsonableError`
translatable.

Fixes #727
2016-05-31 07:40:42 -07:00
Tim Abbott 572c69f3c2 Move REQ and friends to their own module. 2016-05-30 11:24:17 -07:00
Tim Abbott b869be9301 style: Use 'not in' consistently rather than `not foo in`. 2016-05-09 17:00:10 -07:00
Tim Abbott 7a81524c97 event_queue: Fix import from wrong file. 2016-04-28 12:46:21 -07:00
Tim Abbott ffccb572f0 Don't autoreload Tornado when running inside test suite.
The restarted Tornado processes seemed to escape the process group and
thus continue running after run-dev.py finished.

While we're at it, we don't need to dump/reload event queues in the
test suite either.
2016-04-13 13:19:01 -07:00
Varshit 4e1060076d Purge 'from typing import *' from zerver/.
This is a partial implementation of #636.
2016-04-07 14:07:07 -07:00
Tim Abbott 75d134a9b2 Import tornado.autoreload explicitly.
Fixes #623.
2016-04-03 16:52:23 -07:00
Tim Abbott 4a50336476 Add type: Any for tornado.ioloop.add_timeout return value. 2016-04-03 15:40:24 -07:00
Tim Abbott 2059f650ab Add PEP-484 type annotations to zerver/lib/. 2016-04-03 15:40:23 -07:00
Tim Abbott b99313545e Add PEP-484 type annotations to global dictionaties. 2016-04-03 15:40:06 -07:00
Tim Abbott 723d8c288a Add tornado autoreload hook to dump event queues on reload.
It's always been the case that in production, Tornado dumps all the
event queues when shut down so that they can be reloaded by the
replacement Tornado process.  This never worked in development because
the codepath for auto-reload didn't go through either a signal or
sys.exit (it re-execs the process instead).

This meant that we didn't have a mechanism for testing the event queue
dump/load functionality in the development environment.  We fix this
by adding such dumping/loading.  However, this breaks the automatic
reloading of open browser windows on a server restart, so we add that
back in by adjusting the special `restart` events to pass a special
`immediate` flag when used in development.

This also has the benefit of removing the "Bad event queue" errors one
would get on every file save induced restart on the Python console.
2016-03-30 23:09:16 -07:00
Tim Abbott 59e2be2f5f Notify clients when an event queue is garbage collected. 2016-03-30 23:09:14 -07:00
Tim Abbott bf43db0dad Send responses from fetch_events in dict format. 2016-03-30 22:04:09 -07:00
Tim Abbott 485e46f136 Wrap most of fetch_events in a try/except JsonableError.
This is a no-op right now, but we'll want the new structure for the
next commit, and splitting this out makes it a lot easier to read what
is actually changed in the next commit.
2016-03-30 22:02:32 -07:00
Tim Abbott ad1494f8e0 Use dictionaries for passing data into event server subsystem.
This marshalls data in a portable format in preparation for splitting
the event queue server from the Tornado server.
2016-03-30 22:01:08 -07:00
Tim Abbott 6cd14af18f events: Migrate client_type => client_type_name in marshalling code. 2016-03-30 22:00:50 -07:00
Tim Abbott 0419430000 Enhance Tornado logging with Handler stats.
This was useful data for debugging handler/memory leaks.
2016-03-20 16:53:13 -07:00
Tim Abbott e51811aa9e Add comment documenting logic flow of fetch_events. 2016-03-20 16:53:13 -07:00
Tim Abbott 7fabfe9cb9 Add __repr__s for ClientDescriptor and AsyncDjangoHandler. 2016-03-20 16:53:13 -07:00
Tim Abbott 0b96e5e43f Fix connection not being closed cleanly on event queue GC.
Apparently, our event queue garbage collection logic never actually
disconnected any existing handler objects.

We fix this by disconnecting the handler inside cleanup(), adding a
special check to avoid creating a pointless timeout object.
2016-03-20 16:53:13 -07:00
Tim Abbott 12a5a3a6e1 Actually fix main Tornado memory leak of handler objects.
This line appears to have been lost in rebasing from the original
implementation of 1396eb7022faec4c2d91553800a35781a96dd5bd; so the
previous fix actually only addressed the issue in a rare exception
case.
2016-03-20 16:53:12 -07:00
Eklavya Sharma 9e3c3e14f5 Partially apply Python 3 libmodernize.fixes.fix_dict_six.
Refer to #256
2016-03-19 15:52:58 -07:00
Tim Abbott 1396eb7022 Fix Tornado memory leak of handler objects.
In 2ea0daab19, handlers were moved to
being tracked via the handlers_by_id dict, but nothing cleared this
dict, resulting in every handler object being leaked.  Since a Tornado
process uses a different handler object for every request, this
resulted in a significant memory leak.  We fix this by clearing the
handlers_by_id dict in the two code paths that would result in a
Tornado handler being de-allocated: the exception codepath and the
handler disconnect codepath.

Fixes #463.
2016-03-17 18:33:59 -07:00
Tim Abbott 3469fd4bb2 Fix last remaining use of file() -> open(). 2016-02-03 19:31:46 -08:00
Tim Abbott 700055c194 Apply modernize transform libmodernize.fixes.fix_file.
This replaces use of file() with open() which is python 3 compatible,
and also adds it to our python 3 support test suite.
2016-01-26 21:09:42 -08:00
Tim Abbott 83dd51dcd6 Remove now-obsolete get_message_by_id_dbwarn transition code. 2016-01-26 21:05:11 -08:00
Tim Abbott eecd1513b3 Don't access/store full Client objects in Tornado queue servers.
At present, we only do a few simple checks on the client type inside
the event system, and this saves database/memcached queries.

Note that this preserves the structure of the marshalled name in
to_dict/from_dict as client_type to avoid an unnecessary migration.
2016-01-26 21:04:32 -08:00
Tim Abbott e3b6bfa3ca Remove code for pickle event queue store.
Since we've had the JSON store in all environments for some time, we
no longer need this legacy code.
2016-01-26 20:59:34 -08:00
Tim Abbott f6073d1708 Move HTTPResponse construction out of event_queue.py. 2016-01-26 20:59:19 -08:00
Tim Abbott a9bf4b4cc7 Store the client name for the current handler in ClientDescriptor. 2016-01-26 20:58:01 -08:00
Tim Abbott c7e3c3ce38 Look up client descriptors by handler_id.
Previously, client descriptors were referenced directly from the
handler object.  Once we split the Tornado process into separate queue
and connection servers, these will no longer be in the same process,
so we need to reference them by ID instead.
2016-01-26 20:57:25 -08:00
Tim Abbott ae760a351e Move most of ClientDescriptor.finish_current_handler into library. 2016-01-26 20:56:08 -08:00
Tim Abbott 7df61fccbd Add user_profile_email field to ClientDescriptor. 2016-01-26 20:55:59 -08:00
Tim Abbott 2ea0daab19 Track Tornado handlers by uniquely assigned IDs rather than objects.
This is early preparation for splitting apart Tornado into a queue
server and a frontend server.
2016-01-26 20:55:41 -08:00
Tim Abbott 8b42fdd0d7 Move get_events logic into a backend function in event_queue.py.
This commit is somewhat ugly, but its purpose is to be early
preparation for splitting Tornado into a queue server and a frontend
server, and this code belongs, by and large, in the queue server
component.
2016-01-26 20:51:54 -08:00
Tim Abbott 2ea0663a4a Apply Python 3 futurize transform libmodernize.fixes.fix_imports_six. 2015-11-01 09:26:16 -08:00
Tim Abbott b3ac668779 Apply Python 3 futurize transform libmodernize.fixes.fix_filter. 2015-11-01 09:26:16 -08:00
Tim Abbott 0494e40c39 Merge zerver/tornado_callbacks.py into zerver/lib/event_queue.py.
It had stopped being a coherently distinct component a while ago.

(imported from commit 0617957bcfe8dcaf69143c88a96ddd51ecb31a98)
2014-04-23 17:22:31 -07:00
Zev Benjamin fb602da502 Don't log exceptions that we know aren't bugs when adding events to event queues
(imported from commit 55074e68603119bd11cc5218453a4e1eab710feb)
2014-01-03 14:40:55 -05:00
Tim Abbott 2276c6e524 Add support for loading a narrows-only embedded Zulip window.
Features:
* Only shows messages in the narrow
* New messages in the narrow will arrive as they are sent
* Works even for streams you're not subscribed to
* Automatically subscribes you to a stream on send
* Doesn't update your pointer
* All searches etc. automatically have the narrow added

(imported from commit 2e12b76849f6ca0f53dda5985dad477a04f7bbac)
2013-12-16 17:46:02 -05:00