Commit Graph

177 Commits

Author SHA1 Message Date
Tim Abbott 8e4d6fa045 event_queue: Rename IDLE_EVENT_QUEUE_TIMEOUT_SECS.
This is a default value, not an always-used value, and its name should
reflect that.
2018-12-05 14:48:40 -08:00
Tim Abbott 94dfff1c4e event queue: Don't set a minimum for lifespan_secs.
This makes it more convenient for developers to set very short values
for this (e.g. 1 minute) for the purposes of testing/debugging; there
aren't obvious problems with letting users set short values for this.
2018-12-05 14:47:53 -08:00
Tim Abbott a3c2d49f0c event_queue: Change garbage-collection frequency to 1 minute.
This is designed to help make it more convenient to do manual testing
where we need event queues to be garbage-collected.
2018-12-05 14:42:53 -08:00
Tim Abbott 6dd69b9bff event_queue: Rename ClientDescriptor.idle to expired.
This better reflects the situation with these event queues -- they're
not idle, they are expired and to be garbage collected.
2018-12-05 14:42:53 -08:00
Tim Abbott 408af032a0 event_queue: Remove queue_timeout migration code from 2013.
There's never going to be an event queue without a queue_timeout
property anymore.
2018-12-05 14:24:38 -08:00
Tim Abbott d0f71881f4 docs: Add detailed documentation on the process for sending messages.
This has long been something missing from our suite of documentation.
2018-11-29 16:25:35 -08:00
Tim Abbott 64960383e4 mypy: Fix missing type annotation in tornado code. 2018-11-20 19:08:14 -08:00
Tim Abbott 46acb608b1 tornado: Include port number in logging statements. 2018-11-20 18:45:22 -08:00
Tim Abbott 40ff41e135 tornado: Fix populate_db failing to call send_event properly.
This isn't the right long-term fix; theoretically, send_event
shouldn't be doing anything with populate_db, but that's for later.
2018-11-02 17:07:21 -07:00
Tim Abbott 0cac7e1cd3 tornado: Extract functions for Tornado queue names.
This moves all control for what queue to use for which realm in our
Tornado system to just the sharding.py file; no actual sharding is
done yet.
2018-11-02 17:00:10 -07:00
Tim Abbott 152c44b6d2 tornado: Extract function for specifying Tornado URI.
Since TORNADO_PROCESSES is 1 in all default configurations, this
doesn't have any user-facing effect.
2018-11-02 17:00:09 -07:00
Tim Abbott ec065e92ee tornado: Store port on SockJS connection object.
This will make it available for use inside our websockets code.
2018-11-02 16:55:33 -07:00
Tim Abbott ea1ec68899 events: Pass a realm object into send_event.
This is a preparator refactor for supporting hosting different Tornado
processes on different servers; to look up which Tornado server we
should be sending the event to, we'll need the realm object.
2018-11-02 16:47:39 -07:00
Tim Abbott 75e48459b5 tornado: Support using a port-aware file for dumping event queues.
This should make it possible for there to safely be multiple Tornado
processes running on different ports on the same system.

It may also fix a rare race bug in development, where previously, it
was possible for the Tornados processes for Casper and the main
development server to interfere; I haven't investigated whether this
was a real bug or not, but now those two services will use independent
Tornado files.

We still need to add something to direct traffic between the different
Tornado processes.
2018-11-02 16:47:39 -07:00
Tim Abbott 9339c191da tornado: Fix missing mypy annotation. 2018-11-02 14:43:55 -07:00
Tim Abbott cf24a20185 tornado: Refactor logic for persistent queue filenames.
Now, these are computed using a function, which will make it easier to
edit these paths to depend on which Tornado process it is in coming
commits.
2018-11-02 14:19:10 -07:00
Tim Abbott 98f28fa6ce tornado: Remove unused send_notification() function.
This hasn't been used in a long time, probably since
3fddc11cc2.
2018-11-02 14:14:39 -07:00
neiljp (Neil Pilgrim) 482383b6f7 mypy: Add Optional to tornado/descriptors.py; remove from mypy.ini. 2018-10-29 12:53:16 -07:00
Tim Abbott e4813e462b tornado: Rename async_request_{restart,stop} to mention timer.
Previously, these timer accounting functions could be easily mistaken
for referring to starting/stopping the request.  By adding timer to
the name, we make the code easier for the casual observer to read and
understand.
2018-10-16 15:39:10 -07:00
Tim Abbott 58307f80aa event_queue: Stop mocking push notifications in most tests.
Historically, queue_json_publish had a special third argument that was
basically its default mock behavior in the test suite.  We've been
migrating away from that model, because it was confusing and resulted
in poor test coverage of our queue worker code paths; this was one of
the last holdouts.

As it turns out, we don't exercise this code path in a way that
impacts tests much; the main downside of this change is a likely small
penalty to performance of the full test suite when sending private
messages.
2018-08-10 13:58:39 -07:00
Tim Abbott 02ae71f27f api: Stop using API keys for Django->Tornado authentication.
As part of our effort to change the data model away from each user
having a single API key, we're eliminating the couple requests that
were made from Django to Tornado (as part of a /register or home
request) where we used the user's API key grabbed from the database
for authentication.

Instead, we use the (already existing) internal_notify_view
authentication mechanism, which uses the SHARED_SECRET setting for
security, for these requests, and just fetch the user object using
get_user_profile_by_id directly.

Tweaked by Yago to include the new /api/v1/events/internal endpoint in
the exempt_patterns list in test_helpers, since it's an endpoint we call
through Tornado. Also added a couple missing return type annotations.
2018-07-30 12:28:31 -07:00
Tim Abbott 07af59d4cc tornado: Split get_events_backend into two functions.
The lower-layer function, now called get_events_backend, is intended
to be called by multiple code paths (including the upcoming
get_events_internal).
2018-07-30 12:28:31 -07:00
Sarah de947445ca event_queue: Add stream_email_notify.
This adds support to the event queue system for triggering
missed-message notifications (whether push or email) to support the
stream push notifications feature.
2018-07-14 12:19:33 +05:30
Tim Abbott 58a7a390c8 event_queue: Call build_offline_notification unconditionally.
Previously, maybe_enqueue_notifications had this very subtle logic,
where it set the notice variable only inside the block for push
notifications, but then also used it inside the block for email
notifications.

This "worked", because previously the conditions for push
notifications were always true if the conditions for email
notifications were, but the code was unnecessarily confusing.  The
only good reason to write it this way is if build_offline_notification
was expensive; in fact, the most expensive thing it does is calling
time.time(), so that reason does not apply here.

This was further confusing, in that in the original logic, we relied
on the fact that push notification code path edited the "notice"
dictionary for further processing.

Instead, we just call it separately and setup the data separately in
each code path.
2018-07-14 12:01:59 +05:30
Tim Abbott a09ebf0551 event_queue: Remove confusing comment about rabbitmq.
Whatever RabbitMQ check this comment used to be next to, it isn't next
to anymore.
2018-07-14 12:00:37 +05:30
Tim Abbott 08ed1b4605 notifications: Set trigger on offline email notifications.
This data will be required for correctly implementing the upcoming
stream_push_notify feature; it also helps support cleaning up the code
for the existing stream mentions logic.
2018-07-14 11:57:28 +05:30
Vishnu Ks 54a002c2e2 requirements: Upgrade pyflakes to 2.0.0.
We fix a few errors that only the new version finds.
2018-05-24 11:31:36 -07:00
Tim Abbott ae0a929988 tornado: Ensure that tornado doesn't autoreload into syntax errors.
We've for a long time been plagued by run-dev.py needing to be
restarted every time one does a rebase that has merge conflicts,
because the Tornado process restarts itself into a syntax error and
crashes.

This fixes the Tornado autoreload process to check explicitly for
whether files actually syntax-check before trying to actually reload
the Tornado process to run that code.

There are a few things that are a bit janky:
* Ideally, this would go into Tornado upstream
* We removed the `_watched_files` feature, which we weren't using.
* Ideally, we'd use something other than `importlib.reload` that just
  does the syntax-check without adjusting the state within our current
  process.

Fixes #4351.
2018-05-20 16:50:10 -07:00
Tim Abbott 4f4d56b021 tornado: Import autoreload module from upstream Tornado.
This allows to patch things directly.
2018-05-20 16:49:17 -07:00
Tim Abbott 276b78e952 tornado: Extract AsyncDjangoHandlerBase and mark as nocoverage.
We're never going to add tests for this block, which is fundamentally
well-tested code from Django with a since line changed which is hard
to screw up (long-polling will not work at all without it).  The hope
is to remove it entirely and replace it with a cleaner monkey-patch,
but until then, unit tests for it would be redundant.
2018-05-15 18:39:52 -07:00
Aditya Bansal e79a2f2707 zerver/tornado: Change use of typing.Text to str. 2018-05-12 15:22:38 -07:00
Aditya Bansal 1f9244e060 zerver/lib: Change use of typing.Text to str. 2018-05-10 14:19:49 -07:00
Tim Abbott 3872e5521e tornado: Mark setup_tornado_rabbitmq for nocoverage.
We only use this in the direct management command, and it involves
some autoreload process setup stuff that we probably don't want to do
in our unit tests regardless.
2018-05-09 20:49:13 -07:00
Tim Abbott 3fd8d718f4 websockets: Fix race condition in CSRF token initialization.
It appears that we were not always initializing the Socket object
after `setup.js` had the opportunity to set csrf_token.

This should fix #6961.
2018-03-31 09:29:56 -07:00
Steve Howell 431ede77c6 minor: Clean up how we set flags in events.
This is basically a simple fix, where we consistently set
`flags` to an empty array when we pass it around.  The history
here is that we had kind of a nasty bug from setting it to
`None`, which only showed up in the somewhat obscure circumstance
of somebody subscribing to all stream events in our API.

Fixes #7921
2018-03-28 15:56:03 -07:00
neiljp (Neil Pilgrim) 2ed6da77c7 mypy: Rewrite some middleware annotations to use ViewFuncT. 2018-03-17 23:25:05 +00:00
neiljp (Neil Pilgrim) a3d8cc1f86 Mypy: Migrate tornado/descriptors.py to python3 function annotations.
Use quoted form of ClientDescriptor to break presumed circular dependency,
and modify import statement to shorten quotes.
2018-03-15 12:54:43 -07:00
neiljp (Neil Pilgrim) f184249191 mypy: Clarify valid types in tornado views via new variable. 2018-02-13 11:40:52 -08:00
rht 71e883b602 zerver/tornado: Remove u prefix from strings. 2018-02-05 12:11:33 -08:00
rht 9a8d2244ca django-2.0: Shift to resolvers from urlresolvers.
The old name is deprecated.
2018-01-30 10:53:54 -08:00
rht 185fd99816 mypy: Use Python 3 type syntax in several files. 2017-12-30 07:34:51 -05:00
Steve Howell 80477da481 quick fix: Stop tracebacks with missedmessage_hook.
A `None` value is not properly handled in this function, which
indicates some lack of testing or a recent regression we don't
understand.  We were getting lots of tracebacks from this line
of code on our test server:

    mentioned = 'mentioned' in flags and 'read' not in flags
2017-12-28 08:10:56 -05:00
fredfishgames 55801d2feb mypy: Use Python 3 type syntax in zerver/tornado/event_queue.py. 2017-12-26 08:31:09 -05:00
Arseny Chekanov c2c023f0cf mypy: Use Python 3 type syntax in zerver/tornado/websocket_client.py. 2017-12-09 16:50:12 -08:00
Greg Price c32b16715d tornado: Use spiffy new `call_later` rather than `add_timeout`.
This method was new in Tornado 4.0.  It saves us from having to get
the time ourselves and do the arithmetic -- which not only makes the
code a bit shorter, but also easier to get right.  Tornado docs (see
http://www.tornadoweb.org/en/stable/ioloop.html) say we should have
been getting the time from `ioloop.time()` rather than hardcoding
`time.time()`, because the loop could e.g. be running on the
`time.monotonic()` clock.
2017-11-29 16:56:29 -08:00
rht 115f7e6055 zerver/tornado: Use Python 3 syntax for typing (part 2). 2017-11-28 17:02:24 -08:00
Robert Hönig 0e0a8a2b14 queue processor tests: Call consume by default.
This significantly improves the API for queue_json_publish to not be
overly focused on what the behavior of this function should be in our
unit tests.
2017-11-26 11:45:34 -08:00
Tim Abbott 054952a44a docs: Update links from codebase to point to ReadTheDocs. 2017-11-16 10:53:49 -08:00
Tim Abbott dc8dd2333c tornado: Sort imports in files with no merge conflicts. 2017-11-15 15:53:11 -08:00
neiljp (Neil Pilgrim) 790cd5e7c8 mypy: Specify generic type parameters in cachify decorator.
Use of the decorator in event_queue.py suggests concrete return type,
for application of copy() function.
2017-11-08 12:40:40 -08:00