Commit Graph

90 Commits

Author SHA1 Message Date
Greg Price 73886f57d2 zerver/lib/queue: Clean up import order. 2017-11-29 16:56:29 -08:00
Greg Price 3c4e4c14c9 rabbitmq: Add on-close callback atomically in creating the connection.
Adding it afterward is inherently racy, and upstream's API is quite
reasonable for avoiding that -- just like we can pass an on-open
callback up front, we can do the same with the on-close callback.

This is a more thorough version of 4adf2d5c2 from back in 2013-04.
2017-11-29 16:56:29 -08:00
Greg Price e88c2a7ee4 rabbitmq: Cut redundant `stop_ioloop_on_close` parameter.
The default value of this parameter is already False upstream.
(It was already False in pika version 0.9.6, which we were
supposedly using when we introduced this in 4baeaaa52; not sure
what the story was there.)
2017-11-29 16:56:29 -08:00
Robert Hönig 0e0a8a2b14 queue processor tests: Call consume by default.
This significantly improves the API for queue_json_publish to not be
overly focused on what the behavior of this function should be in our
unit tests.
2017-11-26 11:45:34 -08:00
rht 33b1a541d7 zerver/lib: Use python 3 syntax for typing.
With tweaks by tabbott to fix line spacing.
2017-11-18 16:09:04 -08:00
Greg Price 4b5c52fc99 queue: Prevent an AttributeError rather than swallowing it.
When the RabbitMQ server disappears, we log errors like these:

```
Traceback (most recent call last):
  File "./zerver/lib/queue.py", line 114, in json_publish
    self.publish(queue_name, ujson.dumps(body))
  File "./zerver/lib/queue.py", line 108, in publish
    self.ensure_queue(queue_name, do_publish)
  File "./zerver/lib/queue.py", line 88, in ensure_queue
    if not self.connection.is_open:
AttributeError: 'NoneType' object has no attribute 'is_open'

During handling of the above exception, another exception occurred:
[... traceback of connection failure inside the retried self.publish()]
```

That's a type error -- a programming error, not an exceptional
condition from outside the program.  Fix the programming error.

Also move the retry out of the `except:` block, so that if it also
fails we don't get the exceptions stacked on each other.  This is a
new feature of Python 3 which is sometimes indispensable for
debugging, and which surfaced this nit in the logs (on Python 2 we'd
never see the AttributeError part), but in some cases it can cause a
lot of spew if care isn't taken.
2017-11-16 18:33:14 -08:00
rht e311842a1b zerver/lib: Remove inheritance from object. 2017-11-06 08:53:48 -08:00
rht c4fcff7178 refactor: Replace super(.*self) with Python 3-specific super().
We change all the instances except for the `test_helpers.py`
TimeTrackingCursor monkey-patching, which actually needs to specify
the base class.
2017-10-30 14:30:25 -07:00
Tim Abbott fa55d7ed33 retry_event: Remove requirement of failed_tries being already there.
This fixes a bug where retries in the signups queue threw an exception.
2017-10-27 18:15:53 -07:00
derAnfaenger af699500b7 tests: Add option to call queue processor consumer.
This makes tests of queue processors more realistic,
by adding a parameter to `queue_json_publish` that
calls a queue's consumer function if accessed in a test.

Fixes part of #6542.
2017-10-16 23:20:13 -07:00
Umair Khan 7d6ddaad26 mypy: Use Dict instead of Mapping in queues. 2017-10-03 10:35:07 -07:00
rht f43e54d352 zerver/lib: Remove absolute_import. 2017-09-27 10:00:39 -07:00
Umair Khan 62cae23601 queue: Extract event retry into retry_event().
This commit takes the code from zerver.lib.outgoing_webhook.request_retry
and creates a new generic function called retry_event.
2017-08-22 11:16:48 -07:00
Umair Khan 758dbec9e2 tornado: Move setup_tornado_rabbitmq to application.py 2017-07-21 09:55:25 +05:00
neiljp (Neil Pilgrim) 32269ad142 mypy: Set type of rabbitmq_heartbeat to Optional[int] in queue.py. 2017-07-08 10:49:42 -07:00
Aditya Bansal e5bf7e81e6 pep8: Add compliance with rule E261 zerver/lib/queue.py. 2017-05-07 23:21:50 -07:00
Raghav Jajodia a3a03bd6a5 mypy: Added Dict, List and Set imports.
Fixed mypy errors associated with the upgrade.
2017-03-04 14:33:44 -08:00
Tim Abbott 4e171ce787 lint: Clean up E126 PEP-8 rule. 2017-01-23 22:06:13 -08:00
nikolay abc2ff4a06 pep8: Fix many rule E128 violations.
[Tweaked by tabbott to adjust some approaches used in wrapping]
2016-12-03 13:33:31 -08:00
Rafid Aslam 41bd88d5ed pep8: Fix E301 pep8 violations.
Fix "E301: expected (1 or 2) blank line" pep8 violations.
2016-11-29 08:51:44 -08:00
paxapy 6d93b3b60c lint: Fix E703 pep8 violations. 2016-11-09 15:18:35 -08:00
Conrad Dean 9812e676f0 Annotate zerver/lib/queue.py. 2016-07-04 03:01:32 +05:30
Conrad Dean d77c70220c send_event: Remove useless return value and annotate.
Detected by mypy.

[tweaked by tabbott to pass mypy check and remove annotations]
2016-06-07 13:27:40 -07:00
Tim Abbott 43f167849b queue: Disable RabbitMQ heartbeat in BlockingConnection.
Fixes #741.
2016-05-09 10:23:28 -07:00
Tim Abbott 0c42fc2f8f queue: Refactor Pika credentials code to be a bit cleaner. 2016-05-09 10:23:28 -07:00
Tim Abbott f06c8c7cc2 Update *.readthedocs.org => *.readthedocs.io.
ReadTheDocs has moved their hosting of user project websites to the
new readthedocs.io domain.
2016-04-29 16:00:08 -07:00
Tim Abbott a72385246e Fix missing whitespace after '=' in python/js code. 2016-04-20 11:36:14 -07:00
Varshit 4e1060076d Purge 'from typing import *' from zerver/.
This is a partial implementation of #636.
2016-04-07 14:07:07 -07:00
Tim Abbott 2059f650ab Add PEP-484 type annotations to zerver/lib/. 2016-04-03 15:40:23 -07:00
Alexander Trost 84f7a1f1ea Make rabbitmq, redis, and memcached configurable via user settings.py.
Previously these were hardcoded in zproject/settings.py to be accessed
on localhost.

[Modified by Tim Abbott to adjust comments and fix configure-rabbitmq]
2016-01-21 22:07:56 -08:00
Tim Abbott 8c34c40924 Apply Python 3 futurize transform lib2to3.fixes.fix_except. 2015-11-01 08:08:33 -08:00
Steve Howell 70c1c31b3a Simplify callback protocol for register_json_consumer().
The register_json_consumer() function now expects its callback
function to accept a single argument, which is the payload, as
none of the callbacks cared about channel, method, and properties.

This change breaks down as follows:

    * A couple test stubs and subclasses were simplified.
    * All the consume() and consume_wrapper() functions in
      queue_processors.py were simplified.
    * Two callbacks via runtornado.py were simplified.  One
      of the callbacks was socket.respond_send_message, which
      had an additional caller, i.e. not register_json_consumer()
      calling back to it, and the caller was simplified not
      to pass None for the three removed arguments.

(imported from commit 792316e20be619458dd5036745233f37e6ffcf43)
2013-10-31 13:18:11 -04:00
Zev Benjamin 807116e955 Delay connecting to RabbitMQ until it's necessary
Previously we were connecting at import-time.

(imported from commit 9a638f0d238f3b6b00feb4aa524098a64953cb92)
2013-10-28 15:27:25 -04:00
Luke Faraone 1fa65915f2 NACK on consumer failure
(imported from commit 1cad1096dd1a969b2551822575784c338a82f433)
2013-10-25 14:13:30 -04:00
Zev Benjamin ec21275b19 Move rabbitmq username into settings.py
(imported from commit f9a8f80f53b8774781b8558444e601567cb59835)
2013-10-24 16:40:38 -04:00
Zev Benjamin 2c4a1912e8 Fix parameters for on_close callback
Some of the older documentation was apparently wrong.

(imported from commit 9ebf9d1a72c8b230563f1883d63c26212eb80e22)
2013-10-09 10:45:38 -04:00
Tim Abbott 0e23b2240d queue: Log how long it takes to connect to rabbitmq.
(imported from commit be3cd9da5e7c213cfed62feda2e676038ade57a6)
2013-09-30 13:48:48 -04:00
Zev Benjamin d7d17c993c Fix using SimpleQueueClient with multiple queues
(imported from commit 18bda1430d76f8b436ad42f85d0ce3cab8fd8193)
2013-09-23 11:25:58 -04:00
Tim Abbott 7b9305b06f Rename Django project to zproject.
This includes a hack to preserve humbug/backends.py as a symlink, so
that we don't need to regenerate all our old sessions.

(imported from commit b7918988b31c71ec01bbdc270db7017d4069221d)
2013-08-07 11:04:03 -04:00
Tim Abbott e111a2f9a5 [manual] Rename Django app from zephyr to zerver.
This needs to be deployed to both staging and prod at the same
off-peak time (and the schema migration run).

At the time it is deployed, we need to make a few changes directly in
the database:

(1) UPDATE django_content_type set app_label='zerver' where app_label='zephyr';
(2) UPDATE south_migrationhistory set app_name='zerver' where app_name='zephyr';

(imported from commit eb3fd719571740189514ef0b884738cb30df1320)
2013-08-06 07:39:36 -04:00