Mirror
/
zulip
mirror of https://github.com/zulip/zulip.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
39,998 Commits 28 Branches 143 Tags 605 MiB
Commit Graph

15 Commits

Author SHA1 Message Date
Alex Vandiver c2132a4f9c queue: Drop register_json_consumer / json_drain_queue interface. 
Now that all callsites use the same interface, drop the now-unused
ones, and their tests.
 2020-10-11 14:19:42 -07:00
Alex Vandiver 5477b9d9a1 queue: Switch tests to start_json_consumer interface. 2020-10-11 14:19:42 -07:00
Alex Vandiver 571f8b8664 queue: Use low-level queue_purge to empty at the end of tests. 
This is O(1) at the RabbitMQ API level, and doesn't rely on the code
under test to function correctly during test cleanup.
 2020-10-09 20:43:49 -07:00
Alex Vandiver baf882a133 queue: Only ACK drain_queue once it has completed work on the list. 
Currently, drain_queue and json_drain_queue ack every message as it is
pulled off of the queue, until the queue is empty.  This means that if
the consumer crashes between pulling a batch of messages off the
queue, and actually processing them, those messages will be
permanently lost.  Sending an ACK on every message also results in a
significant amount lot of traffic to rabbitmq, with notable
performance implications.

Send a singular ACK after the processing has completed, by making
`drain_queue` into a contextmanager.  Additionally, use the `multiple`
flag to ACK all of the messages at once -- or explicitly NACK the
messages if processing failed.  Sending a NACK will re-queue them at
the front of the queue.

Performance of a no-op dequeue before this change:
```
$ ./manage.py queue_rate --count 50000 --batch
Purging queue...
Enqueue rate: 10847 / sec
Dequeue rate: 2479 / sec
```
Performance of a no-op dequeue after this change (a 25% increase):
```
$ ./manage.py queue_rate --count 50000 --batch
Purging queue...
Enqueue rate: 10752 / sec
Dequeue rate: 3079 / sec
```
 2020-10-06 17:26:14 -07:00
Mohit Gupta 732f1fa44d test_queue: Use assertLogs to verify warning in test_queue_error_json. 
This will avoid spam in test-backend output.
 2020-07-27 11:02:17 -07:00
Anders Kaseorg 365fe0b3d5 python: Sort imports with isort. 
Fixes #2665.

Regenerated by tabbott with `lint --fix` after a rebase and change in
parameters.

Note from tabbott: In a few cases, this converts technical debt in the
form of unsorted imports into different technical debt in the form of
our largest files having very long, ugly import sequences at the
start.  I expect this change will increase pressure for us to split
those files, which isn't a bad thing.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
 2020-06-11 16:45:32 -07:00
Anders Kaseorg 19cc22e5ab queue: Fix types to reflect that Pika channels transmit bytes. 
Signed-off-by: Anders Kaseorg <anders@zulip.com>
 2020-06-07 11:09:24 -07:00
Anders Kaseorg 840cf4b885 requirements: Drop direct dependency on mock. 
mock is just a backport of the standard library’s unittest.mock now.

The SAMLAuthBackendTest change is needed because
MagicMock.call_args.args wasn’t introduced until Python
3.8 (https://bugs.python.org/issue21269).

The PROVISION_VERSION bump is skipped because mock is still an
indirect dev requirement via moto.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
 2020-05-26 11:40:42 -07:00
Rafid Aslam 447f74ae63 Upgrade pika to 1.1.*. 
Upgrade pika to 1.1.* and make some changes accordingly
to comply with the new version.

Fixes #12899.
 2019-10-29 17:01:12 -07:00
Mateusz Mandera 438c4b2935 tests: tearDown overrides should call super().tearDown(). 
ZulipTestCase subclasses that override tearDown() should call the parent
class tearDown() too, as it does some clean up of its own.
 2019-10-18 13:36:59 -07:00
Anders Kaseorg 3127fb4dbd zerver/tests: Remove unused imports. 
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
 2019-02-02 17:43:03 -08:00
Vishnu Ks 54a002c2e2 requirements: Upgrade pyflakes to 2.0.0. 
We fix a few errors that only the new version finds.
 2018-05-24 11:31:36 -07:00
Tim Abbott 930cbabbb7 test_queue: Add a test for register_json_consumer. 
This expands the coverage of the SimpleQueueProcessor logic to be
mostly complete.  We still don't have real tests for
TornadoQueueProcessor.
 2018-05-15 18:24:45 -07:00
Tim Abbott 962c64a1d4 test_queue: Add basic tests for SimpleQueueClient. 2018-05-15 17:56:30 -07:00
Greg Price 3b3154527f queue: Don't blow up when a connection closes quickly. 2018-03-20 16:49:05 -07:00