This saves something like 15ms on our 1000 message get_old_messages
queries, and will save even more when we start sending JSON dumps into
our memcached system.
We need to install python-ujson on servers and dev instances before
pushing this to prod.
(imported from commit 373690b7c056d00d2299a7588a33f025104bfbca)
See PEP 328[1] for details. This feature was introduced in Python 2.5 and
will become mandatory in Python 3.
[1]: http://www.python.org/dev/peps/pep-0328
(imported from commit 7444eeba8a08d5f91b94c7921848f2274979bd76)
Previous we had around 4 copies of the logic for deciding whether we
should publish data via a SimpleQueueClient queue, a
TornadoQueueClient queue, or to directly handle the operation, which
resulted in their getting out of sync and buggy (see e.g. the previous
commit).
We need to add a lock around adding things to the queue to work around
a bug with pika's BlockingConnection.
I should note that the previous logic in some places had a bunch of
tests of the form "elif settings.TEST_SUITE" for doing the work that
would have been done by the queue processor directly; these should
have just been "else" clauses -- since we generally want that code to
run on development environments whether or not the test suite is
currently running.
(imported from commit 16bdbed4fff04b1bda6fde3b16bee7359917720b)
This is required by Pika 0.9.8. We need at least 0.9.6 for the next
commit; I had been testing with 0.9.5 previously. Anyway this way
seems more correct as well.
(imported from commit bfb9e9e78938073001f70c4d28a5e07cc4ebac32)
SimpleQueueClient was not thread-safe as the pika method
calls that we are using might disconnect and end up in a bad
state if two threads enqueue messages at the same time.
In order to avoid this, each user of the rabbitmq queue
gets its own instance of a SimpleQueueClient.
(imported from commit 694083b75cd58a60b8de282a8f40eb92a864c5ce)
Code like this is dangerous:
class SimpleQueue(object):
queues = set()
because all instances will share the same 'queues' set object.
We don't really need multiple instances, but neither is there a reason to break
them. So do the normal Python thing instead.
(imported from commit a56bb8414dd549cfd312c9565df198cf9d20f08a)
Before this is deployed, we need to install rabbitmq and pika on the
target server (see the puppet part of this commit for how).
When this is deployed, we need to start the new user activity bot:
./manage.py process_user_activity
in the screen session on the relevant server, or user_activity logs
won't be processed (which will eventually result in all users getting
notifications about how their mirrors are out of date).
(imported from commit 44d605aca0290bef2c94fb99267e15e26b21673b)
Tim Abbott
Leo Franchi
Luke Faraone
Keegan McAllister