Commit Graph

167 Commits

Author SHA1 Message Date
Jessica McKellar 0c3382fabb Always give hashlib.sha1 and friends bytes.
This fixes an experienced bug where you couldn't subscribe to a stream
with non-ASCII characters (failing with a UnicodeEncodeError), as well
as many other potential bugs.

(imported from commit f084a4b4b597b85935655097a7b5a163811c4d71)
2013-03-23 00:10:10 -04:00
Leo Franchi f64bc59dfb Automatically register consumer once asynchronous connection is completed
(imported from commit 3d860a9b79a42beffbd3b73f49aa2c25986dd3c6)
2013-03-22 15:55:49 -04:00
Leo Franchi 452105e393 Reconnect to rabbitmq in our queue client if the connection is lost
(imported from commit 82ed67f0c0f93e9f79ff65ef55b397815d20aaeb)
2013-03-22 15:55:48 -04:00
Keegan McAllister 4baeaaa532 TornadoQueueClient: Don't stop the Tornado I/O loop on RabbitMQ disconnect
Requires Pika 0.9.6 or later.

(imported from commit 316ab9f2a6b5c2a25ae8584e2604222b33e0dec3)
2013-03-22 15:55:48 -04:00
Keegan McAllister 6ffc2bbe65 TornadoQueueClient: Provide a callback when declaring queues
This is required by Pika 0.9.8.  We need at least 0.9.6 for the next
commit; I had been testing with 0.9.5 previously.  Anyway this way
seems more correct as well.

(imported from commit bfb9e9e78938073001f70c4d28a5e07cc4ebac32)
2013-03-22 15:55:48 -04:00
Keegan McAllister 23b8833ec5 Handle Tornado callback notify in a more generic, less HTTP-centric way
(imported from commit 85a74d1b40461236c4c95ad688e9796ab50f0bbf)
2013-03-22 15:51:23 -04:00
Keegan McAllister 373d3ece1c queue: Implement TornadoQueueClient
(imported from commit 2803bc91e42aa20d6e6dec87bfadd2f890268d05)
2013-03-22 15:39:07 -04:00
Jessica McKellar 37e58b613d Truncate the tutorial stream name to fit the DB max stream name length.
(imported from commit e267656337f97d3ea14e14e957a405cf4fd2d06c)
2013-03-20 10:15:08 -04:00
Leo Franchi c6f9dbdd54 Only sleep if there is more work to be done
(imported from commit f8a1380e0045c9470909c088a9d262f8a714c86e)
2013-03-19 09:55:12 -04:00
Leo Franchi 6d6c3364dc Rewrite message flag handling to fork out to a subprocess for batch handling
(imported from commit 1ef846f542950cabf32f8b176f5591cf5794a0ff)
2013-03-19 09:55:12 -04:00
Tim Abbott 9977cdde54 send_message: Save rendered message content to the database.
(imported from commit e5bd224687d5420f556b341286f03d3829973a3c)
2013-03-19 08:09:26 -04:00
Tim Abbott b38a538f8c Rename userprofile_by_foo cache keys to user_profile_by_foo.
(imported from commit ef398abc48c9b81a3d339ffdce00bae274246d28)
2013-03-18 16:15:11 -04:00
Tim Abbott 8034980cc4 create_mit_user_if_needed: Check memcached before querying database.
This will save at least one database query whenever MIT users send
messages.

(imported from commit 9fd644885a380463d841762b8fafe928431e1263)
2013-03-18 16:15:11 -04:00
Tim Abbott 97d7d31b68 Rewrite internal_send_message to use check_send_message.
This will automatically fix bugs such as one in which
internal_send_message didn't properly strip() the subject argument
before sending a message.

We change the recipient_type argument to internal_send_message to take
the recipient type name (e.g. 'stream') both to better fit the API and
also because the previous code incorrectly handled huddles.

(imported from commit 78c2596d328f6bb1ce2eaa3eed9a9e48146e3b6a)
2013-03-18 16:15:11 -04:00
Tim Abbott bb80d1c58c Move check_send_message and helpers to actions.py.
(imported from commit d74c90e25bd63931955d2ad9b3890be53d674a48)
2013-03-18 16:15:11 -04:00
Tim Abbott 9909c8dc19 send_message: Use a memcached for the UserProfile-by-id cache.
This cache should save 2 database queries whenever we send a private
message.  However, previously it was per-process (which meant it was
mostly useless) and also buggy (it never stored anything in the cache,
so that it was completely useless).  Switching this to our standard
memcached setup will address both problems.

(imported from commit 1d807f30704bccf28de33a80523488aedc58a9be)
2013-03-18 16:15:11 -04:00
Tim Abbott a23f6be28d Cache the results of Recipient queries in our memcached cache.
(imported from commit a6ba25a9c62d2738e4738b076370e4b2bc0ffaba)
2013-03-18 16:15:10 -04:00
Tim Abbott 9c2a9eb288 send_message: Avoid querying the stream object twice.
(imported from commit c23aab232a40195d92ae4b0e68af390c3b6b20ad)
2013-03-18 16:15:10 -04:00
Tim Abbott c6c3cda089 do_send_message: Save a database query on message sending.
(imported from commit c5dc3d9888f9b1867e8d20d84795dde32cf95583)
2013-03-18 16:15:10 -04:00
Tim Abbott 9ae583b910 Use the User/UserProfile caches for Django requests too.
Previously we only used these caches for Tornado requests, because we
were not updating memcached when e.g. the user's pointer changed, and
so functions like update_pointer would not work correctly.

Now that we are updated memcached when the User and UserProfile
objects change, we can use these for all requests.

This saves 2 database queries on every Django request to the server.

(imported from commit aa5bffd885d14bde38b95e80a226bd5ab66f253d)
2013-03-15 18:09:34 -04:00
Tim Abbott 1e6ed16c23 Flush the User cache when saving User objects too.
(imported from commit e595209fe1190e2f3e0d55ad3b9bae0997b91d1a)
2013-03-15 18:09:34 -04:00
Tim Abbott b6f42cb1e1 Update UserProfile objects in memcached when we save them.
(imported from commit 59426e19eeeba131f363a4bc5f74f59b6dbe1066)
2013-03-15 18:09:33 -04:00
Tim Abbott 7e456ddba7 Cache get_status_list output for up to a minute.
This should substantially decrease the amount of server load generated
by the userpresence system.

I tested that this indeed was indeed saving one query on
/json/update_active_status requests on my laptop with 2 users from the
humbughq.com realm logged in.

(imported from commit 03e9d4eb95b9f664d489862684ae162db2076e08)
2013-03-15 16:37:41 -04:00
Tim Abbott ec07184721 cache_helpers: Fill in memcached message cache in batches of 1000.
This cache filling code takes about 5 seconds to run, which means it
will finish before Tornado starts reading from this cache, but if that
were to change, it would be much better to have made at least some
progress.

(imported from commit 60a3420cdb9ddf331d83573a3fdb6be1a5ee5a4f)
2013-03-15 16:37:41 -04:00
Tim Abbott e78ef47487 message_cache: Query related models to avoid more queries later.
(imported from commit c1442165c68e9eef1cb5d310624c1021c1a57350)
2013-03-15 16:37:40 -04:00
Tim Abbott d88591c19c cache: Fix the select_related() query name for sending_client.
Previously we were calling select_related() on Message.client, which
doesn't exist.  It seems kinda poor that this doesn't raise an
exception.

I believe this issue was causing us to do very large numbers of
database queries during get_updates calls during server restarts.

(imported from commit b79bd698820fbd9dd82bd61fc175c32cd5ce6d05)
2013-03-15 16:37:40 -04:00
Waseem Daher 0f621f5f89 Show Twitter previews on production.
This reverts commit 4baed7ba22482eff6ed381d319f6e21da904d9ba.

(imported from commit dcab148b24e95f90edd5f2f3f581b4037deed787)
2013-03-14 18:00:04 -04:00
Leo Franchi 07433b80cc Don't mark your message as read if it comes from the API
(imported from commit 0570d4db590e21b0fadd544b10c9865fa7acd66b)
2013-03-14 15:51:21 -04:00
Tim Abbott 76837ebe2c fill_memcached_caches: Also fill the caches of User/UserProfile objects.
Since we flush memcached when we do a server restart, the flurry of
get_updates requests that fly in afterwards are all cache misses for
getting the User/UserProfile objects, so Tornado ends up spending
around 70ms per get_updates request rather than the usual 1-2ms.

So this should substantially improve our Tornado performance around
server restarts.

(imported from commit 07b8126bdfd4ff14e4c3362f9eda1fe5fd571c5b)
2013-03-14 15:07:42 -04:00
Tim Abbott cf58ee833a Rename lib/message_cache.py to lib/cache_helpers.py
(imported from commit 591e427aee47aea58b50d6c3fbd6a05cfe9378ea)
2013-03-14 15:07:41 -04:00
Tim Abbott adb9c88151 message_cache: Clean up populate_message_cache.
(imported from commit a73e4fc4908d840f4f02aef13a9afe2fb683fd83)
2013-03-14 15:07:41 -04:00
Tim Abbott 0a9a553b5f Fix use of invalid hash keys with memcached.
Email addresses can contain characters that are not valid for
memcached cache keys.

(imported from commit d7cfd6d5292e4fc8f4e2b19f2c01a4097d878f1b)
2013-03-14 15:07:41 -04:00
Tim Abbott c098520bbd Move the key functions for various caches to cache.py.
(imported from commit b04826533c32516cc2eef3b35263a40385ae7be4)
2013-03-14 15:07:41 -04:00
Tim Abbott 1580386946 Replace db_cache_with_key with a parameter on cache_with_key.
(imported from commit f2c600292888ba384ee4acc97c94f1d6f8bc9657)
2013-03-14 15:07:41 -04:00
Waseem Daher 9c45b96b21 Make Dropbox inline image preview more robust to sharing links.
Sometimes Dropbox shares with /s/ and sometimes with /sh/,
and I'm not sure which controls it, but we should deal with both.

(imported from commit 2222450f25c418b5fbd60ab2c30477467e34c0d1)
2013-03-13 17:46:43 -04:00
Leo Franchi 93a3f14c43 Add backend support for handling new 'read' message flag
(imported from commit 6194e9332caa2d279cbc304f0d6a69f969aa9a72)
2013-03-13 14:14:45 -04:00
Tim Abbott 275af189a3 Cache permanent errors from the Twitter API.
This avoids our repeatedly retrying to fetch a tweet that doesn't
exist from the Twitter API.

(imported from commit b4ca1060d03da21e7e59e5b99e682d2e8457df15)
2013-03-13 11:12:25 -04:00
Leo Franchi 2c4c6ba43e [schema][manual] Add South migration for flags on UserMessage
(imported from commit bdf6cf2d5427709e52ef051e3c4a19c5fbb4851c)
2013-03-13 10:20:56 -04:00
Waseem Daher a57afd9a63 Use separate Twitter API key for prod.
(imported from commit ab43b42f69eaa569cf7a1ccba9f15d528050c22b)
2013-03-12 18:28:59 -04:00
Jessica McKellar a8d6bd3322 Remove some unused imports.
(imported from commit bb8311848ca24fbb1fb1b3df8f8b4403eb38ce1c)
2013-03-11 16:47:03 -04:00
Jessica McKellar 2314809d73 Auto-color streams.
(imported from commit b669f864b0574e9e9af282ccb8783ccfaa12adfd)
2013-03-11 16:47:03 -04:00
Waseem Daher 3c6d489064 Don't do Twitter previews on prod.
(imported from commit 4baed7ba22482eff6ed381d319f6e21da904d9ba)
2013-03-11 15:18:57 -04:00
Tim Abbott d679a72952 [manual] Cache results of the Twitter API in the database.
This should substantially improve the repeat-rendering time for pages
with large numbers of tweets since we don't need to go all the way to
twitter.com, which can take like a second, to render tweets properly.

To deploy this commit properly, one needs to run

./manage.py createcachetable third_party_api_results

(imported from commit 01b528e61f9dde2ee718bdec0490088907b6017e)
2013-03-11 13:15:55 -04:00
Waseem Daher 29d504149c twitter: Only render the first Twitter link we see, not all of them.
(imported from commit 68582224962da14795d91a68197db1001ae7b75c)
2013-03-08 17:45:58 -05:00
Tim Abbott f37c6e7a1a do_create_realm: Use internal_send_message.
(imported from commit 92edff133d16f72cf954671392e413066fb22955)
2013-03-08 17:24:33 -05:00
Tim Abbott a6c461e3e0 internal_send_message: Render content before sending.
(imported from commit 00b80c71ccf5b1388d64cf8b6a2cb21b818ece4a)
2013-03-08 17:24:33 -05:00
Tim Abbott 254bc9f361 Attempt to render messages before accepting them from the user.
This fixes trac #407.

(imported from commit e33647eec32266790f864d14ad377d51956d2a6f)
2013-03-08 17:24:33 -05:00
Tim Abbott 2f022ebb34 bugdown: Allow caller of convert to tell whether we succeeded.
(imported from commit 9e3cab73e0d9e9ad31ba770210e535584cca3bb2)
2013-03-08 17:24:33 -05:00
Waseem Daher 0e2e754c3d [manual] Use Twitter API v1.1 for inline previews.
This commit adds a dependency on python-twitter,
whose upstream is at https://github.com/bear/python-twitter,
and which for now needs to manually be installed on our
servers from the Debian package in sid.

(imported from commit 80cd9f4f59a6f0de6b75ac95e412c69e2a2e2490)
2013-03-08 17:10:06 -05:00
Waseem Daher 1df648baa9 Add inline preview of Twitter links.
This uses the unauthed v1 of the Twitter API, which is going to go
away soon, but it's fine as an interim measure.

(imported from commit 709a250271321f5479854a363875c9da43e6382d)
2013-03-08 14:31:18 -05:00