Commit Graph

2587 Commits

Author SHA1 Message Date
Tim Abbott fb77af8f10 Refactor send_message_backend to do checking in new function.
This is preparatory for using this new arguments checking function in
internal_send_message as well.

(imported from commit 578e09c50b8a700c019c7dd235b2d9527af34e39)
2013-03-18 16:15:11 -04:00
Tim Abbott 14efaea0bb send_message_backend: Reorganize subject argument checking.
(imported from commit 80daa46f47e7e5279f746db89b3632eeadb27325)
2013-03-18 16:15:11 -04:00
Tim Abbott 9909c8dc19 send_message: Use a memcached for the UserProfile-by-id cache.
This cache should save 2 database queries whenever we send a private
message.  However, previously it was per-process (which meant it was
mostly useless) and also buggy (it never stored anything in the cache,
so that it was completely useless).  Switching this to our standard
memcached setup will address both problems.

(imported from commit 1d807f30704bccf28de33a80523488aedc58a9be)
2013-03-18 16:15:11 -04:00
Tim Abbott c1d3a5b5e5 Use more caching of UserProfile objects in views.
(imported from commit 3abcceeb4fac1cfa5a9944a6455c60f6512dce6e)
2013-03-18 16:15:10 -04:00
Tim Abbott a23f6be28d Cache the results of Recipient queries in our memcached cache.
(imported from commit a6ba25a9c62d2738e4738b076370e4b2bc0ffaba)
2013-03-18 16:15:10 -04:00
Tim Abbott 3c143db01a Add memcached caching for get_stream lookups.
(imported from commit a089c8a99eaa1fa28ce401859cac1bdf0eb92a69)
2013-03-18 16:15:10 -04:00
Tim Abbott 9c2a9eb288 send_message: Avoid querying the stream object twice.
(imported from commit c23aab232a40195d92ae4b0e68af390c3b6b20ad)
2013-03-18 16:15:10 -04:00
Tim Abbott c0fece1cbd get_stream: Fetch the realm object as well.
(imported from commit 065ae574cb933cfc48371efd4327e70d37184dd4)
2013-03-18 16:15:10 -04:00
Tim Abbott c6c3cda089 do_send_message: Save a database query on message sending.
(imported from commit c5dc3d9888f9b1867e8d20d84795dde32cf95583)
2013-03-18 16:15:10 -04:00
Jessica McKellar 1ca2f6fa7f Add a management command to reset your stream colors to the default.
(imported from commit f6891ad40088bf34686a7d8a2d910a9a0f3be7c2)
2013-03-18 15:49:39 -04:00
Keegan McAllister 45eb9bcf47 tests: Disable tutorial through the server side variable needs_tutorial
This fixes a nondeterministic test failure for me.

The first message sent in the test suite appears to get dropped.  I don't know
why this is, and I'm pretty sure it was an existing bug.  This message used to
be the one disabling the tutorial, which might explain why that didn't always
work.

Regardless, this commit at least makes the test suite usable, and we can work
on fixing that bug later.

(imported from commit 063e40871b9883e3a6dab93a4e0a51c5b2dae4b7)
2013-03-18 13:46:46 -04:00
Waseem Daher 797b5ee63f Add page describing our Humbug apps.
(imported from commit 2e071782a13a497e57225b2a84c41d58f5e120b5)
2013-03-15 19:16:44 -04:00
Tim Abbott 9ae583b910 Use the User/UserProfile caches for Django requests too.
Previously we only used these caches for Tornado requests, because we
were not updating memcached when e.g. the user's pointer changed, and
so functions like update_pointer would not work correctly.

Now that we are updated memcached when the User and UserProfile
objects change, we can use these for all requests.

This saves 2 database queries on every Django request to the server.

(imported from commit aa5bffd885d14bde38b95e80a226bd5ab66f253d)
2013-03-15 18:09:34 -04:00
Tim Abbott 1e6ed16c23 Flush the User cache when saving User objects too.
(imported from commit e595209fe1190e2f3e0d55ad3b9bae0997b91d1a)
2013-03-15 18:09:34 -04:00
Tim Abbott b6f42cb1e1 Update UserProfile objects in memcached when we save them.
(imported from commit 59426e19eeeba131f363a4bc5f74f59b6dbe1066)
2013-03-15 18:09:33 -04:00
Zev Benjamin b08dbdfe48 blueslip: Send the user agent string back with the error report
(imported from commit ed77877465b1497cd3d78753c3398329302075ff)
2013-03-15 17:30:07 -04:00
Zev Benjamin 53a68144e0 Don't do processing for a get_old_messages if the message list it was for goes away
This was biting us before when the user would leave a narrow before a
get_old_messages call associated with it finished.  Specifically,
search.maybe_highlight_message() would assume a message was in the
DOM when it wasn't any more.

We also have to hide the 'loading more messages' indicator when
reseting the 'load more' status because otherwise it wouldn't get
hidden like normal in the load_old_messages() continuation, causing a
load_more_messages() not to fire when re-entering a narrow.

(imported from commit 4a136dd01305b039c0970f897b07e603b87d5d8e)
2013-03-15 16:40:05 -04:00
Tim Abbott 7e456ddba7 Cache get_status_list output for up to a minute.
This should substantially decrease the amount of server load generated
by the userpresence system.

I tested that this indeed was indeed saving one query on
/json/update_active_status requests on my laptop with 2 users from the
humbughq.com realm logged in.

(imported from commit 03e9d4eb95b9f664d489862684ae162db2076e08)
2013-03-15 16:37:41 -04:00
Tim Abbott ec07184721 cache_helpers: Fill in memcached message cache in batches of 1000.
This cache filling code takes about 5 seconds to run, which means it
will finish before Tornado starts reading from this cache, but if that
were to change, it would be much better to have made at least some
progress.

(imported from commit 60a3420cdb9ddf331d83573a3fdb6be1a5ee5a4f)
2013-03-15 16:37:41 -04:00
Tim Abbott e78ef47487 message_cache: Query related models to avoid more queries later.
(imported from commit c1442165c68e9eef1cb5d310624c1021c1a57350)
2013-03-15 16:37:40 -04:00
Tim Abbott d88591c19c cache: Fix the select_related() query name for sending_client.
Previously we were calling select_related() on Message.client, which
doesn't exist.  It seems kinda poor that this doesn't raise an
exception.

I believe this issue was causing us to do very large numbers of
database queries during get_updates calls during server restarts.

(imported from commit b79bd698820fbd9dd82bd61fc175c32cd5ce6d05)
2013-03-15 16:37:40 -04:00
Tim Abbott e080be82fb Add the user's client string to our Django logs.
This should let us distinguish e.g. mobile from API bots.

(imported from commit 192114a0f79bfa7fb918e8c026da63ef772c2449)
2013-03-15 16:37:40 -04:00
Tim Abbott a46eb8a36c Clean up excessive positional arguments to load_old_messages.
(imported from commit 072c0fcbd59256d55364a0fc6be42d5df8ab99e2)
2013-03-15 16:30:03 -04:00
Tim Abbott 43e43a4248 Fix HumbugAdminEmailHandler for non-exception logged errors.
Previously, AdminEmailHandler would crash if the record did not
contain any exception information (e.g. because it came from
logging.error()).

(imported from commit 868ec68f3a57eddbb2c851ecf9894e5eddf21e4c)
2013-03-15 14:53:17 -04:00
Tim Abbott d42e8ecb11 Log handlers: Provide the actual error message on errors.
Previously when we logged errors using logging.error() but didn't
throw an exception, we would send these totally useless error
humbugs/emails that look like this:

Error generated by Anonymous user (not logged in)
No stack trace available

This change makes those messages include the actual text passed to
logging.error(), which is substantially more useful.

(imported from commit 76a8220ffe66d19cb0ca8ba9d1b42d5ecc4fd511)
2013-03-15 14:53:16 -04:00
Tim Abbott 722918aa61 Use humbug_finish() wrapper for finish() with required arguments.
I think this is a little cleaner than the previous approach.

(imported from commit fa6233721f43c94003de643d7801a88a156affd2)
2013-03-15 12:53:07 -04:00
Tim Abbott 419e153422 runtornado: Mark block of get_response modified from the original Django.
(imported from commit 383d6960f0043175c026956e1724ec1d1157767b)
2013-03-15 12:53:06 -04:00
Tim Abbott ade7146cd9 Disable tornado request logging in favor of our Django logging.
(imported from commit 2144b2ac1cdb6062bb32fca4cc69ea3f4570cf69)
2013-03-15 12:53:06 -04:00
Tim Abbott 4df6736225 Call the Django response middleware from Tornado views.
But discard any changes the Django response middleware may have made
to anything other than the content.

This allows us to, for example, output our nice database query logging
for get_updates requests.

(imported from commit e1d2fd38ceb4d73ff50bdfaad7c72ddb24d0fe16)
2013-03-15 12:53:06 -04:00
Tim Abbott 549577a928 Pass the original request object to Tornado's finish().
This is preparatory for running the Django response middleware on
our Tornado responses.

(imported from commit 05da8ea9cb663a928b2f98a928f3992aae4f067c)
2013-03-15 12:53:06 -04:00
Zev Benjamin a495b0dfa6 Re-educate rogue emoticon
He was wrecking havoc by confusing the Javascript mode syntax
highlighter.

(imported from commit 27900cf6b6dc7866f2b2f108e79d28d6785aa466)
2013-03-15 12:43:12 -04:00
Leo Franchi 96041be75c Use 'undefined' instead of -1 as sentinel
(imported from commit 30cc280d89cf0c290459f9d90960cd8eaf76a22f)
2013-03-15 10:51:25 -04:00
Leo Franchi 79dfe57840 Mark all messages between start of scroll and end of scroll as read
If the user scrolls super fast, our scroll handler might not catch
the user  passing by some messages.

(imported from commit 14cebffcd1321f02443971ac5e1c922db19648ab)
2013-03-15 10:43:17 -04:00
Zev Benjamin 22457277f4 Move send_with_safety_check functionality into AsyncDjangoHandler
This will mainly be useful in the event system branch, where we want
to actually send a response from a file other than tornadoviews.py.

(imported from commit b7ae9bb9b062215ab44eb5f0a3a72d6baeee1d07)
2013-03-14 18:48:15 -04:00
Waseem Daher 0f621f5f89 Show Twitter previews on production.
This reverts commit 4baed7ba22482eff6ed381d319f6e21da904d9ba.

(imported from commit dcab148b24e95f90edd5f2f3f581b4037deed787)
2013-03-14 18:00:04 -04:00
Leo Franchi 07433b80cc Don't mark your message as read if it comes from the API
(imported from commit 0570d4db590e21b0fadd544b10c9865fa7acd66b)
2013-03-14 15:51:21 -04:00
Leo Franchi 95c9c6817f Mark messages as read as you scroll by them
(imported from commit 7e4595dcbfe900bde7f062e3fe9e2d53c71b6884)
2013-03-14 15:51:21 -04:00
Leo Franchi d3364eef5d Try a different color for mentions
(imported from commit e1f7e4c03abb54b8e7efbdff84314a902ad32697)
2013-03-14 15:38:09 -04:00
Tim Abbott 76837ebe2c fill_memcached_caches: Also fill the caches of User/UserProfile objects.
Since we flush memcached when we do a server restart, the flurry of
get_updates requests that fly in afterwards are all cache misses for
getting the User/UserProfile objects, so Tornado ends up spending
around 70ms per get_updates request rather than the usual 1-2ms.

So this should substantially improve our Tornado performance around
server restarts.

(imported from commit 07b8126bdfd4ff14e4c3362f9eda1fe5fd571c5b)
2013-03-14 15:07:42 -04:00
Tim Abbott cf58ee833a Rename lib/message_cache.py to lib/cache_helpers.py
(imported from commit 591e427aee47aea58b50d6c3fbd6a05cfe9378ea)
2013-03-14 15:07:41 -04:00
Tim Abbott 60c3cf826d restart-server: Fill memcached caches after clearing the cache.
Our previous code could in theory end up clearing the caches it had
just filled, if Tornado's cache filling work happened to be faster
than the memcached flush.

(imported from commit 48174aadad398fb7a7c917a1df765c1261b12a55)
2013-03-14 15:07:41 -04:00
Tim Abbott adb9c88151 message_cache: Clean up populate_message_cache.
(imported from commit a73e4fc4908d840f4f02aef13a9afe2fb683fd83)
2013-03-14 15:07:41 -04:00
Tim Abbott 0a9a553b5f Fix use of invalid hash keys with memcached.
Email addresses can contain characters that are not valid for
memcached cache keys.

(imported from commit d7cfd6d5292e4fc8f4e2b19f2c01a4097d878f1b)
2013-03-14 15:07:41 -04:00
Tim Abbott c098520bbd Move the key functions for various caches to cache.py.
(imported from commit b04826533c32516cc2eef3b35263a40385ae7be4)
2013-03-14 15:07:41 -04:00
Tim Abbott cc4bcb97c4 Clean up naming for Tornado user_profile cache keys.
(imported from commit 8f401b563c9ef49c739c121c9dc5bffff09c986d)
2013-03-14 15:07:41 -04:00
Tim Abbott 1580386946 Replace db_cache_with_key with a parameter on cache_with_key.
(imported from commit f2c600292888ba384ee4acc97c94f1d6f8bc9657)
2013-03-14 15:07:41 -04:00
Zev Benjamin 719e24a25b blueslip: Use reference counting to try and break the circular references we create
We create a circular reference between handler functions and our
wrappers for them so that we can pass the wrapper to jQuery.off when
users pass the original handler to us.  This reference-counting
system can't break all the circular references we create because
users can unbind event handlers without explicitly naming the
handlers they want to remove (they can remove all bindings on an
element, for example).  For now, we hope that this memory leak isn't
too bad.

(imported from commit 9615b5761b4b09ca7ca52c0d847e9b83330373fa)
2013-03-14 15:07:28 -04:00
Zev Benjamin e8470b9481 blueslip: Also override jQuery.off so that we can pass the correct handlers to the original jQuery.off
Previously, we couldn't actually unbind some event handlers.  The
problem was that when a user called $.off(events, handler), the
passed handler wouldn't match any that were actually bound because
the handler that was actually bound was our wrapper.

This bug specifically caused the handlers for our idle timers to
never be unbound, effectively never cancelling them.

(imported from commit 48efac954994a05c356d326e64a78ab0ace9fe3e)
2013-03-14 15:07:28 -04:00
Zev Benjamin 8709e3f8b4 blueslip: Store the handler wrapper on the original function
We will need this for removing event handlers.  This will
unfortunately create a memory leak, but we'll partially deal with
that later.

(imported from commit e439cb44d245e16d2254d1be053b68015a1f4c79)
2013-03-14 15:07:28 -04:00
Zev Benjamin 122996b656 blueslip: don't double-wrap handlers when invoked with the $.on(types-Object, selector, data) syntax
(imported from commit 458ee9248fc82efc6c2307634b2be6d3ce32246d)
2013-03-14 15:07:28 -04:00