Commit Graph

391 Commits

Author SHA1 Message Date
Luke Faraone 29cb701444 Allow deactivations to cascade to bots
(imported from commit 1c42013dab02a8f264e814fef63c84cd03102458)
2013-07-08 14:30:20 -04:00
Luke Faraone 37edb61b67 Don't set an unusable password during deactivation.
This would have made reactivations hard, and doesn't really buy us much
additional security.

During deactivation, all a user's current sessions are deactivated and
they are marked as not active. This prevents them from logging in via
the web UI, and makes their API key unusable.

Randomizing their password is probably gratuitious, especially as we
start to allow authorized end-users to deactivate others.

(imported from commit c63d23816da0452a1df821f2fa6c1db2761733da)
2013-07-08 14:00:19 -04:00
Luke Faraone 573bb8e5fd Add populate_db support for deactivating users.
Prior to this commit, populate_db would crash if you had ever deactivated
a user in your development instance's message log.

(imported from commit 227b2c0226a46ef5680443d3dbf62a13ce961e64)
2013-07-08 14:00:19 -04:00
Steve Howell 71dec63ad1 Fix regression with JS tests only working the first time.
The JS tests would fail on the second run due to memcache having
dirty data.  This change sets a new KEY_PREFIX whenever you launch
a server in test mode.

(imported from commit 4d41e6b79ab3bb7cb4c96b37050f0b1c9abc6b5e)
2013-07-03 10:43:24 -04:00
Jessica McKellar dbbc555a52 Remove now-unused do_finish_tutorial.
(imported from commit 6511d086376d1e81218cd31268f39e2145cb4d48)
2013-07-02 18:26:17 -04:00
Kevin Mehall dce1f7f729 Parse @-mentions in bugdown and style them.
* This makes bugdown.convert take a `message` parameter. Properties
    for parsed mentions are added to the message object by the `Pattern`
    for use in do_send_messages.

  * Refactor repeated markdown rendering code into `Message` model methods.

(imported from commit 4f0ed5570104c0210f984b6de21e9048e2b53fa0)
2013-07-02 18:20:26 -04:00
Steve Howell 9e64750083 Use memcache in the test suite.
This uses a new configuration that enables memcache, but we have
to be careful to bounce KEY_PREFIX on every new test, since data
gets rolled back in the databases between tests, but not in
memcached.  We had to break up one test to work around UserProfile
objects actually being cached.

(imported from commit f201cf9cd9e0e4c61d3c384fa8d2bbd5134161e8)
2013-07-02 16:50:10 -04:00
Tim Abbott f473cb5fbe Use bulk operations in notify_subscription_{added_removed}.
After fixing the high numbers of database queries earlier in this
branch, I found that sending 500 RabbitMQ messages for a bulk change
in subscriptions was consuming more than half the time for these (and
then we'd end up with 500 events in a queue).  To handle this, we
create a "user X subscribed to these N streams" event, rather than
sending one event for each individual subscription.

(imported from commit 44a34a9fab9b67e9f0da6fee53335d8c5030392b)
2013-07-02 10:50:29 -04:00
Tim Abbott 74fd508b2f Remove subscriptions using bulk queries.
This improves the performance of unsubscribing to N streams by more
than a factor of 10 for large N.

(imported from commit a529e6d3ac4452f49c2294908d275280019bbd05)
2013-07-02 10:50:29 -04:00
Tim Abbott 7f3fded612 Add default stream subscriptions using bulk queries.
(imported from commit f6d7415269cc74836c1c331741665b01cb1090cd)
2013-07-02 10:50:29 -04:00
Tim Abbott a66bb508bb Use bulk queries for adding many subscriptions to a single user.
Previously we only used bulk queries when adding many users to a
single stream, resulting in very slow performance when subscribing
users to large numbers of streams (as happens when setting up a new
MIT realm user).

(imported from commit 849fa7b2a1a146c0a9adc1c727c20c9fbfb7b425)
2013-07-02 10:50:28 -04:00
Tim Abbott 738793a962 bulk_add_subscriptions: Remove incorrect comment.
This comment was only ever accurate for prototype versions of
bulk_add_subscriptions prior to it being committed to master.

(imported from commit 89b9dc49423c45553cb6c810d97eea4583ff0f69)
2013-07-02 10:50:28 -04:00
Tim Abbott c6916faca2 Genericize the bulk cache fetching logic.
(imported from commit 52664ce0ad800efff6b583e80c27701638ff6205)
2013-07-02 10:50:28 -04:00
Steve Howell d94f91fd43 Simplify do_send_messages() from prior commit.
This change removes an "if True:" that was
introduced to make the prior commit a bit more readable.
It also combines two loops, since the second loop is no
longer conditional.

(imported from commit df58f1e5de72d5669f6468fbff54fb62cd22cedb)
2013-07-02 10:50:02 -04:00
Steve Howell a4db7de330 Actually test callbacks in GetUpdatesTest.
The tests in GetUpdatesTest had some callback logic that has
been dead code for at least three months.  We now fully exercise
the callback codepath and make sure that the callbacks do happen.

(imported from commit f5d8fbab28ecc34dc81d3d0c29058b66c10f378f)
2013-07-02 10:50:02 -04:00
Scott Feeney 83cd963c49 Remove unused imports
(imported from commit 9e3050c72a2d1137b9096c6cfa1c3945341b9a56)
2013-06-27 16:22:39 -04:00
Steve Howell 2bacaf2213 Fix % formatting style in miscellaneous places.
(imported from commit 917196024c981f879355c728da9b4590e964eeef)
2013-06-27 14:41:17 -04:00
Steve Howell e904dbc19c Correctly set read flag for messages sent by yourself.
Compare two user objects by id to prevent false negatives
when the objects are fetched thru different paths.

(imported from commit a41f30d27e2b8021600d89f32d6526f48677fd95)
2013-06-27 09:17:47 -04:00
Tim Abbott 0b9e9bca64 gather_subscriptions: Fix some whitespace.
(imported from commit a36c21c1acf2c2fad4aebb78ab1ac07ab67ff285)
2013-06-25 16:34:44 -04:00
Kevin Mehall a06860ca68 Don't perform an additional SELECT just to get a count for statsd.
len() uses the QuerySet as an iterator, which performs a SELECT for all
of those rows.

(imported from commit d362fef375c5270bedade2a7b50aea7b1a559d1e)
2013-06-25 15:25:25 -04:00
Kevin Mehall 953e195be1 Refactor update_message_flags to do the query in the django process.
Trac #1398.

(imported from commit c001747d8e6d78a12fe535c36a81c12592976840)
2013-06-25 15:25:25 -04:00
Tim Abbott d6e04ce2d6 Fix @-mentions.
Trying to check whether a Django model object is inside a set of other
Django models is not correct in general, e.g.:

UserProfile.objects.only("id").get(id=17) in set([UserProfile.objects.get(id=17)])

returns False.

This bug appears twice in the function, once when computing which
users were mentioned and again when pushing the flags through to
Tornado.

(imported from commit b09ed550258f9df2611e1b0a60f87c48a51830f8)
2013-06-25 14:05:20 -04:00
Steve Howell 660ee6571d Disable embedly everywhere.
(Before it had been disabled only on prod/staging, but we are
removing it everywhere, motivated by making tests run faster.
In particular, the call to embedly_client.is_supported() was
expensive, as it went over the Internet.)

(imported from commit ea12bf6e7ae84ce7e8023a0d314ecc4c07cbc0a8)
2013-06-25 08:46:02 -04:00
Steve Howell 67058cc26f Enable back door for clearing user's rate-limit history.
(This is helpful to make the tests run faster.)

(imported from commit 48c1beaed8cca76976bd6f54224c33460b55ceac)
2013-06-25 08:46:02 -04:00
Scott Feeney 6b8bc74b4e Replace SITE_ROOT with DEPLOY_ROOT
Whereas `SITE_ROOT` referred to the directory where settings.py is
located, *all* actual uses of `SITE_ROOT` were joining it with `..` to
get the root of the git checkout, a much more useful value.

`DEPLOY_ROOT` now represents the root of the git checkout.

(imported from commit 351437f9a5801e5c7c08a3a97619e863144e5cc8)
2013-06-24 15:55:03 -04:00
Tim Abbott 04f1b1da66 Don't flush the UserPresence cache for a realm unless status changed.
Previously we had an issue that every other update_active_status
request for a particular realm would result in doing the expensive
query to compute the list of active users in that realm.  It turned
out this was because on every update_active_status request, we'd queue
an event that would have the effect of clearing the cache, even if
nobody's tatus changed.  This fixes that issue, by only clearing the
cache for a realm if someone's status actually changed (or the 60s
timeout expires).

(imported from commit d5b829fe255a31c8cecb58458738f1e72a2cf6de)
2013-06-24 14:34:51 -04:00
Tim Abbott c87d182102 Avoid saving user_presence twice when creating a new object.
(imported from commit 72af41b4e0b85f9a8605f5040c321c80e16e8a97)
2013-06-24 14:34:51 -04:00
Kevin Mehall da26a6a4fd Merge http_link and autolink regexes, as they share many workarounds.
(imported from commit 62615af5bf6d48620ec3f9e2523fecbc1ff01441)
2013-06-24 13:41:01 -04:00
Kevin Mehall c5c47fc921 Simplify and comment AutoLink regex.
Override python-markdown to compile with re.VERBOSE.

(imported from commit 77a797c79fb3f6c11948f431715c8bb624dc8bc7)
2013-06-24 13:41:01 -04:00
Tim Abbott d2d1613153 send_message: Don't fetch unnecessary data on recipients' UserProfiles.
(imported from commit 4e4b5bdb510610cf7198bfb01fc465fe15bd834d)
2013-06-24 11:42:17 -04:00
Tim Abbott edc4f51b00 do_update_message: Check max message and subject lengths.
(imported from commit 48e8b21c7ca122afc40fcd1482d26185f50cadab)
2013-06-24 11:42:17 -04:00
Tim Abbott 631a9048f7 Send message dicts to memcached as JSON dumps.
memcached stores objects sent to it using pickling, which is very
slow.  We work around this by sending memcached strings (i.e. JSON
dumps); pickling doesn't slow things down too much if all it is
getting is a string.

(imported from commit 0f0e534182eccb76c5731198e05a9324a1cef316)
2013-06-24 10:17:10 -04:00
Tim Abbott 222ef672b5 [manual] Use ujson instead of simplejson.
This saves something like 15ms on our 1000 message get_old_messages
queries, and will save even more when we start sending JSON dumps into
our memcached system.

We need to install python-ujson on servers and dev instances before
pushing this to prod.

(imported from commit 373690b7c056d00d2299a7588a33f025104bfbca)
2013-06-24 10:17:10 -04:00
Kevin Mehall 4f5b4fdb67 Bugdown: Elipsis followed by TLD should not be linkified. Trac #1388.
(imported from commit 3929e4bc68872e5f5ed58f35dd1b4eb53a2e5990)
2013-06-21 15:33:17 -04:00
Tim Abbott e55eb53297 Use get_user_profile_by_* functions to get UserProfiles.
We had a few bugs where we were using a raw Django database query to
get a UserProfile object.  This might seem OK, but going through
memcached is more efficient, and also guarantees that we get back the
.select_related() version of the object, so that if we later access
related fields like user_profile.realm.domain, we don't end up doing a
second database query as well.

Fixing these should in practice save a substantial number of database
queries on handling update_status_list requests, which happen very
often and access user_profile.realm.domain.

(imported from commit 0a2027da1b5bbc7a4f6c6927aca498530d7a4977)
2013-06-20 17:57:06 -04:00
Tim Abbott 47255e9cd8 Fix buggy key computation in cache_with_key.
The refactoring to use the cache_get() method incorrectly didn't
remove the addition of KEY_PREFIX inside cache_with_key.  The result
was that the KEY_PREFIX was being added twice, once by cache_with_key
and again inside cache_get.

This had the impact of causing pointer saves to not take effect,
because our attempts to update the memcached cache when we save the
UserProfile object were using the correct cache key, but the actual
code reading values out of the caceh wasn't.

(imported from commit dcea000833f00622bdc0249488de3b186a7417b2)
2013-06-20 10:57:32 -04:00
Tim Abbott 60b8635593 cache: Add a helper function to get the right cache backend.
(imported from commit 3e80178dbaa0fcc77d1075bb3fad29ace7540a02)
2013-06-19 18:40:58 -04:00
Tim Abbott d2e18094d8 cache: Refactor cache_with_key to use the helper functions.
(imported from commit 24f8af8de17d3376c8ee56da5c2a901c2c314e78)
2013-06-19 18:40:58 -04:00
Tim Abbott 46c40c011a cache: Use KEY_PREFIX in cache_set().
(imported from commit 3dbddc9c513b58f692947f0c30a18ec12250e3a5)
2013-06-19 18:40:58 -04:00
Tim Abbott cb4e965fd2 cache: Remove KEY_PREFIX from keys returned by cache_get_many.
Otherwise code paths that use those keys, like get_old_messages, will
incorrectly use the prefix-included keys.

This bug in our KEY_PREFIX system results in our memcached caching for
get_old_messages always missing.

(imported from commit 506c13e06d6f266596ead0b381c324c256e576c3)
2013-06-19 18:40:58 -04:00
Steve Howell b896c587e2 Extracted S3-related code to lib/upload.py
(imported from commit 4f0b0012b89d21b3bd445a2a6f84f3de72190e13)
2013-06-19 18:10:45 -04:00
Steve Howell 2b320f8d57 Allow user to upload an image for their bot's avatar (back end).
(imported from commit 34833b61b935f4eec2b23abbb3532aaa58e13fb6)
2013-06-19 10:21:20 -04:00
Zev Benjamin 80702ccaa0 Add metrics variables to the template context via a context processor
Wrapping render_to_response never actually worked correctly.  On the
login page, mixpanel_token would be missing, but we wouldn't get an
error because it is surrounded by double quotes, which meant that it
was still valid Javascript.

(imported from commit 820ee42fab8f679983e5a3a4309a2feaf690f20f)
2013-06-18 17:07:36 -04:00
Kevin Mehall 334424ae79 bugdown: TLD and parentheses shouldn't make it a link. Trac #1364
(imported from commit d4da7ec080a6a8362b6370ced9ddc10d52443452)
2013-06-17 10:08:13 -04:00
Leo Franchi 625f8427e6 Return both subscribed and unsubscribed stream information when listing streams
(imported from commit 0aee84712eefd93aebc1041b371e471182eb7e9b)
2013-06-13 10:06:34 -04:00
Steve Howell d740d7c082 Show user-uploaded avatars on the website.
Show user-uploaded avatars on the website for users who have
UserProfile.avatar_source == 'U'.  (Continue to show gravatars
for other users.)  This includes the home page, the visible-phone
div, and the settings page.

This fix does NOT address a few things:
* There is no GUI to actually upload user images yet on the website.
* The !gravatar syntax in bugdown will continue to show gravatar images
  only.
* We are not changing identicon behavior.

(imported from commit 9f5ac0bbe21ba56528048233aab2430e4dd431aa)
2013-06-12 15:34:51 -04:00
Zev Benjamin 03b40d180c Add logging filter for checking that the app is actually deployed
(imported from commit 77bd7e008fdea4033e18a91d206999f9714e0f74)
2013-06-10 14:59:29 -04:00
Zev Benjamin 7d999160ab Move logging rate-limiting code
The new file can't be called logging.py because then it would be
annoying to import the system logging module within it.

(imported from commit 71d116e4be98d45b09dda049a43142a82647b727)
2013-06-10 14:59:29 -04:00
Leo Franchi f4132ace48 Fix bankruptcy event sent to graphite
(imported from commit b016ab41ddc2636b76a49c9eb33b7becd387557d)
2013-06-10 09:17:43 -04:00
Tim Abbott 16ffd539af Fix computation of original pre-editing message contents.
(imported from commit ebbdc2580cd502bfb2fb07fa144ab6f815b960e7)
2013-06-06 14:18:49 -04:00