Commit Graph

455 Commits

Author SHA1 Message Date
Tim Abbott d188d829d7 Update UserActivity queries for monitoring Zephyr mirroring.
(imported from commit 04a9536da2891e905c6e14e0d452ca62d632641d)
2013-10-04 16:15:53 -04:00
Zev Benjamin 4129541206 github: Add support for commit comments
(imported from commit 43abd740e563952d1291939e6a530b8366232eb3)
2013-10-04 15:53:21 -04:00
Steve Howell 95321a9845 Clean up a few @slow decorators in tests.py.
A few "slow" tests aren't as slow any more, for whatever reason,
so we're setting a higher bar going forward.

(imported from commit 642137cebb7826f4512b5635da9d7b75bd5c35f4)
2013-10-04 00:25:49 -04:00
Zev Benjamin 87daa8d5ef Disable all github notifications except for pull requests for customer37.invalid
(imported from commit bbfe47f4c463516f1e962291bc50ec79c5b2afb5)
2013-10-03 18:48:51 -04:00
Kevin Mehall 72a0b67c80 Prevent other markdown operators from touching URLs.
The text of manual links are already AtomicStrings, so linkified strings
should be too.

Moves emoji detection to happen after linkification, so the emoji rule
won't look at links.

(imported from commit 9c56bce6a0e873b398255e0762dfb312a4a9a64e)
2013-10-03 17:26:47 -04:00
Kevin Mehall 8842349629 Fix links with invalid emoji in them.
InlinePatterns should return None on failure, not text that may
have placeholders in it.

(imported from commit f9d8d22b2b8cfa7a92ecf3e52a6c76b48e6f0175)
2013-10-03 17:26:47 -04:00
Leo Franchi 7da3084465 Use name attribute in dictionary not property lookup
(imported from commit b2adae82f31b57d4e9e049225a662851b4364789)
2013-10-03 16:59:50 -04:00
Steve Howell 35fce47088 Do not embed the API key in webhook error messages
(imported from commit 12d782b1bde2c9f928e509bd6dd0ae9106616ed6)
2013-10-03 14:38:37 -04:00
Steve Howell 05178349a2 Have rest_dispatch() use name of target function for logging.
We want the UserActivity.query field to reflect the name of the
function for REST calls, not the URL, and we accomplish this by
setting request._query to target_function.__name__.

(imported from commit 9df05fef0dffb34483b182b95f8cbc4409083eed)
2013-10-03 14:23:02 -04:00
Steve Howell 97adbedfca Support request._query in update_user_activity().
If request._query is set in the call to update_user_activity(),
we will use that instead of request.META['PATH_INFO'] for
the query field of the UserActivity row we write.

(imported from commit fcee30098e1c7c5cb4195a1e5905fc7b88af804f)
2013-10-03 14:23:02 -04:00
Steve Howell cab71ab3f9 Apply @api_key_only_webhook_view to api_newrelic_webhook.
This makes the code save UserActivity records for this action.

(imported from commit 1ca37d2ba67b2c0e502fc567f80ebda918adc909)
2013-10-03 14:23:02 -04:00
Steve Howell c990d942ac Apply @api_key_only_webhook_view to api_pivotal_webhook.
This makes the code save UserActivity records for this action.

(imported from commit d11267ad3913a4dfe90ad7bd8b30704c7ddb7cb3)
2013-10-03 14:23:01 -04:00
Steve Howell 1c46eca0ef Apply @api_key_only_webhook_view to api_jira_webhook.
This results in some small behavior changes.  First, if a user
has both malformed JSON and an invalid API key, they will now
be informed of the invalid API key, not the malformed JSON,
because the decorator wrapping code executing first.  Second,
we call process_client(), which basically builds us a
UserActivity record with the client "API".

(imported from commit fadb523db9bdc82984bdae61833c5c99f1ebd1c0)
2013-10-03 14:23:01 -04:00
Steve Howell 8a4a069703 Add api_key_only_webhook_view decorator.
This is for webhook API endpoints that only get passed in an api_key,
not an email.  An example would be api_jira_webhook, and some of
the code is borrowed from there.  The rest of the code is from
authenticated_api_view().

(imported from commit b5b2a4ea52f9b317f00357ef3142c76534fabf20)
2013-10-03 14:22:49 -04:00
Steve Howell f6794ce849 Remove stray print statement in tests.py
(imported from commit 2be8ceb83664cfb9f3158e23894637ac513efa41)
2013-10-03 13:03:33 -04:00
Steve Howell d9afabdf6c Add totals row to the main /activity tab.
(imported from commit 9ab25b8fe3018d6f0b31c65494e0132ca997ca9b)
2013-10-03 09:33:51 -04:00
Steve Howell a51f7949b7 Add minutes column to the main Activity tab.
Add the number of person-minutes for the last 24 hours to the
realm report on the main tab of /activity.

(imported from commit 2ff46eacc4c8276ab0407fc6ff9f28f5137f1ed2)
2013-10-03 09:33:51 -04:00
Tim Abbott 3422b02e52 Add some unit tests for gather_subscriptions.
(imported from commit 10c362e36e2e9044918e22449b6d82acb9c867f3)
2013-10-02 16:37:56 -04:00
Tim Abbott 4ae2ae5005 gather_subscriptions: Avoid fetching full stream objects.
(imported from commit 3052dd9c1df050e217949d36259416bc38469674)
2013-10-02 16:15:25 -04:00
Tim Abbott 356d33cfa0 gather_subscriptions: Use .values() for fetching all the user's subs.
(imported from commit 1f27841527bb5f49fab347071e40d289151fa2c5)
2013-10-02 16:15:25 -04:00
Tim Abbott 1eeb808a79 bulk_get_subscriber_emails: Use .values() to substantially improve performance.
(imported from commit b37738d670930cdf65687a8526d8a8f47835b42b)
2013-10-02 16:15:24 -04:00
Tim Abbott 382dd64ce9 gather_subscriptions: Tighten the query for Subscription objects.
(imported from commit a7caed32e52f68c7d6e33e15a7954df4fe90d797)
2013-10-02 16:15:24 -04:00
Jessica McKellar 6afdfeb1dc Filter obvious footers from email mirror messages.
(imported from commit 7e22b6823b5d0a07f7f350e4af65be66728a8f88)
2013-10-02 14:39:22 -04:00
Leo Franchi 0d6ad56c29 Decode operand in filter.js in operator-aware way
When decoding an operand, a + can be converted to a space
only if the operand is not an email address.

(imported from commit 08fc36a579bbe6409137c60c0fa9579fe3ab2c43)
2013-10-02 12:11:15 -04:00
Steve Howell 00cf695af2 Add "Durations" tab to /activity.
This tab shows how long each user has been on during the last 24
hours, using data from UserActivityInterval.  Much of the code
is borrowed from analyze_user_activity.py, but in this version
we set the time interval to be the last 24 hours and sort by
realm and email.  I also ensure that it only executes one
query to get all the data (and there's test coverage for that).

(imported from commit 7a2b80f52679054b03c5f5f42b2cda07d5599432)
2013-10-02 11:27:23 -04:00
Steve Howell 786c2cf759 Remove client-specific tabs from activity overview page.
Waseem is ok with removing the client-specific tabs on the
main /activity page.  This reduces the number of queries from
25 to 1.  We might eventually restore some of that logic, but
we will do it more efficiently.  A lot of the data for
non-website clients is kind of unreliable, anyway.

The page looks kind of funny with only one tab, but that
will be fixed in the next commit.

(imported from commit 54f08f89d5242ad3e045d8ca0d97b86617c15380)
2013-10-02 11:27:23 -04:00
Steve Howell 374a64acf8 Add test for /activity page.
For now we just verify that we don't explode, and we count
the number of queries (25).

(imported from commit 145c3373e2f6d005827af743729e03e524167904)
2013-10-02 11:27:22 -04:00
Leo Franchi 724f7e54e7 Clean up slow log displays
(imported from commit cab99eeaef14e1f24a6b09b0cdbcdcc45bcb620a)
2013-10-02 11:26:58 -04:00
Leo Franchi 028711baef Handle some JIRA formatting and make it like our own
(imported from commit 1e8369067668b0c375f226c1911814f496637d16)
2013-10-02 11:18:17 -04:00
Steve Howell c23ab96fb9 Add simulated_empty_cache() test decorator
(imported from commit 2dd03a522f9ee01a802c8e5ff4116db04763ed35)
2013-10-02 10:45:18 -04:00
Tim Abbott b567630b5e Fix UserActivityInterval handling when events arrive out of order.
(imported from commit fa1cfbd67ac4c6c2864fb19db8a5221f90d028e7)
2013-10-01 17:41:30 -04:00
Steve Howell 53b5768b14 Double speed of message fetching in get_old_messages_backend().
When we don't already have old messages in cache, we need to
fetch data from the database and create dictionaries for the
cache.  This commit makes that process work in 50ms, instead
of 130ms, for the data set in test_bulk_message_fetching(),
which is 602 records.  Before this commit we had about 132
microseconds of unnecessary churn per message, because we
were fetching DB fields we didn't need and incurring the cost
of the Django ORM.  Now we use values() to get only the columns
we need, and we take advantage of previous commits that make
our code less OO and more function-driven, so we can pass the
values directly to build_message_dict() without having to create
objects.

A couple caveats on this commit:

1) I haven't been able to get good measurements on the overall
effect on get_old_messages_backend().  If you kill the cache to
force DB queries, you introduce noise related to sessions and
user profiles.

2) Look at the long comment in this commit related to
re-rendering messages in this codepath.  The problem precedes
this commit.

(imported from commit dcb64aa9416f0e9583355ddd6dc3adfa746b9fc7)
2013-09-30 18:22:07 -04:00
Steve Howell eb299bf5c8 Extract Message.build_message_dict()
(imported from commit 03395ea8fbffd0a52f85127c9a24f0cc3f4a6969)
2013-09-30 18:22:07 -04:00
Steve Howell 2a3747e21f Isolate message side effects in Message.to_dict_uncached().
Only call a function on the message object in the unfortunate
situation that we are rendering new content in to_dict_uncached().
Long term, it would be nice if this function didn't have side
effects, and we had a better strategy for upgrading rendered
content when bugdown versions change.

(imported from commit 2a323f52af37a6d651c171cb8234fbfa3d25d561)
2013-09-30 18:22:07 -04:00
Steve Howell f65462c891 Extract Message.need_to_render_content()
(imported from commit 466fa133ed1e8fb13b7222ba3bdcd5773d63de14)
2013-09-30 18:22:07 -04:00
Steve Howell 95254398f2 Extract get_avatar_url().
This function doesn't require the whole UserProfile object to
create the avatar url, and we call it from Message.to_dict_uncached().

(imported from commit e814caab101c4fedd1ba66df041a3408014e4085)
2013-09-30 18:22:06 -04:00
Steve Howell e4fb66fdf8 Isolate self-dot use in Message.to_dict_uncached().
For a bunch of self-dot references, move them to the top.  (This
is kind of funny out of context, but it sets us up for future
refactorings.)

(imported from commit 4ebc1c44a633d86772df1828c51180707769c3dc)
2013-09-30 18:20:55 -04:00
Steve Howell 005ad9c8e0 Kill off dead code in Message.to_dict_uncached().
If this line of code were ever called, it would crash anyway,
because it would be an unknown type, and Recipient.type_name()
would raise a KeyError.

(imported from commit db38c5f71fb2f0b044a832eb88e53fceb0d8a9cf)
2013-09-30 18:14:12 -04:00
Steve Howell dc077a708f Call get_display_recipient_by_id() in Message.to_dict_uncached().
This starts to make us less dependent on having full objects.

(imported from commit 5fb9504b077b676ecb697a909f4a0d2737f51efb)
2013-09-30 18:13:53 -04:00
Steve Howell a9fd6ddfce Extract get_display_recipient_by_id().
This is a variation of get_display_recipient that takes
values instead of an object, so that it is decoupled from
the Django object system.

(imported from commit 25bed43ecd62f1fe0176d517b7003e7f4c78bc37)
2013-09-30 18:11:29 -04:00
Steve Howell 3c802e1b02 Add MessageExtractorTest.test_applying_markdown()
(imported from commit 0e88f999a871e93ae14d157e0087a75ae81d60a3)
2013-09-30 18:08:32 -04:00
Steve Howell 8ae789eaf8 Add test_message_extractor_basics().
(imported from commit 37e64cd4db1850e04bd9fdc37de1f38400436736)
2013-09-30 18:08:32 -04:00
Steve Howell 515130396c Remove TEST_SUITE exception for get_display_recipient().
If it's ok for the tests to use memcached, it should be ok
for them to use the in-process cache too.

(imported from commit be43879c3c48f3780317fd5b4139b44d4a1f0ed3)
2013-09-30 18:08:31 -04:00
Steve Howell 826dffbbb4 Extracted Message.save_rendered_content().
This is a harmless extraction designed to allow subclasses override
the behavior of how rendered content gets saved.

(imported from commit 9df4ed9f86c857897fcb5f2b6781bfc5a0813766)
2013-09-30 18:08:31 -04:00
Steve Howell f36938c2f5 Extract Message.extractor
(imported from commit e6ee015cacdb0f6e1f725b7113f750e8b33d316b)
2013-09-30 18:08:29 -04:00
Leo Franchi 2614716fca Log slow queries to zulip so we notice them
(imported from commit 23f311ad881edda4c4495089ea3b55213470a059)
2013-09-30 17:41:56 -04:00
Tim Abbott 002e28d838 Display user's email in logs for requesting hitting /.
(imported from commit d272f22b8b4de9640a86b66d7833336cc8a9822a)
2013-09-30 17:14:05 -04:00
Tim Abbott e9ee05b2b2 gather_subscriptions: Avoid extra queries fetching realm objects.
(imported from commit 591083cfaff454b8ce48aacb5da7913b8fe0a38f)
2013-09-30 17:14:05 -04:00
Tim Abbott 5e420dc5c3 gather_subscriptions: Fetch subscribers using a bulk query.
(imported from commit 13c085d0ec55779279d31d3e301311ef1c9fd2c4)
2013-09-30 17:14:05 -04:00
Tim Abbott 3a3954a8c4 Extract validation for whether the user can access subscribers.
(imported from commit 4bff4c26f98a9c2ebdd11d931c977aae8e4ebffe)
2013-09-30 17:14:05 -04:00