Commit Graph

394 Commits

Author SHA1 Message Date
Jessica McKellar f0911f51a0 email mirror: ignore decoding errors.
Empirically, we only get these for malformed emails where the charset
specified in a message part header does not match the true encoding of
the part. I checked what the resulting Zulip looked like for the
original offender, and it looked find with ignoring errors.

(imported from commit ac6ba65b611cb22d4ec547b75a585abce6fc50b0)
2013-09-27 11:34:57 -04:00
Leo Franchi b2a352da00 Limit Messages to those sent in this realm when prepopulating
(imported from commit af6312705bb43159e4448c16a9e05c00ca696ed5)
2013-09-27 11:12:09 -04:00
Leo Franchi 4102b8e844 Add query count tests for bulk_add_subscriptions
(imported from commit 84a68e3e50178df383e66bb29008da6086709eda)
2013-09-27 10:26:11 -04:00
Leo Franchi 0fe3da55e3 Reduce number of queries done by bulk_add_subscriptions
We now bulk-fetch subscription information once from the database
and use it throughout bulk_add_subscriptions in order to avoid
hitting the db O(streams) times.

On my machine this shaved the accounts_register API call from making
66 queries to making 37 queries.

(imported from commit 5dd5ad3f50b2a6edf85b5f1d55ebd697a1c60647)
2013-09-27 10:26:11 -04:00
Leo Franchi 88242de052 Use bulk_add_subscriptions in account_register
We have a handy bulk_add_subscriptions function to make cases
like this fast, so lets use it.

On my machine this reduces the number of db queries during account_register
from 112 to 66.

(imported from commit 21a6b31d0f229998d095735b8c581a50ca6aab66)
2013-09-27 10:25:23 -04:00
Leo Franchi 1e6b83b6c8 Fetch stream.realm information when fetching default streams
(imported from commit 08eb32e6ecda75b17d6532b0421c371de2327027)
2013-09-27 10:25:23 -04:00
Steve Howell 902185f8fb Add User Profile Count to General tab on /activity.
(imported from commit 231b3d6f2fe48f511d6387e25c2fea5e3accc186)
2013-09-26 22:04:36 -04:00
Steve Howell b84fb319ee Remove all-user counts in realm detail views in /activity.
(imported from commit 2231ef9a5f67e5b951c678d7f0d3f14dc252435d)
2013-09-26 15:41:20 -04:00
Steve Howell e0ceeb3340 Fix blank Desktop tab in /activity.
(imported from commit d1b7daa6db505feeb18baa6a546fd0e3fccf154b)
2013-09-26 15:41:20 -04:00
Steve Howell 85b7c93bd2 Show count of active sites.
(imported from commit 023c7880cd7784bbbb49589acde9f59aed8f1765)
2013-09-26 14:46:35 -04:00
Steve Howell 430aee6f87 Add a "General" tab to the /activity summary report.
It shows domains and how many active users they have.  A
user is consider active if they have done something at least
as active as updating their pointer in the last day.  Domains
with no meaningful activity in the last two weeks are excluded
from the report.

(imported from commit 700cecfc7f1732e9ac3ea590177da18f75b01303)
2013-09-26 13:20:02 -04:00
Steve Howell 6b5d569888 Extract local var "data" in get_activity().
This is partly just cosmetic, but it also sets me up to
build "data" more dynamically.

(imported from commit d436f700b137c1db3e9f50d308944231ce89bb83)
2013-09-26 13:20:02 -04:00
Steve Howell e93dce9ce6 Extract activity_table.html template for /activity.
A small functional change here was to eliminate an enormous "Usage"
headline that was already implicit from the tabs.  It would have
complicated the refactoring to try to preserve it, and I don't think
anyone will miss it.

Extracting this template will give us a little more flexibility
to customize future tabs in the /activity page.

(imported from commit bdb0b7030c8ec1e20d4451dc059830c3f5ea7632)
2013-09-26 13:20:02 -04:00
Steve Howell 0950aa45c4 Eliminate the default_tab attribute in ActivityTable.
(Just have the template automatically select the first tab.)

(imported from commit 4a1877c8d3d6215f8e8ad6a184c79bf4eb001001)
2013-09-26 13:20:01 -04:00
Steve Howell 9110ebd208 Improve the UI for /activity.
We are still showing the same data points, but the logic to drill
down on details for a particular realm is now all server side,
not client side, and we are smarter about omitting fields.  In
summary mode, we don't show empty Name or Email columns.  In
detailed mode, we show the realm as a headline instead of a column.

In this version you do lose the ability to see all system users in
the same view, but Waseem is ok with this.

(imported from commit edd2e646ab4cf5783ea64232d0cd621debece8d4)
2013-09-26 13:20:00 -04:00
Steve Howell fd4601c44d Add a Desktop tab to /activity.
(imported from commit 7a907e0b904be57613d5ec34c335a962ea41db48)
2013-09-26 13:19:59 -04:00
Steve Howell b8f3b38524 Make /activity client tab order be deterministic.
Use an array of tuples, not a dictionary, for the client tabs.

(imported from commit d0ef09bd7885fbb261227786b90ea2949cea7ab4)
2013-09-26 13:19:59 -04:00
Steve Howell a99d2bebab Assert that registration involves <= 53 queries.
This isn't a very strong test, to be sure, but it's something.

(imported from commit 578dbc6601f38186fec9bfb061a8bbc322201ae7)
2013-09-26 09:55:43 -04:00
Steve Howell 6890c89936 Harden the queries_captured() test decorator.
Before, it was trying to use connection.queries, but Django
could pull the rug out from under us.  Now we monkeypatch
the CursorDebugWrapper methods instead.

(imported from commit 25d5bab47673f2b66a6325f48d33e66c31055ab3)
2013-09-26 09:55:42 -04:00
Tim Abbott 4be5d81af1 [manual] Write logs to /var/log/zulip rather than /var/log/humbug.
This requires a puppet apply to update the supervisor configuration.

(imported from commit f2836b6d9c53791af6f6ceb1650d0e0740df70ab)
2013-09-25 16:52:41 -04:00
Tim Abbott e4496521e9 Update logging path for email-mirror.
(imported from commit 75f36dbb963b2b77ecda7ba138023f0f774047e0)
2013-09-25 16:52:41 -04:00
Tim Abbott 8a5fffa7cf Move twitter API credentials to local_settings.py.
(imported from commit 6b95db113b91816fbc5e91db4b1be90d3df8e028)
2013-09-25 15:40:21 -04:00
Steve Howell 93347be4af Send less presence info to Tornado in do_send_messages().
When we send a message, we send some presence information to Tornado
to help it figure out how to generate emails for idle recipients of
a message.  This change limits the presence info to being the
intersection of present users and recipients of the message.  It is
just an internal optimization to avoid queueing up unneeded data.

The history behind this feature is that I implemented it a while
back, but I think I made a rebase mistake that sent all the presence
data over the wire, despite having code to filter on recipients.
It was mostly harmless, just leading to some inefficiency which is
now fixed.

(imported from commit 7c8e97705afb299c67b99053909e952fbc823551)
2013-09-25 14:50:40 -04:00
Jessica McKellar 0d897c469d Add a utility to report some rough recent client activity metrics.
(imported from commit 27b4a70871939b2728fcbe0ce5824326ff4decc2)
2013-09-25 14:32:53 -04:00
Steve Howell 4d5842cdcd Add summary view to the /activity report.
When you load the activity report, it will just show summary
counts for realms, but if you click on a realm, you will see
details about users in the realms.  You can also click "Show all"
to see an interleaved view of realms and users.

(imported from commit b106557b1fae64d525071afc124b5a8aed319086)
2013-09-25 11:33:11 -04:00
Steve Howell b1ef9a686c Sort rows on the activity report by realm/count.
(imported from commit ed1db1b7685dc113acb8a6a8317b7b84d10c440b)
2013-09-25 11:33:11 -04:00
Steve Howell 95ceba2828 Add realm-wide rows to the activity report.
Add rows to the activity report that roll up counts for all
users on each realm, to go along with individual users.

(imported from commit 8104f3ef7fbe406fe0fd2ba1bb00ce76a1ccbee5)
2013-09-25 11:33:09 -04:00
Steve Howell d087b4e4c9 Extract local vars in ActivityTable.
This is mostly cosmetic, but it does DRY up the code a bit.

(imported from commit cb4babe5a06ca936aca90cdf49b7f72ad5fc6ef3)
2013-09-25 11:33:09 -04:00
Steve Howell d08ae832b9 Make /activity queries slimmer.
Just pull back the 5 fields we use in the query used by
ActivityTable.

(imported from commit f147c82997eca3b06ae53ebf52b110e427ebab06)
2013-09-25 11:33:08 -04:00
Zev Benjamin 871afde142 Prepare process_queue to be used with supervisor's numprocs
(imported from commit 5a652b93f1e8b32b5ed89d622035161abaedfb11)
2013-09-24 20:44:15 -04:00
Zev Benjamin 90ee96d456 Split out presence and activity interval events from user_activity queue
(imported from commit 3c22448491867a9e72659629d5d0faaf12b85b2c)
2013-09-24 20:44:15 -04:00
Zev Benjamin e450fc80c9 Move existing queue processing workers to new framework
(imported from commit b448eeefcddb1840344bd337046be30714a7fdbc)
2013-09-24 20:44:15 -04:00
Zev Benjamin 6cf5ebc63d Add new system for defining and running queue-processing workers
(imported from commit 7e77da57b8ad0b70837785c85f546601ba5b1957)
2013-09-24 20:44:15 -04:00
Zev Benjamin f6500d691e github: Make pull request comments go to the user-specified stream
(imported from commit dfce76bcba43dd4a0beba6b6af7642363193dcde)
2013-09-24 16:32:13 -04:00
Steve Howell 49a55ca6fa Call blueslip.warn from stream_data.js functions.
It used to be calling blueslip.warning(), i.e. the wrong name,
and would crash.

(imported from commit ddc953488cdb22dd37ddf674dddf1e05f34eaec9)
2013-09-24 16:19:53 -04:00
Jessica McKellar f058e9cac5 Add backend support for newer realms receiving stream notifications by default.
(imported from commit ea8ee65071423a94a462571aa30013969d9f3f7a)
2013-09-24 16:19:53 -04:00
Jessica McKellar d434bf4bb4 email mirror: use a message part's specific charset rather than the message charset.
The overall message charset may be null or not match the part's
charset. Even though it's unclear from the documentation,
experimentally using the charset for a message part seems to give you
the charset even for non-multipart emails.

(imported from commit 0e1d23073f4c53041f9760e66a6635f8a94893d1)
2013-09-24 16:19:53 -04:00
Jessica McKellar 7bb75e3acb Populate date_created for existing realms.
(imported from commit 6a10150905a23589f098778ccbd8075025ff900f)
2013-09-24 16:19:52 -04:00
Jessica McKellar f22023d4cb [schema] Add a date_created field to Realm.
(imported from commit 1953f7c1f8af7f81d50d7362c704f6a81637d72e)
2013-09-24 16:19:52 -04:00
Jessica McKellar 4e74131008 Add a first pass at Redmine integration.
The integration supports events generated by creating and editing
issues.

(imported from commit 1988013d7a18df173623c0821e807fd4f42af3bc)
2013-09-24 16:19:52 -04:00
Steve Howell 50641c3337 [perf] Reduce the number of DB queries in send_message.
For a 4-person stream, we were hitting the DB 8 times, and 4 of
those queries were to lazily get user.email for the 4 recipients
due to upstream code using only().  I added user_profile__email
to the only() call.

I believe this regression started 9/18, and after pushing this
to prod, we would should look at this graph:

https://stats1.zulip.net/graphs/8274cd84588

(imported from commit 70629cb69fe5955c674ba76482609dfe78e5faaf)
2013-09-24 16:19:52 -04:00
Steve Howell 63de50d6f0 Add StreamMessagesTest.test_not_too_many_queries
(imported from commit bc49d7a0f932aecb1be7cecd0f6c5a9c95b2767d)
2013-09-24 16:19:52 -04:00
Steve Howell 9ee2f186ca Add queries_captured() context manager for tests.
(imported from commit 14a9529aef057e79bc12c979d88b595828a8b3a3)
2013-09-24 16:19:52 -04:00
Leo Franchi edbde0b78b Handle unknown pivotal events, and log
(imported from commit f177d7d0d8f4c09b2b94bb23973cd90ca8930f3b)
2013-09-24 16:19:52 -04:00
Zev Benjamin 233290d578 Add a management command to purge a rabbitmq queue
This is useful in debugging when you just want to discard all the
messages in a queue because they have the wrong structure.

(imported from commit 8559ac74f11841430b4d0c801d5506ebcb74c3eb)
2013-09-23 11:25:58 -04:00
Zev Benjamin 7d0fd364b2 Remove duplicate call to update_user_activity
(imported from commit e9d2dce02cb6863a3ba66045a7b88bd351fd0c5d)
2013-09-23 11:25:58 -04:00
Zev Benjamin d7d17c993c Fix using SimpleQueueClient with multiple queues
(imported from commit 18bda1430d76f8b436ad42f85d0ce3cab8fd8193)
2013-09-23 11:25:58 -04:00
Steve Howell 660475bd0c Hide muted messages.
Instead of collapsing muted messages, just hide them altogether
in view where it makes sense to hide them.

(imported from commit 1c2c987ff302ceb135a025753cf421b4de1aea71)
2013-09-23 10:53:06 -04:00
Steve Howell 022ef6e9b9 Add narrow.muting_enabled()
(imported from commit 8972d1e3ca347397664094a7f150ec86fd2f8f8e)
2013-09-23 10:53:06 -04:00
Waseem Daher 7afd3ecc81 Custom invite text for MIT invitations.
(imported from commit 21b72861e991159eb7faa74809edb8d01027c6c8)
2013-09-20 12:06:26 -04:00