Commit Graph

3688 Commits

Author SHA1 Message Date
Jessica McKellar 4740bc7e3a digest: Don't show PMs sent by you.
You can't have unread PMs sent by you, so we weren't explicitly
checking this, but when testing locally we often ignore the unread
check. Filter PMs sent by you to reduce confusion when testing
locally.

(imported from commit 0205c4a3ed67790b9d60d4f2b927e4cb9e720bf3)
2013-12-13 16:38:09 -05:00
Jessica McKellar 35c2804c13 Use PM coloring for PMs in missed message emails.
Previously it used a blue header and white background for everything.

(imported from commit 03b8b0f8fb33a581d67a249f0496cc27aab30ef3)
2013-12-13 16:38:09 -05:00
acrefoot 1bd420f12c Show auto-scroll forever setting in /#settings
(imported from commit fa66cbe52836f230e551f25c47bd755b705387d2)
2013-12-13 11:50:11 -05:00
Tim Abbott 5c9def5be4 Add script to parse user agents with historical data set.
(imported from commit e529c0b914ed3d3d06e9581a6239676f68c97b3f)
2013-12-13 11:26:36 -05:00
Tim Abbott 2e62cd5a74 Add tests for the events_register event application system.
(imported from commit 6c3dbe55d43f528b6a980bba3598ac30c0307a69)
2013-12-12 17:36:41 -05:00
Tim Abbott a3d67a7fe7 Fix apply_events for subscription events to actually match.
Since we changed the initial subscription data to include
user_profile_ids rather than emails, we need to preserve that when
adding in events generated during the page load.

(imported from commit 4f4071b8ba30e57c6f64c9e7b54c1cc754e8f010)
2013-12-12 17:36:40 -05:00
Tim Abbott 8af0fae335 Fix missing muted topics in apply_events.
These events would previously not be applied.

(imported from commit c3e5b9de559b6f744567a0c78e920d34e4d5ae9a)
2013-12-12 17:36:40 -05:00
Tim Abbott afdf7897e6 Split out helper functions from do_events_register.
(imported from commit d838a3303d3ec960d076e755fc8690da5151ac1a)
2013-12-12 17:36:40 -05:00
Tim Abbott a3fe72642b Add save/reload of narrows of server restart.
(imported from commit 2ad77845ce7d8fe365c893ab5cad06e3c725b421)
2013-12-12 17:36:39 -05:00
Tim Abbott 71ad82b6aa Add support for event queues with an included narrow.
This will allow us to substantially decrease the server-side work that
we do to support our Mirroring systems (since the personal mirrors can
request only messages that user sent) and also is what we need to
support a single-stream Zulip widget that we embed in webpages.

(imported from commit 055f2e9a523920719815181f8fdb44d3384e4a34)
2013-12-12 17:36:39 -05:00
Tim Abbott 48963c1cdf Replace accepts_event_type with accepts_event.
This is in preparation for allowing queues to only accept some
messages.

(imported from commit 6bca11a9084aeff52931762085c4c771edbce56e)
2013-12-12 17:36:38 -05:00
Tim Abbott b8579c6848 Disconnect handlers if the client closes a connection.
Tested using the following procedure (run-dev.py won't pass through
the client connect closing)

tabbott@monastery:~/zulip$ curl http://localhost:9991/api/v1/register  -u email:key  -d 'event_types=["message"]'
{"msg":"","max_message_id":2259,"last_event_id":-1,"result":"success","queue_id":"1386884005:0"}
tabbott@monastery:~/zulip$ curl -G http://localhost:9993/api/v1/events  -u email:key -d "last_event_id=0" -d "queue_id=1386884005:0"
(then hit ctrl-C)

(imported from commit 3c4f3d5caac97b3de53da994ff9cd9ef67b2b9ea)
2013-12-12 16:52:25 -05:00
Tim Abbott cc0ce6b437 Finish event handlers when disconnecting from an event queue.
This should help prevent timeouts from clients whose requests have
been supplanted.

(imported from commit fdb3a89c4ec02bb23d0fba50ea558d48cb786916)
2013-12-12 16:52:25 -05:00
Tim Abbott 46760e91cd rest_dispatch: Don't allow session auth on /api URLs.
This caused problems with our tests suite where we were using a logged
in browser session and actually acting as a different user.

(imported from commit 73b8cb39d5d669e682fbacf2f7e574c228885c2f)
2013-12-12 10:31:04 -05:00
Tim Abbott 291506a2b2 Add logging for when we disconnected an active handler.
(imported from commit 18c21a93bdf14485e5656bd7a38f2069eb8f2aad)
2013-12-11 17:23:56 -05:00
Tim Abbott fcf6ec16aa Fix pruning of event queues that only had virtual events.
(imported from commit 7b43d2164df6e95a6feb073b8f90c0862245d629)
2013-12-11 17:20:34 -05:00
Tim Abbott b10278c8e0 events: Fix incorrectly empty event queues with virtual events.
I'm not sure this had much impact, but it's definitely a subtle bug.

(imported from commit 0260d4c5c35ce8eac683e84f1d939a6704ce4e2b)
2013-12-11 16:59:48 -05:00
Tim Abbott df730ec163 Fix json_unauthorized.
(imported from commit 7537b3bbc21b947f91ec49060935e9e2f92be0ac)
2013-12-11 16:45:29 -05:00
Kevin Mehall 662edc2558 [manual] Backend support for Android GCM push notifications
This adds a dependency on gcmclient:
http://gcm-client.readthedocs.org/en/latest/gcmclient.html

pip install gcm-client

or

apt-get install python-gcm-client

(imported from commit 9f1fbf1f793e4a27baed85c6f1aa7a7b03106a10)
2013-12-11 15:37:48 -05:00
Kevin Mehall e4589700b6 [schema] Modify device token to support both iOS and Android
This replaces the AppleDeviceToken table with a generic
PushDeviceToken with a `kind` field to make it easier to add functionality
like per-device/per-stream settings that share code between Android and
iOS devices.

The schema must continue to work on prod with the old table name, so we
add the new table in parallel and can drop the old table once this code
hits prod and any necessary data is copied.

(imported from commit 0209a7013f2850ac6311f23c3d6f92c65ffd19e3)
2013-12-11 15:37:47 -05:00
Leo Franchi 2c39e28ff6 Use realm filters from the database in bugdown
(imported from commit 8e12bfb1b1fe9cad694f80a189efa64b0f102e81)
2013-12-11 14:39:10 -05:00
Zev Benjamin 874d4282c0 Disconnect handlers from event queues via a finally block
(imported from commit ac63a962f978b05a5d8d6bf81861b908b5fd54b3)
2013-12-10 17:44:35 -05:00
Zev Benjamin b566c0f201 Catch a wider range of exceptions when adding events to event queues
We have observed additional exceptions being thrown from zulip_finish and we
need to make sure that the handler is disconnected from the queue, or else the
event queue will keep throwing exceptions due to the handler being closed.

(imported from commit 59273aa14495216430b9eb1525b2cce230d8913d)
2013-12-10 17:40:30 -05:00
Zev Benjamin 0d989a6972 socket: Close old connections before registering new ones under the same id
(imported from commit f9b75288cbf7e2d9baa3254d57443eea5c1f3fc3)
2013-12-09 16:17:26 -05:00
acrefoot 33ddba3d30 extra debugging info for handle_push_notification
(imported from commit c4f63827fd37e7e99853380eaa268ab2e5ae23ce)
2013-12-09 11:15:20 -05:00
Zev Benjamin 5e19ad31bf socket: Increase redis response cache expiration time
(imported from commit 1b18530f01499af9a555c71f0281e62bd3164542)
2013-12-08 12:13:15 -05:00
Zev Benjamin 96bb739157 socket: Add some server-side logging for bug catching
(imported from commit 8579d8bbb414b063ae9cf36993975adaa86cb132)
2013-12-06 16:22:24 -05:00
Zev Benjamin 8535669d46 socket: Include the user in socket close log lines
(imported from commit fb5b9e54a07b6bfd85e913b8c97105305f8210eb)
2013-12-06 16:22:24 -05:00
Leo Franchi f55067a3db Actually use a UserMessage object instead of Message
(imported from commit 9613af82f62a9ea85a85f1ce662234f410457997)
2013-12-06 15:53:42 -05:00
Leo Franchi 4f2cb1bc1c Read read flags from UserMessage not Message
(imported from commit 4c0f6ec90992c61ba861f5f347655673e769bebc)
2013-12-06 15:08:02 -05:00
Zev Benjamin 657a750550 socket: Fix spelling
We don't ever actually check for this value so nothing else has to
change.

(imported from commit e6ccb153df7bd95c9d195aa83395592b309a2778)
2013-12-05 15:53:50 -05:00
Zev Benjamin 002df64811 Remove extraneous semicolons
(imported from commit 2077705c52e7255ab7ea1b4df71d2ae99f6018dc)
2013-12-05 15:53:50 -05:00
Tim Abbott fed3902fc5 Clean up debugging logs from tracking down slow receive.
(imported from commit e4b1f763522406255ce8be3c8d51b5d855a770cd)
2013-12-05 14:36:29 -05:00
Tim Abbott 55047280c9 Include a newline at the end of our JSON responses.
Otherwise, it's super annoying to work with these using curl.

(imported from commit 0047f07786dc07d7ac5cb671492f8ca40986d8cf)
2013-12-04 13:47:48 -05:00
Leo Franchi 1b53a77f1b Expect an ios client instead of iphone
(imported from commit d5d3cd129d7005fd9294096d3899df86155cd366)
2013-12-03 14:27:51 -05:00
Tim Abbott abded5f886 Upgrade event flag collapsing to support all flag changes.
(imported from commit a2ca74b779d67148aee41e92e4a4cb716620e566)
2013-12-03 12:21:04 -05:00
Tim Abbott d2486670a8 event: Add deduplication of server restart events.
(imported from commit e464aa8b68958fb9f9fe8c199b9e73df510d22bc)
2013-12-03 12:20:36 -05:00
Tim Abbott 54b06d046b events: Add support for not storing redundant events.
This should dramatically improve the speed of the dump/load part of
our restart process, especially with large long-lived event queues.

(imported from commit ae4ae20ba2ca4433e25a5e7beeb4fa4882c53972)
2013-12-03 12:19:42 -05:00
Tim Abbott ad9659ea54 events: Fix zulip_finish to not make assumptions about how queues work.
Previously, we had an issue with the ACKing protocol, where if a
virtualizable event (like a "read" flag) was dispatched to a queue
client immediately, we would not properly ACK the change because it
had been made a virtual event.

(imported from commit ea09812f8a5ba1d5aad65f536022e3dbc77b0f9e)
2013-12-03 12:19:42 -05:00
Jessica McKellar e15cc972ec digest: refactor sending the emails into a function for easier testing.
(imported from commit 2f3c8fb8e812c04ab838b1034ae0811309186625)
2013-12-02 18:58:03 -05:00
Jessica McKellar 207cfc449d Add unsubscribe links to digest e-mails.
(imported from commit 4e2a324decf4bc694752cc24b9085361338a08a5)
2013-12-02 18:58:03 -05:00
acrefoot da1466cfa9 handle case of message getting read while processing missed_message
(imported from commit cd53efe3645f5dc8943becc2b8dbd3d607bb8eba)
2013-12-02 16:00:56 -05:00
Tim Abbott 559f0896a1 Revert "events: Add support for not storing redundant events."
This reverts commit cc493fa50b4c339257e060b3f0c0956c682e449d.

(imported from commit 9e8edfc11d415d61581da482d66bc9fa30cbefdf)
2013-12-02 11:03:04 -05:00
Tim Abbott 5b601d8492 Revert "event: Add deduplication of server restart events."
This reverts commit 05467929ba376a0ce173119c5d90d83671fc544b.

(imported from commit 4f24a0f1fe72e71fa368e1d818d9406a76c131be)
2013-12-02 11:03:04 -05:00
Tim Abbott 2c2f404cb6 Revert "Upgrade event flag collapsing to support all flag changes."
This reverts commit dadf3852f2ca7aaa85fb97b3d265c990b4e5a38a.

(imported from commit de5a95572aa955b4a9419b80a99d2446c51730e7)
2013-12-02 11:03:04 -05:00
Tim Abbott 10bce7ffe8 Revert "Fix loading event queues that don't have a virtual_events attribute"
This reverts commit 99c5ae332d253313ed728883a388c61191ed3021.

(imported from commit d7b601a2897bac6e14b675e689ee63ccc6e8d67d)
2013-12-02 11:03:03 -05:00
Tim Abbott 5bc826067e Revert "Add comment in ClientDescriptor.to_dict and EventQueue.to_dict to avoid future bugs"
This reverts commit 45560539d09e9e1922130349be729005b786e632.

(imported from commit aa266941f03cdbe12f4315e65ec6c472cc207204)
2013-12-02 11:03:03 -05:00
acrefoot 717e06d94c remove log message on push notification
(imported from commit 45fc89e67a79aac3484c911d88aff3e470b4eff6)
2013-11-27 18:00:33 -05:00
acrefoot 8800dcdce4 more helpful error message when missing APNS certs
(imported from commit e21dc627844b9e28f93a924097176481d4135947)
2013-11-27 18:00:33 -05:00
acrefoot 8ee7536012 Send push notifications more aggresively
Unbundle the push notifications from the missed message queue processors
and handlers. This makes notifications more immediate, and sets things up
for better badge count handling, and possibly per-stream filtering.

(imported from commit 11840301751b0bbcb3a99848ff9868d9023b665b)
2013-11-27 18:00:31 -05:00
Zev Benjamin 9d0f801ad6 Exempt a particular client name from heartbeats for testing
(imported from commit 3914130384a7a9854b006405ffd2fe379bbd090d)
2013-11-27 10:39:31 -05:00
Jessica McKellar 2ba9415178 Factor welcome e-mail enqueuing into its own function.
(imported from commit 85054ee3e567d2ba7a4ed94c52a311d02f5e4d43)
2013-11-27 09:30:00 -05:00
Zev Benjamin 8899c62567 Add comment in ClientDescriptor.to_dict and EventQueue.to_dict to avoid future bugs
(imported from commit 45560539d09e9e1922130349be729005b786e632)
2013-11-26 18:39:31 -05:00
Zev Benjamin ce00fe0cd5 Fix loading event queues that don't have a virtual_events attribute
(imported from commit 99c5ae332d253313ed728883a388c61191ed3021)
2013-11-26 18:32:58 -05:00
Tim Abbott e9f4612281 Upgrade event flag collapsing to support all flag changes.
(imported from commit dadf3852f2ca7aaa85fb97b3d265c990b4e5a38a)
2013-11-26 18:12:52 -05:00
Tim Abbott fb5c42707e event: Add deduplication of server restart events.
(imported from commit 05467929ba376a0ce173119c5d90d83671fc544b)
2013-11-26 18:12:52 -05:00
Tim Abbott 1db8fba5aa events: Add support for not storing redundant events.
This should dramatically improve the speed of the dump/load part of
our restart process, especially with large long-lived event queues.

(imported from commit cc493fa50b4c339257e060b3f0c0956c682e449d)
2013-11-26 18:12:52 -05:00
Leo Franchi 501f063426 Replace email_to_domain with split_email_to_domain and resolve_email_to_domain
Now that we support email aliases, we have to be careful when going from
an email address to a domain that we assume we can use to get a Realm
object for. When we care about the Realm's domain, we want to follow
any RealmAliases that exist for a certain domain.

When we just care about the original email address domain itself,
for comparison or other purposes, use split_email_from_domain

This removes the ambiguity of having to decide when to use
email_to_domain + RealmAlias or just email_to_domain

(imported from commit 0e199495502d946ce2e1aae56263e7e8665be4ed)
2013-11-26 10:45:01 -05:00
Kevin Mehall b8a6100d2e bugdown: Open #-relative links in the same window.
Stream notifications shouldn't leave you with tons of Zulip windows.

(imported from commit e5ebfeff51f1e579d30a3a12eaf7341ef82f0449)
2013-11-25 17:34:59 -05:00
Jessica McKellar dbc1b4b501 Add a one-click unsubscribe link to missed message e-mails.
(imported from commit 719eb61c9b7722eb1b0996b4d5f2f8f5929ad701)
2013-11-25 15:18:40 -05:00
Tim Abbott 7cb261a0ed Set default streams to engineering,social when creating realms.
(imported from commit 4e174fec162be4138da099edfc4cfb8d5662b7f1)
2013-11-25 14:00:28 -05:00
Zev Benjamin bc43981ccf socket: Explain why we disable some transports
(imported from commit 670d2f558a11a6f4fcce3f2e107582c4425b9285)
2013-11-25 11:58:02 -05:00
acrefoot 4cd1d69ea0 Add the ability to unsubscribe all users from a stream
This resolves Trac #2022

(imported from commit 23d5129a6bec40c33eaf71953cd41ec387646a39)
2013-11-22 17:12:51 -05:00
Jessica McKellar d0a49e3a80 Replace decentralized topic truncation with a single check when sending messages.
(imported from commit 291c999147dd59a9c7e1d9de446293755c17e99d)
2013-11-22 13:53:03 -05:00
Jessica McKellar baef4b9b00 Truncate long messages instead of rejecting them.
(imported from commit 41e0ce1e91ab7a441f2b21ad445344a86ae18fc7)
2013-11-22 13:53:03 -05:00
Steve Howell ace12bcab6 Support arbitrarily nested fenced quote/code blocks.
Now we can nest fenced code/quote blocks inside of quote
blocks down to arbitrary depths.  Code blocks are always leafs.
Fenced blocks start with at least three tildes or backticks,
and the clump of punctuation then becomes the terminator for
the block.  If the user ends their message without terminators,
all blocks are automatically closed.

When inside a quote block, you can start another fenced block
with any header that doesn't match the end-string of the outer
block.  (If you don't want to specify a language, then you
can change the number of backticks/tildes to avoid amiguity.)

Most of the heavy lifting happens in FencedBlockPreprocessor.run().
The parser works by pushing handlers on to a stack and popping
them off when the ends of blocks are encountered. Parents communicate
with their children by passing in a simple Python list of strings
for the child to append to.  Handlers also maintain their own
lists for their own content, and when their done() method is called,
they render their data as needed.

The handlers are objects returned by functions, and the handler
functions close on variables push, pop, and processor.  The closure
style here makes the handlers pretty tightly coupled to the outer
run() method.  If we wanted to move to a class-based style, the
tradeoff would be that the class instances would have to marshall
push/pop/processor etc., but we could test the components more
easily in isolation.

Dealing with blank lines is very fiddly inside of bugdown.

The new functionality here is captured in the test
BugdownTest.test_complexly_nested_quote().

(imported from commit 53886c8de74bdf2bbd3cef8be9de25f05bddb93c)
2013-11-21 17:13:17 -05:00
Tim Abbott 272e1c2d92 Revert "Don't send heartbeat events to clients that don't request them."
This reverts commit 1147814b22fb9737a807057ddbdbe0e9554086e0.

This seems to with some probability screw up our Zephyr mirroring
script.

(imported from commit 4f82452f1b0ca98e6b895db020e071d2daa325e4)
2013-11-21 12:53:26 -05:00
Tim Abbott ca8225cf47 [manual] Add endpoint to cleanup a finished events queue.
This requires a puppet apply on each of staging and prod0 to update
the nginx configuration to support the new URL when it is deployed.

(imported from commit a35a71a563fd1daca0d3ea4ec6874c5719a8564f)
2013-11-20 18:34:15 -05:00
acrefoot 5973f20b69 fix use of EXTERNAL_URL to EXTERNAL_HOST, and some typos
(imported from commit 3e6e0f337671b289202bc86c6224854b6ee623c5)
2013-11-20 17:39:07 -05:00
Steve Howell 0d920dae87 Extract FencedBlockPreprocessor.placeholder().
(imported from commit cf6dab74ae97a57498e1e1807c7c4f272f3f448b)
2013-11-20 16:06:53 -05:00
Steve Howell 2955cac47e Extract vars in FencedBlockPreprocessor.process_fence().
(imported from commit 092a8e5f7479f1d918219cae058b6a867743c7f9)
2013-11-20 14:30:56 -05:00
Steve Howell b3344c2f49 Move langclass var into FencedBlockPreprocessor.format_code().
(imported from commit e468a8b6a12494facb0dfa97eff90e02cd14f169)
2013-11-20 14:30:55 -05:00
Steve Howell 46838ec6d0 Simplify FencedBlockPreprocessor.format_fence().
(imported from commit 741f4f0640756d545019b6fe6850af3198b351ba)
2013-11-20 14:30:55 -05:00
Steve Howell c0f312ec29 DRY up m.group('code') in FencedBlockPreprocessor.
(imported from commit b23caae1ab63a9adb80025b628bc1a7c6a988ce0)
2013-11-20 14:30:55 -05:00
Steve Howell 6bc25d3d7b DRY up m.group('lang') in FencedBlockPreprocessor.
(imported from commit 9f9ab5604b7df711a3a369247def1277880b0b5f)
2013-11-20 14:30:55 -05:00
Steve Howell 7bb30a96ac Extract FencedBlockPreprocessor.format_fence().
(imported from commit 8a35de7df9ad956496263c82668b118910edd6b3)
2013-11-20 14:30:55 -05:00
Steve Howell 64410fae5e Extract FencedBlockPreprocessor.format_quote().
(imported from commit 59db5415d624402b37ade7fe5547b16f58ae565f)
2013-11-20 14:30:55 -05:00
Steve Howell 9d4b6d0684 Extract FencedBlockPreprocessor.format_code().
(imported from commit 3c98b9a7391a71e9c55a7b5c97abd0ee74178b44)
2013-11-20 14:30:54 -05:00
Zev Benjamin 25c63f08de Use ujson instead of pickle for serializing event queues
(imported from commit 2d90574ae97cc973c9686c2bd557cc199d4b0850)
2013-11-20 14:25:36 -05:00
Zev Benjamin 69e5e2c3d2 Remove old migration code
(imported from commit 5366411706ef296dc14d0d0dd9e2566cb8abb8b7)
2013-11-20 14:25:36 -05:00
Zev Benjamin 06ee83b531 Construct EventQueues externally from ClientDescriptors
(imported from commit 7ef882074ed1c3663939da199d54e4959df8f127)
2013-11-20 14:25:36 -05:00
Zev Benjamin db29e60249 Move the last event queue pickle file to /var/tmp instead of deleting it
(imported from commit b8edaa686e2dadfdb2111ee1da93456591e44a50)
2013-11-19 15:05:19 -05:00
Jessica McKellar 151fbd832c do_update_message: use is_admin to check who can edit a topic.
(imported from commit e96245506c7eecc8aed3fbdb6dfb7c44aca3234e)
2013-11-19 14:31:02 -05:00
Jessica McKellar 2c841042ae Allow admins to rename any topic.
(imported from commit 5b557e8244e5196c9aa0ee29d4efb00986434528)
2013-11-19 14:31:02 -05:00
Zev Benjamin 15a837d12e [manual] socket: Send explicit ACKs
There will be browser errors on staging when this is deployed due to the socket
protocol changing.

(imported from commit f1eda5b5c2ec9c60c23b3ca96277a61debadf5bb)
2013-11-19 11:06:14 -05:00
Zev Benjamin 7870b4d158 socket: Fix a traceback on server restart
I believe there may also be others.  I'm still not sure why clients would be
sending open requests without session or csrf values in their cookies, though.

(imported from commit 7e9660c1c4d5c2abf55ff21b433ba0117180eb82)
2013-11-19 11:06:13 -05:00
Tim Abbott 74c304ae40 Don't send heartbeat events to clients that don't request them.
(imported from commit 1147814b22fb9737a807057ddbdbe0e9554086e0)
2013-11-18 13:57:01 -05:00
Tim Abbott 14f6625569 Don't send server restart events to clients that don't request them.
(imported from commit b3381e904704c18918fcce97d2493241e49004be)
2013-11-18 13:57:01 -05:00
Kevin Mehall b78fa0857f Allow overriding the default avatar image
(imported from commit 66d413682a822e0019f28033f19908bdd9fa0156)
2013-11-18 11:48:53 -05:00
Kevin Mehall a63f4284f0 Allow enterprise deployments to disable the use of Gravatar.
(imported from commit 49c14cc7629f2ffe6cedaab5d42ad5bf7f6569a4)
2013-11-18 11:48:53 -05:00
Zev Benjamin 5c6a85f99a socket: Change path for 'not yet authenticated' error messages
(imported from commit 826d3a0ea9084644810a00a14f2f19936e21118c)
2013-11-18 11:42:57 -05:00
Zev Benjamin 319852c60e socket: Fix keyword argument name for error logging
(imported from commit e61eace1aa48fe17d5d2338f7990e32d3404d390)
2013-11-18 11:42:57 -05:00
Tim Abbott f3fa78d30c Rename do_[rd]activate to do_[rd]activate_user.
For similarity with do_activate_user.

(imported from commit 2874af84dbe1ce666de10e0d8ffe2daa8bad9731)
2013-11-16 11:35:26 -05:00
Steve Howell 1144e0b3f9 Update active_user_dicts_in_realm cache when touching is_active flag.
(imported from commit c9407e2c18dcfcc7a4b1ec79c471f825dff04b1e)
2013-11-16 11:35:25 -05:00
Steve Howell bdfd3ed9c2 Add do_reactivate() action.
(imported from commit fc73d0fdd3a6125b3b31a75c3b108ceb266e528b)
2013-11-16 11:35:25 -05:00
acrefoot daffe049ad Don't use hardcoded noreply@zulip.com, zulip@zulip.com, or https://zulip.com
(imported from commit 1132553b63ae23ebcca746f0f65205b97bfee7dc)
2013-11-15 21:31:37 -05:00
Kevin Mehall fe0dcd4313 Disable camo on enterprise.
CUSTOMER13 doesn't want it, and there's currently no nginx config
or configurable Camo URI, so it wouldn't work if image preview
were enabled.

(imported from commit 615d4a32acbc4d4d590f88cf4e7d45d8f49db1d3)
2013-11-15 14:27:16 -05:00
Jessica McKellar b0209fec05 Don't puke on non-ASCII characters when shelling out to html2text.
(imported from commit a407c9fb2a090075d5e26b5db00388f4f81de1f5)
2013-11-15 12:28:30 -05:00
acrefoot f991fbff9e Fix clear_followup_emails_queue on Zulip Enterprise.
(imported from commit bf211b205bc39a08340306d06fbb76479a02ed9e)
2013-11-14 11:32:29 -05:00
Kevin Mehall b469066e66 Fix emoji on enterprise.
static/ doesn't exist on enterprise, so we can't get the list from the
files there.

(imported from commit ce34a62478abf541feb013da4f970dac3c09d98a)
2013-11-14 11:18:23 -05:00
Tim Abbott 870ae09aae Add setting for disabling inline image preview.
(imported from commit 2321390eb14cfe5701347861b9ae5ad6fdac0cbb)
2013-11-14 08:47:11 -05:00
acrefoot b6751d2119 additional lowercase situations in clear_followup_emails
(imported from commit c0c031e7abb2a166636075596fb0d5ed21a5822c)
2013-11-13 17:53:37 -05:00
Jessica McKellar 338de6715d Fix broken html to markdown conversion on Linux.
(imported from commit 1b62d382e4f2b60972384a6b4b4173161ea58b73)
2013-11-13 17:47:28 -05:00
acrefoot f7b5a10da0 [schema] Add ScheduledJob table, and update mandrill related code
ScheduledJobs with type Email displace the usual mandrill codepaths
in the Zulip Enterprise deploys

* Email-specific helper functions will appear in deliver_email.py
* 0058_auto__add_scheduledjob.py

(imported from commit 8db08d8a279600322acfdbed792dc1a676f7a0ab)
2013-11-13 16:41:36 -05:00
Jessica McKellar 8382e074fe Move use of html2text to a subprocess call.
(imported from commit 36e8a6f030d75196c28fbdc0e58c6968952d95ff)
2013-11-13 16:28:29 -05:00
Jessica McKellar 5c0a624f9e Don't notify on stream messages by default for LevelUp.
(imported from commit c0c21047698612dbcee5eee9e858f37a8f8e29bd)
2013-11-13 16:14:36 -05:00
Leo Franchi 62ba16a829 [schema] Allow a realm to have aliases for multiple other domains
(imported from commit 5d0340afd4c77f50db63550a3ec8d81e885c285c)
2013-11-13 14:17:43 -05:00
Steve Howell 761c0a0266 Support !avatar syntax in bugdown.
(imported from commit 72551c5df96f5b383d3593c9618cc82413d5d95d)
2013-11-13 13:45:09 -05:00
Steve Howell 2086ee8979 Rename Gravatar class to Avatar in bugdown.
(imported from commit 97f1fb52f2582694f5ba43c24649fc58152d3486)
2013-11-13 13:45:09 -05:00
Steve Howell 2d5a19ad1b Have !gravatar markdown serve up /avatar/<email>.
The !gravatar markdown no longer hard codes to Gravatar, but
instead it serves up our generic avatar URL.

(imported from commit 4e3e2baeb3374bcf025a18ff27a8452b975c22b7)
2013-11-13 13:45:08 -05:00
Tim Abbott 1eca4027fa Support filling memcached caches with no messages.
(imported from commit 3dfcdce60f84bfa99820a52b74c637ad849930a8)
2013-11-13 12:07:16 -05:00
Leo Franchi 6274c33f86 Allow disabling of feedback system in UI
(imported from commit ce16b2916784e9cf10a230cb47515ffc65888778)
2013-11-13 11:18:02 -05:00
Leo Franchi 306ce65ea3 Only create initial passwords for local dev setups
(imported from commit 2ef33ebbab0fe21486acbb1a3a78ed434abac2db)
2013-11-12 22:42:05 -05:00
Zev Benjamin cf8cf1fe03 socket: Fix non-rabbitmq codepath
(imported from commit 3ccdcbfb3283fc57ca502878e9c8d6068e7bce40)
2013-11-12 16:42:15 -05:00
Tim Abbott 5d5e662c08 Disable event logs on local server.
(imported from commit eafbd645ae7ce8d868de2fb8c6548325a05b1340)
2013-11-12 15:57:01 -05:00
Zev Benjamin 673f497a6f socket: Only deregister connections that have been registered
(imported from commit bfaed299a129bac1e901883f5c6e3339f5a8b688)
2013-11-12 15:24:30 -05:00
Zev Benjamin eec195f421 socket: Use our full logging infrastructure
We also now separate out the times for the socket overhead, the
request service time, and the queuing delays.

(imported from commit e1683f7f28b968b86ebb701b0ac29b00ac6d67c3)
2013-11-12 15:24:30 -05:00
Zev Benjamin 53ec292022 Store logging data in a dict instead of individual attributes
(imported from commit f7d76670428d5d8298c572a23cbfffc1d9695f23)
2013-11-12 15:24:30 -05:00
Zev Benjamin 24ee4a5d57 socket: Build a real Request object to send through our full stack, including middleware
One quirk here is that the Request object is built in the
message_sender worker, not Tornado.  This means that the request time
only counts time taken for the actual sending and does not account
for socket overhead.  For this reason, I've left the fake logging in
for now so we can compare the two times.

(imported from commit b0c60a3017527a328cadf11ba68166e59cf23ddf)
2013-11-12 15:24:29 -05:00
Leo Franchi cc2d17d7c0 Allow omission of EMAIL_GATEWAY_BOT and stream emails
(imported from commit 51412d2a46bbacd6537ef892e7b6bdf33ac94079)
2013-11-12 10:32:34 -05:00
Tim Abbott b0dc882f7e localserver: Add support for not having an APNS_CERT_FILE.
(imported from commit 42d8caebb3035bd9e6e19db6359cc33e8ab4f317)
2013-11-12 09:34:25 -05:00
Tim Abbott ba15333df9 Add temporary debug logging around Tornado handler protocol.
This logging is kinda excessive since it adds like 4 log lines per
recipient, so I expect we'll end up reverting it once we've debugged
the proximal issue.

(imported from commit 5e6ab3e230f32b65ad9cf0d95f20ffbc0fe7397e)
2013-11-07 15:44:19 -05:00
Tim Abbott 3b41e79074 get_events: Log queue ID in home and get_events requests.
(imported from commit 8be1427c6c9ec3d1eb4a02f3c6ad5c2726542596)
2013-11-07 15:44:19 -05:00
Zev Benjamin 24ae5998e5 socket: Request the status of pending requests on reconnect
This will hopefully help with the send dialog being stuck on
"sending" as well as allowing us to not show errors to the user on
reconnect.

(imported from commit 31ee889853f348e486863073dc130cdfb4e1338d)
2013-11-07 11:43:34 -05:00
Zev Benjamin 48a25211fa socket: Record the request status in redis
(imported from commit 6c3f6dc37d1abdcacf105f865ac24483717438ca)
2013-11-07 11:43:34 -05:00
Zev Benjamin 87fc182645 socket: Move `req_id` parameter to top level of protocol payload
(imported from commit 271d84a0b44a2623bec290f64a83a0423698a1ef)
2013-11-07 11:43:34 -05:00
Zev Benjamin 4595143023 socket: Switch to client ids from connection ids
Clients can only have one connection at a time, anyway, so we can
just keep track of a client id, instead.  This makes reconnections
easier.

It's a little funny to use queue ids for the client id, but we know
they should exist by the time the client is connecting and they are
guaranteed to already be unique and authenticatable.  We will also
eventually be integrating the event system and the socket code closer
anyway.

(imported from commit 1f60e06fb16d31d6c121deafd493fb304d19a6c2)
2013-11-07 11:43:34 -05:00
Zev Benjamin 63924ea742 socket: Reduce code duplication in authentication error cases
(imported from commit e7900b1374c3c0e5ea994ff315d281833e7ecdc6)
2013-11-07 11:37:00 -05:00
Zev Benjamin e0fece4082 socket: Log the transport type on the authentication line
(imported from commit 7b3ae3ab38da35cbd60aca991665d40b41a7f78f)
2013-11-07 11:31:06 -05:00
Kevin Mehall 57aabc112d Add debug for a possible way get_events might forget a request
(imported from commit ca3ac5b942ffdcc8f175cc6b690cb2555e170836)
2013-11-06 18:34:50 -05:00
Jessica McKellar 74d1a56c68 Do not notify on stream messages by default for CUSTOMER19.
(imported from commit 7f737b9068ec39fc53678aa5c0976d3da908ce00)
2013-11-06 10:40:28 -05:00
Tim Abbott e06722657a [manual] Remove /messages/latest API and related legacy code.
This requires doing a puppet apply on our servers to take effect
properly.

(imported from commit 19dc56f071f07a5d2571eef49dd835121b2e82b6)
2013-11-05 14:19:40 -05:00
Tim Abbott 87271b57cd Fix feedback check to correctly use the recipients list.
Even for a PM, the "recipients" list is 2 people -- the sender and the
receiver.

(imported from commit 5503e0ee2abb643601a0a4e61366127914808245)
2013-11-04 12:58:40 -05:00
Tim Abbott 68dcc760c3 Clean up some unused imports.
(imported from commit 0c5d8e2a55ba1b8909ba807fee3afe863dcdc226)
2013-11-04 11:51:17 -05:00
Tim Abbott 29aef33c53 Remove transaction management hacks in workers.
(imported from commit 33b9349bd139907755b52474df8a0da13b73b80d)
2013-11-04 07:31:39 -05:00
Leo Franchi db6550e99a Make built-in bots come from settings.py, and allow localserver-specific ones
(imported from commit e21933e37487314ac986147562817a19227e8960)
2013-11-01 14:13:05 -04:00
Kevin Mehall a09c7e1e0c Make rest_dispatch errors cleaner for requests from browsers.
If you don't have a cookie or basic auth and the request looks like
a top-level page in the browser, redirect to the login page.

(imported from commit fc1bcb1080591522bd1b694664255f7049a5d443)
2013-10-31 14:36:34 -04:00
Steve Howell 70c1c31b3a Simplify callback protocol for register_json_consumer().
The register_json_consumer() function now expects its callback
function to accept a single argument, which is the payload, as
none of the callbacks cared about channel, method, and properties.

This change breaks down as follows:

    * A couple test stubs and subclasses were simplified.
    * All the consume() and consume_wrapper() functions in
      queue_processors.py were simplified.
    * Two callbacks via runtornado.py were simplified.  One
      of the callbacks was socket.respond_send_message, which
      had an additional caller, i.e. not register_json_consumer()
      calling back to it, and the caller was simplified not
      to pass None for the three removed arguments.

(imported from commit 792316e20be619458dd5036745233f37e6ffcf43)
2013-10-31 13:18:11 -04:00
Jessica McKellar f1e90086f5 Remove the onboarding checklist.
Looking at the historical data, fewer than 50% of active users have
completed the checklist, which means that it is just persistent
clutter. We also have other better ways of encouraging people to send
traffic and get the apps now.

This commit removes both the frontend UI and backend work but leaves
the db row for now for the historical data.

(imported from commit e8f5780be37bbc75f794fb118e4dd41d8811f2bf)
2013-10-31 11:34:15 -04:00
Waseem Daher db1f5065a7 Add tooltip for, e.g. !gravatar(wdaher@zulip.com) output.
(imported from commit eff24a2fc7da77edc26449074acf75832d368a17)
2013-10-30 13:29:37 -04:00
Leo Franchi 9be1333c6b Log send_message times through our new socket infrastructure as well
(imported from commit 215be8f7e3d2498ed34e9fcc421498a7cfca4fbc)
2013-10-29 17:49:27 -04:00
Tim Abbott 1292c5dbfe Add backend support for Jabber mirroring.
(imported from commit 12f5c708a6961aded4f4e166af84e87720be4ddc)
2013-10-29 16:49:42 -04:00
Kevin Mehall 06590f4332 Upload to local filesystem on local server.
(imported from commit 6c38a8cae721b2e7a0863470692cc56425006ecb)
2013-10-29 16:19:04 -04:00
Kevin Mehall f9af7ad1cf Resize uploaded avatar images
(imported from commit 47ff86fcd47871916631baa34f99bbddb67f87d0)
2013-10-29 11:42:33 -04:00
Tim Abbott 901a5c342c Use os.path directly rather than sometimes importing it separately.
(imported from commit 48486c4ea64d02a15faeebb0f490d31e9b113d62)
2013-10-28 15:44:13 -04:00
Zev Benjamin 807116e955 Delay connecting to RabbitMQ until it's necessary
Previously we were connecting at import-time.

(imported from commit 9a638f0d238f3b6b00feb4aa524098a64953cb92)
2013-10-28 15:27:25 -04:00
Leo Franchi e1557bef73 Cache realm user basic info dict that is used in bugdown
(imported from commit 688c47daceb73534be90bd98a031c7b4edb5546e)
2013-10-28 12:59:36 -04:00
Leo Franchi b26f158020 Use user_profile instead of leaked object from list comprehension
(imported from commit 7d8467b104fc80d52fa829eb0ff949b1bd0e578f)
2013-10-28 12:59:36 -04:00
Jessica McKellar 2be6c541cf digest emails: use a template filter to format new users, streams, etc.
(imported from commit e61a3dd6753db82da1b30c50e5bbf6273f302645)
2013-10-25 19:11:41 -04:00
Jessica McKellar d4b60e9304 Add a custom template filter for formatting displaying lists of items in emails.
We do this operation several times in the digest e-mails.

(imported from commit 7b906bad582c628d4461637dc7af374df349e09a)
2013-10-25 19:11:40 -04:00
Jessica McKellar 772adc1969 digest: Add functions used to compute interesting traffic.
So far, we gather: "hot conversations", missed PMs, new streams, and
new users.

(imported from commit c3c723d0426cb55bd0e43917c67f93db9052f9ed)
2013-10-25 19:08:46 -04:00
Leo Franchi c12a97577e Rework local_settings_template.py to be clear about what the user needs to do
(imported from commit f69ea0d57e885509abc70dc4f2bee6e36169691f)
2013-10-25 16:35:27 -04:00
Luke Faraone e552307511 Send feedback to a queue to be forwarded to staging.
(imported from commit 4a9a1bfc6c95763a816263a726cc61b3ca90bf15)
2013-10-25 14:13:30 -04:00
Luke Faraone 1fa65915f2 NACK on consumer failure
(imported from commit 1cad1096dd1a969b2551822575784c338a82f433)
2013-10-25 14:13:30 -04:00
Luke Faraone 9d4ef27b97 Make rest_dispatch accept a list of globals and move it into its own file.
We're going to be using this from other views, so make it more generic.

(imported from commit c2c4f64b2a212f8a254f75c21dc93d8fc3f8351a)
2013-10-25 14:13:30 -04:00
Luke Faraone 81d7dd1fda [schema] Support for authenticating Deployments via the API.
Here we introduce a new Django app, zilencer. The intent is to not have
this app enabled on LOCALSERVER instances, and for it to grow to include
all the functionality we want to have in our central server that isn't
relevant for local deployments.

Currently we have to modify functions in zerver/* to match; in the
future, it would be cool to have the relevant shared code broken out
into a separate library.

This commit inclues both the migration to create the models as well as a
data migration that (for non-LOCALSERVER) creates a single default
Deployment for zulip.com.

To apply this migration to your system, run:
   ./manage.py migrate zilencer

(imported from commit 86d5497ac120e03fa7f298a9cc08b192d5939b43)
2013-10-25 14:13:30 -04:00
Luke Faraone 52309b5789 Enable absolute imports for non-third-party code where it was omitted previously
(imported from commit 34856ac32c6b94b614273a6fe46c87a314058d9b)
2013-10-25 14:13:30 -04:00
Leo Franchi 2d276179d0 Refactor email-mirror to handle running on any machine
(imported from commit 2971449ceaacb564770e66874fc095f77e68d445)
2013-10-25 11:37:34 -04:00
Kevin Mehall 19b9b6e2ec Reduce the length of public upload URLs.
(imported from commit dc4d51da39b218068012ebad57e1f92cb50137c3)
2013-10-24 17:01:06 -04:00
Kevin Mehall 0a271f149b Enable authed file uploads for squarespace
(imported from commit c72e76d27f99cd7610eb6c93b57b8ff727b461a9)
2013-10-24 17:01:06 -04:00
Kevin Mehall 5e1caad88a Show upload filename in tooltip without other ugly parts of the URL.
This works for both public and authenticated uploads.

(imported from commit f9be340b53d0f72bf5b3c31a5d51a1fad80759be)
2013-10-24 17:01:06 -04:00
Kevin Mehall 0a3a22cb3d Support authenticated upload URLs.
Trac #1734

This is implemented by bouncing uploaded file links through a view
that checks authentication and redirects to an expiring S3 URL.

This makes file uploads return a domain-relative URI. The client converts
this to an absolute URI when it's in the composebox, then back to relative
when it's submitted to the server.

We need the relative URI because the same message may be viewed across
{staging,www,zephyr}.zulip.com, which have different cookies.

(imported from commit 33acb2abaa3002325f389d5198fb20ee1b30f5fa)
2013-10-24 17:01:06 -04:00
Kevin Mehall 473a98d0a4 Split out common code for existing and authenticated upload systems.
(imported from commit bf513eaff90ae7c4a0be825317690058f9f54621)
2013-10-24 17:01:06 -04:00
Kevin Mehall fb4eae9e2b Include realm ID in S3 metadata on uploaded files.
(imported from commit 7d41ec94991bd257485116791a0d1ef2c22d9b37)
2013-10-24 17:01:05 -04:00
Kevin Mehall e3b2e7ece2 Allow domain-relative and fragment links
(imported from commit 0a15eeb35ad709977f3b2e6c550c3b78f1b4b402)
2013-10-24 17:01:05 -04:00
Zev Benjamin ec21275b19 Move rabbitmq username into settings.py
(imported from commit f9a8f80f53b8774781b8558444e601567cb59835)
2013-10-24 16:40:38 -04:00
Leo Franchi ae104ace25 Only load APNS cert if it exists
(imported from commit 8505c9a32feb4c7470dfde6c4ceaaa0a8e8f33a1)
2013-10-24 15:55:42 -04:00
Leo Franchi 410ee44eb6 Send users push notifications when they miss messages
(imported from commit 6c54fe44a82c5796268e56d3f5577bf4cfc8163a)
2013-10-24 14:54:31 -04:00
Leo Franchi 05564ec0e2 [schema] Add a config option for enabling/disabling push notifications
(imported from commit 5649188cd5e0e4edb6e71559ec7d7fc5e92efaf8)
2013-10-24 14:54:30 -04:00
Leo Franchi 7961d4f6b3 Add a push notification module to handle mobile client notifications
(imported from commit 3061a6e2d845226d3dce5bb262deb3a896e54f07)
2013-10-24 14:54:30 -04:00
Leo Franchi a2a1867f25 Only render tweets if we have credentials
(imported from commit 46c59f430aeae87a832f33d2ccb85209f9caec74)
2013-10-24 10:19:23 -04:00
Tim Abbott 1b381b343d Allow starring historical messages.
This has a small bug where we don't actually filter the message out of
the home view; fixing that requires adding an index on the "flags"
field of UserMessage.

(imported from commit 492c99d0a8e87b253e577be6564bec12099bd8e9)
2013-10-23 18:51:47 -04:00
Zev Benjamin b1eebc0b84 Deliver messages inside Tornado when running tests
This saves us from having to run queue processors while testing.

(imported from commit 927bfa497a47b34603761a148c57d82b2f34f813)
2013-10-22 18:45:11 -04:00
Zev Benjamin 3c73468ba8 Disable unsupported SockJS transports
Because our authentication system reads cookies from the initial
connection attempt, several SockJS transports can't be used.

(imported from commit 34b9571225d39072985b8223fb12c43c7235841f)
2013-10-22 18:45:11 -04:00
Zev Benjamin 5979af3a45 [manual] Add asynchronous message sender via sockjs-tornado
New dependency: sockjs-tornado

One known limitation is that we don't clean up sessions for
non-websockets transports.  This is a bug in Tornado so I'm going to
look at upgrading us to the latest version:
https://github.com/mrjoes/sockjs-tornado/issues/47

(imported from commit 31cdb7596dd5ee094ab006c31757db17dca8899b)
2013-10-22 18:45:11 -04:00
Steve Howell db398d5e41 Send ids, not emails, of subscribers when loading the page.
(imported from commit 26bbf38664d246476df7062432fa20add7d4cb11)
2013-10-22 15:51:18 -04:00
Steve Howell 3b59952ea3 Extract gather_subscriptions_helper().
The gather_subscriptions_helper() does a separate query to
get emails from user_ids, and it returns an email_dict to its
caller.

This may seem like a step backward, since gather_subscriptions()
now needs to do an additional query, but there is some benefit
in passing fewer redundant emails over the wire from the DB.
The real payoff, though, will come in subsequent commits, where
we will reduce the amount of data going over the wire to the browser,
which will benefit users with slow connections.

(imported from commit bf1cc5828a4c5f68cafd052ea29a177837970206)
2013-10-22 15:51:18 -04:00
Steve Howell 58acef4098 Add get_emails_from_user_ids() action.
(imported from commit 887090de16f685373bd2f3548bd1ec94a7ad4a9e)
2013-10-22 15:51:18 -04:00
Tim Abbott b5fbf67511 bugdown: Fix support for python-markdown >= 3.2.1.
(imported from commit c3c61ad3f75ef0019517084278e34641998a7e22)
2013-10-22 14:44:46 -04:00
Tim Abbott b340add1aa bugdown: Disable linebreaks from two spaces at end-of-line.
Arguably the nl2br extension should be doing this for us.  Given that
we're using nl2br, the "two spaces at the end of a line makes a line
break" rule doesn't make any sense (since every newline leads to a
linebreak), so we disable it.

(imported from commit 5ffa2ac8a825642ad31e085c532091e076665710)
2013-10-22 14:04:11 -04:00
Kevin Mehall db71931dab Include is_bot in new user event
(imported from commit 4ae4daaa3ce9b2c2546e1469cc179d209a02c313)
2013-10-22 13:42:22 -04:00
Tim Abbott 8f1fc7c968 Update mirroring backend to support IRC mirrors.
(imported from commit 771aa59b780da0aed2fc7e43d20229e7f42ad06d)
2013-10-21 14:39:21 -04:00
Tim Abbott c34be3a91f Make the forged_timestamp field optional for mirror messages.
(imported from commit 6d03d374bce0d59b0e13c944af7b620a7336a86f)
2013-10-21 14:37:36 -04:00
acrefoot 776feb84e9 Make mandrill actions better
clear_followup_emails_queue now filters by from_email too
send_local_email_template_with_delay passes the template_payload into the subject template

(imported from commit 8044fe2ebad90a9d6d5c67cdfdd08801760fd7f7)
2013-10-21 11:08:38 -04:00
Steve Howell 03c856ce10 Add create-stream management command.
The current version should only be used for testing; for example,
if you want to create a bunch of streams for stress testing, you
can run this in a loop.

(imported from commit ec51a431fb9679fc18379e4c6ecdba66bc75a395)
2013-10-19 09:28:59 -04:00
Kevin Mehall 7b8dea3d54 Add all_public_streams parameter to register API.
It makes the event queue return all messages on public streams, rather
than only the user's subscriptions. It's meant for use with chat bots.

(imported from commit 12d7e9e9586369efa7e7ff9eb060f25360327f71)
2013-10-18 14:55:32 -04:00
Jessica McKellar 6dbf7613a1 Supply a name when creating realms.
For our populate_db bulk creation, just use the domain.

(imported from commit 4fb756f6dfa2d8f90e55822e27891e84168d5d1c)
2013-10-17 13:47:51 -04:00
Leo Franchi 753aace8f8 Add a basic statistics module to analyze active user engagement
(imported from commit 4d7a21b9a09b97c88a80d0446fb16e53012507c9)
2013-10-16 18:39:44 -04:00
Kevin Mehall 71decdbe7a Fix code block auto-closing.
Trac #1162

The process_fence method replaces code blocks with placeholders, so
indexes stored before the replacement are incorrect. However, because
the closed code blocks have been replaced, we can simply search the
whole string for any remaining opening code block markers.

(imported from commit 6a9e6924840f8f3ca5175da7c52a905e27c1fabd)
2013-10-16 10:12:33 -04:00
Kevin Mehall b134c90b6b Enable backtick fenced code blocks.
Trac #1900

(imported from commit 47b3a76488a4285641fd1eb3e68bc72047a8d738)
2013-10-16 10:12:33 -04:00
Steve Howell 8cc82c6cbe Optimize /json/update_message_flags.
I added filter() statements to do_update_message_flags().

Here is some context:

Steve Howell: Case 1, have AND clause to reduce work for DB.

humbug=> update zerver_usermessage set flags = (flags & ~1) where id > 9000;
UPDATE 382
humbug=> select count(*) from zerver_usermessage where (flags & 1) = 0;
 count
-------
   382
(1 row)

humbug=> explain analyze update zerver_usermessage set flags = (flags | 1) where (flags & 1) = 0;
                                                       QUERY PLAN
------------------------------------------------------------------------------------------------------------------------
 Update on zerver_usermessage  (cost=0.00..266.85 rows=47 width=27) (actual time=5.727..5.727 rows=0 loops=1)
   ->  Seq Scan on zerver_usermessage  (cost=0.00..266.85 rows=47 width=27) (actual time=0.045..2.751 rows=382 loops=1)
         Filter: ((flags & 1::bigint) = 0)
         Rows Removed by Filter: 9000
 Total runtime: 5.759 ms
(5 rows)

humbug=> select count(*) from zerver_usermessage where (flags & 1) = 0;
 count
-------
     0
(1 row)
Leo Franchi: Sounds reasonable, but I know way less than zev about DBs so I'll defer to his judgement :)

Steve Howell: Case 2, how the code works now:

humbug=> update zerver_usermessage set flags = (flags & ~1) where id > 9000;
UPDATE 382
humbug=> select count(*) from zerver_usermessage where (flags & 1) = 0;
 count
-------
   382
(1 row)

humbug=> explain analyze update zerver_usermessage set flags = (flags | 1);
                                                        QUERY PLAN
---------------------------------------------------------------------------------------------------------------------------
 Update on zerver_usermessage  (cost=0.00..243.28 rows=9382 width=27) (actual time=362.075..362.075 rows=0 loops=1)
   ->  Seq Scan on zerver_usermessage  (cost=0.00..243.28 rows=9382 width=27) (actual time=0.008..6.138 rows=9382 loops=1)
 Total runtime: 362.105 ms
(3 rows)

humbug=> select count(*) from zerver_usermessage where (flags & 1) = 0;
 count
-------
     0
(1 row)
Steve Howell: In both trials, we set it up so that only 382 of 9382 rows need to be updated. The first trial runs about 63x as fast. The second trial, if my theory is correct, is doing 24x as many writes as it needs. Both trials are reading all 9382 rows.

Steve Howell: The expense of the update statement seems to be proportional to the number of rows you "update", not the number of rows that you actually change.

Steve Howell: For now I created #1869.

Zev Benjamin: That sounds like a reasonable explanation. The disk IO can be expensive

(imported from commit d9090daee1f81cad76c430de0956f9bd504da075)
2013-10-15 11:30:13 -04:00
acrefoot d2b8fe6f57 fix earlier 'linter workaround'
(imported from commit 4ac9f1e57a241f95f53fa65d82c376a61de2c28c)
2013-10-11 11:54:05 -04:00
acrefoot 98897fe5e3 Queue the day 1 and day 2 Zulip followup emails
Handled by the queue processor for signups. Added a management command
that accomplishes the same task, in case it's needed for manually added users,
or in case we goof and need to remove queued emails for a given user.

This addresses Trac #1807

(imported from commit 6727b82a07fa6a3ea3d827860c9e60fd0602297a)
2013-10-10 19:32:21 -04:00
acrefoot f8662c16f7 Add Mandrill decorators, credentials, actions
You can queue email for future delivery or send immediately via mandrill now

(imported from commit e6b6d11a2d94fcdeaffab80793e7ba31955b9031)
2013-10-10 19:32:21 -04:00
acrefoot 851a6e391e change variable name to get linter to stop complaining
(imported from commit b61824c16b1e7ad88375eeebdd945827dfc7fd3c)
2013-10-10 19:24:50 -04:00
Leo Franchi 1f89bf386f Parse markdown tables and show them with some styling
(imported from commit fb3c599b1dbaed2447f1e710ed7202486000ca2a)
2013-10-10 16:39:11 -04:00
Leo Franchi 6e188fb067 Tweak alert_word matches and add some more tests
(imported from commit 63dc4064c15d5d33a7ec0c992b183bf323dd4ee8)
2013-10-10 10:58:21 -04:00
Leo Franchi 08ae641dd2 Pre-fetch data from the DB and hand to markdown thread
We want to avoid opening a DB connection in the markdown thread
as its DB connection might live for a long time

(imported from commit 7700b2ca793ee5e9add7f071b92f22a4bf576b3d)
2013-10-10 10:58:21 -04:00
Zev Benjamin 2c4a1912e8 Fix parameters for on_close callback
Some of the older documentation was apparently wrong.

(imported from commit 9ebf9d1a72c8b230563f1883d63c26212eb80e22)
2013-10-09 10:45:38 -04:00
Tim Abbott 1a7ddd9ea3 Fix UserActivityInterval overlap bug.
The previous version was incorrect in the event that log_time >
last.start + 15.

(imported from commit 56aa619e8c86db1796cf936c2be5224a25ca2e1f)
2013-10-08 17:21:59 -04:00
Jessica McKellar 3b7143d3ec Linkify narrows in missed message emails.
This will hopefully incentivize people to click one and get back into
the app.

We'll also need this for digest emails.

(imported from commit 57191c3fcca3b12df93a81e4692bb7eb8ccc83b2)
2013-10-08 10:46:52 -04:00
Tim Abbott b8b0eb4508 Some zulip => humbug updates in comments and print statements.
(imported from commit 9253569a1df7f96fda81ab162d710cdda03f30ca)
2013-10-08 08:57:29 -04:00
Leo Franchi 72c7c5b836 [schema] Give new realms a default notifications stream, and send signups/creations to it
(imported from commit 19ad761f7d9eabb865ac3526bf66dff8b7ed0224)
2013-10-07 15:46:13 -04:00
Kevin Mehall 72a0b67c80 Prevent other markdown operators from touching URLs.
The text of manual links are already AtomicStrings, so linkified strings
should be too.

Moves emoji detection to happen after linkification, so the emoji rule
won't look at links.

(imported from commit 9c56bce6a0e873b398255e0762dfb312a4a9a64e)
2013-10-03 17:26:47 -04:00
Kevin Mehall 8842349629 Fix links with invalid emoji in them.
InlinePatterns should return None on failure, not text that may
have placeholders in it.

(imported from commit f9d8d22b2b8cfa7a92ecf3e52a6c76b48e6f0175)
2013-10-03 17:26:47 -04:00
Leo Franchi 7da3084465 Use name attribute in dictionary not property lookup
(imported from commit b2adae82f31b57d4e9e049225a662851b4364789)
2013-10-03 16:59:50 -04:00
Tim Abbott 4ae2ae5005 gather_subscriptions: Avoid fetching full stream objects.
(imported from commit 3052dd9c1df050e217949d36259416bc38469674)
2013-10-02 16:15:25 -04:00
Tim Abbott 356d33cfa0 gather_subscriptions: Use .values() for fetching all the user's subs.
(imported from commit 1f27841527bb5f49fab347071e40d289151fa2c5)
2013-10-02 16:15:25 -04:00
Tim Abbott 1eeb808a79 bulk_get_subscriber_emails: Use .values() to substantially improve performance.
(imported from commit b37738d670930cdf65687a8526d8a8f47835b42b)
2013-10-02 16:15:24 -04:00
Tim Abbott 382dd64ce9 gather_subscriptions: Tighten the query for Subscription objects.
(imported from commit a7caed32e52f68c7d6e33e15a7954df4fe90d797)
2013-10-02 16:15:24 -04:00
Tim Abbott b567630b5e Fix UserActivityInterval handling when events arrive out of order.
(imported from commit fa1cfbd67ac4c6c2864fb19db8a5221f90d028e7)
2013-10-01 17:41:30 -04:00
Steve Howell 95254398f2 Extract get_avatar_url().
This function doesn't require the whole UserProfile object to
create the avatar url, and we call it from Message.to_dict_uncached().

(imported from commit e814caab101c4fedd1ba66df041a3408014e4085)
2013-09-30 18:22:06 -04:00
Tim Abbott e9ee05b2b2 gather_subscriptions: Avoid extra queries fetching realm objects.
(imported from commit 591083cfaff454b8ce48aacb5da7913b8fe0a38f)
2013-09-30 17:14:05 -04:00
Tim Abbott 5e420dc5c3 gather_subscriptions: Fetch subscribers using a bulk query.
(imported from commit 13c085d0ec55779279d31d3e301311ef1c9fd2c4)
2013-09-30 17:14:05 -04:00
Tim Abbott 3a3954a8c4 Extract validation for whether the user can access subscribers.
(imported from commit 4bff4c26f98a9c2ebdd11d931c977aae8e4ebffe)
2013-09-30 17:14:05 -04:00
Tim Abbott 4de0ad29b3 Remove confusing realm optional argument to get_subscribers.
The realm should always be the realm of the stream, and we should
always pass in a stream rather than sometimes passing in a stream name
and other times passing in a stream.

(imported from commit a098d6ed3db218a37c1b6b7c956e847c316c2d13)
2013-09-30 17:14:05 -04:00
Tim Abbott 543ba03ff6 gather_subscriptions: Fix confusing overwriting of local variable stream.
(imported from commit cfe369869de5f13579ff0e0b91ec0a735e2fe81c)
2013-09-30 17:14:05 -04:00
Tim Abbott 0e23b2240d queue: Log how long it takes to connect to rabbitmq.
(imported from commit be3cd9da5e7c213cfed62feda2e676038ade57a6)
2013-09-30 13:48:48 -04:00
Steve Howell 49232d9173 Preserve muting preferences across reloads.
We have been persisting muting preferences on the back end for
a while, but we haven't been adding them to page_params for the
client to have at reload/startup time.

(imported from commit d9ca68aa0e4d22bfb0e6ce67fc0bc63981175c8b)
2013-09-28 15:48:37 -04:00
Leo Franchi 0fe3da55e3 Reduce number of queries done by bulk_add_subscriptions
We now bulk-fetch subscription information once from the database
and use it throughout bulk_add_subscriptions in order to avoid
hitting the db O(streams) times.

On my machine this shaved the accounts_register API call from making
66 queries to making 37 queries.

(imported from commit 5dd5ad3f50b2a6edf85b5f1d55ebd697a1c60647)
2013-09-27 10:26:11 -04:00
Leo Franchi 1e6b83b6c8 Fetch stream.realm information when fetching default streams
(imported from commit 08eb32e6ecda75b17d6532b0421c371de2327027)
2013-09-27 10:25:23 -04:00
Tim Abbott 8a5fffa7cf Move twitter API credentials to local_settings.py.
(imported from commit 6b95db113b91816fbc5e91db4b1be90d3df8e028)
2013-09-25 15:40:21 -04:00
Steve Howell 93347be4af Send less presence info to Tornado in do_send_messages().
When we send a message, we send some presence information to Tornado
to help it figure out how to generate emails for idle recipients of
a message.  This change limits the presence info to being the
intersection of present users and recipients of the message.  It is
just an internal optimization to avoid queueing up unneeded data.

The history behind this feature is that I implemented it a while
back, but I think I made a rebase mistake that sent all the presence
data over the wire, despite having code to filter on recipients.
It was mostly harmless, just leading to some inefficiency which is
now fixed.

(imported from commit 7c8e97705afb299c67b99053909e952fbc823551)
2013-09-25 14:50:40 -04:00
Zev Benjamin 90ee96d456 Split out presence and activity interval events from user_activity queue
(imported from commit 3c22448491867a9e72659629d5d0faaf12b85b2c)
2013-09-24 20:44:15 -04:00
Jessica McKellar f058e9cac5 Add backend support for newer realms receiving stream notifications by default.
(imported from commit ea8ee65071423a94a462571aa30013969d9f3f7a)
2013-09-24 16:19:53 -04:00
Steve Howell 50641c3337 [perf] Reduce the number of DB queries in send_message.
For a 4-person stream, we were hitting the DB 8 times, and 4 of
those queries were to lazily get user.email for the 4 recipients
due to upstream code using only().  I added user_profile__email
to the only() call.

I believe this regression started 9/18, and after pushing this
to prod, we would should look at this graph:

https://stats1.zulip.net/graphs/8274cd84588

(imported from commit 70629cb69fe5955c674ba76482609dfe78e5faaf)
2013-09-24 16:19:52 -04:00
Zev Benjamin d7d17c993c Fix using SimpleQueueClient with multiple queues
(imported from commit 18bda1430d76f8b436ad42f85d0ce3cab8fd8193)
2013-09-23 11:25:58 -04:00
Waseem Daher 7afd3ecc81 Custom invite text for MIT invitations.
(imported from commit 21b72861e991159eb7faa74809edb8d01027c6c8)
2013-09-20 12:06:26 -04:00
Steve Howell 6eda9c3314 Rename function to send_pm_if_empty_stream().
This used to have a really long name.

(imported from commit d3027a75f49db428239d432cfaa8821aa733eb04)
2013-09-20 10:19:07 -04:00
Steve Howell a457135a25 Use efficient stream.num_subscribers() for bot PMs.
Use stream.num_subscribers() in check_if_a_bot_is_sending_a_message_to_an_empty_stream().

The num_subscribers() function using Django's count() method, which returns
a single row, vs. len() on an iterator of query rows.

(imported from commit 6157fe248945e9288ee71d8cc39fb6dda4e9a247)
2013-09-20 10:19:07 -04:00
Steve Howell 5ae5e3f881 Disable empty-stream PMs for MIT bots.
(imported from commit a855d612b6ba316fc183200c4f72102831b6179c)
2013-09-20 10:19:07 -04:00
Steve Howell 098720b6d1 Extract check_if_a_bot_is_sending_a_message_to_an_empty_stream().
(imported from commit 3312d70f6b94f4a13a42bbe0320f4bb1062262e7)
2013-09-20 10:19:07 -04:00
Waseem Daher 190390d8ea Allow MIT to invite coworkers.
(imported from commit 273b3abdd3dcc49c01c1a04001450983d4acb000)
2013-09-19 17:14:41 -04:00
Kevin Mehall 6695105bb9 Fix 2499d7b to not break user registration.
Some bots created by us do not have owners. Don't try to send a
message to the nonexistent owner.

(imported from commit ab952eccd7d6c4728e9477a106142214b5c81ca9)
2013-09-19 13:42:20 -04:00
Jessica McKellar 7e733a1b91 Don't rate-limit missed message e-mails.
Instead just rely on the 2-minute delay in the management command to
batch conversations.

We've had people report being confused or thinking the feature was
broken when they didn't get e-mails because of our rate-limiting, so
let's see if this is not too overwhelming.

(imported from commit 706ddb07b906b5c2edea1159c04acc2ee6f06e29)
2013-09-19 11:55:42 -04:00
Steve Howell de0d50d31a [important] Fix recent regression w/maybe_get_subscribers
(imported from commit 8069f729dcb2bfec1c1df8f50a88fe49545a9a87)
2013-09-19 09:37:39 -04:00
Steve Howell 3c69a6dd26 Bulk-send peer_add events in bulk_add_subscriptions().
Send a single peer_add event per stream that has multiple users.

(imported from commit 1027b212557f848d437128a03698ecd54575c3fd)
2013-09-19 08:52:54 -04:00
Steve Howell c8c64984d3 Eliminate redundant peer_add notifications.
Don't send peer_add notifications to users who are already
getting add notifications, because they will already know
about subscribers.

(imported from commit 726b54ae0e30b71440b17d9c51b026872ea96218)
2013-09-19 08:52:54 -04:00
Steve Howell c58d366178 Optimize notifications from do_add_subscription().
Instead of sending one notification per user, bulk them up with
the users parameter.

(imported from commit c296f23599a214a63ef6f09406a9deaea469a7f3)
2013-09-19 08:52:54 -04:00
Steve Howell 8182d60961 Optimize query used by do_add_subscription().
It only grabs the user_profile_id column now.  This leads to a
speedup of about 16x between grabbing large ORM objects vs.
small 1-column dictionaries.

(imported from commit 95150bff3fdcbe250b04f014062224af42a6644f)
2013-09-19 08:52:54 -04:00
Steve Howell e89c6f64bd Extract notify_peers().
Splitting out notify_peers() will give us flexibility for cleaning
up how we notify peers for bulk adds.

(imported from commit e108fa2c432cc1fe54d788c58c82c983e0f2394e)
2013-09-19 08:52:54 -04:00
Steve Howell d24d8e4b36 Optimize showing subscribers on the settings page.
If you expand subscribers on your settings page, you will now see
a query like this in your postgres logs:

SELECT "zerver_userprofile"."email"
FROM "zerver_subscription" INNER JOIN "zerver_recipient" ON ("zerver_subscription"."recipient_id" = "zerver_recipient"."id") INNER JOIN "zerver_userprofile" ON ("zerver_subscription"."user_profile_id" = "zerver_userprofile"."id") WHERE ("zerver_recipient"."type" = 2  AND "zerver_subscription"."active" = true  AND "zerver_recipient"."type_id" = 40  AND "zerver_userprofile"."is_active" = true )

The join's still complicated, but the list of fields is one instead of 40+.

(imported from commit 48de1f888193a4d23fcea52d0b633d134e4a3ff7)
2013-09-19 08:52:53 -04:00
Steve Howell 9cdedf7e6c Optimize adding subscriptions with maybe_get_subscriber_emails().
To get emails, query one field instead of 40-ish fields.

(imported from commit d2dbbe23871df76bc7d431dcd9b19ebd7a58c161)
2013-09-19 08:52:53 -04:00
Steve Howell 986dd8289e Optimize get_subscribers back end.
get_subscribers_backend() now calls the new get_subscriber_emails()
function, which just queries the email field:

  "zerver_userprofile"."email"

...instead of querying about 40 fields that it never uses.

I was able to verify the query slimming by watching my postgres server log.

Also, you can verify that the ORM does roughly 16x less work using values():

>>> def f(): return [sub.user_profile.email for sub in list(Subscription.objects.all().select_related())]
...
>>> def g(): return [row['user_profile__email'] for row in list(Subscription.objects.all().values('user_profile__email'))]
...
>>> def timeit(func): t = time.time(); func(); return time.time() - t
...
>>> timeit(f)
0.045198917388916016
>>> timeit(g)
0.002752065658569336

(imported from commit a69f690a96d076b323fdfc2f4821b0548bdfac7f)
2013-09-19 08:52:53 -04:00
Steve Howell db2c9ff8c2 Extract get_subscribers_query
(imported from commit fea253591093a1f0cf722c1d4bcfb0df4f881984)
2013-09-19 08:52:53 -04:00
Kevin Mehall 8c7f1ecffa bugdown: Don't generate garbage on invalid links.
LinkPattern returned a string which contained a placeholder if the URL was
considered invalid. AtomicLinkPattern wrapped this in an AtomicString,
where the placeholder doesn't get removed properly.

m.group(0) is always incorrect because python-markdown modifies your regex
to include more than you specified (this is why part of the message got
duplicated).

(imported from commit 576bdf09c2b677cf4bc56484c363eb05f2110158)
2013-09-18 16:22:13 -04:00
Kevin Mehall 63cff6eb71 Make populate_db set the is_bot flag on bot accounts.
(imported from commit 93a507faa2b19fde05376fd73ed3a65149cfce68)
2013-09-18 16:16:17 -04:00
Leo Franchi d88266b41d Warn bot owners when a bot sends a message to a nonexistent stream
(imported from commit 2499d7bd9d176a7ab43f751eb855813d0a479bd4)
2013-09-18 15:54:13 -04:00
Leo Franchi 5441e36167 Bulk update messages in deprecated cache when updating message
(imported from commit 5e68e4eec83c0ed02b9997b0cb3f93ce8f0a7fdd)
2013-09-18 13:27:58 -04:00
Leo Franchi c588c7938c Allow message topic changing to propagate backwards
(imported from commit 470178ef5f7aaf09d4528c88ae6e33f17538fcaf)
2013-09-18 13:27:58 -04:00
Steve Howell 30b6f54a36 Don't cache UserPresence info.
(imported from commit ff590bdf5d84bb9b3cedc561c2f2fbf7f7592a66)
2013-09-17 14:51:56 -04:00
Jessica McKellar 97e6709d49 uploads: have upload_image_to_s3 take file data instead of a file object.
We have to read the data anyway, and we don't have a convenient file
handle for uploads from attachments sent through the e-mail gateway.

(imported from commit 86260a4eaceef85c82707929a80558e11dc54ef6)
2013-09-17 10:00:01 -04:00
Jessica McKellar 8a40fb346c Be able to upload images to S3 from outside the web client.
We'll need this to upload attachments sent through the email gateway.

(imported from commit b2f5eb736b65f1478189c3aebd4cb37f1ad540fb)
2013-09-17 10:00:01 -04:00
Jessica McKellar cfeaa2be9b Add backend support for realm admins to rename streams.
(imported from commit eaf419f1f68dfd350b8c99e8a5089ec316f0c409)
2013-09-16 16:16:13 -04:00
Steve Howell 1836d8958e Add comment about sending presence updates
(imported from commit da9c521a6ad652e4dbcaabcc3a24b3e23376f545)
2013-09-16 11:29:13 -04:00
Steve Howell 83e517fcf9 Flush cache on all user presence updates.
(imported from commit 130eac36f9932350a29ed88f11dd3e2f6bf3f979)
2013-09-16 11:29:13 -04:00
Steve Howell 7868787ae0 Fix indentation level in cache_save_user_profile().
(imported from commit d4dff41fa7913b15a65e10fbafb5b7b49d6b6837)
2013-09-16 11:29:13 -04:00
Steve Howell 85f9b6695f Optimize user presence/activity query.
The get_status_dict_by_realm helper gets called whenever our
 realm user_presences cache expires, and it used to query these fields:

    "zerver_userpresence"."id", "zerver_userpresence"."user_profile_id", "zerver_userpresence"."client_id", "zerver_userpresence"."timestamp", "zerver_userpresence"."status", "zerver_userprofile"."id", "zerver_userprofile"."password", "zerver_userprofile"."last_login", "zerver_userprofile"."is_superuser", "zerver_userprofile"."email", "zerver_userprofile"."is_staff", "zerver_userprofile"."is_active", "zerver_userprofile"."is_bot", "zerver_userprofile"."date_joined", "zerver_userprofile"."bot_owner_id", "zerver_userprofile"."full_name", "zerver_userprofile"."short_name", "zerver_userprofile"."pointer", "zerver_userprofile"."last_pointer_updater", "zerver_userprofile"."realm_id", "zerver_userprofile"."api_key", "zerver_userprofile"."enable_desktop_notifications", "zerver_userprofile"."enable_sounds", "zerver_userprofile"."enter_sends", "zerver_userprofile"."enable_offline_email_notifications", "zerver_userprofile"."last_reminder", "zerver_userprofile"."rate_limits", "zerver_userprofile"."avatar_source", "zerver_userprofile"."tutorial_status", "zerver_userprofile"."onboarding_steps", "zerver_userprofile"."invites_granted", "zerver_userprofile"."invites_used", "zerver_userprofile"."alert_words", "zerver_userprofile"."muted_topics", "zerver_client"."id", "zerver_client"."name"

Now it queries just the fields it needs:

  "zerver_client"."name", "zerver_userpresence"."status", "zerver_userpresence"."timestamp", "zerver_userprofile"."email" FROM "zerver_userpresence"

Also, get_status_dict_by_realm is now namespaced under UserPresence as a static method.

(imported from commit be1266844b6bd28b6c615594796713c026a850a1)
2013-09-16 11:29:10 -04:00
Steve Howell 0d2c6f756f Remove code to pre-fetch presence objects.
(imported from commit 546e98bedd809e66f506581434402caad5a984de)
2013-09-16 11:28:47 -04:00
Steve Howell e881fdd2ff Change the cache timeout for get_status_dict() back to a minute.
This function gets user presence information, which changes rapidly
and requires a pretty simple query.

(imported from commit f9b9f0f22277335c76eb4371930a4fff2758a240)
2013-09-16 11:24:29 -04:00
Steve Howell 99dd2ed8df Add presences info to the event data for process_new_message().
The do_send_messages() populates the user_presences data structure
for process_new_message(), so that Tornado code never needs to hit
the database or memcached to get the user presence info.

(imported from commit 194aeaead8fa712297a2ee8aff5aa773b92f1207)
2013-09-16 11:24:12 -04:00
Leo Franchi 6e56342cf6 Send missedmessage notifications if user is idle for >1hr
(imported from commit 573f46a77497cb2f73eae3b4a648e466977e6247)
2013-09-13 17:33:34 -04:00
Leo Franchi 7bb96bd36b Extract data from cache in tornado_callbacks with cache_get_many
This reduces the number of memcached calls we make in our time-
slice-limited tornado event handler.

(imported from commit 8903ce4ac754ba82d57e04d1b0356be7533edee2)
2013-09-13 17:33:34 -04:00
Steve Howell b7ef86bc34 Added /json/set_muted_topics
(imported from commit e9072cd273fabf0e072b6a5e9ed80b07142f1013)
2013-09-11 16:47:37 -04:00
Leo Franchi a408192f92 Don't re-parse alert words more than necessary
(imported from commit 4fee6ad9592c1dc5f844889fcb6586192de6cb35)
2013-09-11 12:00:37 -04:00
Leo Franchi 9ebb536c2d Use json_to_bool in REQ instead of manually using ujson.loads
(imported from commit 5a607f7aba4a8e4e5920a35df06c375cff98ff55)
2013-09-10 13:25:59 -04:00
Tim Abbott 9fcca3df4e [schema] Collect data on when users are active on site.
These engagement data will be useful both for making pretty graphs of
how addicted our users are as well as for allowing us to check whether
a new deployment is actually using the product or not.

This measures "number of minutes during which each user had checked
the app within the previous 15 minutes".  It should correctly not
count server-initiated reloads.

It's possible that we should use something less aggressive than
mousemove; I'm a little torn on that because you really can check the
app for new messages without doing anything active.

This is somewhat tested but there are a few outstanding issues:

* Mobile apps don't report these data.  It should be as easy as having
  them send in update_active_status queries with new_user_input=true.

* The semantics of this should be better documented (e.g. the
  management script should print out the spec above)x.

(imported from commit ec8b2dc96b180e1951df00490707ae916887178e)
2013-09-10 13:25:59 -04:00
Scott Feeney b8dc9cc198 Send you the subscribers list when you add a subscription
(imported from commit 810de1f03c6765ef1c5476df30aff4c467783cab)
2013-09-06 23:13:47 -04:00
Scott Feeney f9cfa6b5f7 Include subscriber list in page_params.stream_list
(imported from commit 7f03eec37c7cbdcdefb590c39a76444bd1b55230)
2013-09-06 23:13:47 -04:00
Scott Feeney c46007e08e When a user subscribes to a stream, notify other stream subscribers
This is a backend-only change. No frontend code that uses this is
included.

(imported from commit 4bc379de3bbfc0975ff939fa5895d716a4d1ff1b)
2013-09-06 23:13:47 -04:00
Scott Feeney 65ed20469e Split out get_subscribers method on backend
(imported from commit b5941ff90bbf5b31a9fa31aa83e5d1856d483dcc)
2013-09-06 23:13:47 -04:00
Leo Franchi cf59f07fed Replace alert_words as event contains full list
(imported from commit 489975f8bcbd0c3bcec07dd5ed62de5cae686140)
2013-09-06 16:05:08 -04:00
Leo Franchi a67d461139 Hoist joining of message content in alert_words processor
(imported from commit 5025b7108839ddc04d6d9e575a9a288223526fe8)
2013-09-06 16:05:08 -04:00
Leo Franchi 7d76c47321 Re-enable alert word bugdown processor for staging
(imported from commit a9fb8284a47d18b97b84d3ce37482132b21d3885)
2013-09-06 14:51:40 -04:00
Leo Franchi 8ad6a0daa9 Cache realm alert_words to avoid database hits when rendering
We found that since bugdown processes are threaded, the cost of
doing a db query in a markdown processor is quite high---each
thread must start up a new db connection including a SSL handshake
etc. We should strive to keep our rendering pipeline free of mandatory
DB queries.

(imported from commit 555066bd03da6c681b74ce6137acc264eb41c55d)
2013-09-06 14:50:25 -04:00
Leo Franchi 7b375634f9 Temporarily disable alert_words markdown processor on staging
(imported from commit ab3eed4ce36ebcd3d928d2d92fc34c9ebacc11f9)
2013-09-06 13:39:26 -04:00
acrefoot 592e1ead6b Actually use the realm specified by the email mirror when doing the markdown
(imported from commit 00b909b1d0c70a3523ab53d2f91815ab14ba2a6f)
2013-09-05 18:24:55 -04:00
Zev Benjamin cb78014eef bugdown: Allow block-level block quotes
It is triggered by specifying the "language" of a code block to
"quote" or "quoted":
    Hamlet said:
    ~~~ quote
    To be or **not** to be.

    That is the question
    ~~~

(imported from commit 847a0602e335e9f2955e32d9955adf8ac8de068c)
2013-09-05 17:34:28 -04:00
Zev Benjamin dd26260884 bugdown: Clarify variable name
(imported from commit 27968536bd6a3fcda6d26d8dde7f82d9f0990dd2)
2013-09-05 17:34:28 -04:00
Zev Benjamin 2f1551e745 Replace comment references to trac.humbughq.com with trac.zulip.net
(imported from commit 231d0884345be525d58ce51f48a76bba6960f09d)
2013-09-05 15:43:08 -04:00
Zev Benjamin 5c81148839 Switch our trac realm filter to use trac.zulip.net
(imported from commit 19163f5be01a1d9370d9ed45c55bedd670772dac)
2013-09-05 15:43:07 -04:00
Tim Abbott ce710292ca Always use external-content.zulipcdn.net for mit.edu realm.
(imported from commit 21d1101185bf52f21fcc345e791217311e9c0a13)
2013-09-05 14:35:42 -04:00
Tim Abbott b557b94d0b bugdown: Rewrite image urls to avoid mixed-content warnings.
(imported from commit fc0a41befb04f2a8aad2937a856366ac3cadb192)
2013-09-05 14:35:29 -04:00
Tim Abbott cdfc6dc10a bugdown: Fetch youtube preview images over HTTPS.
(imported from commit 9dbf3c21083cbbcf1288789f400c33ec735d4416)
2013-09-05 14:35:29 -04:00
Leo Franchi b2ddd670e4 [schema] Add backend support for per-user alert words
(imported from commit 7a9c596a010cbedbddf594c5d9c68bb9ed46d122)
2013-09-05 10:18:40 -04:00
Kevin Mehall 19a835e7d5 Add a checkbox that propagates topic edits to subsequent messages.
Trac #1348

(imported from commit 28e2a8cb3ecda5ec50d17501f4ccbd8644212065)
2013-09-04 14:51:46 -04:00
Leo Franchi 7e6af0a5d0 Mark messages from the desktop app as sent_by_human
(imported from commit 001073221387cbbaa541b17f9a377490562a28f1)
2013-09-04 11:00:11 -04:00
Tim Abbott eeb04ed78e deactivate_user: Only delete sessions that aren't expired.
(imported from commit e3d844a55581a7e7faab8ea9ed6703480fae8f51)
2013-09-03 11:40:42 -04:00
Scott Feeney e3247de572 Factor out get_active_user_profiles_by_realm()
(imported from commit 634ebf265b02303a938f7595de3622e25bc22b49)
2013-08-29 10:28:20 -04:00
Tim Abbott ac1f9b5128 check_message: Allow zephyr mirroring bot to send to invite-only streams.
(imported from commit d45c75915cc4e29328768f1ed0e0ec972e5b7269)
2013-08-28 19:07:05 -04:00
Tim Abbott 5b62101a5f check_message: Clean up confusing and redundant authorization checks.
(imported from commit 1b756c89ec740f988e8f15d21fb413de9947d658)
2013-08-28 19:07:05 -04:00
Tim Abbott 591b42bb1c bugdown: Clarify code around inline_interesting_links processor.
(imported from commit a1d734230135ab9099fafea7814bfec25754ea4e)
2013-08-28 18:56:53 -04:00
Jessica McKellar f67f5b7619 Allow superusers to send messages to even private streams.
(imported from commit b7253174fa25d61b5bda056a5e5353e3b8ea918b)
2013-08-28 16:43:40 -04:00
Jessica McKellar d72ed33dc6 Be more verbose and explicit in the checks on who can send a message.
It was getting hard to follow and is going to get more complicated
with a new super user check in a later commit.

(imported from commit 8d5cfa960824519d87ce0f09aab3a120ba9ef357)
2013-08-28 16:43:40 -04:00
Steve Howell 5b598a6d9c Avoid churning the cache for stream name changes that don't change the cache key.
(imported from commit 1cc1ed01eaaaec4e6ddd6901fb5b877f831b604c)
2013-08-28 13:34:26 -04:00
Steve Howell 6aecdda7f7 Allow stream renames like corporate -> Corporate
(imported from commit 46d5078fb700631ef2a5b847eeeb055ff75d87d1)
2013-08-28 13:34:25 -04:00
Jessica McKellar 97f0320dd1 Add the backend facilities for renaming a stream.
An important part of this is updating the various caches that cache
the display_recipient.

(imported from commit 888bf54fd205516cf31a25ba3f4e45ad11bbd4d5)
2013-08-28 10:23:39 -04:00
Jessica McKellar c1439c8853 actions: factor stream name checks into a function.
(imported from commit e107e895f247f6bfafe2c21de12d8a3c5de9268a)
2013-08-28 10:23:39 -04:00
Jessica McKellar cd4daaa41f Factor the display_recipient cache key into a function.
(imported from commit 4a80c4db6ddcf8104f45b6b068e598378e655bc0)
2013-08-28 10:23:37 -04:00
Jessica McKellar d64c09b1ed Add a cache_delete for deletion of items from memcached.
(imported from commit 4fbdc4c4b230a5ab029dacc2463437f81fb0d023)
2013-08-28 10:23:04 -04:00
Jessica McKellar 21d857e577 Add a cache_delete_many for bulk deletion of items from memcached.
(imported from commit 33fc65dce89e51b99f13b50216d975272a028c16)
2013-08-28 10:23:03 -04:00
Jessica McKellar 2dbe6914d9 Factor out finding active user IDs to a function.
(imported from commit c2d25b24833be900af40d45331e4b9eb9e3a2dc6)
2013-08-28 10:23:03 -04:00
Tim Abbott 648e85160d Fix buggy application of subscription update events.
(imported from commit 4c76dc3ee5d3d1a298591c0c60dbe0e3aa30e0af)
2013-08-27 19:12:20 -04:00
Tim Abbott 77dbbe7f23 zephyr_mirror: Require zcrypt when mirroring to invite-only streams.
(imported from commit 1b88a8fc9bc26f2f9b1bb3f037093f85255feb17)
2013-08-27 18:26:12 -04:00
Tim Abbott 134da30fdf Add webathena authentication button for Zephyr users.
This shows up when you're not running a Zephyr mirroring bot and lets
you use Webathena to have us run it.  Obviously needs more docs.

Current problems include:

* supervisorctl reload ends up recreating /var/run/supervisor.sock
  with the wrong permissions, so it only works once in a row before
  you need to chmod that.

* /etc/supervisor/conf.d needs to be humbug-writeable; this is a clear
  local root vulnerability

* This uses SSH and thus is kinda slow.

(imported from commit 7029979615ffd50b10f126ce2cf9a85a5eefd7a2)
2013-08-26 18:17:25 -04:00
Scott Feeney 0856274fd2 Change empty messages to '(deleted)'
Before this it was [deleted]. Using parens is consistent with how we put
in (no topic) if you don't specify a topic.

(imported from commit 931c06a1096cf7b0d226336cbe82535abd2e6032)
2013-08-26 14:13:42 -04:00
Kevin Mehall 7f1a1dba9d Show realm emoji in typeahead and update the list via events.
(imported from commit bd8b8be5d088d503cac2b72cf228e769827e2308)
2013-08-23 21:39:29 -04:00
Kevin Mehall e2a13637f2 Management command for realm emoji
(imported from commit 4de3ac69c34bc6e92d180c672fa8276338fecf53)
2013-08-23 21:39:29 -04:00
Kevin Mehall d8146141a9 Render realm emoji.
(imported from commit 5a8dca8963bc6e2e16762844aebaefcb85a6aad2)
2013-08-23 21:39:29 -04:00
Kevin Mehall 8b365a5681 Use '[deleted]' instead of failing if you edit a message to empty.
Trac #1571

(imported from commit a51602d72aa2b89ce451946ae7f5a5363a6421f4)
2013-08-21 16:49:24 -04:00
Luke Faraone f17d030272 Send IDLE continuously when idle, interpret a too-old status as offline.
This helps make our statuses more meaningful and should resolve trac #1534.

As part of this, we lower OFFLINE_THRESHOLD_SECS to 1.1̅6 minutes and
mark the user as idle after 5 minutes.

(imported from commit ee6b1ad203554a84b11e16c4c6195be9df5bcf4f)
2013-08-21 11:07:45 -04:00
Kevin Mehall 00fc004be2 Server-side check for empty (or all-whitespace) messages.
(imported from commit 49054dd7a8050705f579b4b7a1bda7eb7bfc25be)
2013-08-21 10:37:36 -04:00
Leo Franchi 8484cac5c9 Add a messages/flags POST api call to change a message's flags
(imported from commit b51ebd94c99f57f1cda19039023013209556e343)
2013-08-20 16:19:07 -04:00
Leo Franchi f8b7b54626 Log message flag changes to our event queues
(imported from commit 72fcea1a98c6a5be81f32963d51453f4bb12ace0)
2013-08-20 16:19:07 -04:00
Leo Franchi 26cd96f132 Let clients specify how long queues shall live, within limits
(imported from commit 86609f6611ed37b45b28c31e541528ce260d62c8)
2013-08-20 16:19:07 -04:00
acrefoot 395aaae994 Allow anyone to edit a "no topic" message
This change would allow anyone in the realm to set a topic for a "no topic"
message. As soon as the message topic is set, only the sender can change it again.

(imported from commit 0a91a93b8fd14549965cedc79f45ffd869d82307)
2013-08-19 16:29:30 -04:00
Jessica McKellar f530e3b930 Display the email address for a stream in its stream page detail.
For now, only show it on staging.

(imported from commit fd07fad1c34578d8ddc2cddd1bb6bdcb72f354de)
2013-08-13 14:28:47 -04:00
Jessica McKellar 15afdf65eb email forwarder: encode stream names so they are safe in email addresses.
(imported from commit 6699d656e3cae58fad062a1403fb9923429fde89)
2013-08-13 14:28:47 -04:00
Luke Faraone af4c34330b Switch to checking database is_bot field instead of using a heuristic.
This has the amusing side effect of showing all the Zulip bots in the
administration view because none of them have the is_bot 	 set.

(imported from commit cdec19d2109c092018c1f331aa32f345d1587683)
2013-08-13 14:20:18 -04:00
Zev Benjamin 307bba5b29 Add comment to some zephyr mirror CC de-duplication logic
(imported from commit 86680f24329475dccc79731639d51a735fa873bf)
2013-08-13 13:48:32 -04:00
Zev Benjamin 91c0be44a1 Raise JsonableErrors while checking messages instead of returning error strings
(imported from commit 986ad1e19dd508b12386c57cf093b32d3fbcf49e)
2013-08-13 13:45:10 -04:00
Zev Benjamin bfebfbbff5 Make error messages a little more consistent
(imported from commit 77baba591628bef6b8b8b0ae28a9e05ec2fac693)
2013-08-13 13:45:10 -04:00
Zev Benjamin 6fcfcf3f32 Fix comment
(imported from commit 6f8eac90e5b011bd25b09e562ebcb9d2cae39a06)
2013-08-13 13:45:09 -04:00
Zev Benjamin bcc03dd4ab Do full-precision time comparison when checking for duplicated messages
We've had microsecond-resolution timestamps since we migrated to Postgres.

(imported from commit 81fe79f00097ceb40105645aa04a2f0ee29d3a19)
2013-08-13 13:45:09 -04:00
Luke Faraone 3129b6374d Fix broken parsing of DNS response.
(imported from commit 3a627d45bc313bf244dc699a59df9423c849bc35)
2013-08-13 10:21:24 -07:00
Waseem Daher abef6be12f bugdown: Unescape HTML entities in Twitter.
This closes Trac #1071.

(imported from commit 521ffa6ed13df7c08e67218564d426ca90080948)
2013-08-12 17:23:49 -04:00
Luke Faraone 0cb741d7cc Removed confusing ALLOW_REGISTER setting.
ALLOW_REGISTER was no longer being used in determining whether you could
register for the app, so I've removed it to avoid additional local-dev /
production issues.

This closes #1613.

(imported from commit c928c6d350602d35f745ae1e60d734e4567885fc)
2013-08-12 16:16:26 -04:00
Luke Faraone 368ace069c Use PyDNS instead of shelling out to `host`
On Debian systems, this is found in the `python-dns` package.

On OS X and others, install "pydns" using your Python package manager.

(imported from commit 17827d0a1d3d72b12945df5563295a1573bfa1ed)
2013-08-12 13:09:19 -07:00
Zev Benjamin 9c8a9ac947 Correctly return the message id for already mirrored zephyr messages
This was previously causing us to generate a traceback every time we
hit a duplicated zephyr due to CC'ing.

(imported from commit 240e1559655d0166dcd864e84649ab97b87a29ad)
2013-08-12 15:49:19 -04:00
Tim Abbott 7fa448cdef Return the message ID when sending a message.
Our API documentation says that we do, and it seems like it could be
useful to clients, so we might as well do it.

(imported from commit c391e4952a09d41df4dc06e3dc6ee094f774822b)
2013-08-09 15:35:33 -04:00
Jessica McKellar 3f8dfc7b9b Move to a common random token generation function instead of several one-offs.
(imported from commit 3217de5384088deff68fbffc6bd481c045a76817)
2013-08-09 14:59:26 -04:00
Waseem Daher 49849a214f New onboarding step: Set up an integration.
(imported from commit 2c6cc6f703e893b44b2871c9510e858743d417ad)
2013-08-08 17:23:27 -04:00
Tim Abbott 7b9305b06f Rename Django project to zproject.
This includes a hack to preserve humbug/backends.py as a symlink, so
that we don't need to regenerate all our old sessions.

(imported from commit b7918988b31c71ec01bbdc270db7017d4069221d)
2013-08-07 11:04:03 -04:00
Tim Abbott d59ba39518 Rename humbug_finish to zulip_finish.
(imported from commit a73ef9302d16a4068c7d050d4882d2eba699488d)
2013-08-07 10:00:08 -04:00
Tim Abbott d63a398716 Change Humbug => Zulip in name of error limiter module.
(imported from commit 55c5deba463faecc50da22e9745fd13ad8b11fd8)
2013-08-07 10:00:07 -04:00
Tim Abbott 1f1af26048 Change Humbug => Zulip in text/comments.
(imported from commit 2f9d73431ae40e1b9e9e11bc2f4f62f566ae758a)
2013-08-07 10:00:07 -04:00
Steve Howell 9f9c7c3e3b Loosen up emoji regex. (i.e. More emojis! 👍)
Now parsed: 🍺,🍺;🍺!

 If \w characters surround :foo:, we still say it's NOT an
 emoji, but we used to do this for \S characters, so it's loosened up.

(imported from commit 49b33d2f0ffdcfde8947ae411a4addcf4c24af9c)
2013-08-06 18:20:30 -04:00
Tim Abbott e111a2f9a5 [manual] Rename Django app from zephyr to zerver.
This needs to be deployed to both staging and prod at the same
off-peak time (and the schema migration run).

At the time it is deployed, we need to make a few changes directly in
the database:

(1) UPDATE django_content_type set app_label='zerver' where app_label='zephyr';
(2) UPDATE south_migrationhistory set app_name='zerver' where app_name='zephyr';

(imported from commit eb3fd719571740189514ef0b884738cb30df1320)
2013-08-06 07:39:36 -04:00