Commit Graph

79 Commits

Author SHA1 Message Date
Leo Franchi 29413867b7 Use just one cache key for all tweet lookups
Since we log to statsd our cache time lookups by cache key, using a unique
tweet id for each lookup was just filling up our cache without being useful.

Also, log database cache lookups in a further namespace to distinguish between
memcached caches

(imported from commit a2a16b777fb7ab8cd066feee7344f9c8a3c107f5)
2013-05-11 15:51:40 -04:00
Luke Faraone 3c08781764 Limit the number of URLs we parse at once to 5.
(imported from commit 876c0d0713b4a4d328e4a69e46c984694f5bce1b)
2013-05-09 10:44:08 -04:00
Luke Faraone 2c44ed1096 Log URLs that cause Embedly timeouts.
(imported from commit 84fdca163f6fefae5adc5ea4849d4d707aa7049e)
2013-05-09 10:44:08 -04:00
Luke Faraone f8d443c3c9 Time out embed.ly calls after 2.5 seconds.
(imported from commit 903b800a8444ff5c0b463dba17b2653e1775ae81)
2013-05-09 10:44:08 -04:00
Waseem Daher 47ac589f87 Make inline previews smaller.
(imported from commit 228f6f6f296877810824f98ea8001e47cf955cf1)
2013-05-09 10:44:08 -04:00
Luke Faraone 0513d9760a Batch-process and cache responses from the Embedly API
(imported from commit c427e2bf8f0f32dc632b05f41fa18ab10d0b6d03)
2013-05-09 10:44:08 -04:00
Luke Faraone 9fec3cd3c0 Explicitly skip Twitter links.
(imported from commit 2452dd4d92febbfc2b83ab0e09d7158f4bda76e3)
2013-05-09 10:44:08 -04:00
Luke Faraone fac83ed2b7 [manual] Add support for Embedly Extract
For sites that are supported, we now grab thumbnails for images + video
embed code for videos and use them in lieu of our existing embed code.

We also embed rich non-script content.

Special casing is done so that we don't embed images twice.

Some testcases were modified to avoid triggering Embed.ly

The manual step is to install python-embedly.

(imported from commit d725bab91675c61953116c5ca741055fce49724e)
2013-05-09 10:42:49 -04:00
Luke Faraone 8601c1670f bugdown: Factor out creating a link into its own method
(imported from commit 869cec9927570c4126b78f90aeedc2d5b542d097)
2013-05-09 10:42:19 -04:00
Tim Abbott d45527864c bugdown: Include server name in markdown failure emails.
(imported from commit 5b090480a66ca04f7b065215c8c5da698b6c5358)
2013-05-09 10:35:45 -04:00
Jeff Arnold 105fdc076b Reserve height=100 for images so that collapsing takes them into account
Addresses #1204

(imported from commit 9a1f4b13ab14c4b8fc4eab50178b9c28f477fcd7)
2013-05-02 18:53:34 -04:00
Tim Abbott e3bb1bc8ec bugdown: Fix tweet ID extraction from twitter urls.
(imported from commit 88b9882527a5317bf30bcc5f0d1255e819ea149c)
2013-04-30 10:43:17 -04:00
Waseem Daher 20233cfc96 Time out on Twitter rendering if it takes too long.
Timing out within the Twitter portion of the render causes the message
to still go through (without a preview). If we don't timeout here, it
causes the entire Markdown render to timeout, which rejects the
message in its entirety -- a far worse outcome.

(imported from commit f510a56f48afa46da8ec6277496fa03374cdb042)
2013-04-23 12:56:34 -04:00
Leo Franchi 0055107cfd Use IANA's TLD list for auto-linkification detecting
(imported from commit 9103fdc92405b92300a793bd1d4f493df64b5b9c)
2013-04-03 09:58:17 -04:00
Leo Franchi a5643efa14 Allow @ in urls
(imported from commit cb2ffe4a8f050e732bb06ab4609997be35577417)
2013-04-02 18:38:38 -04:00
Leo Franchi d127d6f19f Support up to one level deep of nested parens in urls
(imported from commit 3f314b16a47b5267ddb0d18aa6c5456656895f77)
2013-04-02 18:38:38 -04:00
Keegan McAllister 191231ab3d bugdown: Whitelist URL schemes
(imported from commit 76e22cec3918c00faaa903baae74915cc5e64264)
2013-04-02 18:38:38 -04:00
Keegan McAllister 5d538d7a2a bugdown: Allow colons in URLs
(imported from commit b57fc21f4508f2bff3cbc32a6359de686aa3a96e)
2013-04-02 18:38:38 -04:00
Tim Abbott 9977cdde54 send_message: Save rendered message content to the database.
(imported from commit e5bd224687d5420f556b341286f03d3829973a3c)
2013-03-19 08:09:26 -04:00
Tim Abbott 97d7d31b68 Rewrite internal_send_message to use check_send_message.
This will automatically fix bugs such as one in which
internal_send_message didn't properly strip() the subject argument
before sending a message.

We change the recipient_type argument to internal_send_message to take
the recipient type name (e.g. 'stream') both to better fit the API and
also because the previous code incorrectly handled huddles.

(imported from commit 78c2596d328f6bb1ce2eaa3eed9a9e48146e3b6a)
2013-03-18 16:15:11 -04:00
Waseem Daher 0f621f5f89 Show Twitter previews on production.
This reverts commit 4baed7ba22482eff6ed381d319f6e21da904d9ba.

(imported from commit dcab148b24e95f90edd5f2f3f581b4037deed787)
2013-03-14 18:00:04 -04:00
Tim Abbott 1580386946 Replace db_cache_with_key with a parameter on cache_with_key.
(imported from commit f2c600292888ba384ee4acc97c94f1d6f8bc9657)
2013-03-14 15:07:41 -04:00
Waseem Daher 9c45b96b21 Make Dropbox inline image preview more robust to sharing links.
Sometimes Dropbox shares with /s/ and sometimes with /sh/,
and I'm not sure which controls it, but we should deal with both.

(imported from commit 2222450f25c418b5fbd60ab2c30477467e34c0d1)
2013-03-13 17:46:43 -04:00
Tim Abbott 275af189a3 Cache permanent errors from the Twitter API.
This avoids our repeatedly retrying to fetch a tweet that doesn't
exist from the Twitter API.

(imported from commit b4ca1060d03da21e7e59e5b99e682d2e8457df15)
2013-03-13 11:12:25 -04:00
Waseem Daher a57afd9a63 Use separate Twitter API key for prod.
(imported from commit ab43b42f69eaa569cf7a1ccba9f15d528050c22b)
2013-03-12 18:28:59 -04:00
Waseem Daher 3c6d489064 Don't do Twitter previews on prod.
(imported from commit 4baed7ba22482eff6ed381d319f6e21da904d9ba)
2013-03-11 15:18:57 -04:00
Tim Abbott d679a72952 [manual] Cache results of the Twitter API in the database.
This should substantially improve the repeat-rendering time for pages
with large numbers of tweets since we don't need to go all the way to
twitter.com, which can take like a second, to render tweets properly.

To deploy this commit properly, one needs to run

./manage.py createcachetable third_party_api_results

(imported from commit 01b528e61f9dde2ee718bdec0490088907b6017e)
2013-03-11 13:15:55 -04:00
Waseem Daher 29d504149c twitter: Only render the first Twitter link we see, not all of them.
(imported from commit 68582224962da14795d91a68197db1001ae7b75c)
2013-03-08 17:45:58 -05:00
Tim Abbott 2f022ebb34 bugdown: Allow caller of convert to tell whether we succeeded.
(imported from commit 9e3cab73e0d9e9ad31ba770210e535584cca3bb2)
2013-03-08 17:24:33 -05:00
Waseem Daher 0e2e754c3d [manual] Use Twitter API v1.1 for inline previews.
This commit adds a dependency on python-twitter,
whose upstream is at https://github.com/bear/python-twitter,
and which for now needs to manually be installed on our
servers from the Debian package in sid.

(imported from commit 80cd9f4f59a6f0de6b75ac95e412c69e2a2e2490)
2013-03-08 17:10:06 -05:00
Waseem Daher 1df648baa9 Add inline preview of Twitter links.
This uses the unauthed v1 of the Twitter API, which is going to go
away soon, but it's fine as an interim measure.

(imported from commit 709a250271321f5479854a363875c9da43e6382d)
2013-03-08 14:31:18 -05:00
Tim Abbott 4ab5cabb80 bugdown: Split out iteration through a tree into helper function.
(imported from commit ba73832c61eccdefe45f8e0d935216e6cc431176)
2013-03-08 14:31:17 -05:00
Reid Barton bc0dbbb566 Detect image-ness by end of path component, not end of entire URL
(imported from commit 9dd2e9c7273d3c7d071a8e5f82014e28b2a99ee4)
2013-03-06 19:02:29 -05:00
Waseem Daher 0dee6235cc Don't image-ify :( and other smileys 😞
(imported from commit 25323f19572c24da36267064a1ca820d80c9db79)
2013-03-05 16:22:51 -05:00
Waseem Daher 9f0877a7f2 Add emoji support 👍 🎩 💩 😀
(imported from commit 5f15ddb4492aad7cc4a9b3249acac8b8edc0a177)
2013-03-04 11:08:00 -05:00
Waseem Daher 1650dba322 Intelligently preview Dropbox image links.
Even though they look like images, they're not -- you need to
append ?dl=1 to get the image version.

(imported from commit 2a05e7c58f475c908687110d9191f8709425c660)
2013-03-04 10:38:42 -05:00
Tim Abbott 02bf89c3c3 Add inline previews for images and youtube videos.
(imported from commit 13711a7821318c8bff7c483e874756a63d761c15)
2013-03-01 14:21:35 -05:00
Keegan McAllister 29b98e5546 Display invalid links as plain text rather than linkifying with empty href
Fixes #1001.

(imported from commit 116df72f37e45dd87c34cd836918dd559630fe76)
2013-02-27 11:29:05 -05:00
Leo Franchi 5d82dc5ce2 Don't match ** for strong across newlines
(imported from commit e3bc0b409bd99c3707746c23eb73e11342cfe178)
2013-02-14 11:56:53 -05:00
Leo Franchi 168f8623e4 Add https?: greedy url matching before falling back to our url guesser
(imported from commit 9e6e5a0522e6501b354a56223c2639841d290d4b)
2013-02-12 12:24:15 -05:00
Leo Franchi c637b9cf44 Handle chars at end of link better
(imported from commit 53842bc17bbb9cfb555738ee6b72291d7ce61d67)
2013-02-06 11:51:52 -05:00
Leo Franchi 054420fc38 Match .co.uk before .co in linkification
(imported from commit 2461cb4d49eef15431dde30dda646b25dc860a7b)
2013-02-05 13:04:45 -05:00
Leo Franchi 3569e8e9b2 Sanitize links to prevent XSS, and handle emails
(imported from commit 622396efde50d9f5e3501f5d780c344ad0692662)
2013-02-01 18:42:27 -05:00
Leo Franchi cc58044234 Rework linkify regex to match more urls
(imported from commit 0e1a1df88363374ffbc802f83f43eb0fac8c99ea)
2013-02-01 16:14:29 -05:00
Keegan McAllister d348e0e9d6 bugdown: Disable italics and __foo__-style bold
(imported from commit c35d6980db6c59828514a90eec199a7494625495)
2013-01-31 16:28:57 -05:00
Leo Franchi 071ea3b9a4 Send humbug & email on markdown parse failure
(imported from commit c132ab91f1ff9ffdbe5f740980bf362b16c6bba6)
2013-01-31 14:48:40 -05:00
Keegan McAllister e12f10ec1f bugdown: Spend at most 5 seconds rendering a message
(imported from commit bc092acc8b2b9f8a63af669de06c6f7512ccf8c9)
2013-01-30 11:13:48 -05:00
Keegan McAllister f2659b7dcd bugdown: Remove dead code
The user of this function was removed in 97dd20ec.

(imported from commit c3823df65172f3799658e36d7ba183f661e9fe65)
2013-01-28 18:37:48 -05:00
Leo Franchi c39c6f643c Handle lists that start immediately after paragraphs
(imported from commit 055593d7a324598e133d53db0c33103016426c8f)
2013-01-25 17:13:13 -05:00
Leo Franchi 969d89d070 Don't accept - as valid list delimiters
(imported from commit 287353a29289ee536a59f47f87ff66893bf261ec)
2013-01-25 17:13:13 -05:00