Commit Graph

93 Commits

Author SHA1 Message Date
Tim Abbott 2de2491021 Disabled embed.ly support, for now.
We get too many error reports from it, which is bad for us actually
fixing the other errors that we do have.

(imported from commit 8442fe4251adb15a01b4e61ebcd07bc270b08631)
2013-05-28 11:50:56 -04:00
Tim Abbott c98b37550e Record how much time is spent in the markdown parser.
(imported from commit c382f4f88ae253a47e05b3d15c6d21bc0cbb2bb7)
2013-05-22 14:41:57 -04:00
Tim Abbott 5b8dc0d09a Unify our various link processors into a single processor.
There was no benefit to our various link processors all doing
independent scans through the list of messages, and this makes it much
easier to understand the logic of how each link will be handled, and
also makes policies like "don't process links if there are more than 5
of then" easier to implement coherently.

(imported from commit 4affdeab889ba89b99eec905fdf871e78bbc3dd4)
2013-05-22 13:43:20 -04:00
Tim Abbott c325794f3c Send Markdown parser failures to the errors stream.
(imported from commit 8638744ec45d6505fe3a7be87df0a70943f23996)
2013-05-20 23:29:54 -04:00
Luke Faraone 1bb9def157 Only contact embed.ly if we have items that are not cached to fetch.
(imported from commit 8016d1f5dac6a75ced63bec8f0f11d9e4020cccb)
2013-05-14 08:00:37 -07:00
Leo Franchi 29413867b7 Use just one cache key for all tweet lookups
Since we log to statsd our cache time lookups by cache key, using a unique
tweet id for each lookup was just filling up our cache without being useful.

Also, log database cache lookups in a further namespace to distinguish between
memcached caches

(imported from commit a2a16b777fb7ab8cd066feee7344f9c8a3c107f5)
2013-05-11 15:51:40 -04:00
Luke Faraone 3c08781764 Limit the number of URLs we parse at once to 5.
(imported from commit 876c0d0713b4a4d328e4a69e46c984694f5bce1b)
2013-05-09 10:44:08 -04:00
Luke Faraone 2c44ed1096 Log URLs that cause Embedly timeouts.
(imported from commit 84fdca163f6fefae5adc5ea4849d4d707aa7049e)
2013-05-09 10:44:08 -04:00
Luke Faraone f8d443c3c9 Time out embed.ly calls after 2.5 seconds.
(imported from commit 903b800a8444ff5c0b463dba17b2653e1775ae81)
2013-05-09 10:44:08 -04:00
Waseem Daher 47ac589f87 Make inline previews smaller.
(imported from commit 228f6f6f296877810824f98ea8001e47cf955cf1)
2013-05-09 10:44:08 -04:00
Luke Faraone 0513d9760a Batch-process and cache responses from the Embedly API
(imported from commit c427e2bf8f0f32dc632b05f41fa18ab10d0b6d03)
2013-05-09 10:44:08 -04:00
Luke Faraone 9fec3cd3c0 Explicitly skip Twitter links.
(imported from commit 2452dd4d92febbfc2b83ab0e09d7158f4bda76e3)
2013-05-09 10:44:08 -04:00
Luke Faraone fac83ed2b7 [manual] Add support for Embedly Extract
For sites that are supported, we now grab thumbnails for images + video
embed code for videos and use them in lieu of our existing embed code.

We also embed rich non-script content.

Special casing is done so that we don't embed images twice.

Some testcases were modified to avoid triggering Embed.ly

The manual step is to install python-embedly.

(imported from commit d725bab91675c61953116c5ca741055fce49724e)
2013-05-09 10:42:49 -04:00
Luke Faraone 8601c1670f bugdown: Factor out creating a link into its own method
(imported from commit 869cec9927570c4126b78f90aeedc2d5b542d097)
2013-05-09 10:42:19 -04:00
Tim Abbott d45527864c bugdown: Include server name in markdown failure emails.
(imported from commit 5b090480a66ca04f7b065215c8c5da698b6c5358)
2013-05-09 10:35:45 -04:00
Jeff Arnold 105fdc076b Reserve height=100 for images so that collapsing takes them into account
Addresses #1204

(imported from commit 9a1f4b13ab14c4b8fc4eab50178b9c28f477fcd7)
2013-05-02 18:53:34 -04:00
Tim Abbott e3bb1bc8ec bugdown: Fix tweet ID extraction from twitter urls.
(imported from commit 88b9882527a5317bf30bcc5f0d1255e819ea149c)
2013-04-30 10:43:17 -04:00
Waseem Daher 20233cfc96 Time out on Twitter rendering if it takes too long.
Timing out within the Twitter portion of the render causes the message
to still go through (without a preview). If we don't timeout here, it
causes the entire Markdown render to timeout, which rejects the
message in its entirety -- a far worse outcome.

(imported from commit f510a56f48afa46da8ec6277496fa03374cdb042)
2013-04-23 12:56:34 -04:00
Keegan McAllister 3c40dd3bf3 bugdown: Fix fenced_code for Python-Markdown 2.3
(imported from commit 3954444708e222217407df228f07d2cad402a02b)
2013-04-05 13:14:00 -04:00
Leo Franchi 0055107cfd Use IANA's TLD list for auto-linkification detecting
(imported from commit 9103fdc92405b92300a793bd1d4f493df64b5b9c)
2013-04-03 09:58:17 -04:00
Leo Franchi a5643efa14 Allow @ in urls
(imported from commit cb2ffe4a8f050e732bb06ab4609997be35577417)
2013-04-02 18:38:38 -04:00
Leo Franchi d127d6f19f Support up to one level deep of nested parens in urls
(imported from commit 3f314b16a47b5267ddb0d18aa6c5456656895f77)
2013-04-02 18:38:38 -04:00
Keegan McAllister 191231ab3d bugdown: Whitelist URL schemes
(imported from commit 76e22cec3918c00faaa903baae74915cc5e64264)
2013-04-02 18:38:38 -04:00
Keegan McAllister 5d538d7a2a bugdown: Allow colons in URLs
(imported from commit b57fc21f4508f2bff3cbc32a6359de686aa3a96e)
2013-04-02 18:38:38 -04:00
Tim Abbott 9977cdde54 send_message: Save rendered message content to the database.
(imported from commit e5bd224687d5420f556b341286f03d3829973a3c)
2013-03-19 08:09:26 -04:00
Tim Abbott 97d7d31b68 Rewrite internal_send_message to use check_send_message.
This will automatically fix bugs such as one in which
internal_send_message didn't properly strip() the subject argument
before sending a message.

We change the recipient_type argument to internal_send_message to take
the recipient type name (e.g. 'stream') both to better fit the API and
also because the previous code incorrectly handled huddles.

(imported from commit 78c2596d328f6bb1ce2eaa3eed9a9e48146e3b6a)
2013-03-18 16:15:11 -04:00
Waseem Daher 0f621f5f89 Show Twitter previews on production.
This reverts commit 4baed7ba22482eff6ed381d319f6e21da904d9ba.

(imported from commit dcab148b24e95f90edd5f2f3f581b4037deed787)
2013-03-14 18:00:04 -04:00
Tim Abbott 1580386946 Replace db_cache_with_key with a parameter on cache_with_key.
(imported from commit f2c600292888ba384ee4acc97c94f1d6f8bc9657)
2013-03-14 15:07:41 -04:00
Waseem Daher 9c45b96b21 Make Dropbox inline image preview more robust to sharing links.
Sometimes Dropbox shares with /s/ and sometimes with /sh/,
and I'm not sure which controls it, but we should deal with both.

(imported from commit 2222450f25c418b5fbd60ab2c30477467e34c0d1)
2013-03-13 17:46:43 -04:00
Tim Abbott 275af189a3 Cache permanent errors from the Twitter API.
This avoids our repeatedly retrying to fetch a tweet that doesn't
exist from the Twitter API.

(imported from commit b4ca1060d03da21e7e59e5b99e682d2e8457df15)
2013-03-13 11:12:25 -04:00
Waseem Daher a57afd9a63 Use separate Twitter API key for prod.
(imported from commit ab43b42f69eaa569cf7a1ccba9f15d528050c22b)
2013-03-12 18:28:59 -04:00
Waseem Daher 3c6d489064 Don't do Twitter previews on prod.
(imported from commit 4baed7ba22482eff6ed381d319f6e21da904d9ba)
2013-03-11 15:18:57 -04:00
Tim Abbott d679a72952 [manual] Cache results of the Twitter API in the database.
This should substantially improve the repeat-rendering time for pages
with large numbers of tweets since we don't need to go all the way to
twitter.com, which can take like a second, to render tweets properly.

To deploy this commit properly, one needs to run

./manage.py createcachetable third_party_api_results

(imported from commit 01b528e61f9dde2ee718bdec0490088907b6017e)
2013-03-11 13:15:55 -04:00
Waseem Daher 29d504149c twitter: Only render the first Twitter link we see, not all of them.
(imported from commit 68582224962da14795d91a68197db1001ae7b75c)
2013-03-08 17:45:58 -05:00
Tim Abbott 2f022ebb34 bugdown: Allow caller of convert to tell whether we succeeded.
(imported from commit 9e3cab73e0d9e9ad31ba770210e535584cca3bb2)
2013-03-08 17:24:33 -05:00
Waseem Daher 0e2e754c3d [manual] Use Twitter API v1.1 for inline previews.
This commit adds a dependency on python-twitter,
whose upstream is at https://github.com/bear/python-twitter,
and which for now needs to manually be installed on our
servers from the Debian package in sid.

(imported from commit 80cd9f4f59a6f0de6b75ac95e412c69e2a2e2490)
2013-03-08 17:10:06 -05:00
Waseem Daher 1df648baa9 Add inline preview of Twitter links.
This uses the unauthed v1 of the Twitter API, which is going to go
away soon, but it's fine as an interim measure.

(imported from commit 709a250271321f5479854a363875c9da43e6382d)
2013-03-08 14:31:18 -05:00
Tim Abbott 4ab5cabb80 bugdown: Split out iteration through a tree into helper function.
(imported from commit ba73832c61eccdefe45f8e0d935216e6cc431176)
2013-03-08 14:31:17 -05:00
Reid Barton bc0dbbb566 Detect image-ness by end of path component, not end of entire URL
(imported from commit 9dd2e9c7273d3c7d071a8e5f82014e28b2a99ee4)
2013-03-06 19:02:29 -05:00
Waseem Daher 0dee6235cc Don't image-ify :( and other smileys 😞
(imported from commit 25323f19572c24da36267064a1ca820d80c9db79)
2013-03-05 16:22:51 -05:00
Waseem Daher 9f0877a7f2 Add emoji support 👍 🎩 💩 😀
(imported from commit 5f15ddb4492aad7cc4a9b3249acac8b8edc0a177)
2013-03-04 11:08:00 -05:00
Waseem Daher 1650dba322 Intelligently preview Dropbox image links.
Even though they look like images, they're not -- you need to
append ?dl=1 to get the image version.

(imported from commit 2a05e7c58f475c908687110d9191f8709425c660)
2013-03-04 10:38:42 -05:00
Tim Abbott 02bf89c3c3 Add inline previews for images and youtube videos.
(imported from commit 13711a7821318c8bff7c483e874756a63d761c15)
2013-03-01 14:21:35 -05:00
Keegan McAllister 29b98e5546 Display invalid links as plain text rather than linkifying with empty href
Fixes #1001.

(imported from commit 116df72f37e45dd87c34cd836918dd559630fe76)
2013-02-27 11:29:05 -05:00
Keegan McAllister 75704ad1e8 Consistently use #!/usr/bin/env python
At Ksplice we used /usr/bin/python because we shipped dependencies as Debian /
Red Hat packages, which would be installed against the system Python.  We were
also very careful to use only Python 2.3 features so that even old system
Python would still work.

None of that is true at Humbug.  We expect users to install dependencies
themselves, so it's more likely that the Python in $PATH is correct.  On OS X
in particular, it's common to have five broken Python installs and there's no
expectation that /usr/bin/python is the right one.

The files which aren't marked executable are not interesting to run as scripts,
so we just remove the line there.  (In general it's common to have libraries
that can also be executed, to run test cases or whatever, but that's not the
case here.)

(imported from commit 437d4aee2c6e66601ad3334eefd50749cce2eca6)
2013-02-20 16:02:30 -05:00
Leo Franchi 5d82dc5ce2 Don't match ** for strong across newlines
(imported from commit e3bc0b409bd99c3707746c23eb73e11342cfe178)
2013-02-14 11:56:53 -05:00
Leo Franchi 168f8623e4 Add https?: greedy url matching before falling back to our url guesser
(imported from commit 9e6e5a0522e6501b354a56223c2639841d290d4b)
2013-02-12 12:24:15 -05:00
Leo Franchi c637b9cf44 Handle chars at end of link better
(imported from commit 53842bc17bbb9cfb555738ee6b72291d7ce61d67)
2013-02-06 11:51:52 -05:00
Leo Franchi 054420fc38 Match .co.uk before .co in linkification
(imported from commit 2461cb4d49eef15431dde30dda646b25dc860a7b)
2013-02-05 13:04:45 -05:00
Leo Franchi 3569e8e9b2 Sanitize links to prevent XSS, and handle emails
(imported from commit 622396efde50d9f5e3501f5d780c344ad0692662)
2013-02-01 18:42:27 -05:00