Commit Graph

192 Commits

Author SHA1 Message Date
Steve Howell 152cebf33d bugdown: Extract log_bugdown_error().
(It makes it easier to mock out the logging.)
2016-09-16 10:01:54 -07:00
umkay 7eb6924841 Wrap dict view object in list for python3 compatibility.
Deleting items from a dictionary being iterated through directly
doesn't work properly in Python 3.
2016-09-13 21:21:45 -07:00
Taranjeet Singh 5e5ad9de0c bugdown/__init__.py: Fix line with length greater than 120.
This updates file zerver/lib/bugdown/__init__.py.
2016-08-19 11:56:43 -07:00
Tim Abbott e7aac717e8 Fix markdown ordering bugs with some custom syntax.
!avatar, !modal_link, !gravatar, etc. were incorrectly being processed
before the escape character for code blocks.

While we're at it, we add tests for these special syntaxes.
2016-08-18 14:47:37 -07:00
Tim Abbott 70543e059a bugdown: Remove hardcoding of mit.edu for zephyr_mirror realm config. 2016-07-26 20:30:12 -07:00
Eklavya Sharma c389d22b5f Fix twitter mock data. 2016-07-17 10:23:25 -07:00
Eklavya Sharma fb1d00fc6c Upgrade python-twitter to latest version.
Fixes #1145.
2016-07-17 10:23:25 -07:00
Eklavya Sharma 4130170da8 bugdown: Fix string handling.
Use unicode strings where needed and correctly encode/decode strings.
2016-07-17 10:23:25 -07:00
Eklavya Sharma abe7faa2f9 bugdown: Annotate some variables. 2016-07-17 10:23:25 -07:00
Taranjeet a8a4caf2c0 zerver: Fix lines with length greater than 120. 2016-07-08 11:41:43 -07:00
Eklavya Sharma 6bb266d262 bugdown's __init__.py: Add python 3 compatibility.
* Use Response.text instead of Response.content.
* Make unescaping work on python 3.
2016-07-07 09:52:24 -07:00
Eklavya Sharma 05046d9288 bugdown's codehilite.py: Fix annotations in python 3 mode.
Many stubs in xml.etree.ElementTree use Union[str, bytes] as
return type.  Mypy wants us to correctly handle each case.  This
is correct, but not useful for us since we know that we'll always
get str.  So force the return value to text_type, to supress mypy
errors.
2016-07-07 09:52:24 -07:00
Eklavya Sharma b902c1ae45 bugdown: Replace raw unicode literal with non-raw.
Raw unicode literals are disallowed in python 3.
2016-07-02 10:38:48 -07:00
Alex Wilson 8c62cff1b7 bugdown: Clean up paths to emojis. 2016-06-30 15:48:02 -07:00
Alex Wilson 5ec29101eb Add unicode emoji support to bugdown.
Fixes half of #1011.
2016-06-30 15:48:02 -07:00
Tim Abbott ce7c7d3510 Wrap youtube_re. 2016-06-25 10:52:03 -07:00
Eklavya Sharma 598fb1ff28 zerver/lib/bugdown/fenced_code.py: Add BaseHandler.
Add a class 'BaseHandler' and make it a base class of OuterHandler,
QuoteHandler and CodeHandler.  This will help annotate some functions
and improve type checking.
2016-06-16 15:21:17 -07:00
Eklavya Sharma 142bcadb68 Annotate zerver/lib/bugdown/fenced_code.py. 2016-06-16 15:21:17 -07:00
Eklavya Sharma 9b72b7b37e Annotate zerver/lib/bugdown/codehilite.py.
Also change some string literals to unicode.
2016-06-16 15:21:17 -07:00
Eklavya Sharma 744007f33f zerver/lib/bugdown/__init__.py: Fix string types.
Change important string constants from str to text_type.
Replace str by text_type in annotations where relevant.
2016-06-16 15:21:17 -07:00
Eklavya Sharma 04f44b12ad zerver/lib/bugdown/testing_mocks.py: Fix types.
Also fix clashing annotations in zerver/lib/bugdown/__init__.py.
2016-06-16 15:21:17 -07:00
Eklavya Sharma 7db0765a18 zerver/lib/bugdown/fenced_code.py: Fix types.
Change some important string literals from str to unicode.
Annotate format_code and codehilite_conf in FencedBlockProcessor.
2016-06-16 15:21:17 -07:00
Eklavya Sharma 48c5b299b6 zerver/lib/bugdown/codehilite.py: Add type annotation.
Mypy incorrectly infers a dict's type as `Dict[str, List[object]]`.
Change that to `Dict[str, List[Any]]`.
2016-06-16 15:21:17 -07:00
Eklavya Sharma a485d63975 zerver/lib/bugdown/__init__.py: Fix non-string annotations. 2016-06-16 15:21:17 -07:00
Tim Abbott eafb91719c get_tweet_id: Fix return type to by None. 2016-06-16 15:21:01 -07:00
Eklavya Sharma 72ed1f4187 Fix python 3 pyflakes error.
Change raw unicode literal to non-raw.  A raw unicode literal is a
syntax error in python 3.  This error was detected by pyflakes.
2016-06-13 19:44:56 -07:00
Tim Abbott 2219ef7bef bugdown: Rename upload_re to upload_title_re for clarity. 2016-06-13 19:44:56 -07:00
Tim Abbott 5b1cfbc977 bugdown: Fix extraction of titles for uploaded files.
The previous code was associated with a previous version of the upload
URL naming scheme, and thus never triggered in practice.
2016-06-13 19:44:51 -07:00
Umair Khan c8d139b2b1 Fix emoji urls interaction with i18n.
When accessing emojis with relative urls we should start the urls with
a slash so that language code doesn't become part of these urls.

Fixes #1014.
2016-06-13 09:10:11 -07:00
Eklavya Sharma 8acc51218e Fix realm emoji and realm filter annotations.
Change `str` to `text_type` in annotations in zerver/models.py
related to realm emoji and realm filters.
Also fix clashing annotations in zerver/lib/bugdown/__init__.py.
2016-06-12 09:25:42 -07:00
Tim Abbott 1ca7c3378b Annotate zerver.lib.testing_mocks.
Also fix some annotations in bugdown to match.
2016-06-04 22:54:49 -07:00
Tim Abbott 6f69053911 Annotate most of the rest of bugdown. 2016-06-04 11:35:29 -07:00
Tim Abbott e89730dc8f subject_links: Remove useless RealmFilter.DoesNotExist case. 2016-06-03 23:18:39 -07:00
Tim Abbott 654bd663aa bugdown: Add annotations for a few more functions.
This resolves the issue with Typeshed #244 by making `upload_re` start
with `ur`.
2016-06-03 18:11:53 -07:00
Oren Leaffer 2916fb30cb bugdown: add some type annotations.
Had to add some "type: ignore" because the pattern used in match
doesn't affect the type returned. A fix for this issue has been pushed
to typeshed - https://github.com/python/typeshed/pull/244
2016-06-03 17:03:52 -07:00
Nathan Florea 04c71fadc6 More removal of mutable default arguments.
I've left a few that clearly aren't being passed and aren't being mutated, but
I think I've gotten the rest of them.
2016-06-03 09:17:04 -07:00
Eklavya Sharma 149938d468 Change shebangs from python2.7 to python. 2016-05-29 05:03:08 -07:00
Eklavya Sharma 0dcd8b387d Make zerver/lib/bugdown/fenced_code.py pass mypy check. 2016-05-18 17:10:17 +05:30
Tim Abbott b869be9301 style: Use 'not in' consistently rather than `not foo in`. 2016-05-09 17:00:10 -07:00
Tim Abbott 762a3188ee Fix missing whitespace after # in comments. 2016-05-02 22:10:47 -07:00
Tim Abbott 5bd94c15c7 Use camo to avoid mixed content warnings when displaying emoji. 2016-05-02 17:21:31 -07:00
Tim Abbott 65207477c4 bugdown: Annotate emoji handleMatch function. 2016-05-02 17:01:09 -07:00
Tim Abbott 4644967afc dropbox preview: Remove preview_fail.png error condition.
Since we don't have a stable way to get the Dropbox preview failure
image (and it was sorta a weird setup anyway), it seems best to just
remove the condition.
2016-04-29 15:40:52 -07:00
Tim Abbott 755695d3c0 bugdown: Add type: ignore for fenced_code import.
This is a workaround to allow us to type-check files that depend on
this.  Ideally in the future we'll fix the type errors in
fenced_code.py.
2016-04-28 12:50:47 -07:00
Tim Abbott 3e5ad69ffc Extract camo encoding to a library. 2016-04-27 22:23:40 -07:00
Varshit 4e1060076d Purge 'from typing import *' from zerver/.
This is a partial implementation of #636.
2016-04-07 14:07:07 -07:00
Tim Abbott 2059f650ab Add PEP-484 type annotations to zerver/lib/. 2016-04-03 15:40:23 -07:00
Tim Abbott 96eb81e5d5 Add documentation of Zulip's markdown implementation. 2016-04-01 09:56:59 -07:00
Eklavya Sharma 9e3c3e14f5 Partially apply Python 3 libmodernize.fixes.fix_dict_six.
Refer to #256
2016-03-19 15:52:58 -07:00
Eklavya Sharma 186efc6a6d Handle unicode properly in camo.
In Bugdown's InlineHttpsProcessor.run, hex_encoded_url was not being
decoded to utf-8. This was causing problems in python 3.

Refer to #256.
2016-03-19 15:46:07 -07:00
Tim Abbott df0d2a726d python3: Add missing utf-8 encoding/decoding in various places. 2016-03-08 09:14:15 -08:00
Tim Abbott df1670ef59 Fix various float initialization to use 0.0 instead of 0.
This is needed to type-check these values.
2016-02-03 19:29:07 -08:00
Tim Abbott 8654b57c7b bugdown: Move definition of current_node above set_text function using it. 2016-02-03 19:25:18 -08:00
Tim Abbott aad3bff193 Harden style rule for % comprehensions and fix existing errors. 2016-02-02 23:08:19 -08:00
Tim Abbott 5bacda3662 python3: Fix usage of .keys()/.values() to handle iterators.
This fixes the places where we use the result of .keys(), .items(),
and .values() that wouldn't work with an iterator to wrap them with
list().
2016-01-26 21:11:25 -08:00
Tim Abbott 6528b18ad3 Switch all urllib/urlparse usage to six.moves.urllib.
This provides Python 2+3 compatibility for our use of urllib.

Also add a test to avoid future regressions.
2016-01-26 21:09:43 -08:00
Tim Abbott f6f8f1fe36 Use new-style classes consistently for Python 3 support.
Also add the fixer for this to our list of fixers we check.
2015-11-04 08:01:52 -08:00
Tim Abbott f7878a61e1 Apply Python 3 futurize transform libmodernize.fixes.fix_xrange_six. 2015-11-01 09:35:06 -08:00
Tim Abbott 2ea0663a4a Apply Python 3 futurize transform libmodernize.fixes.fix_imports_six. 2015-11-01 09:26:16 -08:00
Tim Abbott 651b011514 Apply Python 3 futurize transform libmodernize.fixes.fix_basestring. 2015-11-01 09:26:16 -08:00
Steven Oud d5435fad1d Consistently use /usr/bin/env python2.7 in shebangs and commands. 2015-10-21 22:58:21 +00:00
Darren Worrall 8b002040e0 Correct twitter library in requirements.
This also requires updating the required version of oauthlib; previously an
appropriate version was being installed only because it was a dependency of
the wrong twitter library.

This only affects development environments and/or hand-built
installations relying on the contents of requirements.txt.

To fix existing environments, the incorrect api needs to be explicitly
removed with `pip uninstall twitter`.

Fixes #86.
2015-09-30 13:49:33 -07:00
Darren Worrall bec3c0943a Fix validation that twitter creds are present.
They are looked up as secrets which initialize to `None`, but the code
was checking for empty strings.

This, along with #80, fixes #81.
2015-09-30 09:27:37 -07:00
Tim Abbott 3770142635 Revert "Host "preview failed" image locally."
This reverts commit 39f2908a32c0276b1d87ecedc876c71dd35a9b2f.

We're not including the preview_fail.png image in the release.

(imported from commit 2de1451de2f9b1727fc3a7e64c380b71c0f2caa8)
2015-09-25 00:44:20 -07:00
David Roe edf7e732a2 ENTERPRISE => VOYAGER.
(imported from commit 4f8080b9f506a87ca40bef32e39de5218cba916a)
2015-08-21 10:33:35 -07:00
David Roe 13f3b448e5 Kill Embed.ly integration.
(imported from commit dfe42a26e57100108c3c50298041bd520e0d8b1b)
2015-08-20 19:46:48 -07:00
Waseem Daher 569abda653 Host "preview failed" image locally.
Who knows when this Cloudfront URL will stop working!

(imported from commit 39f2908a32c0276b1d87ecedc876c71dd35a9b2f)
2014-07-29 21:46:35 -07:00
Paul Ruan 12cb8ceae0 Time out when waiting for OpenGraph links that take too long.
One common place that this happens (for us) is on a local
Dropbox .dev.corp.dropbox.com instance, which can't be reached
by the Zulip servers.

This commit also:
* Fixes the test suite
* Properly previews /photos/ links

(imported from commit b4788b6236e7a9d390e1efc4673be34d9ba5e091)
2014-07-29 21:46:35 -07:00
Jason Michalski 94ec1f3741 Better previews for dropbox shmodel.
Getting previews for docs using open-graph protocol images.

(imported from commit 11a9b446eeb5786fdbf536681a100f43ad6db978)
2014-07-29 21:46:34 -07:00
Steve Howell 8f37c3f5fd Tolerate spaces after language in markdown fences.
See #2357.  We now support `~~~ .py ` with that trailing space.

Note that the test coverage is Python-side only due to
bugdown_matches_marked being set to false, since we don't yet
support language syntax on the client side.

(imported from commit ccd5fcb0eee01478d349161400103480678d7486)
2014-03-06 10:15:20 -05:00
Waseem Daher a9f01e382a Switch streams page over to use consistent buttons.
(imported from commit c4d7f658f361a759a38951349e027ac3e279fe9b)
2014-03-05 17:23:23 -05:00
Leo Franchi 05d911b913 Allow auto-linked URLs to end with a !, without including the ! in the url
(imported from commit 26fbf8d8318dc32f31f94f366750521afab94ac5)
2014-03-05 15:22:05 -05:00
Waseem Daher 243c327d0a Properly preview images from Dropbox photo albums.
Reported by jhurwitz (sorta).

(imported from commit 8ae4c5f7af86c5492842142b89be8cab1a2c3fae)
2014-02-27 20:25:35 -05:00
Jason Michalski fa285fd1a9 Use case insensitive matching when linkifying tweets
(imported from commit c8a897f05a79a2b23998ffe186c77a9f7af4ae7f)
2014-02-15 15:33:22 -05:00
Luke Faraone 89eef10d44 Match alert words inside markup.
We now will match an alert word even if it is used at the boundry of
bolding, backtick escaping, or caret quoting.

Closes trac #2186.

(imported from commit 984bc63eb621772c95a01ca5c5bfeb190767f71f)
2014-02-13 17:14:54 -05:00
Luke Faraone 4ebc4a4ef1 Expand characters allowed after an alertword.
We now support .';[ before a word and ';]! after.

Closes trac #2283.

(imported from commit b3e3cb3d8d49a8e74d0b6a108c5ec73e9d9b5f32)
2014-02-13 17:14:53 -05:00
Tim Abbott 4891da83ad Fix autolinkification in Zephyr mirrored content.
Apparently the "inline" treeprocessor is what runs the inline
patterns.  Also re-enable the rewriting-to-https support.

(imported from commit 2fde2c1f15217a784f26b16db25ee745f424f2f0)
2014-02-10 13:23:27 -05:00
Leo Franchi 8ad2ca15cd bugdown: Match up to 8 groups of nested parens
(imported from commit e3d61ec644fdb293322cab0a4449d8e1a27a513a)
2014-02-06 17:26:01 -05:00
Tim Abbott ca3430862c More correctly disable markdown processing for Zephyr mirrored messages.
(imported from commit 688eead11c329e93c649ea1633dbf56328b5587b)
2014-02-06 10:25:35 -05:00
Tim Abbott 7151af5119 Fix disabling of markdown for mit.edu.
(imported from commit 952456ddab8112bf62e6d17e29325c6d974be7a5)
2014-01-29 14:39:53 -05:00
Jason Michalski a028bda264 Allow up to three twitter previews per message
(imported from commit 566b786f9ac0582f15f8f77363f63932ee5ceec8)
2014-01-29 11:00:41 -05:00
Jessica McKellar 086eab8bb5 bugdown: support modal in-app links in messages.
(imported from commit 0e3c0ff61cf9a84bae81e37b0a6d79831dea9224)
2014-01-23 17:06:29 -05:00
Jason Michalski c30a411c10 Add custom markdown tag to render a stream subscribe button
When new streams are created we now send a message with a custom
markdown tag that renders a subscribe button.

(imported from commit 9dfba280b3b4ff4f32f6431ef9227867c8bf4b40)
2014-01-22 11:28:31 -05:00
Waseem Daher fe748f7633 Make Twitter preview look like other inline image previews.
(imported from commit ca4ff9146679a5348bf53d06e992972e9a4b628a)
2014-01-10 21:39:04 -05:00
Jason Michalski 9e822403bb Linkify twitter media links
Image and video links in the twitter API are media and need to be
handed on separately. We also include a preview image if the media link
is a to a picture.

(imported from commit 2bd00d267e51b29ad0ba681195b2bfea9b991d8c)
2014-01-10 21:39:02 -05:00
Jason Michalski 1352dccea5 Add support to linkify links and mentions on twitter
This converts links in tweets to a tags. We also convert the displayed
text to the target of the twitter short URL. Mentions are linked to the
users twitter page.

(imported from commit 192d5546a7eea82759f9ae30d82c102aed15ff71)
2014-01-10 21:39:01 -05:00
Waseem Daher 14a5c6bf66 twitter: Detect link styles more robustly.
* Deal with shorter tweet IDs
  (some old tweets don't have a full 18-character ID)
* Allow trailing slash
* Deal with old-style #! syntax
* Deal with links that link to a photo

(imported from commit 008a98c806f3b8dddd9e2f18a8f002af6932766f)
2013-12-13 18:13:23 -05:00
Kevin Mehall ebab4a59c9 Don't proxy https images through Camo on MIT
These images at least load now, but that's because Camo redirects
the browser to the origin server, so the only effect is an extra
round-trip time.

(imported from commit 0d6b9c888a5cdfaa9299272d74a085e872dfa434)
2013-12-13 17:27:01 -05:00
Leo Franchi 2c39e28ff6 Use realm filters from the database in bugdown
(imported from commit 8e12bfb1b1fe9cad694f80a189efa64b0f102e81)
2013-12-11 14:39:10 -05:00
Kevin Mehall b8a6100d2e bugdown: Open #-relative links in the same window.
Stream notifications shouldn't leave you with tons of Zulip windows.

(imported from commit e5ebfeff51f1e579d30a3a12eaf7341ef82f0449)
2013-11-25 17:34:59 -05:00
Steve Howell ace12bcab6 Support arbitrarily nested fenced quote/code blocks.
Now we can nest fenced code/quote blocks inside of quote
blocks down to arbitrary depths.  Code blocks are always leafs.
Fenced blocks start with at least three tildes or backticks,
and the clump of punctuation then becomes the terminator for
the block.  If the user ends their message without terminators,
all blocks are automatically closed.

When inside a quote block, you can start another fenced block
with any header that doesn't match the end-string of the outer
block.  (If you don't want to specify a language, then you
can change the number of backticks/tildes to avoid amiguity.)

Most of the heavy lifting happens in FencedBlockPreprocessor.run().
The parser works by pushing handlers on to a stack and popping
them off when the ends of blocks are encountered. Parents communicate
with their children by passing in a simple Python list of strings
for the child to append to.  Handlers also maintain their own
lists for their own content, and when their done() method is called,
they render their data as needed.

The handlers are objects returned by functions, and the handler
functions close on variables push, pop, and processor.  The closure
style here makes the handlers pretty tightly coupled to the outer
run() method.  If we wanted to move to a class-based style, the
tradeoff would be that the class instances would have to marshall
push/pop/processor etc., but we could test the components more
easily in isolation.

Dealing with blank lines is very fiddly inside of bugdown.

The new functionality here is captured in the test
BugdownTest.test_complexly_nested_quote().

(imported from commit 53886c8de74bdf2bbd3cef8be9de25f05bddb93c)
2013-11-21 17:13:17 -05:00
Steve Howell 0d920dae87 Extract FencedBlockPreprocessor.placeholder().
(imported from commit cf6dab74ae97a57498e1e1807c7c4f272f3f448b)
2013-11-20 16:06:53 -05:00
Steve Howell 2955cac47e Extract vars in FencedBlockPreprocessor.process_fence().
(imported from commit 092a8e5f7479f1d918219cae058b6a867743c7f9)
2013-11-20 14:30:56 -05:00
Steve Howell b3344c2f49 Move langclass var into FencedBlockPreprocessor.format_code().
(imported from commit e468a8b6a12494facb0dfa97eff90e02cd14f169)
2013-11-20 14:30:55 -05:00
Steve Howell 46838ec6d0 Simplify FencedBlockPreprocessor.format_fence().
(imported from commit 741f4f0640756d545019b6fe6850af3198b351ba)
2013-11-20 14:30:55 -05:00
Steve Howell c0f312ec29 DRY up m.group('code') in FencedBlockPreprocessor.
(imported from commit b23caae1ab63a9adb80025b628bc1a7c6a988ce0)
2013-11-20 14:30:55 -05:00
Steve Howell 6bc25d3d7b DRY up m.group('lang') in FencedBlockPreprocessor.
(imported from commit 9f9ab5604b7df711a3a369247def1277880b0b5f)
2013-11-20 14:30:55 -05:00
Steve Howell 7bb30a96ac Extract FencedBlockPreprocessor.format_fence().
(imported from commit 8a35de7df9ad956496263c82668b118910edd6b3)
2013-11-20 14:30:55 -05:00
Steve Howell 64410fae5e Extract FencedBlockPreprocessor.format_quote().
(imported from commit 59db5415d624402b37ade7fe5547b16f58ae565f)
2013-11-20 14:30:55 -05:00
Steve Howell 9d4b6d0684 Extract FencedBlockPreprocessor.format_code().
(imported from commit 3c98b9a7391a71e9c55a7b5c97abd0ee74178b44)
2013-11-20 14:30:54 -05:00