Commit Graph

257 Commits

Author SHA1 Message Date
Tim Abbott 4b709de9c9 python3: Remove six.PY2/six.PY3 checks. 2017-08-25 11:06:43 -07:00
Tim Abbott 7d49ce13b1 bugdown: Fix mypy error with empty return. 2017-08-16 11:29:12 -07:00
Tim Abbott d2288154f6 bugdown: Fully remove mentions matching @name.
Given typeahed and the fact that this only worked if the person had a
full name that didn't contain whitespace, this side effect of the
original @shortname mentionfeature that we removed was experienced by
users as a bug.

Fixes #6142.
2017-08-16 11:18:09 -07:00
digi0ps 2d92f9dc0b emoji: Change emoji image titles to use spaces instead of underscores.
Previously, the emoji images title display `🍼`.
This commit changes the title to display `baby bottle`.
2017-07-26 09:27:17 -07:00
Harshit Bansal a03ce01688 bugdown: Fix the rendering of emojis in twitter link previews.
Process the unicode emojis in twitter link previews and render them
properly. Before this we were not processing the unicode emojis in
twitter link previews and hence on the systems which don't have
fonts for displaying them they were rendered as blank boxes.

Fixes: #5427.
2017-07-24 17:32:05 -07:00
Harshit Bansal 9256ff0f7d bugdown: Rename `to_linkify` to `to_process` and add `type` field.
This commit renames list named `to_linkify` in twitter link processor
to `to_process` and adds a `type` field to each entry in it to
indicate the type of data represented by that particular entry.
2017-07-24 17:31:56 -07:00
neiljp (Neil Pilgrim) 8611a2cafa tidying: Combine multiple 'import typing' lines onto one line. 2017-07-16 16:43:21 -07:00
Aditya Bansal 73ad834129 pep8: Add compliance with rule E261 bugdown/__init__.py. 2017-07-11 11:53:33 -07:00
Harshit Bansal 7387e6fe4b bugdown/__init__.py: Close opened file objects. 2017-07-01 11:14:24 -07:00
Harshit Bansal 4cb8ac100a emoji: Fix the title and alt text for unicode emojis.
Unicode emojis when rendered should display canonical short name.
Similarly, the alt text should be of the format `:<short_name>:`.
For both of these we currently display the actual unicode symbol.
As some systems don't have the fonts necessary for displaying them
properly, they are rendered as empty square blocks. This commit also
ensures that the markup generated for emoji generated by canonical
name and by an unicode emoji is same.

Fixes: #5555.
2017-07-01 11:14:24 -07:00
Harshit Bansal fc6eedb831 bugdown: Make `unicode_emoji_regex` global. 2017-06-29 13:26:07 -07:00
Harshit Bansal dfbf8b8086 bugdown: Remove shortname mention syntax.
We used shortnames for mentioning users before we had autocomplete
feature. Since we now have autocomplete typeahead, this syntax is
no more useful and just causes problems. This commit removes the
shortname mention syntax.

Fixes: #4189.
2017-06-19 16:57:59 -04:00
Harshit Bansal 7950f82f27 realm_emoji: Make sure that a deactivated realm emoji can't be used.
A deactivated realm emoji should neither be accepted further as a
reaction nor its further occurences in a message be rendered as an
emoji. However, all the old occurences should continue to render
normally.
2017-06-15 02:59:57 -07:00
Tommy Ip 8d6af030c0 bugdown: Fix modal_link.
The `data-toggle` property prevented the new style of overlay modals
from launching, and regardless, isn't a future-proof options for how
this should work.
2017-06-03 18:41:19 -07:00
Nathan Miller 2311e169ec mypy: Various strict-optional fixes in zerver. 2017-05-26 10:10:20 -07:00
umkay ccc70445d6 mypy: Fix strict-optional errors for test files.
Fix mypy --strict-optional errors in zerver/tests
2017-05-24 12:43:28 -07:00
Konstantin Gukov dd76222a3f Fetch system bots using new get_system_bot function.
This eliminate a bunch of uninteresting calls to
get_user_profile_by_email.
2017-05-23 10:30:40 -07:00
Harshit Bansal c549dea9ac bugdown: Fix the regex used for unicode emoji matching.
The regex we were using didn't cover all the unicode blocks
to which our emojis belong. This commit fixes the regex to
include all the unicode blocks and also updates the
corresponding JS regex in marked.js.

Fixes: #3460.
2017-05-16 09:05:42 -07:00
Harshit Bansal f8824ea623 bugdown: Fix the `unicode_emoji_to_codepoint()` function.
Unicode codepoints are of minimum length 4, padded with extra
zeroes if the length is less than 4. This commit fixes the
`unicode_emoji_to_codepoint()` to ensure that the codepoint
it generates are of correct length.
2017-05-16 09:05:42 -07:00
Tim Abbott 9f7236eec1 message: Remove unused old gravatar_hash field from message dicts.
This was deprecated and replaced some 4 years ago.
2017-05-09 22:33:27 -07:00
Rohitt Vashishtha 0414ac6df5 bugdown: Convert GitHub image-preview urls to image urls.
This makes inline image previews work for links to image files' pages
on GitHub.

Fixes #4658.
2017-05-09 11:22:37 -07:00
Rohitt Vashishtha a2c1e10986 bugdown: Add domain-name to relative image-url in open graph data.
fixes #4608.
2017-05-04 16:54:10 -07:00
K.Kanakhin f13d6a18eb realm-emoji: Add realm emoji uploading instead url providing.
- Add file_name field to `RealmEmoji` model and migration.
- Add emoji upload supporting to Upload backends.
- Add uploaded file processing to emoji views.
- Use emoji source url as based for display url.
- Change emoji form for image uploading.
- Fix back-end tests.
- Fix front-end tests.
- Add tests for emoji uploading.

Fixes #1134
2017-05-01 14:50:20 -07:00
Tim Abbott 85805cf380 bugdown: Simplify logic for constructing unicode_emoji_list. 2017-04-30 16:40:58 -07:00
Tim Abbott 8b3be3cd3a bugdown: Pass a codepoint, not a URL, to make_emoji. 2017-04-30 16:40:58 -07:00
Tim Abbott 565370a365 bugdown: Create make_realm_emoji function.
Currently, this is a duplicate of the existing make_emoji, but the
plan is to change that in some upcoming refactoring.
2017-04-30 16:40:58 -07:00
Tim Abbott 18bd566958 bugdown: Extract unicode_emoji_to_codepoint. 2017-04-30 16:40:58 -07:00
Tim Abbott e0a8f383b3 bugdown: Remove unnecessary first argument to make_emoji. 2017-04-30 16:40:58 -07:00
Tim Abbott cfeff245c1 emoji: Manage Zulip emoji as a custom/extra realm emoji.
This is prerequisite for converting our emoji system to work off of
iamcal sprite sheets.

Also, fixes #4308.
2017-04-30 16:40:31 -07:00
Sampriti Panda 0e3d694df8 bugdown: Use queue for processing of links in image previews.
Earlier, a stack was being used to go through the message and search
for links.  Because of this, in some cases the images were added to
the preview in reverse.  Using a queue will keep the image previews in
the same order as they appeared in the message.

Fixes #4453.
2017-04-17 20:48:16 -07:00
hollywoodno 401fae30f1 markdown: Render backend second level markdown.
By default, Python markdown tab length for indents is 4 spaces, which
require using 4 spaces or a tab to create nested elements. This
modifies that setting to specify 2-space indentation for nesting
elements only.

Modified significantly by tabbott to limit the change to just list
indentation.

Fixes #4252.
2017-04-07 16:14:57 -07:00
Yago González 34a9e1ae11 markdown: Add TeX typesetting support.
Co-authored-by: Reid Barton <rwbarton@gmail.com>

Fixes #2056.
2017-03-21 16:40:00 -07:00
Ayush Jain bddcfb1c96 Add realm-level settings to control inline image and url preview.
This gives users more control in case they don't want previews,
especially for the "previews of linked websites" feature.

Fixes: #2640.
2017-03-21 15:46:17 -07:00
Cynthia Lin 589551b6ee settings: Change "Custom alert words" to "Alert words". 2017-03-15 11:25:37 -07:00
sinwar 6f0564e9f4 python: Fix remaining bare excepts in codebase.
Fixes #2862.
2017-03-05 16:17:04 -08:00
Tim Abbott 464928fdcc bugdown: Fix mypy error with python 3 coverage. 2017-03-05 00:53:26 -08:00
Tim Abbott ff9f827c9a coverage: Don't require coverage in six.PY2/PY3 branches. 2017-03-04 22:51:35 -08:00
Raghav Jajodia a3a03bd6a5 mypy: Added Dict, List and Set imports.
Fixed mypy errors associated with the upgrade.
2017-03-04 14:33:44 -08:00
Rishi Gupta 28d3af0965 Fix several new errors caught by mypy 0.501.
Clear out a bunch of easy to review errors, so we can focus on the more
complicated ones.
2017-03-03 14:12:52 -08:00
Igor Tokarev 31dff09efa Support email changes for !avatar syntax.
Significantly modified by tabbott to avoid calling
get_user_profile_by_email in bugdown, and have 100% test coverage of
the views code.

Fixes #2041.
2017-02-28 21:56:04 -08:00
Harshit Bansal 40d137d621 bugdown: Change rendered emoji image to `unicode/<codepoint>.png.`
Use `name_to_codepoint.json` file (and the similar structure in
emoji_codes.js) to map emoji names directly to codepoints and change
the rendered emoji image to `unicode/<codepoint.png>` rather than
`<emoji_name>.png`.

Fixes: #3539.
2017-02-26 18:30:15 -08:00
Tim Abbott 6d00d4d2b1 mypy: Clean bugdown use of Optional. 2017-02-10 23:53:44 -08:00
Tim Abbott e746868375 mypy: Fix optional typing usage in rendering code path. 2017-02-10 23:53:44 -08:00
Tim Abbott 4e20b622ad bugdown: Remove unused height argument to add_a. 2017-02-10 23:53:44 -08:00
Steve Howell 6e2189f134 bugdown: Disable link previews when bots send messages.
We do not use `get_link_embed_data` for messsages sent by
bots, as bots often repeat the same URL over and over again
and are generally either text-focused or have their own
mechanisms to provide preview content.

Fixes #2968.
2017-02-03 17:08:23 -08:00
Steve Howell 709493cd75 Pass in sent_by_bot flag to bugdown parser.
We will use this flag to suppress certain url previews
for bots.
2017-02-03 17:07:38 -08:00
Steve Howell 5b8cc03c61 bugdown: Restore data-user-email to user mention spans.
(The commit q7ef4e40258280e202325c9295579c93fb948b replaced
data-user-email with data-user-id, but we still need to
support data-user-email for old clients like non-updated
androids and we still want to start the migration forward
to data-user-id.)
2017-02-02 12:32:18 -08:00
Tim Abbott 4e171ce787 lint: Clean up E126 PEP-8 rule. 2017-01-23 22:06:13 -08:00
Tim Abbott 9cc83f87fc lint: Clean up E241 PEP-8 rule. 2017-01-23 21:21:14 -08:00
Tim Abbott 6f0d2a9445 lint: Clean up E115 PEP-8 rule. 2017-01-23 20:55:37 -08:00
Steve Howell 1137ef4e40 Mark up mention buttons with data-user-id on the backend.
We use data-user-id now in the buttons for at-mentions when
we render messages.  A previous commit already made the front
end support this new style.
2017-01-21 21:45:12 -08:00
Tim Abbott 5d52f1ec17 bugdown: Move realm_filters_key logic out of callers.
This gets rid of the confusing duplicate realm_filters_key and
message_realm arguments that previously were passed to bugdown.
2017-01-21 21:37:57 -08:00
Sampriti Panda 34a4a1378d bugdown: Use specified realm, not sender realm, for rendering.
This changes bugdown to use the realm passed in by the caller (if any)
for rendering, fixing a problem where bots such as the notification
bot would have their messages rendering using the admin realm's
settings, not the settings of the realm their messages are being sent
into.

Also adds a test for the notification bot case.

Fixes #3215.
2017-01-21 21:37:57 -08:00
Tim Abbott 99c5563bc6 internal_send_message: Make realm argument mandatory.
A lot of care has been taken to ensure we're using the realm that the
message is being sent into, not the realm of the sender, to correctly
handle the logic for cross-realm bot users such as the notifications
bot.
2017-01-21 21:37:30 -08:00
Tim Abbott 19b89eb050 bugdown: Rename realm_id to realm_filters_key.
This should substantially improve the clarity of the code, since
inside bugdown, this is only being used as a hash key that happens to
usually be a realm ID, not used as a Realm ID.
2017-01-16 21:48:55 -08:00
Tim Abbott 3f8d4193da lint: Fix % comprehensions being used without a tuple. 2017-01-09 11:45:11 -08:00
Brock Whittaker b976e179e6 css: Restyle Open-Graph Links.
This styles open-graph links to be cleaner and smaller.
2017-01-05 16:15:01 -08:00
Rishi Gupta caccf8ee17 Remove models.get_realm(domain). 2017-01-03 16:46:15 -08:00
Rishi Gupta c6e12e74be Change domain to realm_id in bugdown and realm filter dicts and caches. 2017-01-03 16:25:20 -08:00
Tim Abbott b36e94c954 bugdown: Blacklist linkifying the .md TLD.
Fixes #2065.
2016-12-29 18:17:50 -08:00
Tim Abbott 39f0ffdedd Move static/third/gemoji to static/generated/emoji.
Fixes #1153.
2016-12-27 20:16:23 -08:00
Robert Hönig 0917493588 mypy: Convert zerver/lib to use typing.Text. 2016-12-25 10:33:45 -08:00
reallyly 94b28e0afb pep8: Fix E128 violations.
With some line-wrapping tweaks by tabbott.
2016-12-13 10:50:14 -08:00
Igor Tokarev c93f1d4eda Add oembed/Open Graph/Meta tags data retrieval from inline links.
This change adds support for displaying inline open graph previews for
links posted into Zulip.

It is designed to interact correctly with message editing.

This adds the new settings.INLINE_URL_EMBED_PREVIEW setting to control
whether this feature is enabled.

By default, this setting is currently disabled, so that we can burn it
in for a bit before it impacts users more broadly.

Eventually, we may want to make this manageable via a (set of?)
per-realm settings.  E.g. I can imagine a realm wanting to be able to
enable/disable it for certain URLs.
2016-12-07 17:40:18 -08:00
Mikebarson 690d72d35f mypy: Convert more zerver/lib files to typing.Text.
This commit touches files in zerver/lib/bugdown/ and
zerver/lib/webhooks.
2016-12-05 21:17:24 -08:00
bulat22101 a6f91064a2 pep8: Fix E129 violations 2016-12-03 10:56:36 -08:00
Alex Huang 007b693cc7 pep8: Fix E131. 2016-12-01 23:16:47 -08:00
AZtheAsian c23f9e0df7 pep8: Fix E111 violations 2016-12-01 14:18:11 -08:00
Rafid Aslam c5316b4002 lint: Fix E127 pep8 violations.
Fix pep8: E127 continuation line over-indented for visual indent
style issue.
2016-12-01 10:23:55 -08:00
Bickio 6b0df43463 pep8: Fix E125. 2016-11-30 20:03:29 -08:00
Rafid Aslam 41bd88d5ed pep8: Fix E301 pep8 violations.
Fix "E301: expected (1 or 2) blank line" pep8 violations.
2016-11-29 08:51:44 -08:00
Rafid Aslam 7a2282986a pep8: Fix E225 pep8 violations. 2016-11-28 15:21:15 -08:00
Vladislav Manchev d7e1e4a2c0 Add initial implementation of custom realm filters.
This PR was abandoned by Vladislav and then substantially modified by
Igor Tokarev and Tim Abbott to complete it and fix a number of bugs.

Fixes #544.
2016-11-17 17:11:25 -08:00
hackerkid cdb716aa63 bugdown: Add explanation for unicode-emoji JS regex. 2016-11-17 10:44:07 -08:00
Umair Khan 29254142f6 Django 1.10: Run maybe_update_realm_filter when it's needed. 2016-11-15 09:34:08 -08:00
hackerkid 67120d7a9d Support backend unicode-emoji rendering without whitespace. 2016-11-14 21:02:18 -08:00
hackerkid 8c83b34ad5 Support backend emoji rendering without whitespace. 2016-11-11 16:18:57 -08:00
Igor Tokarev 852bc6b491 bugdown: Re-enable support for italics in bugdown.
Fixes: #1103.
2016-11-10 19:14:40 -08:00
Meena Rajan 30acb23314 bugdown: Add support for strikethrough in markdown processor.
[Tweaked to move tests to bugdown_data.json, add additional tests, and
add frontend processor support by tabbott]
2016-11-07 22:26:38 -08:00
Kevin Chen 6107c877e8 bugdown: Add option to support "file:///" as hyperlink.
This contains contributions from Tim Abbott and Igor Tokarev.

Fixes #380.
2016-11-06 22:30:05 -08:00
paxapy 1d1c41551b bugdown: Auto-linkify #StreamName. 2016-11-01 19:59:21 -07:00
paxapy d965ff16fd bugdown: refactor AutoLink to use VerbosePattern.
This makes it possible to reuse the verbose pattern logic for other
regexs as well.
2016-11-01 19:50:59 -07:00
umkay 1d077d1a4f bugdown: Remove safe_mode dependencies.
Now that we have updated python-markdown, we remove the deprecated
safe_mode. We used safe_mode to escape raw html, so now instead we
pass in an EscapeHtml markdown extension to the markdown engine.

See https://pythonhosted.org/Markdown/release-2.6.html for details on
the deprecation.

Fixes: #2037 (also addresses the remaining piece of #2043).
2016-10-25 15:22:18 -07:00
K.Kanakhin 7bc0c7f0d5 bugdown: Fix patterns ordering in Markdown extension.
- Put 'modal_link' and 'link' inline patterns after avatar inline pattern
  to correct applying them to parsed text.

Fixes:  #1654
2016-10-20 11:18:41 -07:00
Brock Whittaker 77233c7e3b bugdown: Distinguish between YouTube Videos and Image Previews.
This distinguishes between YouTube Videos and Image Previews by adding
a particular “youtube-video” class to the preview along with changing
the title to the video ID rather than the link. This serves to allow
the lightbox to ID when a lightbox preview should be treated like a
YouTube video rather than an image preview.

This also modifies the tests in bug down to expect a youtube-video class
along with the title to just be the video ID on YouTube rather than the
entire URL link.
2016-10-18 14:45:47 -07:00
Mohsen Ibrahim e9899e4dc9 bugdown: Fix some Python-Markdown deprecation warnings.
Python-Markdown changed how one is supposed to declare extensions
being used.
2016-10-17 23:13:41 -07:00
Tim Abbott e37b6488b8 Bugdown: Remove checked-in CodeHilite extension.
The changes that required us to fork this extension had been merged
into upstream CodeHilite, so we can remove it and switch to using the
version that comes with python-markdown.
2016-10-16 00:40:08 -07:00
umkay c028815385 bugdown: Update python-markdown to version 2.6.7.
This updates Bugdown to reflect the changes in the updated
markdown. In particular, we now pass a default config object in the
__init__ for the Bugdown extension, update the make_md_engine function
to take kwargs as opposed to a config list, and have UListProcessor
inherit from ulist as opposed to olist (which no longer works).

We update the (forked from upstream) fenced_code extension's
makeExtension to take args and kwargs, and update
FencedBlockPreprocessor __init__ method with updated Codehilite
arguments.

We update the (forked from upstream) Codehilite extension to
mirror the logic with the latest upstream Codehilite:
	Add parse_hl_lines function
	update makeExtension to take args and kwarfs instead of config
	list
	Add regex for highlight lines
	use linenums instead of linenos
	use get_formatter_by_name instead of HtmlFormatter
	user get_lexer_by_name instead of TextLexer
	add hl_lines and use_pygments arguments to the codehlite
	constructor
2016-10-15 23:38:32 -07:00
Eklavya Sharma 41587edd2e bugdown: Change parameter name.
Change the parameter name of some functions from 'md' to 'content',
since the name 'md' seems to be the reason why this parameter was
wrongly annotated.
2016-10-11 11:31:01 -07:00
Eklavya Sharma f35d0f1c75 bugdown: Fix type annotations.
In some functions the first parameter was wrongly annotated as
`markdown.Markdown`. Change that to `text_type`.
2016-10-11 11:31:01 -07:00
Tim Abbott 22fd7ba02a avatar: Move avatar hash computations to their own file. 2016-10-02 21:19:10 -07:00
Brock Whittaker 98937ac539 bugdown: Set "title" attribute to actual image title when available.
This sets the “title” attribute on the image to the actual title of
the image specified by the user in their markdown, rather than just
the URL of the full link to it.
2016-09-27 17:34:05 -07:00
Steve Howell 40b18094ec alert words: Refactor alert word detection.
The bugdown parser no longer has a concept of which users need which
alert words, since it can't really do anything actionable with that info
from a rendering standpoint.

Instead, our calling code passes in a set of search words to the parser.
The parser returns the list of words it finds in the message.
Then the model method builds up the list of user ids that should be
flagged as having alert words in the message.

This refactoring is a little more involved than I'd like, but there are
still some circular dependency issues with rendering code, so I need to
pass in the rather complicated realm_alert_words data structure all the way
from the action through the model to the renderer.

This change shouldn't change the overall behavior of the system, except
that it does remove some duplicate regex checks that were occurring when
multiple users may have had the same alert word.
2016-09-23 15:21:17 -07:00
Steve Howell a036a72db6 bugdown: Raise BugdownRenderingException.
We now raise an exception in bugdown.do_convert() if rendering
fails, to avoid silent failures, and then calling code can convert
the exception to a JsonableError.
2016-09-16 10:01:55 -07:00
Steve Howell 152cebf33d bugdown: Extract log_bugdown_error().
(It makes it easier to mock out the logging.)
2016-09-16 10:01:54 -07:00
umkay 7eb6924841 Wrap dict view object in list for python3 compatibility.
Deleting items from a dictionary being iterated through directly
doesn't work properly in Python 3.
2016-09-13 21:21:45 -07:00
Taranjeet Singh 5e5ad9de0c bugdown/__init__.py: Fix line with length greater than 120.
This updates file zerver/lib/bugdown/__init__.py.
2016-08-19 11:56:43 -07:00
Tim Abbott e7aac717e8 Fix markdown ordering bugs with some custom syntax.
!avatar, !modal_link, !gravatar, etc. were incorrectly being processed
before the escape character for code blocks.

While we're at it, we add tests for these special syntaxes.
2016-08-18 14:47:37 -07:00
Tim Abbott 70543e059a bugdown: Remove hardcoding of mit.edu for zephyr_mirror realm config. 2016-07-26 20:30:12 -07:00
Eklavya Sharma fb1d00fc6c Upgrade python-twitter to latest version.
Fixes #1145.
2016-07-17 10:23:25 -07:00