Commit Graph

361 Commits

Author SHA1 Message Date
rht 9161f8c39b zerver/lib: Remove u prefix from strings. 2018-02-05 12:12:58 -08:00
Eeshan Garg fe98a59f5c bugdown/api_code_examples: Add macro for admin zuliprc lines.
Some of our code examples can only be run with administrator
credentials (such as create-user). Thus, the Markdown extension
for generating code examples should have an option to include
the lines that recommend using an admin zuliprc instead of a
non-admin one.
2018-01-31 07:30:54 -05:00
Eeshan Garg c158869096 Rename tools/lib/api_tests.py to zerver/lib/api_test_helpers.py.
Now that the Markdown extension defined in
zerver/lib/bugdown/api_generate_examples depended on code in the
tools/lib/* directory, it caused the production tests to fail since
the tools/ directory wouldn't exist in a production environment.
2018-01-31 07:30:54 -05:00
Eeshan Garg c89550cfa0 api docs: Add Markdown extension for generating examples/fixtures.
This commit adds a Markdown extension that allows the following
syntax,

{generate_code_example|<md_file_name>|<fixture or method>}

to generate code examples and fixtures found in tools/lib/api_tests.py
and templates/zerver/api/fixtures.json, respectively.
2018-01-31 07:30:54 -05:00
rht ee06da9350 bugdown: Linkify Bitcoin URI. 2018-01-19 06:07:43 -05:00
Shreyansh Dwivedi b0fb7aa6b2 bugdown: Add preview for vimeo videos.
This also amends a commit from Brock Whittaker <brock@zulipchat.com>
that merges two separate functions for YouTube videos and Vimeo videos
into a generic video recall function.

Fixes #7550.
2018-01-17 09:11:43 -05:00
Andy Perez c209069fcd bugdown: Improve image inlining logic.
Fix #7537
2018-01-09 16:43:22 -05:00
Eeshan Garg 89f622937d api docs: Generate Arguments table from JSON.
This commit does the following:
* Move the Arguments table data from stream-message.md and
  private-message.md to a JSON file.
* Add a Markdown extension that allows one to include and render
  a table from a JSON file like so:
    {generate_arguments_table|arguments.json|private-stream.md}
* Use Bootstrap's .table class to format the table instead of
  relying on custom CSS.
2017-12-29 17:03:36 -05:00
akashnimare ef51a39552 markdown: Update regex for strikethrough.
This uses the correct regex for strikethrough. Also, added
a test to make sure that strikethrough works when it contains
link with whitespace.

Fixes #7596.
2017-12-28 15:59:04 -05:00
Greg Price b15231dfc2 logging: Rename AdminZulipHandler to AdminNotifyHandler.
This name hasn't been right since f7f2ec0ac back in 2013; this handler
sends the log record to a queue, whose consumer will not only maybe
send a Zulip message but definitely send an email.  I found this
pretty confusing when I first worked on this logging code and was
looking for how exception emails got sent; so now that I see exactly
what's actually happening here, fix it.
2017-12-22 11:52:13 -05:00
Tommy Ip 9ea7b05cfe bugdown: Rewrite markdown link to relative link conditionally. 2017-12-22 08:10:42 -05:00
Tommy Ip efe2d497f4 bugdown: Avoid matching uri with regex. 2017-12-22 08:10:42 -05:00
Tommy Ip 9ce797d68c bugdown: Extract rewrite_if_relative_link function. 2017-12-22 08:10:42 -05:00
Andy Perez f0190333b8 bugdown: Remove whitespace strip from inline code.
Fixes #4507
2017-12-14 07:48:10 -05:00
Weronika Grzybowska b54f78968f markdown: Add automatic numbering of ordered lists.
Adds a markdown preprocessor that finds ordered lists where all items
use the same number and change them to be in normal increasing order,
starting with that number.

Fixes #5159.
2017-12-12 18:35:38 -06:00
Xavier Cooney a831acd97f mypy: Use Python 3 type syntax in bugdown/__init__.py. 2017-12-08 18:34:26 -08:00
rht ee546a33a3 zerver/lib: Use python 3 syntax for typing.
Edited by tabbott to improve various line-wrapping decisions.
2017-11-28 17:15:14 -08:00
Rohitt Vashishtha 6734b00c0b markdown: Hide URL if message is only an image.
Hides URL if the message content == image url so that sending gifs or
images feels less cluttered. Uses the url_to_a() function to generate
the expected url string for matching.

Fixes #7324.
2017-11-27 13:30:18 -08:00
Andy Perez 0449dc8b8a bugdown: Fix handling of nested fenced math blocks.
Change any fence delimiters (backticks and tildes) to be
interpreted literally inside of math fenced_code blocks.

Fixes #5190.
2017-11-22 12:19:43 -08:00
rht 3f4bf2d22f zerver/lib: Use python 3 syntax for typing.
Extracted from a larger commit by tabbott because these changes will
not create significant merge conflicts.
2017-11-21 20:56:40 -08:00
Andy Perez 83f563aeff bugdown: Properly parse empty blockquotes.
The character ">" now only starts a blockquote if the resulting
blockquote would be non-empty.  Thus, by itself, ">" is now
interpreted literally by bugdown, fixing #687.  The message
with contents consisting of ">>>" is now parsed as a doubly
(not triply) nested blockquote with contents ">".  Properly
formed blockquotes have identical behavior as before, but now
bugdown can no longer produce empty blockquotes as output.

Fixes #2886, #687.
2017-11-17 11:41:44 -08:00
David Rosa Tamsen 7072fa5b37 docs: Reorganize developer docs to improve navigation.
This commit helps reduce clutter on the navigation sidebar.
Creates new directories and moves relevant files into them.
Modifies index.rst, symlinks, and image paths accordingly.

This commit also enables expandable/collapsible navigation items,
renames files in docs/development and docs/production,
modifies /tools/test-documentation so that it overrides a theme setting,
Also updates links to other docs, file paths in the codebase that point
to developer documents, and files that should be excluded from lint tests.

Note that this commit does not update direct links to
zulip.readthedocs.io in the codebase; those will be resolved in an
upcoming follow-up commit (it'll be easier to verify all the links
once this is merged and ReadTheDocs is updated).

Fixes #5265.
2017-11-16 09:45:08 -08:00
Harshit Bansal 3b95a6d2d9 bugdown: Fix a bug in the updation logic of markdown engines.
While fixing an issue related to email gateway messages not getting
rendered properly, I unknowingly introduced a bug in the markdown
engine updation code. This commit fixes it. The issue was that for
a realm having email gateway setup, updation of realm filters would
lead to the updation of only one of the markdown engines not both.
2017-11-15 22:47:11 -08:00
Harshit Bansal b849c5e551 bugdown: Remove an unnecessary check in `do_convert()`. 2017-11-15 22:47:11 -08:00
Harshit Bansal 5e5bb02e39 bugdown: Remove `make_realm_filters()`.
This commit removes `make_realm_filters()` and merges its functionality
in `make_md_engine()` and `maybe_update_markdown_engines()`.
2017-11-15 22:47:11 -08:00
Harshit Bansal f9f9e92443 bugdown: Correct the descriptions of `Bugdown` config. 2017-11-15 22:47:11 -08:00
Harshit Bansal e615a2ab4f bugdown: Simplify the "opts" parameter of `make_md_engine()`. 2017-11-15 22:47:11 -08:00
Harshit Bansal e3f2f67d0e bugdown: Rename `maybe_update_realm_filters()`.
This commit renames `maybe_update_realm_filters()` to
`maybe_update_markdown_engines()` which sounds closer
to the semantics.
2017-11-15 22:47:11 -08:00
Andy Perez 4d51a6cfac bugdown: Fix $$ whitespace handling.
The intended use of $$ is for inline expressions, not for multiline
ones; ```math is an acceptable alternative for the latter.  Hence,
the $$-syntax for inline TeX no longer permits newlines within it.

This was also necessary for the next change to be sensible; namely
allowing for spaces around both $$ when crafting inline TeX instead of
forcing everything to be crammed together, e.g. $$x=7$$.  In order to
avoid uninentionally creating inline expressions, the opening and
closing $$'s of an inline expression must now both exactly consist of
two dollar signs, no more and no less.

Fixes: #6488.
2017-11-10 16:54:27 -08:00
rht 8e87eb4084 zerver/lib/bugdown: Text-wrap long lines exceeding 110. 2017-11-10 16:29:32 -08:00
Umair Khan 18aecb82d7 user-groups: Get members from membership. 2017-11-09 16:18:40 -08:00
Tommy Ip bcd575d8de bugdown: Rewrite links to local server to be relative links.
Fixes #7247
2017-11-09 15:59:31 -08:00
Steve Howell 531a3ee9de refactor: Fix parse.quote(force_str(s)) calls in bugdown. 2017-11-09 10:43:20 -08:00
Steve Howell 17cfead19a Remove force_text() in twitter_text(). 2017-11-09 10:43:20 -08:00
Steve Howell 585829f665 Remove force_text() in list_of_tlds(). 2017-11-09 10:43:19 -08:00
Steve Howell 786e90d774 Removed unused imports of force_bytes. 2017-11-09 10:32:14 -08:00
Steve Howell a77cdaa527 Remove force_text() in get_url_data(). 2017-11-09 10:32:14 -08:00
Harshit Bansal 65838bb825 email_gateway: Disable code block processor for email gateway.
Generally emails are not written with markdown in mind and hence
sometimes render in strange ways. This commit fixes a particular
issue that was causing whitespace before paragraphs to be treated
as code block due to which email content was being rendered in a
box that scrolls in right direction a lot.

Fixes: #7045.
2017-11-09 09:56:35 -08:00
derAnfaenger 19bc55aa45 Fix various typos.
The typos and their corrections were found with the
aid of https://github.com/lucasdemarchi/codespell.
2017-11-09 16:26:38 +01:00
rht 8242c15b48 zerver/lib: Remove unused imports (F401). 2017-11-07 16:37:13 -08:00
rht 01eb22abd9 refactor: Remove all the remaining six.moves import. 2017-11-07 10:51:44 -08:00
rht 19bd335cbb Change urllib import to be Python 3-specific. 2017-11-07 10:46:42 -08:00
derAnfaenger 0709033858 bugdown: Fetch complete data for tweet.
This makes an extended instead of a normal query,
resulting in the full text of a tweet being fetched
instead of just a truncated version.
2017-11-07 09:13:39 -08:00
rht e311842a1b zerver/lib: Remove inheritance from object. 2017-11-06 08:53:48 -08:00
neiljp (Neil Pilgrim) b693f32c0b mypy: Add Dict parameters for dropbox_image return type.
Also a note re using TypedDict in future, as the Dict looks constrained.
2017-11-04 19:47:44 -07:00
rht fef7d6ba09 zerver/lib: Remove u prefix from strings.
License: Apache-2.0
Signed-off-by: rht <rhtbot@protonmail.com>
2017-11-03 15:34:37 -07:00
Umair Khan 636046aec9 user-groups: Add basic backend for UserGroup model.
This adds the data model and bugdown support for the new UserGroup
mention feature.

Before it'll be fully operational, we'll still need:
* A backend API for making these.
* A UI for interacting with that API.
* Typeahead on the frontend.
* CSS to make them look pretty and see who's in them.
2017-10-31 15:16:14 -07:00
rht c4fcff7178 refactor: Replace super(.*self) with Python 3-specific super().
We change all the instances except for the `test_helpers.py`
TimeTrackingCursor monkey-patching, which actually needs to specify
the base class.
2017-10-30 14:30:25 -07:00
Tim Abbott 6c242379f4 bugdown: Remove unused caching imports. 2017-10-28 09:02:36 -07:00
Steve Howell 65b6676844 minor: Remove unused import (internal_send_message). 2017-10-27 10:48:11 -07:00
Steve Howell 87b48f722d Add MentionData.get_user_ids().
This prepares us to take advantage of the user_ids
in subsequent commits related to `do_send_messages()`.
2017-10-26 22:16:47 -07:00
Steve Howell 8ac26dfb9b refactor: Introduce bugdown.MentionData class.
We now have a MentionData class that encapsulates
the users who are possibly mentioned in a message.

Not that the rendering code may not keep all the mentions,
since things like backticks will suppress the mention.

We populate this now in do_send_messages, so that we can use
the info earlier in the message-sending process.  This info
now gets passed down the call stack as an optional parameter.

Note that bugdown.convert() still populates the data when its
callers decline to pass in a MentionData object.

This is mostly a preparatory commit, as we don't take advantage
of the data yet in do_send_messages.
2017-10-26 22:16:47 -07:00
Tim Abbott be619fe881 lint: Wrap many very long lines in the Python codebase.
This decreases the maximum line length in our Python codebase to 130.
2017-10-26 17:31:58 -07:00
rht 691598a88b py3: Remove "from six.moves import range".
This is no longer required, since in Python 3, this is what the range
built-in does.
2017-10-17 23:28:14 -07:00
Tim Abbott 1d314c3bf4 bugdown: Remove ERROR_BOT markdown rendering notices.
Nobody has used this feature in years, and it causes certain types of
markdown issues in development to completely DoS the development
environment by making it possible for the "Bugdown timeout" exception
handler to timeout in bugdown.

Since we already send an email to the server administrators, there's
no need to replace this feature with anything.
2017-10-12 17:45:33 -07:00
Steve Howell e0bc1b114e Make sure mentions refer only to active users.
An active user can share the same full name as a deactivated
user.  We now only allow mention syntax to find users who are
activated.

Fixed #6978
2017-10-12 17:11:36 -07:00
Tim Abbott b3b5d5b7cd report: Avoid sending raw message content in error reporting.
This fixes a violation of Zulip's privacy policies (that error
reporting never contain message content) in the previous commit.
2017-10-11 17:44:05 -07:00
Harshit Bansal 5b5bcce098 emoji: Migrate bugdown emoji to use sprite sheets.
This commit switches to use sprite sheets for rendering emojis
in all the remaining places, i.e., message bodies and composebox
typeahead. This commit also includes some changes to notifications.py
file so that the spans used for rendering emojis can be converted
to corresponding image tags so that we don't break the emoji rendering
in missed message emails since we can't use sprite sheets there.

As part of switching the bugdown system to use sprite sheets, we need
to switch the name_to_codepoint mappings to match the new sprite
sheets.  This has the side effect of fixing a bunch of emoji like
numbers and flag emoji in the emoji pickers.

Fixes: #3895.
Fixes: #3972.
2017-09-29 11:14:34 -07:00
rht 9f944d9449 zerver/lib: Remove from __future__ import *. 2017-09-28 10:14:03 -07:00
rht 035ed93111 zerver/lib: remove `import six`. 2017-09-27 19:10:28 -07:00
rht f43e54d352 zerver/lib: Remove absolute_import. 2017-09-27 10:00:39 -07:00
Juliana Bacelar 928dd06cc8 linter: Add lint rule banning 'import os.path' 2017-09-22 10:32:21 -07:00
Tim Abbott f1a0f23d23 bugdown: Document the mention fetches code block. 2017-09-15 01:21:04 -07:00
Steve Howell 7159f34bed Avoid getting realm emoji unless there is emoji syntax. 2017-09-15 01:09:08 -07:00
Steve Howell 8a941d03ae Avoid querying streams unless there are stream links. 2017-09-15 01:09:08 -07:00
Steve Howell c3032a7fe8 Avoid looking up emails when rendering messages.
We now fetch email -> id mappings for messages ONLY if
it potentially uses the !avatar(foo@example.com) syntax.
2017-09-15 01:09:08 -07:00
Steve Howell 4e7fce60ee Add possible_mentions() to speed up rendering.
We now triage message content for possible mentions before
going to the cache/DB to get name info.  This will create an
extra data hop for messages with mentions, but it will save
a fairly expensive cache lookup for most messages.  (This will
be especially helpful for large realms.)

[Note that we need a subsequent commit to actually make the speedup
happen here, since avatars also cause us to look up all users in
the realm.]
2017-09-15 01:09:08 -07:00
Steve Howell 6a625eef66 Clean up model imports for bugdown/__init__.py.
We broke some circular dependencies a while back, so we can
move a bunch of imports to the top of the file.
2017-09-15 01:09:08 -07:00
Tim Abbott e5da9966c2 bugdown: Remove now-unnecessary short_names.
This field hasn't been used since we removed the related mention
syntax.
2017-08-27 08:45:02 -07:00
Tim Abbott 4b709de9c9 python3: Remove six.PY2/six.PY3 checks. 2017-08-25 11:06:43 -07:00
Tim Abbott 7d49ce13b1 bugdown: Fix mypy error with empty return. 2017-08-16 11:29:12 -07:00
Tim Abbott d2288154f6 bugdown: Fully remove mentions matching @name.
Given typeahed and the fact that this only worked if the person had a
full name that didn't contain whitespace, this side effect of the
original @shortname mentionfeature that we removed was experienced by
users as a bug.

Fixes #6142.
2017-08-16 11:18:09 -07:00
digi0ps 2d92f9dc0b emoji: Change emoji image titles to use spaces instead of underscores.
Previously, the emoji images title display `🍼`.
This commit changes the title to display `baby bottle`.
2017-07-26 09:27:17 -07:00
Harshit Bansal a03ce01688 bugdown: Fix the rendering of emojis in twitter link previews.
Process the unicode emojis in twitter link previews and render them
properly. Before this we were not processing the unicode emojis in
twitter link previews and hence on the systems which don't have
fonts for displaying them they were rendered as blank boxes.

Fixes: #5427.
2017-07-24 17:32:05 -07:00
Harshit Bansal 9256ff0f7d bugdown: Rename `to_linkify` to `to_process` and add `type` field.
This commit renames list named `to_linkify` in twitter link processor
to `to_process` and adds a `type` field to each entry in it to
indicate the type of data represented by that particular entry.
2017-07-24 17:31:56 -07:00
neiljp (Neil Pilgrim) 8611a2cafa tidying: Combine multiple 'import typing' lines onto one line. 2017-07-16 16:43:21 -07:00
Aditya Bansal 73ad834129 pep8: Add compliance with rule E261 bugdown/__init__.py. 2017-07-11 11:53:33 -07:00
Harshit Bansal 7387e6fe4b bugdown/__init__.py: Close opened file objects. 2017-07-01 11:14:24 -07:00
Harshit Bansal 4cb8ac100a emoji: Fix the title and alt text for unicode emojis.
Unicode emojis when rendered should display canonical short name.
Similarly, the alt text should be of the format `:<short_name>:`.
For both of these we currently display the actual unicode symbol.
As some systems don't have the fonts necessary for displaying them
properly, they are rendered as empty square blocks. This commit also
ensures that the markup generated for emoji generated by canonical
name and by an unicode emoji is same.

Fixes: #5555.
2017-07-01 11:14:24 -07:00
Harshit Bansal fc6eedb831 bugdown: Make `unicode_emoji_regex` global. 2017-06-29 13:26:07 -07:00
Harshit Bansal dfbf8b8086 bugdown: Remove shortname mention syntax.
We used shortnames for mentioning users before we had autocomplete
feature. Since we now have autocomplete typeahead, this syntax is
no more useful and just causes problems. This commit removes the
shortname mention syntax.

Fixes: #4189.
2017-06-19 16:57:59 -04:00
Cory Lynch 6255636923 Support special characters for code block languages.
Fixes #5412
2017-06-19 06:53:25 -04:00
Harshit Bansal 7950f82f27 realm_emoji: Make sure that a deactivated realm emoji can't be used.
A deactivated realm emoji should neither be accepted further as a
reaction nor its further occurences in a message be rendered as an
emoji. However, all the old occurences should continue to render
normally.
2017-06-15 02:59:57 -07:00
Tommy Ip 8d6af030c0 bugdown: Fix modal_link.
The `data-toggle` property prevented the new style of overlay modals
from launching, and regardless, isn't a future-proof options for how
this should work.
2017-06-03 18:41:19 -07:00
Nathan Miller 2311e169ec mypy: Various strict-optional fixes in zerver. 2017-05-26 10:10:20 -07:00
umkay ccc70445d6 mypy: Fix strict-optional errors for test files.
Fix mypy --strict-optional errors in zerver/tests
2017-05-24 12:43:28 -07:00
Konstantin Gukov dd76222a3f Fetch system bots using new get_system_bot function.
This eliminate a bunch of uninteresting calls to
get_user_profile_by_email.
2017-05-23 10:30:40 -07:00
Harshit Bansal c549dea9ac bugdown: Fix the regex used for unicode emoji matching.
The regex we were using didn't cover all the unicode blocks
to which our emojis belong. This commit fixes the regex to
include all the unicode blocks and also updates the
corresponding JS regex in marked.js.

Fixes: #3460.
2017-05-16 09:05:42 -07:00
Harshit Bansal f8824ea623 bugdown: Fix the `unicode_emoji_to_codepoint()` function.
Unicode codepoints are of minimum length 4, padded with extra
zeroes if the length is less than 4. This commit fixes the
`unicode_emoji_to_codepoint()` to ensure that the codepoint
it generates are of correct length.
2017-05-16 09:05:42 -07:00
Tim Abbott 9f7236eec1 message: Remove unused old gravatar_hash field from message dicts.
This was deprecated and replaced some 4 years ago.
2017-05-09 22:33:27 -07:00
Rohitt Vashishtha 0414ac6df5 bugdown: Convert GitHub image-preview urls to image urls.
This makes inline image previews work for links to image files' pages
on GitHub.

Fixes #4658.
2017-05-09 11:22:37 -07:00
Aditya Bansal c23901c3a8 pep8: Add compliance with rule E261 to fenced_code.py. 2017-05-07 23:21:50 -07:00
Rohitt Vashishtha a2c1e10986 bugdown: Add domain-name to relative image-url in open graph data.
fixes #4608.
2017-05-04 16:54:10 -07:00
K.Kanakhin f13d6a18eb realm-emoji: Add realm emoji uploading instead url providing.
- Add file_name field to `RealmEmoji` model and migration.
- Add emoji upload supporting to Upload backends.
- Add uploaded file processing to emoji views.
- Use emoji source url as based for display url.
- Change emoji form for image uploading.
- Fix back-end tests.
- Fix front-end tests.
- Add tests for emoji uploading.

Fixes #1134
2017-05-01 14:50:20 -07:00
Tim Abbott 85805cf380 bugdown: Simplify logic for constructing unicode_emoji_list. 2017-04-30 16:40:58 -07:00
Tim Abbott 8b3be3cd3a bugdown: Pass a codepoint, not a URL, to make_emoji. 2017-04-30 16:40:58 -07:00
Tim Abbott 565370a365 bugdown: Create make_realm_emoji function.
Currently, this is a duplicate of the existing make_emoji, but the
plan is to change that in some upcoming refactoring.
2017-04-30 16:40:58 -07:00
Tim Abbott 18bd566958 bugdown: Extract unicode_emoji_to_codepoint. 2017-04-30 16:40:58 -07:00
Tim Abbott e0a8f383b3 bugdown: Remove unnecessary first argument to make_emoji. 2017-04-30 16:40:58 -07:00
Tim Abbott cfeff245c1 emoji: Manage Zulip emoji as a custom/extra realm emoji.
This is prerequisite for converting our emoji system to work off of
iamcal sprite sheets.

Also, fixes #4308.
2017-04-30 16:40:31 -07:00