Commit Graph

221 Commits

Author SHA1 Message Date
Anders Kaseorg 4f748fb627 markdown: Stop setting target="_blank".
This setting is being overridden by the frontend since the last
commit, and the security model is clearer and more robust if we don't
make it appear as though the markdown processor is handling this
issue.

Co-authored-by: Tim Abbott <tabbott@zulipchat.com>
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-01 14:01:45 -07:00
Tim Abbott e3a4aeeffa CVE-2020-9445: Remove unused and insecure modal_link feature.
Zulip's modal_link markdown feature has not been used since 2017; it
was a hack used for a 2013-era tutorial feature and was never used
outside that use case.

Unfortunately, it's sloppy implementation was exposed in the markdown
processor for all users, not just the tutorial use case.

More importantly, it was buggy, in that it did not validate the link
using the standard validation approach used by our other code
interacting with links.

The right solution is simply to remove it.
2020-04-01 14:01:45 -07:00
Rohitt Vashishtha 4a0fe4606a tests: Print test-case name in all test failures in fixtures.
Previously, we only printed the test-case when we had an assertion error.
With this change, we also include timeout errors as well as any other
causes for failure.
2020-03-31 14:57:18 -07:00
Anders Kaseorg 39f9abeb3f python: Convert json.loads(f.read()) to json.load(f).
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-03-24 10:46:32 -07:00
Steve Howell 1306239c16 tests: Use email/delivery_email more explicitly.
We try to use the correct variation of `email`
or `delivery_email`, even though in some
databases they are the same.

(To find the differences, I temporarily hacked
populate_db to use different values for email
and delivery_email, and reduced email visibility
in the zulip realm to admins only.)

In places where we want the "normal" realm
behavior of showing emails (and having `email`
be the same as `delivery_email`), we use
the new `reset_emails_in_zulip_realm` helper.

A couple random things:

    - I fixed any error messages that were leaking
      the wrong email

    - a test that claimed to rely on the order
      of emails no longer does (we sort user_ids
      instead)

    - we now use user_ids in some place where we used
      to use emails

    - for IRC mirrors I just punted and used
      `reset_emails_in_zulip_realm` in most places

    - for MIT-related tests, I didn't fix email
      vs. delivery_email unless it was obvious

I also explicitly reset the realm to a "normal"
realm for a couple tests that I frankly just didn't
have the energy to debug.  (Also, we do want some
coverage on the normal case, even though it is
"easier" for tests to pass if you mix up `email`
and `delivery_email`.)

In particular, I just reset data for the analytics
and corporate tests.
2020-03-19 16:04:03 -07:00
Steve Howell c235333041 test performance: Pass in users to api_* helpers.
This reduces query counts in some cases, since
we no longer need to look up the user again. In
particular, it reduces some noise when we
count queries for O(N)-related tests.

The query count is usually reduced by 2 per
API call.  We no longer need to look up Realm
and UserProfile.  In most cases we are saving
these lookups for the whole tests, since we
usually already have the `user` objects for
other reasons.  In a few places we are simply
moving where that query happens within the
test.

In some places I shorten names like `test_user`
or `user_profile` to just be `user`.
2020-03-11 14:18:29 -07:00
Steve Howell 5e2a32c936 tests: Use users in send_*_message.
This commit mostly makes our tests less
noisy, since emails are no longer an important
detail of sending messages (they're not even
really used in the API).

It also sets us up to have more scrutiny
on delivery_email/email in the future
for things that actually matter.  (This is
a prep commit for something along those
lines, kind of hard to explain the full
plan.)
2020-03-07 18:30:13 -08:00
Rohitt Vashishtha 2fab45e530 bugdown: Use AtomicString in UserMentionPattern.
This fixes the user-mention counterpart of #14080.
2020-03-06 11:35:56 -08:00
Rohitt Vashishtha 7f9d8e1907 bugdown: Use AtomicString in UserGroupMentionPattern.
This fixes the user-group counterpart of #14080.
2020-03-06 11:35:56 -08:00
Rohitt Vashishtha 62a7e464fb bugdown: Use AtomicString in StreamPattern.
This fixes the stream counterpart of #14080.
2020-03-02 00:03:33 -08:00
Rohitt Vashishtha 245de9e1e2 bugdown: Use AtomicString in StreamTopicPattern.
Fixes #14080.
2020-03-02 00:03:33 -08:00
Rohitt Vashishtha 630c564fc7 bugdown: Rewrite List Preprocessor logic to properly parse fences.
Previously, we didn't track opening and closing fences separately,
with led to bugs like not parsing a list that was immediately after
a quoted fence; we treated each ``` as a new fence.

This commit rewrites the function to maintain a stack of currently
open fences. If any of the parent fences is a code fence, we do not
insert a new line before a list.

We also add some test cases specifically to test this behavior with
complexly nested lists.

Fixes #13745.
2020-01-27 17:14:27 -08:00
Rohitt Vashishtha 85c669e366 markdown: Remove redundant checks from /me.
If a message begins with /me, we do not have any cases where the
rendered content would not begin with `<p>/me`. Thus, we can safely
remove the redundant checks both on the backend and frontend.
2019-12-03 17:17:10 -08:00
Rohitt Vashishtha 3df18c365d markdown: Bring /me python code up to date with frontend.
It appears we forgot to make identical changes to the backend
in #11089 while adding support for multiline /me messages,
resulting in any messages that didn't end in a paragraph getting
rendered as a regular message instead.

Fixes #13454.
2019-12-03 19:46:38 +05:30
Rohitt Vashishtha 9174c636ce bugdown: Store if message has wildcards in MentionData.
We also switch the underlying exctact_mention_text method to use
a regular for loop, as well as make the related methods return
tuples of (names, is_wildcard). This abstraction is hidden from the
MentionData callers behind mention_data.message_has_wildcards().

Concerns #13430.
2019-12-02 12:12:35 -08:00
Mateusz Mandera bbf2474bd0 tests: setUp overrides should call super().setUp().
MigrationsTestCase is intentionally omitted from this, since migrations
tests are different in their nature and so whatever setUp()
ZulipTestCase may do in the future, MigrationsTestCase may not
necessarily want to replicate.
2019-10-19 17:27:01 -07:00
Wyatt Hoodes 4a64a487d3 bugdown: Remove print statement from fixtures test. 2019-09-13 11:54:14 -07:00
Thomas Ip 574c35c0b8 markdown: Render ordered lists using <ol> markup.
This brings us in line, and also allows us to style these more like
unordered lists, which is visually more appealing.

On the backend, we now use the default list blockprocessor + sane list
extension of python-markdown to get proper list markup; on the
frontend, we mostly return to upstream's code as they have followed
CommonMark on this issue.

Using <ol> here necessarily removes the behaviour of not renumbering
on lists written like 3, 4, 7; hopefully users will be OK with the
change.

Fixes #12822.
2019-09-08 16:42:20 -07:00
Mateusz Mandera 6b20006aea test_bugdown: Prepare for moving system bots to zulipinternal. 2019-07-24 16:44:16 -07:00
Wyatt Hoodes e331a758c3 python: Migrate open statements to use with.
This is low priority, but it's nice to be consistently using the best
practice pattern.

Fixes: #12419.
2019-07-20 15:48:52 -07:00
Rohitt Vashishtha 726d5003e1 bugdown: Force absolute urls in topic links.
If a url doesn't have a scheme, browsers would treat it as a relative
url and open something like: https://chat.zulip.org/google.com instead.

This PR fixes the issue on the backend; the frontend implementation
remains out of sync and the user sending the message wouldn't see
any linkification for urls without a scheme.

Fixes #12791.
2019-07-19 12:02:52 -07:00
Puneeth Chaganti 865bc24f67 url preview: Avoid showing previews for URLs in blockquotes.
Messages with links embedded in blockquotes turn out to be replies to
messages with links, more often than not. Showing previews for links in
replies seems like clutter, and it seems reasonable to turn off previews for
such links.
2019-07-12 19:14:00 -07:00
Rohitt Vashishtha 0ba332bcc0 topic-mention: Add Bugdown implementation as StreamTopicPattern. 2019-07-11 14:53:10 -07:00
Aayush Agrawal 54584f6c16 url preview: Create a single preview for each URL in a message.
Modified by punchagan to:
* Add a separate markdown test for de-duplicating inline previews
* Check for number of unique URLs to see if per limit message is crossed
* Use a set for processed URLs instead of a list

Fixes #8379.
2019-07-11 13:37:15 -07:00
Anders Kaseorg 5f3d36ae87 tex: Bundle KaTeX CLI using Webpack.
We don’t need a hacked copy anymore.  We run the installed version out
of node_modules in development, and a Webpack-bundled version of that
in production.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-02 16:50:03 -07:00
Rohitt Vashishtha c03efd2971 markdown: Make backend status message detection logic mimic frontend.
In 93914d8cd8, we intended to change our
markdown processor to add support for multi-line /me messages.
However, we neglected to change the backend processor, resulting in
the change only taking effect for the user sending the message :(.

We fix this by changing the backend processor too.

Fixes #12450.
2019-06-28 10:31:20 -07:00
Rohitt Vashishtha 047086b81c markdown: Make raw urls in topic names navigable.
We reuse the link regexes we use elsewhere inn markdown
for parsing links in topic names and add a button to open
them in new tabs similar to our behavior with linkifiers
in topic names.

Fixes #12391.
2019-06-27 15:18:42 -07:00
Puneeth Chaganti 30dcf805ea url preview: Use oEmbed preview for Vimeo, instead of custom code. 2019-06-02 14:31:39 -07:00
Puneeth Chaganti 8c0c9ca7a4 url preview: Turn Realm.inline_url_embed_preview off by default. 2019-05-31 15:28:32 -07:00
Eeshan Garg cecea75457 api_docs: Detect missing arguments in curl examples.
This commit adds automated tests that make sure that every curl
example command in our API docs has the '-X (POST|GET)' argument.

Fixes: #11927
2019-05-28 16:53:48 -07:00
Puneeth Chaganti 5dee17dca0 bugdown: Show previews for manually created youtube playlists.
Youtube playlists can be created by manually listing video_ids, as follows:
https://youtube.com/watch_videos?video_ids=vid1,vid2,vid3. This commit adds
previews for URLs of this type, using the first video ID.
2019-05-12 22:24:42 -07:00
Puneeth Chaganti a1f0713b2c bugdown: Show previews for youtube playlist URLs, if possible.
If a youtube playlist URL has a video-id, we show a preview for the URL.

Closes #8562
2019-05-12 22:24:42 -07:00
Puneeth Chaganti 4de261c2de bugdown: Don't show previews for youtube URLs without video ids.
`youtube.com/playlist?list=<list-id>` incorrectly matches the regex since the
change in 8afda1c1bb. The regex was modified to
match URLs of the form `youtu.be/<id>` and this playlist URL incorrectly matches
with the `<id>` set to `playlist`.

This commit avoids this match by verifying that the ID is not playlist.
2019-05-12 22:24:42 -07:00
Anders Kaseorg 643bd18b9f lint: Fix code that evaded our lint checks for string % non-tuple.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-04-23 15:21:37 -07:00
Puneeth Chaganti 8afda1c1bb bugdown: Show preview for urls copied from the Youtube share widget. 2019-04-13 20:25:37 -07:00
Rohitt Vashishtha 51986c27a5 markdown: Use common testcases for realmfilters in topics and messages.
The actual bug in #11791 was caused by code reverted in
3ed85f4cd7, so technically #11791 is
already fixed.  However, it makes sense to add tests to ensure that it
doesn't regress in the future as part of closing out the issue.

Fixes #11791.
2019-03-11 11:48:47 -07:00
YashRE42 a724a38c03 markdown: Improve handling of broken img urls.
Some urls which end with image file extensions (eg .jpg) may link to
html pages. This adds handling for linx.li, wikipedia.org and
pasteboard.co. If it is possible, we redirect to the actual image url
otherwise we do not attempt to render it as an image.

Fixes #10438.
2019-03-08 13:39:34 -08:00
Rohitt Vashishtha 3ed85f4cd7 Revert "bugdown: Process word boundaries properly in realm_filters."
This reverts commit ff90c0101c but keeps
the test cases added for reference.

This was reverted because it was both not a clean solution and created
other realm filters bugs involving dashes (etc.).
2019-03-07 11:03:35 -08:00
Bennet Sunder 7c5f316cb8 alert_words: Performance improvements in looking for alert_words.
This commit leverages the ahocorasick algorithm to build a set of user_ids
that have their alert_words present in the message. It runs in linear time
of the order of length of the input message as opposed to number of
alert_words. This is after building a ahocorasick Automaton which runs
in O(number of alert_words in entire realm) which is usually cached.
2019-03-01 15:36:39 -08:00
Tim Abbott d6c09eac51 bugdown: Add support for no_previews argument.
This allows us to have some features using bugdown rendering where
inline image previews will not be rendered (which would be problematic
for e.g. stream descriptions).
2019-02-28 16:54:04 -08:00
Rohitt Vashishtha 44ec83ef28 markdown: Render silent mentions as **name**.
This change should help people discover to distinguish
silent mentions in text as a part of Zulip syntax while
differentiating them from regular mentions.
2019-02-20 10:41:42 -08:00
Rohitt Vashishtha 57b9991396 markdown: Change syntax of silent mentions ( _@person -> @_person). 2019-02-20 10:41:42 -08:00
savish 9c292fbeab tests: Use subTest in test_bugdown_fixtures. 2019-02-13 10:49:13 -08:00
Anders Kaseorg 3127fb4dbd zerver/tests: Remove unused imports.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-02-02 17:43:03 -08:00
Steve Howell 643d4cacd3 bugdown tests: Pass message stub to bugdown_convert().
This makes the calling sequence slightly more
realistic (without any additional overhead).
2019-01-29 12:49:56 -08:00
Rohitt Vashishtha ff90c0101c bugdown: Process word boundaries properly in realm_filters.
Earlier, our realm filters didn't render for languages that do not
use spaces (eg: Japanese) since we used to check for the presence
of an actual space character. This commit replaces that logic with
a complex scheme to detect word boundaries.

Also, we convert the RealmFilterPattern to subclass InlineProcessor
and make use of the new no-op feature in py-markdown 3.0.1 where we
can tell py-markdown that our pattern didn't find a match despite
the initial regex getting matched.

Fixes #9883.
2019-01-28 14:48:15 -08:00
Rohitt Vashishtha 0451c42e5c minor: Add test for mentions using silent syntax inside quotes. 2019-01-22 15:28:08 -08:00
Rohitt Vashishtha 96aa1d4b37 markdown: Reduce mentions inside blockquotes to silent-mentions.
On the backend, we extend the BlockQuoteProcessor's clean function that
just removes '>' from the start of each line to convert each mention to
have the silent mention syntax, before UserMentionPattern is invoked.

The frontend, however, has an edge case where if you are mentioned in
some message and you quote it while having mentioned yourself above
the quoted message, you wouldn't see the red highlight till we get the
final rendered message from the backend.

This is such a subtle glitch that it's likely not worth worrying about.

Fixes #8025.
2019-01-16 16:08:37 -08:00
Rohitt Vashishtha f993fdd480 markdown: Add _@**Name** syntax for silent mentions.
These mentions look like regular mentions except they do not
trigger any notification for the person mentioned. These are
primarily to be used when you make a bot take an action and
the bot mentions you, or when you quote a message that mentions
you.

Fixes #11221.
2019-01-16 16:01:06 -08:00
Harshit Bansal 5f76a65b1d emoji: Make unicode/span emojis more accessible.
This commit adds `aria-label="<title_text>"` and `role="img"` to
the generated HTML.

Fixes: #5975.
2019-01-16 09:07:19 -08:00