Commit Graph

186 Commits

Author SHA1 Message Date
Harshit Bansal a49655e0d4 emoji: Migrate realm emoji to be addressed by `id` rather than `name`.
This commit migrates realm emoji to be addressed by their `id` rather
than their name. This fixes a long standing issue which was causing
an error on uploading an emoji with same name as a deactivated realm
emoji.

Fixes: #6977.
2018-03-20 22:24:44 -07:00
Nikhil Kumar Mishra 9e17692d94 markdown: Add unit test for render_tex.
Test for invalid path of Katex.
2018-03-19 10:36:26 -07:00
Aditya Bansal 9eeb1c59f6 bugdown: Remove email from rendered content of messages with mentions.
This field has been unused by clients for some time, and isn't great
for our public archive feature plans (where we'll not want to be
including email addresses in messages).
2018-03-04 20:04:27 -08:00
Marco Burstein bdb86f1b5e emoji: Add support for translating emoticons.
Add `translate_emoticons` to `prop_types` and `expected_keys`.
Furthermore, create a emoji-translating Markdown inline pattern.

Also use a JavaScript version of `translate_emoticons` and then use
this function during Markdown previews and as a preprocessor. This
is only needed for previews, because usually emoticon translation
happens on the backend after sending.

Add tests for emoticon translation, a settings UI, and a /help/ page
as well.

Tweaked by tabbott to fix various test failurse as well as how this
handles whitespace, requiring emoticons to not have adjacent
characters.

Fixes #1768.
2018-03-04 15:37:24 -08:00
neiljp (Neil Pilgrim) 354d552a10 mypy: Add two mypy-pacifying asserts in upload & bugdown tests. 2018-02-19 09:24:50 -08:00
Steve Howell ffb7a371ed minor: Make relative link tests slightly more realistic.
The two tests changed here are exercising some URL rewrite
logic, and now the URL for stream narrows is slightly more
realistic.
2018-02-19 09:03:11 -08:00
Steve Howell 46a49777c4 Add stream ids to urls for stream-related narrows.
This commit prefixes stream names in urls with stream ids,
so that the urls don't break when we rename streams.

strean name: foo bar.com%
before: #narrow/stream/foo.20bar.2Ecom.25
after: #narrow/stream/20-foo-bar.2Ecom.25

For new realms, everything is simple under the new scheme, since
we just parse out the stream id every time to figure out where
to narrow.

For old realms, any old URLs will still work under the new scheme,
assuming the stream hasn't been renamed (and of course old urls
wouldn't have survived stream renaming in the first place).  The one
exception is the hopefully rare case of a stream name starting with
something like "99-" and colliding with another stream whose id is 99.

The way that we enocde the stream name portion of the URL is kind
of unimportant now, since we really only look at the stream id, but
we still want a safe encoding of the name that is mostly human
readable, so we now convert spaces to dashes in the stream name.  Also,
we try to ensure more code on both sides (frontend and backend) calls
common functions to do the encoding.

Fixes #4713
2018-02-19 09:03:11 -08:00
Shubham Dhama a32e1eb913 markdown: Require double-asterisk around all mentions.
This enforces `**` around all the mentions including "at-all" and
"at-everyone" mentions. Hence this makes `@all` and `@everyone`
invalid mentions, resulting into proper syntax for these mentions as
`@**all**` and `@**everyone**` respectively.

Note from tabbott: This removes an old feature/syntax, which made
sense back when @Tim was also a way to mention a user with Tim as
their first name.  Given how nice typeahead is now, the user part of
the feature was removed a while ago; this should have gone at the same
time.

Fixes: #8143.
2018-02-16 11:45:08 -08:00
Tim Abbott 77addc5456 bugdown: Fix handling of ultra-long renderings.
If some bug in Bugdown results in a rendered message content that is
bigger than twice the message size, we now just throw an exception
from Bugdown.  This is considerably better than the old behavior,
which might result in an enormous message being placed in the database
(potentially, bigger than the 1MB limit to store in memcached), which
would in turn result in tragic consequences.

This fixes #8322, in that it prevents the super bad outcome seen there
(where basically Zulip became unusable for everyone on the stream
where the message is posted).  Now, the failure mode is just the
message failing to send.  Still not ideal (and requires further work
on the URL embed feature), but not a minor problem, not a major one.
2018-02-09 10:57:55 -08:00
Weronika Grzybowska 7ac7100a1d messages: Make checking for status message consistent with backend.
Adds a check for newline that was present on backend, but missing in the
frontend markdown implementation. Updating messages uses is_me_message flag
received from server instead of its own partial test. Similarly, rendering
previews uses markdown code.

Fixes #6493.
2018-01-23 09:26:41 -05:00
rht ee06da9350 bugdown: Linkify Bitcoin URI. 2018-01-19 06:07:43 -05:00
Shreyansh Dwivedi b0fb7aa6b2 bugdown: Add preview for vimeo videos.
This also amends a commit from Brock Whittaker <brock@zulipchat.com>
that merges two separate functions for YouTube videos and Vimeo videos
into a generic video recall function.

Fixes #7550.
2018-01-17 09:11:43 -05:00
Andy Perez c209069fcd bugdown: Improve image inlining logic.
Fix #7537
2018-01-09 16:43:22 -05:00
Tommy Ip 9ea7b05cfe bugdown: Rewrite markdown link to relative link conditionally. 2017-12-22 08:10:42 -05:00
Andy Perez 44e6d9b54b testing: Add nice diffs to markdown python tests. 2017-12-18 19:03:38 -05:00
Alena Volkova 26eb23afc7 tests: Replace api_auth usages with auth methods in zerver/tests. 2017-12-18 09:24:09 -05:00
Rohitt Vashishtha 6734b00c0b markdown: Hide URL if message is only an image.
Hides URL if the message content == image url so that sending gifs or
images feels less cluttered. Uses the url_to_a() function to generate
the expected url string for matching.

Fixes #7324.
2017-11-27 13:30:18 -08:00
Rohitt Vashishtha 3550467899 markdown: Insert text in link-only inline image preview tests.
Appends "Test: " text to some tests to make changes to the image preview
rendering. In the future, if the message is only a link to an image,
the link will be hidden.
2017-11-27 13:30:18 -08:00
rht 6c28c7e879 zerver/tests: Use python 3 syntax for typing. 2017-11-18 15:35:57 -08:00
rht 4f5b1c0a5a zerver/tests: Use python 3 syntax for typing in most files. 2017-11-16 21:52:01 -08:00
Harshit Bansal 5e5bb02e39 bugdown: Remove `make_realm_filters()`.
This commit removes `make_realm_filters()` and merges its functionality
in `make_md_engine()` and `maybe_update_markdown_engines()`.
2017-11-15 22:47:11 -08:00
Harshit Bansal e615a2ab4f bugdown: Simplify the "opts" parameter of `make_md_engine()`. 2017-11-15 22:47:11 -08:00
Harshit Bansal e3f2f67d0e bugdown: Rename `maybe_update_realm_filters()`.
This commit renames `maybe_update_realm_filters()` to
`maybe_update_markdown_engines()` which sounds closer
to the semantics.
2017-11-15 22:47:11 -08:00
Tommy Ip bcd575d8de bugdown: Rewrite links to local server to be relative links.
Fixes #7247
2017-11-09 15:59:31 -08:00
Steve Howell 531a3ee9de refactor: Fix parse.quote(force_str(s)) calls in bugdown. 2017-11-09 10:43:20 -08:00
Harshit Bansal 65838bb825 email_gateway: Disable code block processor for email gateway.
Generally emails are not written with markdown in mind and hence
sometimes render in strange ways. This commit fixes a particular
issue that was causing whitespace before paragraphs to be treated
as code block due to which email content was being rendered in a
box that scrolls in right direction a lot.

Fixes: #7045.
2017-11-09 09:56:35 -08:00
Tim Abbott 417f92c8e1 markdown: Remove now-obsolete bugdown_matches_marked.
Fixes #7099.
2017-11-07 12:01:57 -08:00
rht 19bd335cbb Change urllib import to be Python 3-specific. 2017-11-07 10:46:42 -08:00
derAnfaenger 0709033858 bugdown: Fetch complete data for tweet.
This makes an extended instead of a normal query,
resulting in the full text of a tweet being fetched
instead of just a truncated version.
2017-11-07 09:13:39 -08:00
rht c7fa1d4146 zerver/tests: Remove inheritance from object. 2017-11-06 08:53:48 -08:00
rht dcc831f767 refactor: Replace all __unicode__ method with __str__.
Close #6627.
2017-11-02 11:01:47 -07:00
Umair Khan 636046aec9 user-groups: Add basic backend for UserGroup model.
This adds the data model and bugdown support for the new UserGroup
mention feature.

Before it'll be fully operational, we'll still need:
* A backend API for making these.
* A UI for interacting with that API.
* Typeahead on the frontend.
* CSS to make them look pretty and see who's in them.
2017-10-31 15:16:14 -07:00
Tim Abbott 8f9463bd72 test_bugdown: Check for unknown keys in markdown_test_cases.json.
This should help protect us from typos.
2017-10-31 08:58:42 -07:00
Steve Howell 4372bfb4a5 tests: Fix send_message calls in test_bugdown.py. 2017-10-28 10:20:59 -07:00
Steve Howell 87b48f722d Add MentionData.get_user_ids().
This prepares us to take advantage of the user_ids
in subsequent commits related to `do_send_messages()`.
2017-10-26 22:16:47 -07:00
Steve Howell e0bc1b114e Make sure mentions refer only to active users.
An active user can share the same full name as a deactivated
user.  We now only allow mention syntax to find users who are
activated.

Fixed #6978
2017-10-12 17:11:36 -07:00
Tim Abbott 234f7960a1 tests: Add @slow decorator to various methods needing it.
Also, fix the test_message_editing test having way too many cases.

Fixes #1478.
2017-10-06 15:45:04 -07:00
Harshit Bansal 5b5bcce098 emoji: Migrate bugdown emoji to use sprite sheets.
This commit switches to use sprite sheets for rendering emojis
in all the remaining places, i.e., message bodies and composebox
typeahead. This commit also includes some changes to notifications.py
file so that the spans used for rendering emojis can be converted
to corresponding image tags so that we don't break the emoji rendering
in missed message emails since we can't use sprite sheets there.

As part of switching the bugdown system to use sprite sheets, we need
to switch the name_to_codepoint mappings to match the new sprite
sheets.  This has the side effect of fixing a bunch of emoji like
numbers and flag emoji in the emoji pickers.

Fixes: #3895.
Fixes: #3972.
2017-09-29 11:14:34 -07:00
rht 26f5d9a32c zerver/tests: Remove print_function. 2017-09-27 18:05:45 -07:00
rht daef6a7dbc zerver/tests: Remove `import six`. 2017-09-27 17:07:40 -07:00
rht 1e87a4b68c zerver/tests: Remove absolute_import. 2017-09-27 10:00:39 -07:00
Steve Howell 7159f34bed Avoid getting realm emoji unless there is emoji syntax. 2017-09-15 01:09:08 -07:00
Steve Howell 8a941d03ae Avoid querying streams unless there are stream links. 2017-09-15 01:09:08 -07:00
Steve Howell c3032a7fe8 Avoid looking up emails when rendering messages.
We now fetch email -> id mappings for messages ONLY if
it potentially uses the !avatar(foo@example.com) syntax.
2017-09-15 01:09:08 -07:00
Steve Howell 4e7fce60ee Add possible_mentions() to speed up rendering.
We now triage message content for possible mentions before
going to the cache/DB to get name info.  This will create an
extra data hop for messages with mentions, but it will save
a fairly expensive cache lookup for most messages.  (This will
be especially helpful for large realms.)

[Note that we need a subsequent commit to actually make the speedup
happen here, since avatars also cause us to look up all users in
the realm.]
2017-09-15 01:09:08 -07:00
Steve Howell ba397b5109 Use user_ids, not full objects, in render path.
There is no reason for either render_incoming_message() or
render_markdown() to require full UserProfile objects just to
triage alert words.

By only asking for user_ids, we save extra queries in two
callpaths and we make it easier to start using user_ids in
do_send_messages().
2017-09-12 04:22:55 -07:00
Tim Abbott 8600efcf36 test_bugdown: Fix duplicated test name. 2017-08-16 11:29:12 -07:00
Tim Abbott d2288154f6 bugdown: Fully remove mentions matching @name.
Given typeahed and the fact that this only worked if the person had a
full name that didn't contain whitespace, this side effect of the
original @shortname mentionfeature that we removed was experienced by
users as a bug.

Fixes #6142.
2017-08-16 11:18:09 -07:00
Umair Khan 0e119c0ec2 test_bugdown: Upgrade to HttpResponse.json(). 2017-08-16 07:06:58 -07:00
Tim Abbott 4b7c23f133 markdown: Rename bugdown-data.json to a more reasonable name. 2017-07-28 17:54:01 -07:00
Harshit Bansal 4737363f0c test_bugdown: Migrate tests to use the default test realm emoji. 2017-07-27 18:59:02 +00:00
Tim Abbott c9b2ef82c9 test_bugdown: Fix test for Twitter emoji.
Our work on the Twitter emoji added a new instance of our emoji syntax
between when we wrote 2d92f9dc0b and
rebase+merged it.
2017-07-26 10:20:25 -07:00
digi0ps 2d92f9dc0b emoji: Change emoji image titles to use spaces instead of underscores.
Previously, the emoji images title display `🍼`.
This commit changes the title to display `baby bottle`.
2017-07-26 09:27:17 -07:00
Harshit Bansal a03ce01688 bugdown: Fix the rendering of emojis in twitter link previews.
Process the unicode emojis in twitter link previews and render them
properly. Before this we were not processing the unicode emojis in
twitter link previews and hence on the systems which don't have
fonts for displaying them they were rendered as blank boxes.

Fixes: #5427.
2017-07-24 17:32:05 -07:00
Harshit Bansal 127d1f995d test_bugdown.py: Add test for `url_to_a()`. 2017-07-24 17:31:08 -07:00
Aditya Bansal ee97ea1356 pep8: Add compliance with rule E261 to tests/test_bugdown.py. 2017-07-11 15:17:32 -07:00
Harshit Bansal 4cb8ac100a emoji: Fix the title and alt text for unicode emojis.
Unicode emojis when rendered should display canonical short name.
Similarly, the alt text should be of the format `:<short_name>:`.
For both of these we currently display the actual unicode symbol.
As some systems don't have the fonts necessary for displaying them
properly, they are rendered as empty square blocks. This commit also
ensures that the markup generated for emoji generated by canonical
name and by an unicode emoji is same.

Fixes: #5555.
2017-07-01 11:14:24 -07:00
Harshit Bansal dfbf8b8086 bugdown: Remove shortname mention syntax.
We used shortnames for mentioning users before we had autocomplete
feature. Since we now have autocomplete typeahead, this syntax is
no more useful and just causes problems. This commit removes the
shortname mention syntax.

Fixes: #4189.
2017-06-19 16:57:59 -04:00
Cory Lynch 552c39a036 fenced_code.py: Improve test coverage for special chars. 2017-06-19 06:53:25 -04:00
Harshit Bansal 7950f82f27 realm_emoji: Make sure that a deactivated realm emoji can't be used.
A deactivated realm emoji should neither be accepted further as a
reaction nor its further occurences in a message be rendered as an
emoji. However, all the old occurences should continue to render
normally.
2017-06-15 02:59:57 -07:00
Tommy Ip 8d6af030c0 bugdown: Fix modal_link.
The `data-toggle` property prevented the new style of overlay modals
from launching, and regardless, isn't a future-proof options for how
this should work.
2017-06-03 18:41:19 -07:00
Vishnu Ks bb98e35aa0 Replace othello@zulip.com with example_email('othello'). 2017-05-24 19:37:36 -07:00
Vishnu Ks 5230eaef1c Replace hamlet@zulip.com with example_email('hamlet'). 2017-05-24 19:37:36 -07:00
Tim Abbott e5f5e5c69f tests: Remove unnecessary maxDiff settings.
Now that ZulipTestCase does this, we don't need it in all these
individual test modules.
2017-05-24 12:43:28 -07:00
umkay ccc70445d6 mypy: Fix strict-optional errors for test files.
Fix mypy --strict-optional errors in zerver/tests
2017-05-24 12:43:28 -07:00
Vishnu Ks 075b0cae70 test_bugdown.py: Remove unused get_user_profile_by_email import. 2017-05-23 15:27:21 -07:00
Vishnu Ks c680c5f1e8 Replace sipbtest@mit.edu with mit_user('sipbtest'). 2017-05-22 19:02:42 -07:00
Rohitt Vashishtha 0414ac6df5 bugdown: Convert GitHub image-preview urls to image urls.
This makes inline image previews work for links to image files' pages
on GitHub.

Fixes #4658.
2017-05-09 11:22:37 -07:00
Steve Howell 942db9b6c5 tests: Added ZulipTestCase.example_user() function.
The example_user() function is specifically designed for
AARON, hamlet, cordelia, and friends, and it allows a concise
way of using their built-in user profiles. Eventually, the
widespread use of example_user() should help us with refactorings
such as moving the tests users out of the "zulip.com" realm
and deprecating get_user_profile_by_email.
2017-05-08 11:57:38 -07:00
K.Kanakhin f13d6a18eb realm-emoji: Add realm emoji uploading instead url providing.
- Add file_name field to `RealmEmoji` model and migration.
- Add emoji upload supporting to Upload backends.
- Add uploaded file processing to emoji views.
- Use emoji source url as based for display url.
- Change emoji form for image uploading.
- Fix back-end tests.
- Fix front-end tests.
- Add tests for emoji uploading.

Fixes #1134
2017-05-01 14:50:20 -07:00
Sampriti Panda 0e3d694df8 bugdown: Use queue for processing of links in image previews.
Earlier, a stack was being used to go through the message and search
for links.  Because of this, in some cases the images were added to
the preview in reverse.  Using a queue will keep the image previews in
the same order as they appeared in the message.

Fixes #4453.
2017-04-17 20:48:16 -07:00
Rishi Gupta 30024d0a8f models: Remove Realm.domain. 2017-03-25 19:55:48 -07:00
Ayush Jain bddcfb1c96 Add realm-level settings to control inline image and url preview.
This gives users more control in case they don't want previews,
especially for the "previews of linked websites" feature.

Fixes: #2640.
2017-03-21 15:46:17 -07:00
Rishi Gupta 0559afb928 models: Use Realm.string_id instead of domain in RealmFilter.__unicode__. 2017-03-13 09:27:21 -07:00
Tim Abbott 39704bac6c test_bugdown: Remove unused common_bugdown_test. 2017-03-04 23:35:46 -08:00
Rishi Gupta 1453a5bfda Change string_id of test zephyr realm from mit to zephyr.
Also changes Realm.is_zephyr_mirror_realm to use string_id=zephyr instead of
domain=mit.edu.
Part of a larger migration away from Realm.domain.
2017-03-04 12:18:01 -08:00
Igor Tokarev 31dff09efa Support email changes for !avatar syntax.
Significantly modified by tabbott to avoid calling
get_user_profile_by_email in bugdown, and have 100% test coverage of
the views code.

Fixes #2041.
2017-02-28 21:56:04 -08:00
Steve Howell 5b8cc03c61 bugdown: Restore data-user-email to user mention spans.
(The commit q7ef4e40258280e202325c9295579c93fb948b replaced
data-user-email with data-user-id, but we still need to
support data-user-email for old clients like non-updated
androids and we still want to start the migration forward
to data-user-id.)
2017-02-02 12:32:18 -08:00
Tim Abbott 700089f386 bugdown: Fix need for Internet in test_inline_dropbox_bad. 2017-01-29 22:07:14 -08:00
Tim Abbott d6e38e2a5c lint: Clean up E123 PEP-8 rule. 2017-01-23 21:34:26 -08:00
Tim Abbott 9cc83f87fc lint: Clean up E241 PEP-8 rule. 2017-01-23 21:21:14 -08:00
Steve Howell 1137ef4e40 Mark up mention buttons with data-user-id on the backend.
We use data-user-id now in the buttons for at-mentions when
we render messages.  A previous commit already made the front
end support this new style.
2017-01-21 21:45:12 -08:00
Tim Abbott 5d52f1ec17 bugdown: Move realm_filters_key logic out of callers.
This gets rid of the confusing duplicate realm_filters_key and
message_realm arguments that previously were passed to bugdown.
2017-01-21 21:37:57 -08:00
Tim Abbott 19b89eb050 bugdown: Rename realm_id to realm_filters_key.
This should substantially improve the clarity of the code, since
inside bugdown, this is only being used as a hash key that happens to
usually be a realm ID, not used as a Realm ID.
2017-01-16 21:48:55 -08:00
Sampriti Panda 196cf4367b urls: Move /messages/render to POST endpoint 2017-01-13 16:11:51 -08:00
Rishi Gupta 494c1a2b55 Remove unnecessary uses of Realm.domain in zerver/tests. 2017-01-09 11:26:08 -08:00
Rishi Gupta 2b0a7fd0ba Rename models.get_realm_by_string_id to get_realm.
Finishes the refactoring started in c1bbd8d. The goal of the refactoring is
to change the argument to get_realm from a Realm.domain to a
Realm.string_id. The steps were

* Add a new function, get_realm_by_string_id.

* Change all calls to get_realm to use get_realm_by_string_id instead.

* Remove get_realm.

* (This commit) Rename get_realm_by_string_id to get_realm.

Part of a larger migration to remove the Realm.domain field entirely.
2017-01-04 17:12:23 -08:00
Rishi Gupta cf762eaf84 Change X.realm.id to X.realm_id across codebase.
This makes it more clearly the pattern in the Zulip codebase, and thus
decreases the risk of accidentally doing database queries.
2017-01-03 16:46:26 -08:00
Rishi Gupta c6e12e74be Change domain to realm_id in bugdown and realm filter dicts and caches. 2017-01-03 16:25:20 -08:00
Tim Abbott 39f0ffdedd Move static/third/gemoji to static/generated/emoji.
Fixes #1153.
2016-12-27 20:16:23 -08:00
Tim Abbott bf80873d4f tests: s/assertRaisesRegexp/assertRaisesRegex/ due to deprecation. 2016-12-15 17:11:42 -08:00
Igor Tokarev c93f1d4eda Add oembed/Open Graph/Meta tags data retrieval from inline links.
This change adds support for displaying inline open graph previews for
links posted into Zulip.

It is designed to interact correctly with message editing.

This adds the new settings.INLINE_URL_EMBED_PREVIEW setting to control
whether this feature is enabled.

By default, this setting is currently disabled, so that we can burn it
in for a bit before it impacts users more broadly.

Eventually, we may want to make this manageable via a (set of?)
per-realm settings.  E.g. I can imagine a realm wanting to be able to
enable/disable it for certain URLs.
2016-12-07 17:40:18 -08:00
bulat22101 9f68efa47b mypy: Convert zerver/tests/ to use typing.Text. 2016-12-04 14:47:21 -08:00
nikolay abc2ff4a06 pep8: Fix many rule E128 violations.
[Tweaked by tabbott to adjust some approaches used in wrapping]
2016-12-03 13:33:31 -08:00
Sidhant Bhavnani 8c0c12c1d9 pep8: Fix E303 violations. 2016-12-02 15:34:11 -08:00
Rafid Aslam c5316b4002 lint: Fix E127 pep8 violations.
Fix pep8: E127 continuation line over-indented for visual indent
style issue.
2016-12-01 10:23:55 -08:00
Rafid Aslam 7a2282986a pep8: Fix E225 pep8 violations. 2016-11-28 15:21:15 -08:00
Vladislav Manchev d7e1e4a2c0 Add initial implementation of custom realm filters.
This PR was abandoned by Vladislav and then substantially modified by
Igor Tokarev and Tim Abbott to complete it and fix a number of bugs.

Fixes #544.
2016-11-17 17:11:25 -08:00
reyha 9f8630fed8 tests: Access realms by string_id.
Replaces get_realm() function by get_realm_by_string_id() function
in zerver/tests.

Fixes #2226.
2016-11-15 13:32:20 -08:00
sinwar 4582a98c09 tests: Split out ZulipTestCase and WebhookTestCase to a separate file.
Fixes #1671.
2016-11-10 19:29:43 -08:00