This adds the data model and bugdown support for the new UserGroup
mention feature.
Before it'll be fully operational, we'll still need:
* A backend API for making these.
* A UI for interacting with that API.
* Typeahead on the frontend.
* CSS to make them look pretty and see who's in them.
We now have a MentionData class that encapsulates
the users who are possibly mentioned in a message.
Not that the rendering code may not keep all the mentions,
since things like backticks will suppress the mention.
We populate this now in do_send_messages, so that we can use
the info earlier in the message-sending process. This info
now gets passed down the call stack as an optional parameter.
Note that bugdown.convert() still populates the data when its
callers decline to pass in a MentionData object.
This is mostly a preparatory commit, as we don't take advantage
of the data yet in do_send_messages.
Nobody has used this feature in years, and it causes certain types of
markdown issues in development to completely DoS the development
environment by making it possible for the "Bugdown timeout" exception
handler to timeout in bugdown.
Since we already send an email to the server administrators, there's
no need to replace this feature with anything.
This commit switches to use sprite sheets for rendering emojis
in all the remaining places, i.e., message bodies and composebox
typeahead. This commit also includes some changes to notifications.py
file so that the spans used for rendering emojis can be converted
to corresponding image tags so that we don't break the emoji rendering
in missed message emails since we can't use sprite sheets there.
As part of switching the bugdown system to use sprite sheets, we need
to switch the name_to_codepoint mappings to match the new sprite
sheets. This has the side effect of fixing a bunch of emoji like
numbers and flag emoji in the emoji pickers.
Fixes: #3895.
Fixes: #3972.
We now triage message content for possible mentions before
going to the cache/DB to get name info. This will create an
extra data hop for messages with mentions, but it will save
a fairly expensive cache lookup for most messages. (This will
be especially helpful for large realms.)
[Note that we need a subsequent commit to actually make the speedup
happen here, since avatars also cause us to look up all users in
the realm.]
Given typeahed and the fact that this only worked if the person had a
full name that didn't contain whitespace, this side effect of the
original @shortname mentionfeature that we removed was experienced by
users as a bug.
Fixes#6142.
Process the unicode emojis in twitter link previews and render them
properly. Before this we were not processing the unicode emojis in
twitter link previews and hence on the systems which don't have
fonts for displaying them they were rendered as blank boxes.
Fixes: #5427.
This commit renames list named `to_linkify` in twitter link processor
to `to_process` and adds a `type` field to each entry in it to
indicate the type of data represented by that particular entry.
Unicode emojis when rendered should display canonical short name.
Similarly, the alt text should be of the format `:<short_name>:`.
For both of these we currently display the actual unicode symbol.
As some systems don't have the fonts necessary for displaying them
properly, they are rendered as empty square blocks. This commit also
ensures that the markup generated for emoji generated by canonical
name and by an unicode emoji is same.
Fixes: #5555.
We used shortnames for mentioning users before we had autocomplete
feature. Since we now have autocomplete typeahead, this syntax is
no more useful and just causes problems. This commit removes the
shortname mention syntax.
Fixes: #4189.
A deactivated realm emoji should neither be accepted further as a
reaction nor its further occurences in a message be rendered as an
emoji. However, all the old occurences should continue to render
normally.
The `data-toggle` property prevented the new style of overlay modals
from launching, and regardless, isn't a future-proof options for how
this should work.
The regex we were using didn't cover all the unicode blocks
to which our emojis belong. This commit fixes the regex to
include all the unicode blocks and also updates the
corresponding JS regex in marked.js.
Fixes: #3460.
Unicode codepoints are of minimum length 4, padded with extra
zeroes if the length is less than 4. This commit fixes the
`unicode_emoji_to_codepoint()` to ensure that the codepoint
it generates are of correct length.
- Add file_name field to `RealmEmoji` model and migration.
- Add emoji upload supporting to Upload backends.
- Add uploaded file processing to emoji views.
- Use emoji source url as based for display url.
- Change emoji form for image uploading.
- Fix back-end tests.
- Fix front-end tests.
- Add tests for emoji uploading.
Fixes#1134
Earlier, a stack was being used to go through the message and search
for links. Because of this, in some cases the images were added to
the preview in reverse. Using a queue will keep the image previews in
the same order as they appeared in the message.
Fixes#4453.
By default, Python markdown tab length for indents is 4 spaces, which
require using 4 spaces or a tab to create nested elements. This
modifies that setting to specify 2-space indentation for nesting
elements only.
Modified significantly by tabbott to limit the change to just list
indentation.
Fixes#4252.
Use `name_to_codepoint.json` file (and the similar structure in
emoji_codes.js) to map emoji names directly to codepoints and change
the rendered emoji image to `unicode/<codepoint.png>` rather than
`<emoji_name>.png`.
Fixes: #3539.
We do not use `get_link_embed_data` for messsages sent by
bots, as bots often repeat the same URL over and over again
and are generally either text-focused or have their own
mechanisms to provide preview content.
Fixes#2968.
(The commit q7ef4e40258280e202325c9295579c93fb948b replaced
data-user-email with data-user-id, but we still need to
support data-user-email for old clients like non-updated
androids and we still want to start the migration forward
to data-user-id.)
This changes bugdown to use the realm passed in by the caller (if any)
for rendering, fixing a problem where bots such as the notification
bot would have their messages rendering using the admin realm's
settings, not the settings of the realm their messages are being sent
into.
Also adds a test for the notification bot case.
Fixes#3215.
A lot of care has been taken to ensure we're using the realm that the
message is being sent into, not the realm of the sender, to correctly
handle the logic for cross-realm bot users such as the notifications
bot.
This should substantially improve the clarity of the code, since
inside bugdown, this is only being used as a hash key that happens to
usually be a realm ID, not used as a Realm ID.
This change adds support for displaying inline open graph previews for
links posted into Zulip.
It is designed to interact correctly with message editing.
This adds the new settings.INLINE_URL_EMBED_PREVIEW setting to control
whether this feature is enabled.
By default, this setting is currently disabled, so that we can burn it
in for a bit before it impacts users more broadly.
Eventually, we may want to make this manageable via a (set of?)
per-realm settings. E.g. I can imagine a realm wanting to be able to
enable/disable it for certain URLs.
Now that we have updated python-markdown, we remove the deprecated
safe_mode. We used safe_mode to escape raw html, so now instead we
pass in an EscapeHtml markdown extension to the markdown engine.
See https://pythonhosted.org/Markdown/release-2.6.html for details on
the deprecation.
Fixes: #2037 (also addresses the remaining piece of #2043).
This distinguishes between YouTube Videos and Image Previews by adding
a particular “youtube-video” class to the preview along with changing
the title to the video ID rather than the link. This serves to allow
the lightbox to ID when a lightbox preview should be treated like a
YouTube video rather than an image preview.
This also modifies the tests in bug down to expect a youtube-video class
along with the title to just be the video ID on YouTube rather than the
entire URL link.
The changes that required us to fork this extension had been merged
into upstream CodeHilite, so we can remove it and switch to using the
version that comes with python-markdown.
This updates Bugdown to reflect the changes in the updated
markdown. In particular, we now pass a default config object in the
__init__ for the Bugdown extension, update the make_md_engine function
to take kwargs as opposed to a config list, and have UListProcessor
inherit from ulist as opposed to olist (which no longer works).
We update the (forked from upstream) fenced_code extension's
makeExtension to take args and kwargs, and update
FencedBlockPreprocessor __init__ method with updated Codehilite
arguments.
We update the (forked from upstream) Codehilite extension to
mirror the logic with the latest upstream Codehilite:
Add parse_hl_lines function
update makeExtension to take args and kwarfs instead of config
list
Add regex for highlight lines
use linenums instead of linenos
use get_formatter_by_name instead of HtmlFormatter
user get_lexer_by_name instead of TextLexer
add hl_lines and use_pygments arguments to the codehlite
constructor
Change the parameter name of some functions from 'md' to 'content',
since the name 'md' seems to be the reason why this parameter was
wrongly annotated.
This sets the “title” attribute on the image to the actual title of
the image specified by the user in their markdown, rather than just
the URL of the full link to it.
The bugdown parser no longer has a concept of which users need which
alert words, since it can't really do anything actionable with that info
from a rendering standpoint.
Instead, our calling code passes in a set of search words to the parser.
The parser returns the list of words it finds in the message.
Then the model method builds up the list of user ids that should be
flagged as having alert words in the message.
This refactoring is a little more involved than I'd like, but there are
still some circular dependency issues with rendering code, so I need to
pass in the rather complicated realm_alert_words data structure all the way
from the action through the model to the renderer.
This change shouldn't change the overall behavior of the system, except
that it does remove some duplicate regex checks that were occurring when
multiple users may have had the same alert word.
We now raise an exception in bugdown.do_convert() if rendering
fails, to avoid silent failures, and then calling code can convert
the exception to a JsonableError.
!avatar, !modal_link, !gravatar, etc. were incorrectly being processed
before the escape character for code blocks.
While we're at it, we add tests for these special syntaxes.
Change `str` to `text_type` in annotations in zerver/models.py
related to realm emoji and realm filters.
Also fix clashing annotations in zerver/lib/bugdown/__init__.py.
Had to add some "type: ignore" because the pattern used in match
doesn't affect the type returned. A fix for this issue has been pushed
to typeshed - https://github.com/python/typeshed/pull/244
Since we don't have a stable way to get the Dropbox preview failure
image (and it was sorta a weird setup anyway), it seems best to just
remove the condition.
This also requires updating the required version of oauthlib; previously an
appropriate version was being installed only because it was a dependency of
the wrong twitter library.
This only affects development environments and/or hand-built
installations relying on the contents of requirements.txt.
To fix existing environments, the incorrect api needs to be explicitly
removed with `pip uninstall twitter`.
Fixes#86.
This reverts commit 39f2908a32c0276b1d87ecedc876c71dd35a9b2f.
We're not including the preview_fail.png image in the release.
(imported from commit 2de1451de2f9b1727fc3a7e64c380b71c0f2caa8)
One common place that this happens (for us) is on a local
Dropbox .dev.corp.dropbox.com instance, which can't be reached
by the Zulip servers.
This commit also:
* Fixes the test suite
* Properly previews /photos/ links
(imported from commit b4788b6236e7a9d390e1efc4673be34d9ba5e091)
We now will match an alert word even if it is used at the boundry of
bolding, backtick escaping, or caret quoting.
Closes trac #2186.
(imported from commit 984bc63eb621772c95a01ca5c5bfeb190767f71f)
Apparently the "inline" treeprocessor is what runs the inline
patterns. Also re-enable the rewriting-to-https support.
(imported from commit 2fde2c1f15217a784f26b16db25ee745f424f2f0)
When new streams are created we now send a message with a custom
markdown tag that renders a subscribe button.
(imported from commit 9dfba280b3b4ff4f32f6431ef9227867c8bf4b40)
Image and video links in the twitter API are media and need to be
handed on separately. We also include a preview image if the media link
is a to a picture.
(imported from commit 2bd00d267e51b29ad0ba681195b2bfea9b991d8c)
This converts links in tweets to a tags. We also convert the displayed
text to the target of the twitter short URL. Mentions are linked to the
users twitter page.
(imported from commit 192d5546a7eea82759f9ae30d82c102aed15ff71)
* Deal with shorter tweet IDs
(some old tweets don't have a full 18-character ID)
* Allow trailing slash
* Deal with old-style #! syntax
* Deal with links that link to a photo
(imported from commit 008a98c806f3b8dddd9e2f18a8f002af6932766f)
These images at least load now, but that's because Camo redirects
the browser to the origin server, so the only effect is an extra
round-trip time.
(imported from commit 0d6b9c888a5cdfaa9299272d74a085e872dfa434)
CUSTOMER13 doesn't want it, and there's currently no nginx config
or configurable Camo URI, so it wouldn't work if image preview
were enabled.
(imported from commit 615d4a32acbc4d4d590f88cf4e7d45d8f49db1d3)
The !gravatar markdown no longer hard codes to Gravatar, but
instead it serves up our generic avatar URL.
(imported from commit 4e3e2baeb3374bcf025a18ff27a8452b975c22b7)
Arguably the nl2br extension should be doing this for us. Given that
we're using nl2br, the "two spaces at the end of a line makes a line
break" rule doesn't make any sense (since every newline leads to a
linebreak), so we disable it.
(imported from commit 5ffa2ac8a825642ad31e085c532091e076665710)
We want to avoid opening a DB connection in the markdown thread
as its DB connection might live for a long time
(imported from commit 7700b2ca793ee5e9add7f071b92f22a4bf576b3d)
The text of manual links are already AtomicStrings, so linkified strings
should be too.
Moves emoji detection to happen after linkification, so the emoji rule
won't look at links.
(imported from commit 9c56bce6a0e873b398255e0762dfb312a4a9a64e)
InlinePatterns should return None on failure, not text that may
have placeholders in it.
(imported from commit f9d8d22b2b8cfa7a92ecf3e52a6c76b48e6f0175)
LinkPattern returned a string which contained a placeholder if the URL was
considered invalid. AtomicLinkPattern wrapped this in an AtomicString,
where the placeholder doesn't get removed properly.
m.group(0) is always incorrect because python-markdown modifies your regex
to include more than you specified (this is why part of the message got
duplicated).
(imported from commit 576bdf09c2b677cf4bc56484c363eb05f2110158)
Now parsed: 🍺,🍺;🍺!
If \w characters surround :foo:, we still say it's NOT an
emoji, but we used to do this for \S characters, so it's loosened up.
(imported from commit 49b33d2f0ffdcfde8947ae411a4addcf4c24af9c)
This needs to be deployed to both staging and prod at the same
off-peak time (and the schema migration run).
At the time it is deployed, we need to make a few changes directly in
the database:
(1) UPDATE django_content_type set app_label='zerver' where app_label='zephyr';
(2) UPDATE south_migrationhistory set app_name='zerver' where app_name='zephyr';
(imported from commit eb3fd719571740189514ef0b884738cb30df1320)
derAnfaenger
rht
neiljp (Neil Pilgrim)
Umair Khan
Tim Abbott
Steve Howell
Harshit Bansal
Juliana Bacelar
digi0ps
Aditya Bansal
Tommy Ip
Nathan Miller
umkay
Konstantin Gukov
Rohitt Vashishtha
K.Kanakhin
Sampriti Panda
hollywoodno
Yago González
Ayush Jain
Cynthia Lin
sinwar
Raghav Jajodia
Rishi Gupta
Igor Tokarev
Steve Howell
Brock Whittaker
reallyly
Mikebarson
bulat22101
Alex Huang
AZtheAsian
Rafid Aslam
Bickio
Vladislav Manchev
hackerkid
Meena Rajan
Kevin Chen
paxapy
umkay
Mohsen Ibrahim
Eklavya Sharma
Taranjeet Singh
Tim Abbott
Alex Wilson
Oren Leaffer
Varshit
Darren Worrall
Tim Abbott
David Roe
Waseem Daher
Paul Ruan
Jason Michalski
Leo Franchi
Luke Faraone
Jessica McKellar
Kevin Mehall
Steve Howell
Zev Benjamin
Tim Abbott
Steve Howell