Commit Graph

402 Commits

Author SHA1 Message Date
Lauryn Menard b64d8e8a52 templates: Remove `base_path` argument from api_arguments_table_generator.
Removes `base_path` argument when making the markdown extension for
parameters in documentation for API endpoints.

This seems to have been originally included for API parameters that
were documented in JSON files, which is no longer in use. Now all
API endpoints in the documentation are documented in
`zerver/openapi/zulip.yaml`.
2023-02-01 13:12:53 -08:00
Lauryn Menard f38d5a6a26 templates: Remove `base_path` argument from api_return_values_generator.
Removes `base_path` argument when making the markdown extension for
return values in documentation for API endpoints.

This seems to have been a copy and paste error in commit d2ee99a2fd
when `zerver/lib/markdown/api_return_values_generator.py` was created.
2023-02-01 13:12:53 -08:00
David Rosa 8d1db6482f help center: Update relative gear menu macro for organization settings.
Renames "Manage organization" to "Organization settings" to reflect
changes in the previous commit.
2023-01-26 10:17:45 -08:00
Anders Kaseorg 4eda29bd86 ruff: Fix RUF005 Consider spread instead of concatenation.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-26 10:16:30 -08:00
Anders Kaseorg 7a7513f6e0 ruff: Fix SIM201 Use `… != …` instead of `not … == …`.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-23 11:18:36 -08:00
Anders Kaseorg b8b29dc3ad ruff: Fix SIM110 Use `return any(…)` instead of `for` loop.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-23 11:18:36 -08:00
Anders Kaseorg ff1971f5ad ruff: Fix SIM105 Use `contextlib.suppress` instead of try-except-pass.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-23 11:18:36 -08:00
Anders Kaseorg b0e569f07c ruff: Fix SIM102 nested `if` statements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-23 11:18:36 -08:00
David Rosa f70b321773 help center: Document bot name in org settings link to bot's user card.
Documents link to the bot's user card from the bot's name in
Organization settings > Bots, and information in the bot's user card.

Fixes part of #23970.
2023-01-19 11:13:33 -08:00
Trident Pancake c6ea673cc9 markdown: Update max inline preview from 10 to 24.
The max inline preview limit was previously increased to 10 by #20789.
However, as issue #23624 shows, it's still causing confusion for users
when they include more than 10 links.

Bump this limit up to 24, which is a multiple of the 4 image preview
per line logic.
2023-01-18 14:58:00 -05:00
Anders Kaseorg 17300f196c ruff: Fix ISC003 Explicitly concatenated string.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-04 16:25:07 -08:00
Anders Kaseorg 2c5e114f8b ruff: Fix ISC001 Implicitly concatenated string literals on one line.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-04 16:25:07 -08:00
Anders Kaseorg b5cad938b8 ruff: Fix DTZ006 `datetime.datetime.fromtimestamp()` without `tz` argument.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-04 16:25:07 -08:00
Anders Kaseorg f7e97b1180 ruff: Fix PLW0602 Using global but no assignment is done.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-04 16:25:07 -08:00
David Rosa b870d85634 help center: Update "Import from Mattermost".
Updates the help center article to match the style and formatting
of "Import from Slack" and replaces existing content with its
corresponding Markdown macro.
2022-12-30 18:06:44 -08:00
Zixuan James Li a3a0103d86 markdown: Calculate linkifier precedence in topics.
This uses the linkifier index among the list of linkifiers in the
replacement as the priority to order the replacement order for
patterns in the topic. This avoids having multiple overlapping matches
that each produce a link.

The linkifier with the lowest id will be prioritized when its pattern
overlaps with another. Linkifiers are prioritized over raw URLs.

Note that the same algorithm is used for local echoing and the
backend markdown processor.

Fixes #23715.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-12-13 15:16:20 -08:00
Zixuan James Li 4602c34108 markdown: Correctly retrieve indices for repeated matches.
The same pattern being matched multiple times in a topic cannot be
properly ordered using topic_name.find(match_text) and etc. when there
are multiple matches of the same pattern in the topic.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
2022-12-13 15:16:20 -08:00
Anders Kaseorg e634e3276a ruff: Fix PLC0414 Import alias does not rename original package.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-12-04 22:11:24 -08:00
Anders Kaseorg 73c4da7974 ruff: Fix N818 exception name should be named with an Error suffix.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-17 16:52:00 -08:00
Anders Kaseorg 924d530292 ruff: Fix N813 camelcase imported as lowercase.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-16 09:29:11 -08:00
Anders Kaseorg 2876ae8e48 ruff: Fix N803 argument name should be lowercase.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-16 09:29:11 -08:00
Anders Kaseorg 46955da3a0 ruff: Fix ANN204 missing return type annotation for __init__.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-11-16 09:29:11 -08:00
Lauryn Menard 9aacb15530 tabbed-sections: Update "via-user-profile" to be "via-user-card". 2022-11-14 12:16:22 -08:00
Alya Abbott 8f6cf65f28 help center: Reorganize and extend the "Logging in" help page.
Also move some instructions from related pages into shared /include files,
with minor tweaks.
2022-10-26 12:28:29 -07:00
Lauryn Menard c9044f7acd recent-topics: Rename to "Recent conversations" in web-app and docs.
Replaces instances of "recent topics" in the web-app and documentation
to be "recent conversations".

Renames both `recent-topics.md` files in the help center to be
`recent-conversations.md` and updates/redirects links to new URL.

Does not update instances of "recent topics" in frontend code comments
and does not update the main overview changelog, for now.

Does not change case study text where "recent topics" was referenced
in a quote, but does change generic text references to be "recent
conversations".
2022-10-21 10:28:19 -07:00
David Rosa d9dd272a78 help: Update documentation on bots.
Adds new tab to `zerver/lib/markdown/tabbed_sections.py` to document
managing bots from both personal settings and organization settings.

Documents adding bots from the organization settings Bots panel.

Separates instructions for deactivating and reactivating a bot from
both personal settings and organization settings.

Fixes a few formatting issues such as missing bold formatting and
heading level.

Fixes: #23066.
2022-10-18 10:16:42 -07:00
Alex Vandiver 5d42a0cb00 linkifiers: Support %20 in URLs for topic links.
9381a3bd45 added support for linkifier pattern URLs containing
`%20`-style escapes, but only did so for the codepath which is used in
the message body -- topic links did not understand them.

Expand the support to include when they are substituted into topics.
2022-10-11 14:31:13 -07:00
Anders Kaseorg 8230324068 markdown: Store ZulipMarkdown in members with the right type.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-10-06 15:15:10 -07:00
Anders Kaseorg 2bd81dd5c9 fenced_code: Avoid sloppy AttributeError handler.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-10-06 15:15:10 -07:00
Anders Kaseorg 3cf91e9e45 markdown: Rename our Markdown subclass to ZulipMarkdown.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-10-06 15:15:10 -07:00
Anders Kaseorg 97be895cf0 markdown: Remove Optional from zulip_rendering_result type.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-10-06 15:15:10 -07:00
Anders Kaseorg d01c99d2ee markdown: Add missing None check in InlineInterestingLinkProcessor.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-10-06 15:15:10 -07:00
Mateusz Mandera 95bc77ae4f help center: Update "SAML authentication" to follow help center guidelines.
Co-authored-by: Alya Abbott <alya@zulip.com>
2022-09-09 10:30:05 -07:00
Alya Abbott ff82c69480 help center: Rewrite Import from Slack documentation. 2022-08-25 15:17:43 -07:00
Anders Kaseorg 4a61e36def CVE-2022-36048: Rewrite only specific local links to relative.
Due to mismatches between the URL parsers in Python and browsers, it
was possible to hoodwink rewrite_local_links_to_relative into
generating links that browsers would interpret as absolute.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-08-24 16:29:09 -07:00
N-Shar-ma ef044b8697 markdown: Update characters allowed before @ and stream mentions.
Now the following characters are allowed before @-mentions and stream
references (starting with #) for proper rendering - {, [, /.

This commit makes the markdown rendering consistent with autocomplete
(anything that is autocompleted is also rendered properly).
2022-08-06 19:29:39 -07:00
Kartik Srivastava 6d514c7f78 markdown: Fix rendering of nested objects in API return values.
`render_table` calls itself recursively when it finds nested
`additionalProperties` (i.e. nested objects) in response schema,
to render their properties.

This fixes `render_table` to call `render_desc` along with
calling itself, to render the description of the nested
`additionalProperties` as well.
2022-08-04 16:34:13 -07:00
Anders Kaseorg 946a0565c6 openapi: Fuse generate_api_title with generate_api_description.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-07-22 16:41:55 -07:00
Mateusz Mandera 2299aa3382 docs: Remove some outdated references to thumbnailing.md doc.
The doc was removed in 405bc8dabf
2022-07-12 17:44:24 -07:00
Anders Kaseorg 8246ee7c57 mypy: Add links to specific mypy bugs.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-07-05 17:54:58 -07:00
Anders Kaseorg feff1d0411 requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-07-05 17:54:17 -07:00
Anders Kaseorg 869fe60689 markdown: Parse included blocks in a new parser state.
This fixes inclusion of a multi-paragraph file into a list item.

Followup to commit dc33a0ae67 (#22315).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-06-27 17:37:00 -07:00
Anders Kaseorg dc33a0ae67 markdown: Rewrite include plugin without markdown-include.
markdown-include is GPL licensed.

Also, rewrite it as a block processor, so that it works correctly
inside indented blocks.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-06-26 17:36:31 -07:00
Anders Kaseorg 7f0e11bd06 markdown: Rename preprocessor_priorities module to priorities.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-06-26 17:36:31 -07:00
David Rosa 52ef574d3e help-docs: Fix minor errors in "Link to a message or conversation".
Removes the ":" which have accidentally ended up in the "Get a link
to a specific topic" and "Get a link to a specific stream" headings.

Renames the "Via browser's address bar" tab to "Web" so that it
stays consistent with other help center articles.

Fixes part of #22147.
2022-06-08 12:20:32 -07:00
Anders Kaseorg 6331a314d4 Correctly hyphenate “non-”.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-27 22:10:31 -07:00
Anders Kaseorg a2825e5984 python: Use Python 3.8 typing.{Protocol,TypedDict}.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-27 12:57:49 -07:00
Anders Kaseorg fd16f97d6b python: Excise None from pointlessly nullable booleans.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-04-27 12:40:14 -07:00
Alex Vandiver 351bdfaf78 preview: Use cache only as a non-durable cache, not an IPC.
The `get_link_embed_data` / `link_embed_data_from_cache` pair as
introduced in c93f1d4eda uses the cache
as a temporary store inside of the `embed_links` worker; this means
that it must be durable storage, or the worker will stall and re-fetch
the same links to preview them.

Switch to plumbing through the fetched URL embed data as an parameter
to the Markdown evaluation which uses them, rather than using the
cache as an intermediary.  This frees up the cache to be merely a
non-durable cache.

As a side-effect, this removes get_cache_with_key, and
link_embed_data_from_cache which was its only callsite.
2022-04-15 14:48:12 -07:00
Alex Vandiver 327ff9ea0f preview: Use a dataclass for the embed data.
This is significantly cleaner than passing around `Dict[str, Any]` all
of the time.
2022-04-15 14:48:12 -07:00
Alex Vandiver 661c333377 markdown: Use named parameters to add_a helper.
This has enough parameters that it benefits from making which is which
explicit.
2022-04-15 14:48:12 -07:00
Alex Vandiver 452a30305d markdown: Clarify url parameter of "add_a" helper. 2022-04-15 14:48:12 -07:00
Lauryn Menard 9f8022de5e help_docs: Add 'About Zulip' to relative help links.
Adds the 'About Zulip' gear menu option to the available
relative link patterns used in help center documentation.
2022-04-12 15:22:58 -07:00
Alya Abbott f04fb51ecc help: Rewrite pages on inviting users and related permissions.
Fixes #21520.

Co-authored-by: Lauryn Menard <lauryn@zulip.com>
2022-03-28 17:00:10 -07:00
Lauryn Menard cbfe2707f4 help_docs: Clarify relative link text for Subscribed streams tab. 2022-03-28 15:13:39 -07:00
Lauryn Menard 182c00248d help_docs: Update various docs related to permissions.
Adds tab for web-public streams in documentation for setting
who can create new streams, as well as some text about why
this is limited to certain roles.

Removes list of actions that can be restricted to full members
due to maintainability concerns for that type of list in the
documentation and replaces it with a short descriptive text
explaining that many settings in Zulip support this restriction.
2022-03-24 11:37:35 -07:00
Alex Vandiver 1ac0035f8c markdown: Allow whitespace overlaps in topic linkifiers.
`prepare_linkifier_pattern`, as of db934be064, adds a match to the
end of the regex, of either the end of string, or a non-word character
-- this is in place of a negative look-ahead, which is no longer
possible in re2.  This causes the regex to consume trailing
whitespace, and thus not be able to match twice in succession with
`pattern.finditer` -- "#1234 #5678" fails to match because the space
is consumed by the first match of the regex.

Rather than use `pattern.finditer`, write own own version, which
rewinds over the non-word character consumed after the match, if any.
This allows the same "after" non-word character to also satisfy the
"before" of the next match.

Fixes #21502.
2022-03-22 15:40:03 -07:00
Lauryn Menard b67288db67 help_docs: Extend options for getting links to Zulip content.
Extends the linking to Zulip documentation to cover:

- Getting URLs to messages via the message timestamp.
- Getting links to topics via the three-dots menu.
- Getting links to streams via right-click context menu.

Creates a new tabbed section for using the browser
address bar to copy URLs.
2022-03-22 12:38:09 -07:00
Tim Abbott f8146cfaa1 help: Fix label for Subscribed tab in stream settings. 2022-03-18 14:28:20 -07:00
Lauryn Menard 6cc818b672 help_docs: Document user management via user profile modal.
Adds documentation for admins to manage users via the user profile
modal for these actions:
- Deactivating a user
- Changing a user's role
- Changing a user's name

Creates two new tab sections because we still want to document
the ability to do these actions through the users section in
the organizational settings modal.

Also cleans up some text in the help center article for changing
a user's role.

Fixes #21318.
Fixes #21415.
2022-03-15 12:35:03 -07:00
Anders Kaseorg 1629d6bfb3 python: Reformat with Black 22 (stable).
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-18 18:03:13 -08:00
Anders Kaseorg b0ce4f1bce docs: Fix many spelling mistakes.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-07 18:51:06 -08:00
Anders Kaseorg df304c40da markdown: Use built-in hex formatting for unicode_emoji_to_codepoint.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-03 11:00:04 -08:00
Lauryn Menard cc712dbade api_docs: Add check for extra `additionalProperties` in return vals.
Adds a check for `additionalProperties: true` when there are no
properties listed in the schema.

This currently only happens in one place, but will be helpful for
deduplicating text between the `register-queue` and `get-events`
endpoints.
2022-02-02 17:25:58 -08:00
Puneeth Chaganti d55c137277 emoji: Add yellow_large_square and green_large_square emojis.
Wordle has recently become a thing and it uses green, yellow and white (or
black in dark mode) large square unicode characters to let people share their
gameplay. Zulip converts the white and black large square unicode characters to
emojis, but not the green and yellow ones. This causes the Wordle grid to be
misaligned when shared on Zulip.

This commit adds green and yellow large square emojis to our emoji list to fix
the problem.
2022-02-02 16:26:31 -08:00
Puneeth Chaganti 6beb84b553 emoji: Use str.rjust to pad codepoint strings instead of a loop. 2022-02-02 16:26:30 -08:00
Puneeth Chaganti 0eeb74b3c2 emoji: Fix minor typo in unicode_emoji_to_codepoint comment. 2022-02-02 16:26:28 -08:00
Lauryn Menard a832a8a3af api_docs: Fix enum strings in parameters to have quotes.
Fixes the rendering of enums to show strings with quotation marks,
while integers will continue to be rendered without quotation marks.

This allows for an empty string to be passed as an enum value and be
rendered as such in the documentation. Null will be rendered without
quotation marks, like integer values.
2022-01-27 13:03:20 -08:00
Lauryn Menard 7b1597d4cf api_docs: Format required or optional detail of param object field.
Formats and moves whether a field of an object in a request
parameter is required or optional to be in the same location
and have the same formatting as the general api parameter
documentation.

Also formats any examples within the object detailed
description to be the same as the general api parameter
documentation.

Follow up to #20409.
2022-01-25 11:20:18 -08:00
Lauryn Menard e479acc809 api_docs: Add line break before return value description text.
Adds a line break before the descriptive text for return
values and events in the api documentation in order to
help with readability of descriptions with multiple
paragraphs of descriptive text.

Adjustments made to the CSS of list items in unordered
lists to visually group the first paragraph of text
to any following paragraphs or unordered lists.
2022-01-24 10:02:02 -08:00
Lauryn Menard 7993aed530 api_documentation: Render object details in parameter description.
Adds a check for object in parameter type that will render the details
of the object in the parameter description if they are in the object
definition in the OpenAPI documentation.

Fixes #19424.
2022-01-19 13:04:04 -08:00
Alex Vandiver 19f891968d markdown: Increase the maximum number of image previews per message.
The limit here is purely to prevent breakage in case of a pathological
number of images in a single message; 5 images is entirely possible in
a reasonable message, and causes user confusion when they are not
expended.

Increase the limit to 10 per message.
2022-01-14 11:30:07 -08:00
Steve Howell 4adcaf92f7 refactor: Attach get_stream_name_map to MentionData.
This diff looks slightly noisy, but the main chunk of
code that we moved here has the same logic as before,
and it just gets realm_id from MentionBackend now, instead
of having our markdown processor have to supply it.

We basically want MentionData to be the gatekeeper of
mention data, and then we delegate backend tasks to
MentionBackend.

Soon we will add a cache to MentionBacked, which will
justify this change a bit more.
2021-12-30 11:28:15 -08:00
Steve Howell c6448263c3 refactor: Add MentionBackend.
We will eventually use this to avoid redundant
queries.

The diff is slightly noisy here, but there are no
logic changes.
2021-12-30 11:28:15 -08:00
Steve Howell ea252ab53e refactor: Convert FullNameInfo to a dataclass.
As part of this we no longer query for email, which
is a vestige of when we used emails to identify users
on the frontend.
2021-12-30 11:28:15 -08:00
Steve Howell f5fc348786 mypy: Add explicit types for dbdata references.
When our handlers specifically reference self.md.zulip_db_data,
we now use an explicit type.

We probably want a more robust solution here, such as a semgrep
rule.
2021-12-30 11:28:15 -08:00
Steve Howell df84892aad markdown: Convert DbData to a dataclass. 2021-12-30 11:28:15 -08:00
Steve Howell 4e551f8279 refactor: Introduce get_stream_name_map.
We only need a name -> id map, and the FullNameInfo
type was a lie.
2021-12-30 11:28:15 -08:00
Steve Howell c04a8097f3 mypy: Add EmojiInfo type.
We now serialize still_url as None for non-animated emojis,
instead of omitting the field. The webapp does proper checks
for falsiness here.  The mobile app does not yet use the field
(to my knowledge).

We bump the API version here. More discussion here:

https://chat.zulip.org/#narrow/stream/378-api-design/topic/still_url/near/1302573
2021-12-30 11:28:14 -08:00
Lauryn Menard 5300ad8ca0 api_documentation: Display `null` data type if potential value.
If null is a potential value of data type for a return value or
parameter in the API endpoint, then it is rendered as an option.

This currently relies on the 'nullable' setting in the OpenAPI spec
that was removed in the 3.1.0 release. If/when the OpenAPI version
is updated, then how the `data_type` for parameters and return values
is rendered will need to be reworked.

Fixes #20264.
2021-11-17 13:20:24 -08:00
Alex Vandiver 6a40c17ccf markdown: CSS-escape preview links.
This adds `soupsieve` as an explicit dependency, but intentionally
does not adjust the provision version, as it was already an indirect
dependency.
2021-10-26 18:17:23 -07:00
Alex Vandiver 52f74bbd9b markdown: Run URL preview links through camo.
Not proxying these requests through camo is a security concern.
Furthermore, on the desktop client, any embed image which is hosted on
a server with an expired or otherwise invalid certificate will trigger
a blocking modal window with no clear source and a confusing error
message; see zulip/zulip-desktop#1119.

Rewrite all `message_embed_image` URLs through camo, if it is enabled.
2021-10-26 18:17:23 -07:00
Anders Kaseorg 58920affd4 python: Remove re.UNICODE flag (redundant in Python 3).
https://docs.python.org/3/library/re.html#re.A

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-10-22 13:42:29 -07:00
Alex Vandiver 9381a3bd45 linkifiers: Support URL percent-encoded bytes.
Supporting URL percent-encoded bytes is possible using `%%20`, but this
is not necessarily very understandable to end-users, even those that
understand percent encoding.

Allow `%20` in linkifier URL format strings, and transform them into
`%%20` in the pattern just before they are applied in markdown
translation.  Care must be taken here, such that already-escaped `%`s
are not escaped an extra time.

We do this before rendering, and not before storage, as
a simplification; the JS-side linkifier at present only understands
`%(foo)s` and thus needs no changes, and to avoid an un-escaping pass
before showing in the admin UI.
2021-10-22 13:00:20 -07:00
Anders Kaseorg 4839b7ed27 url_preview: Interpret og:image relative to full page URL.
og:image is supposed to be an absolute URL, but some sites incorrectly
provide a relative URL.  In this case, it makes more sense to
interpret it relative to the full page URL after redirects, rather
than relative to just the domain part of the page URL before
redirects.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-10-21 12:20:37 -07:00
Alya Abbott 8045355eee user docs: Document public vs. private stream creation permissions. 2021-10-06 17:33:19 -07:00
Alex Vandiver db934be064 CVE-2021-41115: Use re2 for user-supplied linkifier patterns.
Zulip attempts to validate that the regular expressions that admins
enter for linkifiers are well-formatted, and only contain a specific
subset of regex grammar.  The process of checking these
properties (via a regex!) can cause denial-of-service via
backtracking.

Furthermore, this validation itself does not prevent the creation of
linkifiers which themselves cause denial-of-service when they are
executed.  As the validator accepts literally anything inside of a
`(?P<word>...)` block, any quadratic backtracking expression can be
hidden therein.

Switch user-provided linkifier patterns to be matched in the Markdown
processor by the `re2` library, which is guaranteed constant-time.
This somewhat limits the possible features of the regular
expression (notably, look-head and -behind, and back-references);
however, these features had never been advertised as working in the
context of linkifiers.

A migration removes any existing linkifiers which would not function
under re2, after printing them for posterity during the upgrade; they
are unlikely to be common, and are impossible to fix automatically.

The denial-of-service in the linkifier validator was discovered by
@erik-krogh and @yoff, as GHSL-2021-118.
2021-10-04 21:26:24 +00:00
Pradyumna Sinha 18b36e5b8c markdown/tabbed_sections: Raise exception for missing tab name.
This fixes the issue  where 'None' would appear in the rendered
html in case of a missing tab display_name. Now,
'test-help-documentation' will fail in case of any tab display_name
being missing.

In case of a tab_section with no tabs, currently a single tab with
the name 'null_tab' gets added. Added the display name 'None' for
'null_tab', to keep in line with the existing behaviour.

Fixes #19822
2021-10-01 16:25:17 -07:00
Pradyumna Sinha 78692e9c14 markdown/tabbed_sections: Don't set tab label as 'None' in case of no tabs. 2021-10-01 16:22:11 -07:00
Pradyumna Sinha fe5de4a128 markdown/tabbed_sections: Rename TAB_DISPLAY_NAMES to TAB_SECTION_LABELS. 2021-10-01 16:22:11 -07:00
Kevin Scott 09851ff290 markdown: Document built-in preprocessor priorities for convenience.
Fixes #19810
2021-09-27 10:42:09 -07:00
Alya Abbott c7544d620d user docs: Fix missing tab name on /help/getting-your-organization-started-with-zulip. 2021-09-22 14:29:44 -07:00
Alya Abbott b773048031 user docs: Add documentation for configuring new user defaults.
Also remove a redundant link on customize-settings-for-new-users page.
2021-09-21 16:43:35 -07:00
Eeshan Garg 5839f16969 markdown: Run settings_tab/relative links extensions after include.
Recently, we discovered that our settings_tab/relative Markdown
directives didn't work when they were in a macro that was included
in another Markdown file. Note that without this commit, the
/help/create-your-organization-profile page is broken. This commit
changes the respective priorities of these two extensions such that
these directives are rendered *after* the macro is included in
another file.

Thanks to Alya Abbott for reporting this bug!
2021-09-20 16:57:43 -07:00
Eeshan Garg bfbd77ca5c markdown: Organize preprocessor priorities in one place.
All of our custom Markdown extensions have priorities that govern
the order in which the preprocessors will be run. It is more
convenient to have these all in one file so that you can easily
discern the order at first glance.

Thanks to Alya Abbott for reporting the bug that led to this
refactoring!
2021-09-20 16:57:43 -07:00
Alya Abbott 310b8736af user docs: Add a bunch of "getting started" content. 2021-09-16 11:41:43 -07:00
Mateusz Mandera 05fe62c96c docs: Adjust phrasing of "From your desktop, click on the gear...".
We actually mean the main Zulip UI here, so "from your desktop" sounds
potentially misleading; reodering the phrasing makes the meaning
unambiguous.
2021-09-16 10:47:56 -07:00
Tim Abbott 545911b051 markdown: Remove useless locless_schemes check.
This check was copied from upstream python-markdown's "safe mode"
before they removed that feature.  The upstream history is that they
introduced this check in
2db5d1c8e4,
which was not a complete security check, and then added the
immediately following check (with an allowlist of schemes) in
0b4ffbb60e.

Their first, incomplete check provides no security benefit and makes
the code hard to reason about, so we remove it.
2021-09-09 09:03:40 -07:00
rht c24ab8c4d3 markdown: Expand list of safelisted URL schemes to match HTML spec. 2021-09-09 09:03:40 -07:00
Anders Kaseorg 66ad6a4583 docs: Inline code spans are not blocks.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-09-07 16:12:39 -07:00
Anders Kaseorg 646c04eff2 Rename default branch to ‘main’.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-09-06 12:56:35 -07:00
Alex Vandiver 4d428490fd outgoing_http: Use OutgoingSession subclasses in more places.
This adds the X-Smokescreen-Role header to proxy connections, to track
usage from various codepaths, and enforces a timeout.  Timeouts were
kept consistent with their previous values, or set to 5s if they had
none previously.
2021-09-01 05:34:13 -07:00
Priyansh Garg 1e51c23494 markdown: Remove unnecessary checks for zulip_message.
This commits removes some unnecessary checks for `self.md.zulip_message`,
which were put there historically, as earlier we used to add the additional
properties like mentions_user_ids, alert_words, etc. to Message dict
only. These were later moved to MessageRenderingResult class in commit
75cea329b but the checks weren't removed.

This is important because while rendering the messages imported from
other chat tools (like Rocket.Chat), the Message dict is not passed to
the markdown, due to which the checks for `self.md.zerver_message` fails
and hence, things like user mentions, stream/topic mentions are not
rendered in the imported messages properly.
2021-08-31 16:53:42 -07:00
Anders Kaseorg 4206e5f00b python: Remove locally dead code.
These changes are all independent of each other; I just didn’t feel
like making dozens of commits for them.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-19 01:51:37 -07:00
Anders Kaseorg 806494da06 markdown: Stream and parse incrementally in fetch_open_graph_image.
This way we can stop reading as soon as we get to the body.  Also,
send an Accept header, check that the request was actually successful,
use lxml.etree.iterparse instead of a broken hand-rolled state
machine, and support XHTML, all for negative 28 lines of code.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-05 09:17:32 -07:00
Priyansh Garg 0a875c1c4c markdown: Fix jpeg extension in `IMAGE_EXTENSIONS`. 2021-08-05 08:54:02 -07:00
Anders Kaseorg 42fa62e563 Revert "time_widget: Make the generated time string more readable."
This reverts commit 1965584eec.

This syntax has a bad interaction with table syntax and needs to be
rethought.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-03 16:45:31 -07:00
Ganesh Pawar 1965584eec time_widget: Make the generated time string more readable.
Before: <time:2021-07-14T00:14:00-07:00>
After: <time:2021-07-14|00:14:00|UTC-07:00>

Fixes #19205
2021-08-02 23:17:01 -07:00
Anders Kaseorg 498d2b48d9 fenced_code: Use find_lexer_class_by_name.
This is more efficient than get_lexer_by_name, since we don’t need to
instantiate the class just to get its name.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-02 22:31:46 -07:00
Anders Kaseorg 3665deb93a python: Remove unnecessary intermediate lists.
Generated automatically by pyupgrade.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-02 15:53:52 -07:00
Anders Kaseorg 162e9d6c0b fenced_code: Optimize FENCE_RE to fix cubic worst-case complexity.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-07-22 16:40:44 -07:00
Anders Kaseorg dea935f26f fenced_code: Write FENCE_RE with a raw string.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-07-22 16:40:43 -07:00
akshatdalton 7d9e71be21 fenced_code: Add `process_contents` flag to de-duplicate code. 2021-07-22 10:57:23 -07:00
akshatdalton 6b5812082e markdown: Fix shebang line eliminating behaviour of Codehilite.
See the block comment explaining the motivation for this change, but
basically, the shebang feature of Python-Markdown's Codehilite
extension could be really confusing and is not part of the CommonMark
standard.

1. https://python-markdown.github.io/extensions/code_hilite/#shebang-no-path
2. eacff473a2/markdown/extensions/codehilite.py (L164-L180)

Fixes: #18591.
2021-07-15 15:18:33 -07:00
Anders Kaseorg 8486499314 fenced_code: Fix processor type annotation.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-07-09 17:10:31 -07:00
akshatdalton db1cf3b521 refactor: Add class `ZulipBaseHandler` to de-duplicate code. 2021-07-07 17:53:22 -07:00
Anders Kaseorg ee8724e436 docs: Correct Big Blue Button to BigBlueButton.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-07-05 16:59:00 -07:00
Anders Kaseorg c56440ded0 requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-07-05 12:23:06 -07:00
Anders Kaseorg 24c239d991 openapi: Add missing __init__.py; fix type errors hidden by its absence.
The absence of __init__.py was preventing mypy from following any of
the zerver.openapi imports.  These errors were being silenced by
ignore_missing_imports.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-07-05 12:20:39 -07:00
Priyansh Garg 94a2be06f3 markdown: Use a shared variable for IMAGE_EXTENSION. 2021-07-02 11:22:55 -07:00
sahil839 437f4c2288 settings: Rename 'Your bots' to 'Bots' in settings sidebar.
As we have changed the tab selector above from "Settings" to "Personal
settings", we can simply change "Your bots" to "Bots" as "Bots" is
clear enough given the personal settings context.

We also need to update the API documentation for bots accordingly.
2021-07-01 12:14:48 -07:00
sahil839 37620ffef0 gear_menu: Rename "Settings" option to "Personal settings".
Also includes change in markdown used in docs.
2021-07-01 11:47:14 -07:00
sahil839 0c80d23857 help: Update help pages according to profile page redesign.
This commit fixes the documentation of settings as we have
replaced "Your account" section into two new sections -
"Profile" and "Account & privacy".

This commit also fixes a comment in the test for settings
documentation in test_middleware.py.
2021-06-30 16:56:03 -07:00
akshatdalton 44a298b671 minor: Use `OUTER_CAPTURE_GROUP` variable instead of string value. 2021-06-25 17:43:27 -07:00
akshatdalton 490f6b6880 markdown: Extract regex in local variables. 2021-06-25 17:43:01 -07:00
PIG208 75cea329b4 markdown: Refactor out additional properties added to Message.
This adds a new class called MessageRenderingResult to contain the
additional properties we added to the Message object (like alert_words)
as well as the rendered content to ensure typesafe reference. No
behavioral change is made except changes in typing.

This is a preparatory change for adding django-stubs to the backend.

Related: #18777
2021-06-24 18:14:53 -07:00
Suyash Vardhan Mathur fab6a5192c openapi: Fix 'no parameters found' with x-parameter-description.
Currently, the message that no parameters are accepted by
the endpoint is displayed if there are no parameters in
OpenAPI data, but it is possible that information is
encoded in x-parameter-description (example in upload-file
endpoint), and we want to display that information rather
than the message.

Added an if condition to check the same.
2021-06-24 10:38:33 -07:00
Suyash Vardhan Mathur ad9d1c0f80 openapi: Fix return values in delete-queue endpoint.
The check for whether to do the special GET /events logic was
incorrectly also covering DELETE /events.
2021-06-24 10:37:56 -07:00
akshatdalton c507931ac8 refactor: Export non-markdown logic in mention.py. 2021-06-14 13:26:30 -07:00
Suyash Vardhan Mathur 5be92ad30d openapi: Auto-render heading with return values.
The headings for return values were currently hardcoded
in cases where they occur, but they can be rendered directly
in the markdown extension if the return values exist.
2021-06-10 11:25:13 -07:00
Wesley Aptekar-Cassels d5ba94082a markdown: Increase max rendered message length to 1MB.
This should help with #17425, where messages with lots of LaTeX are
lost, due to the large expansion factor.

This isn't a total fix for this - large messages with lots of LaTeX
can still end up larger than 1MB, and rendering could timeout, but
this fix should help significantly.

1MB is still small enough that I don't expect we'll run into any DOS
problems - my testing didn't show any problems rendering messages that
contain ~1MB of LaTeX.
2021-06-03 10:10:35 -07:00
akshatdalton 7df62ebbaf settings: Make `MAX_MESSAGE_LENGTH` a server-level setting.
This will offer users who are self-hosting to adjust
this value. Moreover, this will help to reduce the
overall time taken to test `test_markdown.py` (since
this can be now overridden with `override_settings`
Django decorator).

This is done as a prep commit for #18641.
2021-06-03 09:26:28 -07:00
akshatdalton 832c763c38 minor: Remove unnecessary `__init__` method in `InlineInterestingLinkProcessor`.
Subclass `Treeprocessor` takes care of the `__init__` method.
2021-05-26 17:13:03 -07:00
Anders Kaseorg bac96cae80 markdown: Fix Dropbox image previews.
?dl=1 causes Dropbox to send Content-Type: application/binary, which
can’t be interpreted by Camo.  Use ?raw=1 instead.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-25 13:42:29 -07:00
akshatdalton 503247ebfa refactor: Add class `CompiledInlineProcessor` to de-duplicate code. 2021-05-23 14:30:22 -07:00
akshatdalton 78f26b6031 minor: Use `super` to initialize subclass. 2021-05-23 14:30:22 -07:00
akshatdalton 18203d8af3 markdown: Silence user group mention inside blockquotes. 2021-05-18 17:31:25 -07:00
akshatdalton 0245b590e9 markdown: Add support for user group silent mention.
Prior to this, we only supported direct mention to
the user groups. This commit extends that support
to silent mention for the user groups.
A related test case is also added.

Fixes: #11711.
2021-05-18 17:31:25 -07:00
akshatdalton f56fca308a mention: Refactor `USER_GROUP_MENTIONS_RE` and simplify its related code path.
Earlier, USER_GROUP_MENTIONS_RE was:
r"(?<![^\s\'\"\(,:<])@(\*[^\*]+\*)"

For the syntax: *foo*, this was unnecessarily capturing it as
*foo* and the extraction of `foo` was done using another helper
function: `extract_user_group`.

This is now changed as:
r"(?<![^\s\'\"\(,:<])@(\*(?P<match>[^\*]+)\*)"

and extraction of `foo` can be done just by using the named capture
group `match`.

This change also helps to simplify its related code path.
2021-05-18 17:31:25 -07:00
akshatdalton d5a36ac5e2 mention: Refactor `MENTIONS_RE` and simplify its related code path.
Earlier, MENTIONS_RE was:
r"(?<![^\s\'\"\(,:<])@(?P<silent>_?)(?P<match>\*\*[^\*]+\*\*)"

For the syntax: **foo**, this was unnecessarily capturing it as
**foo** and adding extra operation for the extraction of `foo`.

This is now changed as:
r"(?<![^\s\'\"\(,:<])@(?P<silent>_?)(\*\*(?P<match>[^\*]+)\*\*)"

and extraction of `foo` can be done just by using the named capture
group `match`.

This change also helps to simplify its related code path.
2021-05-18 17:31:25 -07:00
akshatdalton a9d89b3c56 minor: Convert `unicode_emoji_regex` to uppercase.
Following the convention, we use uppercase for
regex. Also, `unicode_emoji_regex` is given a
conventional name ending with `*_RE`: `UNICODE_EMOJI_RE`.
2021-05-18 17:31:25 -07:00
akshatdalton ffc4724287 minor: Convert `emoticon_regex` to uppercase.
Following the convention, we use uppercase for
regex. Also, `emoticon_regex` is given a
conventional name ending with `*_RE`: `EMOTICON_RE`.
2021-05-18 17:31:25 -07:00
akshatdalton 9f6e6709d3 minor: Convert `user_group_mentions` to uppercase.
Following the convention, we use uppercase for
regex. Also, `user_group_mentions` is given a
conventional name ending with `*_RE`: `USER_GROUP_MENTIONS_RE`.
2021-05-18 17:31:25 -07:00
akshatdalton 0a01b1b28e minor: Convert `find_mentions` to uppercase.
Following the convention, we use uppercase for
regex. Also, `find_mentions` is given a
conventional name ending with `*_RE`: `MENTIONS_RE`.
2021-05-18 17:31:25 -07:00
Tim Abbott 06f1ea657e navbar: Adjust label for usage statistics. 2021-05-12 10:34:12 -07:00
Abhijeet Prasad Bodas 67cd385cec mute user: Add user documentation page.
Also link to it from the API documentation page,
other help pages, and the confirmation dialog for
muting a user.

With substantial edits by tabbott and alya.
2021-05-10 17:10:40 -07:00
Ganesh Pawar 529f72fa3f markdown: Add support for sms and tel links.
Fixes #18390
2021-05-10 15:15:34 -07:00
akshatdalton 55f4996f16 markdown: Fix silent wildcard mentions bug.
A message containing wildcard mention when quoted (which
is turned into a silent mention) or message with silent
wildcard mention notifies the users by sending desktop,
sound, and missed message email notifications. This
is clearly a bug which is fixed by this commit.

Fixes: #18354.
2021-05-10 12:19:40 -07:00
Anders Kaseorg d0c6f4f400 python: Strip leading and trailing spaces from docstrings.
This is enforced by Black ≥ 21.4b0.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-07 22:42:39 -07:00
Alya Abbott 279f7b1c8d user docs: Create separate "Code blocks" user documentation page.
* Move the extended documentation of code blocks to a separate page.
* Merge "code playgrounds" documentation to be a section of that page.
* Document copy widget on code blocks.
* This commit changes how we refer to "```python" type syntax for code
  blocks. Instead of being called a syntax highlighting label, this is
  now referred to as a "language tag", since it serves both syntax
  highlighting and playgrounds.
* Remap all the links.
* Advertise this new page in various places that previously did not have a link.
2021-05-05 16:11:21 -07:00
Sumanth V Rao a40fe26586 help: Document playgrounds on /help, /features and /for/open-source.
Linked the Help Center document in places like
    - zulip.yaml (/events, /register/, realm/playgrounds,
                  /realm/playgrounds/{playground_id})
    - /help/format-your-message-using-markdown (Linked to make
      users reading the markdown code block style, aware of this
      feature)
    - /templates/settings/playground_settings_admin.hbs (Linked
      as a reference to read more about playgrounds before
      configuring one)

Also showcase the feature on /features and /for/open-source.
2021-05-04 12:19:01 -07:00
Anders Kaseorg 995389b4c1 markdown: Don’t apply further Markdown processing to KaTeX output.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-01 15:43:35 -07:00
Wesley Aptekar-Cassels 6b7a3fb74a markdown: Rewrite all external images to use Camo.
Requesting external images is a privacy risk, so route all external
images through Camo.

Tweaked by tabbott for better test coverage, more comments, and to fix
bugs.
2021-04-30 10:36:16 -07:00
Anders Kaseorg e3f2ffa681 docs: Capitalize “Markdown” consistently.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-04-26 09:31:08 -07:00
Anders Kaseorg 178736c8eb docs: Fix spelling errors caught by codespell.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-04-26 09:31:08 -07:00
Tim Abbott 2e928a0853 markdown: Remove logic for creating markdown engines for all realms.
This logic likely never ran due to a combination of bugs.

* Running `maybe_update_markdown_engines` unconditionally meant that
  `if md_engine_key in md_engines` was likely always true.
* Introduced in 65838bb: DEFAULT_MARKDOWN_KEY could never be in
  md_engines, so should we have ever reached that code path, we'd have
  tried to rebuild all markdown engines every time.

And it also wasn't clearly helpful -- because we fetch all linkifiers
for a realm on every request anyway, we don't really save database
queries by doing a bulk fetch on startup, and doing so would likely
result in a material regression to Zulip's overall startup time that
we were creating markdown engines for large numbers of realms in bulk
during process startup.
2021-04-13 09:18:18 -07:00
Tim Abbott c6f27440c6 openapi: Fix display for boolean example values.
The comments explain in some detail, but basically we were displaying
the types for booleans incorrectly, and the types for strings in a
somewhat confusing fashion.  Fix this with comments explaining the logic.

Using JSON dumping also results in our showing strings inside
quotation marks in our examples, which seems net helpful.

Thanks to ArunSankarKs for finding where we needed to change the
codebase.

Fixes #18021.
2021-04-07 09:52:42 -07:00
Abhijeet Prasad Bodas 52a86d9604 linkifiers: Use dictionaries for internal structures.
This change does not affect the API in anyway.
All internal code now uses dictionaries to
denote a linkifier, instead of tuples.
2021-04-05 18:16:08 -07:00
Abhijeet Prasad Bodas 68fe912c63 refactor: Rename most of "filter" to "linkifier".
After this only the database table, events,
and API endpoints remain.
2021-04-05 18:14:07 -07:00
Abhijeet Prasad Bodas f896a7667f refactor: Update some uses of "filter" to "linkifier".
This updates some comments and local variables
which could be changed without breaking other
stuff.
2021-04-05 18:14:07 -07:00
Anders Kaseorg ceb7e2d2bd Revert "markdown: Add support to shorten GitHub links."
This reverts commit 9c6d8d9d81 (#16916).

This feature has known bugs, and also wants some design changes to
make it customizable like linkifiers, so we’re retargeting this to
post-4.x.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-04-02 15:52:34 -07:00
Sumanth V Rao e12f682e2e markdown: Include text & url in `topic_links` parameter of our API.
The linkifier code now includes both the shortened text and the expanded
URL, sorted by the order of the occurrence in a topic. This list is passed
back in the `topic_links` parameter of the /messages and the /events APIs.

topic_links earlier vs now:

earlier: ['https://www.google.com', 'https://github.com/zulip/zulip/32']

now: [{'url': 'https://www.google.com', 'text': 'https://www.google/com},
      {'url': 'https://github.com/zulip/zulip/32', 'text': '#32'}]

Similarly, the topic_links local echo logic in the frontend now returns
back an object.

Fixes: #17109.
2021-03-30 15:53:07 -07:00
m-e-l-u-h-a-n 1b8a5a3344 markdown: Refactor backend logic for handling user mention.
Backend logic for handling user mention was cluttered
because it was handled at two stages first in
get_possible_mentions_info while fetching mention data
based on the messsage and then later in UserMentionPattern
which handles processing of text for mention.

Ideally UserMentionPattern should depend on
get_possible_mentions_info only for data but there was a
shared logic between these two that made it hard to debug
any possible bugs.

Updates in this commit make both of these functions
coherent in terms of logic and also add appropiate
comments to improve readability of these functions.

There was also a hidden bug that if a user A is
mentioned in with @**name|id** then @**invalid|id**
again mentioned A because of the way we handled mentions
earlier. It is solved as a result of this refactor and
appropiate test has been added for this.

This has been tested manually as well as by adding new
test to address missing case.
2021-03-28 16:52:48 -07:00
m-e-l-u-h-a-n 2699048208 markdown: Extend user mention syntax to support user_id for mentioning.
Extend our markdown system to support mentioning of users
by id also. Following these changes, it would be possible
to mention users with @**|user_id** and silently mention
using @_**|user_id**.

Main intention for extending the mention syntax is to make
it convenient for bots to mention a users using their ids. It
is to be noted that previous syntax are also supported.

Documentation tweaked by tabbott for better readability.

The changes were tested manually in development server, and also
by adding some new backend and frontend tests.

Fixes: #17487.
2021-03-25 00:44:56 -07:00
akshatdalton 9c6d8d9d81 markdown: Add support to shorten GitHub links.
We add support to shorten links and test their shortening in
well-organized, clean manner that makes it trivial to extend the
GitHub approach for GitLab and perhaps other services.

We only shorten basic types of GitHub links (issue, PR, commit) that
fit a set of simple common patterns; the default behaviour of Autolink
is kept for everything else.

Logic added in frontend and backend Markdown Processor is identical.
This makes easy to extend the logic for other services like GitLab.

Fixes #11895.
2021-03-25 00:39:44 -07:00
m-e-l-u-h-a-n 830c4acedc markdown: Fix invalid mention bug for stream and stream topic mention.
Modifies `StreamPattern` and `StreamTopicPattern` to inherit
from InlineProcessor instead of Pattern. This change is done
because Pattern stopped checking for matching patterns as soon
as it found a match which was not a valid stream. Due to this
all the subsequent mention failed, even if they were valid.
This bug was only present in backend renderring due to
markdown.inlinepatterns.Pattern.

Due to above changes verbose_compile is no longer used for
precompiling STREAM_LINK_REGEX, STREAM_TOPIC_LINK_REGEX as
adds ^(.*?) and (.*?)$ which cause extra overhead of matching
pattern which is not required. With new InlineProcessor these
extra patterns at beggining and end are not required.
So, StreamPattern and StreamTopicPattern now define their own
__init__ method for precompiling the regex.

Fixes #17535.

These changes were tested locally in dev server and by adding
some new markdown tests to test these.
2021-03-23 01:28:30 -07:00
m-e-l-u-h-a-n dadbba0c25 markdown: Fix invalid mention bug for user group mention.
Modifies `UserGroupMentionPattern` to inherit from InlineProcessor
instead of Pattern. This change is done because Pattern
stopped checking for matching patterns as soon as it found
a match which was not a valid user group. Due to this all
the subsequent user group mention failed, even if they were
valid. This bug was only present in backend renderring due to
markdown.inlinepatterns.Pattern.

This was reported as issue #17535.

These changes were tested locally in dev server and by adding
some new markdown tests to test these.
2021-03-23 01:28:30 -07:00
m-e-l-u-h-a-n c8979a5100 markdown: Fix invalid mention bug for user mention.
Modifies `UserMentionPattern` to inherit from InlineProcessor
instead of Pattern. This change is done because Pattern
stopped checking for matching patterns as soon as it found
a match which was not a valid user. Due to this all the
subsequent user mention failed. This bug was only present in
backend renderring due to markdown.inlinepatterns.Pattern.

This was reported as issue #17535.

These changes were tested locally in dev server and by adding
some new markdown tests to test these.
2021-03-23 01:28:30 -07:00
Anders Kaseorg 23088b5d78 markdown: Fix some Any annotations.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-03-17 18:41:46 -07:00
Anders Kaseorg 9864907985 mypy: Correct typing.re imports to typing.
Although typing.re exists in the standard library, mypy has never
recognized it.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-03-17 18:41:46 -07:00
Abhijeet Prasad Bodas 9223dced3b refactor: Rename filter to linkifier in frontend code and docs.
This only leaves `page_params.realm_filters`, which
will be changed in further commits along with the
API change.
2021-03-15 11:19:59 -07:00
Anders Kaseorg 0a09c9dfd7 markdown: Re-enable typeshed stub for Python-Markdown.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-03-10 11:49:59 -08:00
Suyash Vardhan Mathur 82f6bff0c4 api docs: Sort response keys in /get-events.
Currently, the keys were not sorted in example responses of events
making them unreadable. Added sort_keys parameter to sort them.
2021-02-26 15:01:37 -08:00
Suyash Vardhan Mathur dd8964a31f api docs: Fix id and type fields of events and display them.
Currently, the ID and Type fields didn't have a description,
and weren't being displayed. Added a schema component to add
descriptions, and display on the api page. Fixes part of #15967.
2021-02-23 15:22:53 -08:00
Suyash Vardhan Mathur 96bfeeb9e6 api docs: Expand checking for deprecated fields.
Added assertion to check that if a deprecated flag is in a field's
schema, then it should have deprecated mentioned in description
as well, and moved these checks to a separate function.
Fixes part of #15967.
2021-02-16 15:34:52 -08:00
Anders Kaseorg b728727d9d timeout: Remove unnecessary varargs support.
Mypy can check it this way.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-02-15 17:05:28 -08:00
Anders Kaseorg 6e4c3e41dc python: Normalize quotes with Black.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-02-12 13:11:19 -08:00
Anders Kaseorg 11741543da python: Reformat with Black, except quotes.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-02-12 13:11:19 -08:00
Suyash Vardhan Mathur c9c40d4fd2 api docs: Cleaned up CSS for parameter classes.
Deduplicated CSS classes of data types of response and
request parameters in API Documentation to use a single
class.
2021-02-09 10:31:36 -08:00
Suyash Vardhan Mathur 9d74c7001d api docs: Fix non-rendering response parameter data types.
The current logic doesn't display data types when the additionalProperties
variables are not object, but are array of strings, etc. Changed the if
condition to allow rendering in such cases.
2021-02-09 10:29:25 -08:00
Suyash Vardhan Mathur 26a81ab3aa api docs: Display data type of responses in API Documentation.
Previously, the data type of responses wasn't displayed in the API
Documentation, even though that OpenAPI data is carefully validated
against the implementation. Here we add a recursive function to
render the data types visibly in API Documentation.
Fixes part of #15967.
2021-02-05 10:41:42 -08:00
Anders Kaseorg ae0afa2390 markdown: Explode config dict.
Commit 434094e599 (#11321) changed this
from an Extension to a subclass of Markdown, so it no longer has any
reason to use a config dict structured like that of an Extension.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-02-05 10:52:31 -05:00
Anders Kaseorg 4ca66e7278 timezone: Correct common_timezones dictionary.
The changes are as follows:

• Fix one day offset in all western zones.
• Correct CST from -64800 to -21600 and CDT from -68400 to -18000.
• Disambiguate PST in favor of -28000 over +28000.
• Add GMT, UTC, WET, previously excluded for being at offset 0.
• Add ACDT, AEDT, AKST, MET, MSK, NST, NZDT, PKT, which the previous
  code did not find.
• Remove numbered abbreviations -12, …, +14, which are unnecessary.
• Remove MSD and PKST, which are no longer used.

Hardcode the dict and verify it with a test, so that future
discrepancies won’t go silently unnoticed.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-01-27 15:23:15 -08:00
Suyash Vardhan Mathur f4cf5166bb api docs: Display data type of parameters in API documentation.
Previously, the data type of parameters wasn't displayed in the API
Documentation, even though that OpenAPI data is carefully validated
against the implementation.  Here we add a recursive function to
render the data types visibly in the API documentation.

This only covers the request parameters; we'll want to do something
similar for response parameters in a follow-up PR.

Fixes part of #15967.
2021-01-21 15:56:07 -08:00
Aman Agrawal c685d36821 hipchat_import: Remove tool from codebase.
Remove functions and scripts used by HipChat import tool and
those which will no longer be required in future.
2020-12-23 08:28:49 -08:00
Anders Kaseorg e7e1fde6ec fenced_code: Use immutable type for codehilite_conf.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-11-10 15:54:28 -08:00
Anders Kaseorg fbf8ce0305 markdown: Add types for extra Markdown members.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-11-10 15:54:27 -08:00
Anders Kaseorg b48bdc65b9 markdown: Fix AlertWordNotificationProcessor.run type.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-11-10 15:54:27 -08:00
Anders Kaseorg 9573f6dc00 markdown: Fix build_block_parser type.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-11-10 15:54:27 -08:00
Anders Kaseorg 4398eecd2b markdown: Use immutable type for extension config.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-11-10 15:54:27 -08:00
Anders Kaseorg 060036dfd5 markdown: Merge build_engine into Markdown constructor.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-11-10 15:54:27 -08:00
Anders Kaseorg 08c64f5cfa markdown: Fix imports for compatibility with typeshed stubs.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-11-10 15:54:27 -08:00
Anders Kaseorg ac5cbf7693 Revert "markdown: Escape lang when echoing back custom non-pygments languages."
This reverts commit 564b199fe6, which
was part of #16308.

Escaping is either required or incorrect; it is never “defensive”.
This escaping is incorrect.  lxml already escapes attributes during
serialization (any other behavior would be a serious bug), and
additional escaping just results in double escaping.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-11-02 16:23:48 -08:00
akshatdalton 620e9cbf72 markdown: Fix merging of separate quotations.
Initally, when writing two or more quotes, having
a blank line in between them, merges those quotes.
This created confusion especially in "quote and reply".

This commit fixes such issues. Now two or more quotes
having a blank line in between them, will not get merged.

This change is correct both for usability and for improving our
compatibility with CommonMark.

Fixes #14379.
2020-10-30 15:21:15 -07:00
Anders Kaseorg 9281dccae4 python: Serialize lxml elements directly to str.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-30 11:36:38 -07:00
Anders Kaseorg 1352f2f233 python: Replace manual quote_plus usage with urlencode.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-27 13:47:02 -07:00
Anders Kaseorg 72d6ff3c3b docs: Fix more capitalization issues.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-23 11:46:55 -07:00
Anders Kaseorg e513b75e86 markdown: Remove handler for old bug with incompatible twitter library.
See commit 8b002040e0 and #86.  The
development environment bug that necessitated this handler has long
been irrelevant.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-23 11:30:26 -07:00
Anders Kaseorg 254b904965 markdown: Migrate off deprecated extension registration interface.
Fixes #15205.

https://python-markdown.github.io/change_log/release-3.0/#homegrown-ordereddict-has-been-replaced-with-a-purpose-built-registry
https://python-markdown.github.io/change_log/release-3.0/#md_globals-keyword-deprecated-from-extension-api

The priority numbers are arbitrarily chosen to preserve the existing
order.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-19 18:31:12 -07:00
akshatdalton 287c4ed2bb markdown: Fix Youtube and Vimeo preview overriding markdown link titles bug.
Initially markdown titles were overridden by Youtube and Vimeo preview titles.
But now it will check if any markdown title is present to replace Youtube or
Vimeo preview titles, if preview of linked websites is enabled.
Fixes #16100
2020-10-19 12:06:13 -07:00