Mirror
/
zulip
mirror of https://github.com/zulip/zulip.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity

linkifier: Support URL templates for linkifiers. 

This swaps out url_format_string from all of our APIs and replaces it
with url_template. Note that the documentation changes in the following
commits  will be squashed with this commit.

We change the "url_format" key to "url_template" for the
realm_linkifiers events in event_schema, along with updating
LinkifierDict. "url_template" is the name chosen to normalize
mixed usages of "url_format_string" and "url_format" throughout
the backend.

The markdown processor is updated to stop handling the format string
interpolation and delegate the task template expansion to the uri_template
library instead.

This change affects many test cases. We mostly just replace "%(name)s"
with "{name}", "url_format_string" with "url_template" to make sure that
they still pass. There are some test cases dedicated for testing "%"
escaping, which aren't relevant anymore and are subject to removal.
But for now we keep most of them as-is, and make sure that "%" is always
escaped since we do not use it for variable substitution any more.

Since url_format_string is not populated anymore, a migration is created
to remove this field entirely, and make url_template non-nullable since
we will always populate it. Note that it is possible to have
url_template being null after migration 0422 and before 0424, but
in practice, url_template will not be None after backfilling and the
backend now is always setting url_template.

With the removal of url_format_string, RealmFilter model will now be cleaned
with URL template checks, and the old checks for escapes are removed.

We also modified RealmFilter.clean to skip the validation when the
url_template is invalid. This avoids raising mulitple ValidationError's
when calling full_clean on a linkifier. But we might eventually want to
have a more centric approach to data validation instead of having
the same validation in both the clean method and the validator.

Fixes #23124.

Signed-off-by: Zixuan James Li <p359101898@gmail.com>
This commit is contained in:
Zixuan James Li 2022-10-05 14:55:31 -04:00 committed by Tim Abbott
parent e855be0f9a
commit b7bfa5801c
41 changed files with 533 additions and 307 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

16
api_docs/changelog.md
View File

@ -20,6 +20,21 @@ format used by the Zulip server that they are interacting with.
## Changes in Zulip 7.0
**Feature level 176**
* [`POST /realm/filters`](/api/add-linkifier), [`realm/filters/<int:filter_id>`](/api/update-linkifier):
The parameter `url_format_string` is replaced by `url_template`.
The linkifiers now accept only [RFC 6570][rfc6570] compliant URL Templates.
The old URL format strings are no longer supported.
* [`GET /events`](/api/get-events), [`POST /register`](/api/register-queue):
The key `url_format_string` is replaced by `url_template` for the `realm_linkifiers`
event type. For backwards-compatibility, clients that do not support the
`linkifier_url_template`
[client capability](/api/register-queue#parameter-client_capabilities)
will get an empty list in the response of `/register` and not receive `realm_linkifiers`
events. Unconditionally, the deprecated event type `realm_filters` gives an empty list in the
response of `/register` and is no longer sent the clients otherwise.
**Feature level 175**
* [`POST /register`](/api/register-queue), [`PATCH /settings`](/api/update-settings),
@ -1463,3 +1478,4 @@ No changes; feature level used for Zulip 3.0 release.
[server-changelog]: https://zulip.readthedocs.io/en/latest/overview/changelog.html
[release-lifecycle]: https://zulip.readthedocs.io/en/latest/overview/release-lifecycle.html
[rfc6570]: https://www.rfc-editor.org/rfc/rfc6570.html

12
help/add-a-custom-linkifier.md
View File

@ -7,6 +7,10 @@ party issue trackers, like GitHub, Salesforce, Zendesk, and others.
For instance, you can add a linkifier that automatically turns `#2468`
into a link to `https://github.com/zulip/zulip/issues/2468`.
Linkifiers use [RFC 6570](https://www.rfc-editor.org/rfc/rfc6570.html)
compliant URL templates to represent the link to be generated from the
pattern.
If the pattern appears in a topic, Zulip adds an **Open**
(<i class="fa fa-external-link-square"></i>) button to the right of the
topic in the message recipient bar that links to the appropriate URL.
@ -18,7 +22,7 @@ topic in the message recipient bar that links to the appropriate URL.
{settings_tab|linkifier-settings}
1. Under **Add a new linkifier**, enter a **Pattern** and
**URL format string**.
**URL template**.
1. Click **Add linkifier**.
@ -31,21 +35,21 @@ This is best explained by example.
Hash followed by a number of any length.
* Pattern: `#(?P<id>[0-9]+)`
* URL format string: `https://github.com/zulip/zulip/issues/%(id)s`
* URL template: `https://github.com/zulip/zulip/issues/{id}`
* Original text: `#2468`
* Automatically links to: `https://github.com/zulip/zulip/issues/2468`
String of hexadecimal digits between 7 and 40 characters long.
* Pattern: `(?P<id>[0-9a-f]{7,40})`
* URL format string: `https://github.com/zulip/zulip/commit/%(id)s`
* URL template: `https://github.com/zulip/zulip/commit/{id}`
* Original text: `abdc123`
* Automatically links to: `https://github.com/zulip/zulip/commit/abcd123`
Generic GitHub `org/repo#ID` format:
* Pattern: `(?P<org>[a-zA-Z0-9_-]+)/(?P<repo>[a-zA-Z0-9_-]+)#(?P<id>[0-9]+)`
* URL format string: `https://github.com/%(org)s/%(repo)s/issues/%(id)s`
* URL template: `https://github.com/{org}/{repo}/issues/{id}`
* Original text: `zulip/zulip#2468`
* Automatically links to: `https://github.com/zulip/zulip/issues/2468`

1
package.json
View File

@ -77,6 +77,7 @@
"tippy.js": "^6.3.7",
"turndown": "^7.0.0",
"url-loader": "^4.1.1",
"url-template": "2.0.8",
"webfonts-loader": "^8.0.0",
"webpack": "^5.61.0",
"webpack-bundle-tracker": "^1.2.0",

7
pnpm-lock.yaml
View File

@ -229,6 +229,9 @@ dependencies:
url-loader:
specifier: ^4.1.1
version: 4.1.1(webpack@5.77.0)
url-template:
specifier: 2.0.8
version: 2.0.8
webfonts-loader:
specifier: ^8.0.0
version: 8.0.1
@ -11640,6 +11643,10 @@ packages:
requires-port: 1.0.0
dev: true
/url-template@2.0.8:
resolution: {integrity: sha512-XdVKMF4SJ0nP/O7XIPB0JwAEuT9lDIYnNsK8yGVe43y0AWoKeJNdv3ZNWh7ksJ6KqQFjOO6ox/VEitLnaVNufw==}
dev: false
/util-deprecate@1.0.2:
resolution: {integrity: sha512-EPD5q1uXyFxJpCrLnCc1nHnq3gOa6DZBocAIiI2TaSCA7VCJ1UJDMagCzIkXNsUYfD1daK//LTEQ8xiIbrHtcw==}

1
tools/test-js-with-node
View File

@ -210,6 +210,7 @@ EXEMPT_FILES = make_set(
"web/src/unread_ui.js",
"web/src/upload.js",
"web/src/upload_widget.ts",
"web/src/url-template.d.ts",
"web/src/user_group_create.js",
"web/src/user_group_create_members.js",
"web/src/user_group_create_members_data.js",

2
version.py
View File

@ -48,4 +48,4 @@ API_FEATURE_LEVEL = 175
# historical commits sharing the same major version, in which case a
# minor version bump suffices.
PROVISION_VERSION = (233, 3)
PROVISION_VERSION = (234, 0)

16
web/e2e-tests/realm-linkifier.test.ts
View File

@ -8,7 +8,7 @@ async function test_add_linkifier(page: Page): Promise<void> {
await page.waitForSelector(".admin-linkifier-form", {visible: true});
await common.fill_form(page, "form.admin-linkifier-form", {
pattern: "#(?P<id>[0-9]+)",
url_format_string: "https://trac.example.com/ticket/%(id)s",
url_template: "https://trac.example.com/ticket/{id}",
});
await page.click("form.admin-linkifier-form button.button");
@ -30,7 +30,7 @@ async function test_add_linkifier(page: Page): Promise<void> {
page,
".linkifier_row span.linkifier_url_format_string",
),
"https://trac.example.com/ticket/%(id)s",
"https://trac.example.com/ticket/{id}",
);
}
@ -46,7 +46,7 @@ async function test_add_invalid_linkifier_pattern(page: Page): Promise<void> {
await page.waitForSelector(".admin-linkifier-form", {visible: true});
await common.fill_form(page, "form.admin-linkifier-form", {
pattern: "(foo",
url_format_string: "https://trac.example.com/ticket/%(id)s",
url_template: "https://trac.example.com/ticket/{id}",
});
await page.click("form.admin-linkifier-form button.button");
@ -62,7 +62,7 @@ async function test_edit_linkifier(page: Page): Promise<void> {
await common.wait_for_micromodal_to_open(page);
await common.fill_form(page, "form.linkifier-edit-form", {
pattern: "(?P<num>[0-9a-f]{40})",
url_format_string: "https://trac.example.com/commit/%(num)s",
url_template: "https://trac.example.com/commit/{num}",
});
await page.click(".dialog_submit_button");
@ -78,7 +78,7 @@ async function test_edit_linkifier(page: Page): Promise<void> {
page,
".linkifier_row span.linkifier_url_format_string",
),
"https://trac.example.com/commit/%(num)s",
"https://trac.example.com/commit/{num}",
);
}
@ -87,7 +87,7 @@ async function test_edit_invalid_linkifier(page: Page): Promise<void> {
await common.wait_for_micromodal_to_open(page);
await common.fill_form(page, "form.linkifier-edit-form", {
pattern: "#(?P<id>d????)",
url_format_string: "????",
url_template: "{id",
});
await page.click(".dialog_submit_button");
@ -108,7 +108,7 @@ async function test_edit_invalid_linkifier(page: Page): Promise<void> {
page,
edit_linkifier_format_status_selector,
);
assert.strictEqual(edit_linkifier_format_status, "Failed: Enter a valid URL.");
assert.strictEqual(edit_linkifier_format_status, "Failed: Invalid URL template.");
await page.click(".dialog_cancel_button");
await page.waitForSelector("#dialog_widget_modal", {hidden: true});
@ -123,7 +123,7 @@ async function test_edit_invalid_linkifier(page: Page): Promise<void> {
page,
".linkifier_row span.linkifier_url_format_string",
),
"https://trac.example.com/commit/%(num)s",
"https://trac.example.com/commit/{num}",
);
}

36
web/src/linkifiers.ts
View File

@ -1,29 +1,39 @@
import url_template_lib from "url-template";
import * as blueslip from "./blueslip";
const linkifier_map = new Map(); // regex -> url
type LinkifierMap = Map<
RegExp,
{url_template: url_template_lib.Template; group_number_to_name: Record<number, string>}
>;
const linkifier_map: LinkifierMap = new Map();
type Linkifier = {
pattern: string;
url_format: string;
url_template: string;
id: number;
};
export function get_linkifier_map(): Map<RegExp, string> {
export function get_linkifier_map(): LinkifierMap {
return linkifier_map;
}
function python_to_js_linkifier(pattern: string, url: string): [RegExp | null, string] {
function python_to_js_linkifier(
pattern: string,
url: string,
): [RegExp | null, url_template_lib.Template, Record<number, string>] {
// Converts a python named-group regex to a javascript-compatible numbered
// group regex... with a regex!
const named_group_re = /\(?P<([^>]+?)>/g;
let match = named_group_re.exec(pattern);
let current_group = 1;
const group_number_to_name: Record<number, string> = {};
while (match) {
const name = match[1];
// Replace named group with regular matching group
pattern = pattern.replace("(?P<" + name + ">", "(");
// Replace named reference in URL to numbered reference
url = url.replace("%(" + name + ")s", `\\${current_group}`);
// Map numbered reference to named reference for template expansion
group_number_to_name[current_group] = name;
// Reset the RegExp state
named_group_re.lastIndex = 0;
@ -73,20 +83,28 @@ function python_to_js_linkifier(pattern: string, url: string): [RegExp | null, s
throw error;
}
}
return [final_regex, url];
const url_template = url_template_lib.parse(url);
blueslip.info(`Linkifier info ${String(final_regex)} ${url}`, group_number_to_name);
return [final_regex, url_template, group_number_to_name];
}
export function update_linkifier_rules(linkifiers: Linkifier[]): void {
linkifier_map.clear();
for (const linkifier of linkifiers) {
const [regex, final_url] = python_to_js_linkifier(linkifier.pattern, linkifier.url_format);
const [regex, url_template, group_number_to_name] = python_to_js_linkifier(
linkifier.pattern,
linkifier.url_template,
);
if (!regex) {
// Skip any linkifiers that could not be converted
continue;
}
linkifier_map.set(regex, final_url);
linkifier_map.set(regex, {
url_template,
group_number_to_name,
});
}
}

16
web/src/markdown.js
View File

@ -294,19 +294,19 @@ export function get_topic_links({topic, get_linkifier_map}) {
// The lower the precedence is, the more prioritized the pattern is.
let precedence = 0;
for (const [pattern, url] of get_linkifier_map().entries()) {
for (const [pattern, {url_template, group_number_to_name}] of get_linkifier_map().entries()) {
let match;
while ((match = pattern.exec(topic)) !== null) {
let link_url = url;
const matched_groups = match.slice(1);
let i = 0;
const template_context = {};
while (i < matched_groups.length) {
const matched_group = matched_groups[i];
const current_group = i + 1;
const back_ref = "\\" + current_group;
link_url = link_url.replace(back_ref, matched_group);
template_context[group_number_to_name[current_group]] = matched_group;
i += 1;
}
const link_url = url_template.expand(template_context);
// We store the starting index as well, to sort the order of occurrence of the links
// in the topic, similar to the logic implemented in zerver/lib/markdown/__init__.py
links.push({url: link_url, text: match[0], index: match.index, precedence});
@ -410,17 +410,17 @@ function handleEmoji({emoji_name, get_realm_emoji_url, get_emoji_codepoint}) {
}
function handleLinkifier({pattern, matches, get_linkifier_map}) {
let url = get_linkifier_map().get(pattern);
const {url_template, group_number_to_name} = get_linkifier_map().get(pattern);
let current_group = 1;
const template_context = {};
for (const match of matches) {
const back_ref = "\\" + current_group;
url = url.replace(back_ref, match);
template_context[group_number_to_name[current_group]] = match;
current_group += 1;
}
return url;
return url_template.expand(template_context);
}
function handleTimestamp(time) {

12
web/src/settings_linkifiers.js
View File

@ -42,7 +42,7 @@ function sort_pattern(a, b) {
}
function sort_url(a, b) {
return compare_values(a.url_format, b.url_format);
return compare_values(a.url_template, b.url_template);
}
function open_linkifier_edit_form(linkifier_id) {
@ -51,7 +51,7 @@ function open_linkifier_edit_form(linkifier_id) {
const html_body = render_admin_linkifier_edit_form({
linkifier_id,
pattern: linkifier.pattern,
url_format_string: linkifier.url_format,
url_format_string: linkifier.url_template,
});
function submit_linkifier_form() {
@ -118,8 +118,8 @@ function handle_linkifier_api_error(xhr, pattern_status, format_status, linkifie
xhr.responseText = JSON.stringify({msg: errors.pattern});
ui_report.error($t_html({defaultMessage: "Failed"}), xhr, pattern_status);
}
if (errors.url_format_string !== undefined) {
xhr.responseText = JSON.stringify({msg: errors.url_format_string});
if (errors.url_template !== undefined) {
xhr.responseText = JSON.stringify({msg: errors.url_template});
ui_report.error($t_html({defaultMessage: "Failed"}), xhr, format_status);
}
if (errors.__all__ !== undefined) {
@ -140,7 +140,7 @@ export function populate_linkifiers(linkifiers_data) {
return render_admin_linkifier_list({
linkifier: {
pattern: linkifier.pattern,
url_format_string: linkifier.url_format,
url_format_string: linkifier.url_template,
id: linkifier.id,
},
can_modify: page_params.is_admin,
@ -151,7 +151,7 @@ export function populate_linkifiers(linkifiers_data) {
predicate(item, value) {
return (
item.pattern.toLowerCase().includes(value) ||
item.url_format.toLowerCase().includes(value)
item.url_template.toLowerCase().includes(value)
);
},
onupdate() {

19
web/src/url-template.d.ts vendored Normal file
View File

@ -0,0 +1,19 @@
// We need to use an older version of url-template because our test suite setup does not
// support testing with ESM-only modules. Unfortunately, the older version also happens to
// not have type definitions, so we maintain our own copy of it. This is adapted from:
// https://github.com/bramstein/url-template/blob/a8e204a92de3168a56ef2e528ae4d841287636fd/lib/url-template.d.ts
declare module "url-template" {
export type PrimitiveValue = string | number | boolean | null;
export interface Template {
expand(
context: Record<
string,
PrimitiveValue | PrimitiveValue[] | Record<string, PrimitiveValue>
>,
): string;
}
export function parse(template: string): Template;
}

4
web/templates/settings/admin_linkifier_edit_form.hbs
View File

@ -6,8 +6,8 @@
<div class="alert" id="edit-linkifier-pattern-status"></div>
</div>
<div class="input-group name_change_container">
<label for="edit-linkifier-url-format-string" >{{t "URL format string" }}</label>
<input type="text" autocomplete="off" id="edit-linkifier-url-format-string" name="url_format_string" placeholder="https://github.com/zulip/zulip/issues/%(id)s" value="{{ url_format_string }}" />
<label for="edit-linkifier-url-format-string" >{{t "URL template" }}</label>
<input type="text" autocomplete="off" id="edit-linkifier-url-format-string" name="url_template" placeholder="https://github.com/zulip/zulip/issues/{id}" value="{{ url_format_string }}" />
<div class="alert" id="edit-linkifier-format-status"></div>
</div>
</form>

8
web/templates/settings/linkifier_settings_admin.hbs
View File

@ -18,7 +18,7 @@
{{t "Pattern" }}: <span class="rendered_markdown"><code>#(?P&lt;id&gt;[0-9]+)</code></span>
</li>
<li>
{{t "URL format string" }}: <span class="rendered_markdown"><code>https://github.com/zulip/zulip/issues/%(id)s</code></span>
{{t "URL template" }}: <span class="rendered_markdown"><code>https://github.com/zulip/zulip/issues/{id}</code></span>
</li>
</ul>
<p>
@ -44,8 +44,8 @@
<div class="alert" id="admin-linkifier-pattern-status"></div>
</div>
<div class="input-group">
<label for="linkifier_format_string">{{t "URL format string" }}</label>
<input type="text" id="linkifier_format_string" name="url_format_string" placeholder="https://github.com/zulip/zulip/issues/%(id)s" />
<label for="linkifier_format_string">{{t "URL template" }}</label>
<input type="text" id="linkifier_format_string" name="url_template" placeholder="https://github.com/zulip/zulip/issues/{id}" />
<div class="alert" id="admin-linkifier-format-status"></div>
</div>
<button type="submit" class="button rounded sea-green">
@ -66,7 +66,7 @@
<table class="table table-condensed table-striped wrapped-table admin_linkifiers_table">
<thead class="table-sticky-headers">
<th class="active" data-sort="pattern">{{t "Pattern" }}</th>
<th data-sort="url">{{t "URL format string" }}</th>
<th data-sort="url">{{t "URL template" }}</th>
{{#if is_admin}}
<th class="actions">{{t "Actions" }}</th>
{{/if}}

2
web/tests/lib/events.js
View File

@ -505,7 +505,7 @@ exports.fixtures = {
realm_linkifiers: [
{
pattern: "#[123]",
url_format: "ticket %(id)s",
url_template: "ticket {id}",
id: 55,
},
],

8
web/tests/linkifiers.test.js
View File

@ -20,12 +20,12 @@ run_test("python_to_js_linkifier", () => {
linkifiers.update_linkifier_rules([
{
pattern: "/a(?im)a/g",
url_format: "http://example1.example.com",
url_template: "http://example1.example.com",
id: 10,
},
{
pattern: "/a(?L)a/g",
url_format: "http://example2.example.com",
url_template: "http://example2.example.com",
id: 20,
},
]);
@ -36,7 +36,7 @@ run_test("python_to_js_linkifier", () => {
linkifiers.update_linkifier_rules([
{
pattern: "#cf(?P<contest>\\d+)(?P<problem>[A-Z][\\dA-Z]*)",
url_format: "http://example3.example.com",
url_template: "http://example3.example.com",
id: 30,
},
]);
@ -51,7 +51,7 @@ run_test("python_to_js_linkifier", () => {
linkifiers.update_linkifier_rules([
{
pattern: "!@#@(!#&((!&(@#(",
url_format: "http://example4.example.com",
url_template: "http://example4.example.com",
id: 40,
},
]);

31
web/tests/markdown.test.js
View File

@ -12,19 +12,31 @@ const {page_params, user_settings} = require("./lib/zpage_params");
const example_realm_linkifiers = [
{
pattern: "#(?P<id>[0-9]{2,8})",
url_format: "https://trac.example.com/ticket/%(id)s",
url_template: "https://trac.example.com/ticket/{id}",
id: 1,
},
{
pattern: "ZBUG_(?P<id>[0-9]{2,8})",
url_format: "https://trac2.zulip.net/ticket/%(id)s",
url_template: "https://trac2.zulip.net/ticket/{id}",
id: 2,
},
{
pattern: "ZGROUP_(?P<id>[0-9]{2,8}):(?P<zone>[0-9]{1,8})",
url_format: "https://zone_%(zone)s.zulip.net/ticket/%(id)s",
url_template: "https://zone_{zone}.zulip.net/ticket/{id}",
id: 3,
},
{
// For example, this linkifier matches:
// FOO_abcde;e;zulip;luxembourg;foo;23;testing
// which expands to:
// https://zone_e.zulip.net/ticket/luxembourg/abcde?name=foo&chapter=23#testi
// This exercises different URL template supported syntax.
pattern:
"FOO_(?P<id>[a-f]{5});(?P<zone>[a-f]);(?P<domain>[a-z]+);(?P<location>[a-z]+);(?P<name>[a-z]{2,8});(?P<chapter>[0-9]{2,3});(?P<fragment>[a-z]{2,8})",
url_template:
"https://zone_{zone}{.domain}.net/ticket{/location}{/id}{?name,chapter}{#fragment:5}",
id: 4,
},
];
user_settings.translate_emoticons = false;
@ -431,6 +443,11 @@ test("marked", () => {
expected:
'<p>This is a linkifier <a href="https://trac.example.com/ticket/1234" title="https://trac.example.com/ticket/1234">#1234</a> with text after it</p>',
},
{
input: "This is a complicated linkifier FOO_abcde;e;zulip;luxembourg;foo;23;testing with text after it",
expected:
'<p>This is a complicated linkifier <a href="https://zone_e.zulip.net/ticket/luxembourg/abcde?name=foo&amp;chapter=23#testi" title="https://zone_e.zulip.net/ticket/luxembourg/abcde?name=foo&amp;chapter=23#testi">FOO_abcde;e;zulip;luxembourg;foo;23;testing</a> with text after it</p>',
},
{input: "#1234is not a linkifier.", expected: "<p>#1234is not a linkifier.</p>"},
{
input: "A pattern written as #1234is not a linkifier.",
@ -652,6 +669,14 @@ test("topic_links", () => {
message = {type: "not-stream"};
markdown.add_topic_links(message);
assert.equal(message.topic_links.length, 0);
message = {type: "stream", topic: "FOO_abcde;e;zulip;luxembourg;foo;23;testing"};
markdown.add_topic_links(message);
assert.equal(message.topic_links.length, 1);
assert.deepEqual(message.topic_links[0], {
url: "https://zone_e.zulip.net/ticket/luxembourg/abcde?name=foo&chapter=23#testi",
text: "FOO_abcde;e;zulip;luxembourg;foo;23;testing",
});
});
test("message_flags", () => {

27
web/tests/markdown_parse.test.js
View File

@ -2,6 +2,8 @@
const {strict: assert} = require("assert");
const url_template_lib = require("url-template");
const {zrequire} = require("./lib/namespace");
const {run_test} = require("./lib/test");
@ -114,7 +116,10 @@ function get_realm_emoji_url(emoji_name) {
const regex = /#foo(\d+)(?!\w)/g;
const linkifier_map = new Map();
linkifier_map.set(regex, "http://foo.com/\\1");
linkifier_map.set(regex, {
url_template: url_template_lib.parse("http://foo.com/{id}"),
group_number_to_name: {1: "id"},
});
function get_linkifier_map() {
return linkifier_map;
@ -226,7 +231,7 @@ function assert_topic_links(topic, expected_links) {
}
run_test("topic links", () => {
linkifiers.initialize([{pattern: "#foo(?P<id>\\d+)", url_format: "http://foo.com/%(id)s"}]);
linkifiers.initialize([{pattern: "#foo(?P<id>\\d+)", url_template: "http://foo.com/{id}"}]);
assert_topic_links("progress on #foo101 and #foo102", [
{
text: "#foo101",
@ -243,7 +248,7 @@ run_test("topic links repeated", () => {
// Links generated from repeated patterns should preserve the order.
const topic =
"#foo101 https://google.com #foo102 #foo103 https://google.com #foo101 #foo102 #foo103";
linkifiers.initialize([{pattern: "#foo(?P<id>\\d+)", url_format: "http://foo.com/%(id)s"}]);
linkifiers.initialize([{pattern: "#foo(?P<id>\\d+)", url_template: "http://foo.com/{id}"}]);
assert_topic_links(topic, [
{
text: "#foo101",
@ -282,10 +287,10 @@ run_test("topic links repeated", () => {
run_test("topic links overlapping", () => {
linkifiers.initialize([
{pattern: "[a-z]+(?P<id>1\\d+) #[a-z]+", url_format: "http://a.com/%(id)s"},
{pattern: "[a-z]+(?P<id>1\\d+)", url_format: "http://b.com/%(id)s"},
{pattern: ".+#(?P<id>[a-z]+)", url_format: "http://wildcard.com/%(id)s"},
{pattern: "#(?P<id>[a-z]+)", url_format: "http://c.com/%(id)s"},
{pattern: "[a-z]+(?P<id>1\\d+) #[a-z]+", url_template: "http://a.com/{id}"},
{pattern: "[a-z]+(?P<id>1\\d+)", url_template: "http://b.com/{id}"},
{pattern: ".+#(?P<id>[a-z]+)", url_template: "http://wildcard.com/{id}"},
{pattern: "#(?P<id>[a-z]+)", url_template: "http://c.com/{id}"},
]);
// b.com's pattern should be matched while it overlaps with c.com's.
assert_topic_links("#foo100", [
@ -342,13 +347,13 @@ run_test("topic links overlapping", () => {
run_test("topic links ordering by priority", () => {
// The same test case is also implemented in zerver/tests/test_markdown.py
linkifiers.initialize([
{pattern: "http", url_format: "http://example.com/"},
{pattern: "b#(?P<id>[a-z]+)", url_format: "http://example.com/b/%(id)s"},
{pattern: "http", url_template: "http://example.com/"},
{pattern: "b#(?P<id>[a-z]+)", url_template: "http://example.com/b/{id}"},
{
pattern: "a#(?P<aid>[a-z]+) b#(?P<bid>[a-z]+)",
url_format: "http://example.com/a/%(aid)s/b/%(bid)",
url_template: "http://example.com/a/{aid}/b/%(bid)",
},
{pattern: "a#(?P<id>[a-z]+)", url_format: "http://example.com/a/%(id)s"},
{pattern: "a#(?P<id>[a-z]+)", url_template: "http://example.com/a/{id}"},
]);
// There should be 5 link matches in the topic, if ordered from the most priortized to the least:

37
zerver/actions/realm_linkifiers.py
View File

@ -12,7 +12,6 @@ from zerver.models import (
UserProfile,
active_user_ids,
linkifiers_for_realm,
realm_filters_for_realm,
)
from zerver.tornado.django_api import send_event
@ -21,13 +20,6 @@ def notify_linkifiers(realm: Realm, realm_linkifiers: List[LinkifierDict]) -> No
event: Dict[str, object] = dict(type="realm_linkifiers", realm_linkifiers=realm_linkifiers)
transaction.on_commit(lambda: send_event(realm, event, active_user_ids(realm.id)))
# Below is code for backwards compatibility. The now deprecated
# "realm_filters" event-type is used by older clients, and uses
# tuples.
realm_filters = realm_filters_for_realm(realm.id)
legacy_event = dict(type="realm_filters", realm_filters=realm_filters)
transaction.on_commit(lambda: send_event(realm, legacy_event, active_user_ids(realm.id)))
# NOTE: Regexes must be simple enough that they can be easily translated to JavaScript
# RegExp syntax. In addition to JS-compatible syntax, the following features are available:
@ -35,11 +27,15 @@ def notify_linkifiers(realm: Realm, realm_linkifiers: List[LinkifierDict]) -> No
# * Inline-regex flags will be stripped, and where possible translated to RegExp-wide flags
@transaction.atomic(durable=True)
def do_add_linkifier(
realm: Realm, pattern: str, url_format_string: str, *, acting_user: Optional[UserProfile]
realm: Realm,
pattern: str,
url_template: str,
*,
acting_user: Optional[UserProfile],
) -> int:
pattern = pattern.strip()
url_format_string = url_format_string.strip()
linkifier = RealmFilter(realm=realm, pattern=pattern, url_format_string=url_format_string)
url_template = url_template.strip()
linkifier = RealmFilter(realm=realm, pattern=pattern, url_template=url_template)
linkifier.full_clean()
linkifier.save()
@ -54,7 +50,7 @@ def do_add_linkifier(
"realm_linkifiers": realm_linkifiers,
"added_linkifier": LinkifierDict(
pattern=pattern,
url_format=url_format_string,
url_template=url_template,
id=linkifier.id,
),
}
@ -80,7 +76,7 @@ def do_remove_linkifier(
realm_linkifier = RealmFilter.objects.get(realm=realm, id=id)
pattern = realm_linkifier.pattern
url_format = realm_linkifier.url_format_string
url_template = realm_linkifier.url_template
realm_linkifier.delete()
realm_linkifiers = linkifiers_for_realm(realm.id)
@ -92,10 +88,7 @@ def do_remove_linkifier(
extra_data=orjson.dumps(
{
"realm_linkifiers": realm_linkifiers,
"removed_linkifier": {
"pattern": pattern,
"url_format": url_format,
},
"removed_linkifier": {"pattern": pattern, "url_template": url_template},
}
).decode(),
)
@ -107,17 +100,17 @@ def do_update_linkifier(
realm: Realm,
id: int,
pattern: str,
url_format_string: str,
url_template: str,
*,
acting_user: Optional[UserProfile],
) -> None:
pattern = pattern.strip()
url_format_string = url_format_string.strip()
url_template = url_template.strip()
linkifier = RealmFilter.objects.get(realm=realm, id=id)
linkifier.pattern = pattern
linkifier.url_format_string = url_format_string
linkifier.url_template = url_template
linkifier.full_clean()
linkifier.save(update_fields=["pattern", "url_format_string"])
linkifier.save(update_fields=["pattern", "url_template"])
realm_linkifiers = linkifiers_for_realm(realm.id)
RealmAuditLog.objects.create(
@ -130,7 +123,7 @@ def do_update_linkifier(
"realm_linkifiers": realm_linkifiers,
"changed_linkifier": LinkifierDict(
pattern=pattern,
url_format=url_format_string,
url_template=url_template,
id=linkifier.id,
),
}

24
zerver/lib/event_schema.py
View File

@ -808,7 +808,7 @@ def check_realm_export(
realm_linkifier_type = DictType(
required_keys=[
("pattern", str),
("url_format", str),
("url_template", str),
("id", int),
]
)
@ -822,28 +822,6 @@ realm_linkifiers_event = event_dict_type(
check_realm_linkifiers = make_checker(realm_linkifiers_event)
# This is a legacy event type to ensure backwards compatibility
# for old clients. Newer clients should handle only the
# "realm_linkifiers" event above.
realm_filter_type = TupleType(
[
# we should make this an object
# see realm_filters_for_realm_remote_cache
str, # pattern
str, # format string
int, # id
]
)
realm_filters_event = event_dict_type(
[
# force vertical
("type", Equals("realm_filters")),
("realm_filters", ListType(realm_filter_type)),
]
)
check_realm_filters = make_checker(realm_filters_event)
plan_type_extra_data_type = DictType(
required_keys=[
# force vertical

31
zerver/lib/events.py
View File

@ -70,7 +70,6 @@ from zerver.models import (
get_realm_domains,
get_realm_playgrounds,
linkifiers_for_realm,
realm_filters_for_realm,
)
from zerver.tornado.django_api import get_user_events, request_event_queue
from zproject.backends import email_auth_enabled, password_auth_enabled
@ -115,6 +114,7 @@ def fetch_initial_state_data(
include_streams: bool = True,
spectator_requested_language: Optional[str] = None,
pronouns_field_type_supported: bool = True,
linkifier_url_template: bool = False,
) -> Dict[str, Any]:
"""When `event_types` is None, fetches the core data powering the
web app's `page_params` and `/api/v1/register` (for mobile/terminal
@ -385,11 +385,20 @@ def fetch_initial_state_data(
state["realm_emoji"] = realm.get_emoji()
if want("realm_linkifiers"):
state["realm_linkifiers"] = linkifiers_for_realm(realm.id)
if linkifier_url_template:
state["realm_linkifiers"] = linkifiers_for_realm(realm.id)
else:
# When URL template is not supported by the client, return an empty list
# because the new format is incompatible with the old URL format strings
# and the client would not render it properly.
state["realm_linkifiers"] = []
# Backwards compatibility code.
if want("realm_filters"):
state["realm_filters"] = realm_filters_for_realm(realm.id)
# Always return an empty list because the new URL template format is incompatible
# with the old URL format string, because legacy clients that use the
# backwards-compatible `realm_filters` event would not render the it properly.
state["realm_filters"] = []
if want("realm_playgrounds"):
state["realm_playgrounds"] = get_realm_playgrounds(realm)
@ -648,6 +657,7 @@ def apply_events(
client_gravatar: bool,
slim_presence: bool,
include_subscribers: bool,
linkifier_url_template: bool,
) -> None:
for event in events:
if event["type"] == "restart":
@ -669,6 +679,7 @@ def apply_events(
client_gravatar=client_gravatar,
slim_presence=slim_presence,
include_subscribers=include_subscribers,
linkifier_url_template=linkifier_url_template,
)
@ -680,6 +691,7 @@ def apply_event(
client_gravatar: bool,
slim_presence: bool,
include_subscribers: bool,
linkifier_url_template: bool,
) -> None:
if event["type"] == "message":
state["max_message_id"] = max(state["max_message_id"], event["message"]["id"])
@ -1266,10 +1278,12 @@ def apply_event(
state["muted_topics"] = event["muted_topics"]
elif event["type"] == "muted_users":
state["muted_users"] = event["muted_users"]
elif event["type"] == "realm_filters":
state["realm_filters"] = event["realm_filters"]
elif event["type"] == "realm_linkifiers":
state["realm_linkifiers"] = event["realm_linkifiers"]
# We only send realm_linkifiers event to clients that indicate
# support for linkifiers with URL templates. Otherwise, silently
# ignore the event.
if linkifier_url_template:
state["realm_linkifiers"] = event["realm_linkifiers"]
elif event["type"] == "realm_playgrounds":
state["realm_playgrounds"] = event["realm_playgrounds"]
elif event["type"] == "update_display_settings":
@ -1425,6 +1439,7 @@ def do_events_register(
)
stream_typing_notifications = client_capabilities.get("stream_typing_notifications", False)
user_settings_object = client_capabilities.get("user_settings_object", False)
linkifier_url_template = client_capabilities.get("linkifier_url_template", False)
if fetch_event_types is not None:
event_types_set: Optional[Set[str]] = set(fetch_event_types)
@ -1445,6 +1460,7 @@ def do_events_register(
queue_id=None,
# Force client_gravatar=False for security reasons.
client_gravatar=client_gravatar,
linkifier_url_template=linkifier_url_template,
user_avatar_url_field_optional=user_avatar_url_field_optional,
user_settings_object=user_settings_object,
# slim_presence is a noop, because presence is not included.
@ -1479,6 +1495,7 @@ def do_events_register(
stream_typing_notifications=stream_typing_notifications,
user_settings_object=user_settings_object,
pronouns_field_type_supported=pronouns_field_type_supported,
linkifier_url_template=linkifier_url_template,
)
if queue_id is None:
@ -1495,6 +1512,7 @@ def do_events_register(
include_subscribers=include_subscribers,
include_streams=include_streams,
pronouns_field_type_supported=pronouns_field_type_supported,
linkifier_url_template=linkifier_url_template,
)
# Apply events that came in while we were fetching initial data
@ -1508,6 +1526,7 @@ def do_events_register(
client_gravatar=client_gravatar,
slim_presence=slim_presence,
include_subscribers=include_subscribers,
linkifier_url_template=linkifier_url_template,
)
except RestartEventError:
# This represents a rare race condition, where Tornado

1
zerver/lib/home.py
View File

@ -138,6 +138,7 @@ def build_page_params_for_home_page_load(
"user_avatar_url_field_optional": True,
"stream_typing_notifications": False, # Set this to True when frontend support is implemented.
"user_settings_object": True,
"linkifier_url_template": True,
}
if user_profile is not None:

14
zerver/lib/markdown/__init__.py
View File

@ -41,6 +41,7 @@ import markdown.treeprocessors
import markdown.util
import re2
import requests
import uri_template
from django.conf import settings
from markdown.blockparser import BlockParser
from markdown.extensions import codehilite, nl2br, sane_lists, tables
@ -1788,7 +1789,7 @@ class LinkifierPattern(CompiledInlineProcessor):
def __init__(
self,
source_pattern: str,
format_string: str,
url_template: str,
zmd: "ZulipMarkdown",
) -> None:
# Do not write errors to stderr (this still raises exceptions)
@ -1796,7 +1797,8 @@ class LinkifierPattern(CompiledInlineProcessor):
options.log_errors = False
compiled_re2 = re2.compile(prepare_linkifier_pattern(source_pattern), options=options)
self.format_string = percent_escape_format_string(format_string)
self.prepared_url_template = uri_template.URITemplate(url_template)
super().__init__(compiled_re2, zmd)
@ -1806,7 +1808,7 @@ class LinkifierPattern(CompiledInlineProcessor):
db_data: Optional[DbData] = self.zmd.zulip_db_data
url = url_to_a(
db_data,
self.format_string % m.groupdict(),
self.prepared_url_template.expand(**m.groupdict()),
markdown.util.AtomicString(m.group(OUTER_CAPTURE_GROUP)),
)
if isinstance(url, str):
@ -2260,7 +2262,7 @@ class ZulipMarkdown(markdown.Markdown):
for linkifier in self.linkifiers:
pattern = linkifier["pattern"]
registry.register(
LinkifierPattern(pattern, linkifier["url_format"], self),
LinkifierPattern(pattern, linkifier["url_template"], self),
f"linkifiers/{pattern}",
45,
)
@ -2368,7 +2370,7 @@ def topic_links(linkifiers_key: int, topic_name: str) -> List[Dict[str, str]]:
options.log_errors = False
for linkifier in linkifiers:
raw_pattern = linkifier["pattern"]
url_format_string = percent_escape_format_string(linkifier["url_format"])
prepared_url_template = uri_template.URITemplate(linkifier["url_template"])
try:
pattern = re2.compile(prepare_linkifier_pattern(raw_pattern), options=options)
except re2.error:
@ -2396,7 +2398,7 @@ def topic_links(linkifiers_key: int, topic_name: str) -> List[Dict[str, str]]:
# don't have to implement any logic of their own to get back the text.
matches += [
TopicLinkMatch(
url=url_format_string % match_details,
url=prepared_url_template.expand(**match_details),
text=match_text,
index=m.start(),
precedence=precedence,

2
zerver/lib/types.py
View File

@ -57,7 +57,7 @@ DisplayRecipientT = Union[str, List[UserDisplayRecipient]]
class LinkifierDict(TypedDict):
pattern: str
url_format: str
url_template: str
id: int

18
zerver/migrations/0442_remove_realmfilter_url_format_string.py Normal file
View File

@ -0,0 +1,18 @@
from django.db import migrations, models
from zerver.models import url_template_validator
class Migration(migrations.Migration):
dependencies = [
("zerver", "0441_backfill_realmfilter_url_template"),
]
operations = [
migrations.RemoveField(model_name="realmfilter", name="url_format_string"),
migrations.AlterField(
model_name="realmfilter",
name="url_template",
field=models.TextField(validators=[url_template_validator], null=False),
),
]

54
zerver/models.py
View File

@ -1300,10 +1300,7 @@ def filter_format_validator(value: str) -> None:
def url_template_validator(value: str) -> None:
"""Verifies URL-ness, and then validates as a URL template"""
# URLValidator is assumed to catch anything which is malformed as a URL
URLValidator()(value)
"""Validate as a URL template"""
if not uri_template.validate(value):
raise ValidationError(_("Invalid URL template."))
@ -1315,19 +1312,16 @@ class RealmFilter(models.Model):
realm = models.ForeignKey(Realm, on_delete=CASCADE)
pattern = models.TextField()
url_format_string = models.TextField(
validators=[filter_format_validator], null=True, blank=True
)
url_template = models.TextField(validators=[url_template_validator], null=True)
url_template = models.TextField(validators=[url_template_validator])
class Meta:
unique_together = ("realm", "pattern")
def __str__(self) -> str:
return f"{self.realm.string_id}: {self.pattern} {self.url_format_string}"
return f"{self.realm.string_id}: {self.pattern} {self.url_template}"
def clean(self) -> None:
"""Validate whether the set of parameters in the URL Format string
"""Validate whether the set of parameters in the URL template
match the set of parameters in the regular expression.
Django's `full_clean` calls `clean_fields` followed by `clean` method
@ -1338,33 +1332,33 @@ class RealmFilter(models.Model):
pattern = filter_pattern_validator(self.pattern)
group_set = set(pattern.groupindex.keys())
# Extract variables used in the URL format string. Note that
# this regex will incorrectly reject patterns that attempt to
# escape % using %%.
found_group_set: Set[str] = set()
group_match_regex = r"(?<!%)%\((?P<group_name>[^()]+)\)s"
for m in re.finditer(group_match_regex, self.url_format_string):
group_name = m.group("group_name")
found_group_set.add(group_name)
# Do not continue the check if the url template is invalid to begin with.
# The ValidationError for invalid template will only be raised by the validator
# set on the url_template field instead of here to avoid duplicates.
if not uri_template.validate(self.url_template):
return
# Extract variables used in the URL template.
template_variables_set = set(uri_template.URITemplate(self.url_template).variable_names)
# Report patterns missing in linkifier pattern.
missing_in_pattern_set = found_group_set - group_set
missing_in_pattern_set = template_variables_set - group_set
if len(missing_in_pattern_set) > 0:
name = min(missing_in_pattern_set)
raise ValidationError(
_("Group %(name)r in URL format string is not present in linkifier pattern."),
_("Group %(name)r in URL template is not present in linkifier pattern."),
params={"name": name},
)
missing_in_url_set = group_set - found_group_set
# Report patterns missing in URL format string.
missing_in_url_set = group_set - template_variables_set
# Report patterns missing in URL template.
if len(missing_in_url_set) > 0:
# We just report the first missing pattern here. Users can
# incrementally resolve errors if there are multiple
# missing patterns.
name = min(missing_in_url_set)
raise ValidationError(
_("Group %(name)r in linkifier pattern is not present in URL format string."),
_("Group %(name)r in linkifier pattern is not present in URL template."),
params={"name": name},
)
@ -1387,18 +1381,6 @@ def linkifiers_for_realm(realm_id: int) -> List[LinkifierDict]:
return per_request_linkifiers_cache[realm_id]
def realm_filters_for_realm(realm_id: int) -> List[Tuple[str, str, int]]:
"""
Processes data from `linkifiers_for_realm` to return to older clients,
which use the `realm_filters` events.
"""
linkifiers = linkifiers_for_realm(realm_id)
realm_filters: List[Tuple[str, str, int]] = []
for linkifier in linkifiers:
realm_filters.append((linkifier["pattern"], linkifier["url_format"], linkifier["id"]))
return realm_filters
@cache_with_key(get_linkifiers_cache_key, timeout=3600 * 24 * 7)
def linkifiers_for_realm_remote_cache(realm_id: int) -> List[LinkifierDict]:
linkifiers = []
@ -1406,7 +1388,7 @@ def linkifiers_for_realm_remote_cache(realm_id: int) -> List[LinkifierDict]:
linkifiers.append(
LinkifierDict(
pattern=linkifier.pattern,
url_format=linkifier.url_format_string,
url_template=linkifier.url_template,
id=linkifier.id,
)
)

2
zerver/openapi/curl_param_value_generators.py
View File

@ -274,7 +274,7 @@ def remove_realm_filters() -> Dict[str, object]:
filter_id = do_add_linkifier(
get_realm("zulip"),
"#(?P<id>[0-9]{2,8})",
"https://github.com/zulip/zulip/pull/%(id)s",
"https://github.com/zulip/zulip/pull/{id}",
acting_user=None,
)
return {

13
zerver/openapi/python_examples.py
View File

@ -430,12 +430,17 @@ def create_realm_profile_field(client: Client) -> None:
@openapi_test_function("/realm/filters:post")
def add_realm_filter(client: Client) -> None:
# TODO: Switch back to using client.add_realm_filter when python-zulip-api
# begins to support url_template.
# {code_example|start}
# Add a filter to automatically linkify #<number> to the corresponding
# issue in Zulip's server repo
result = client.add_realm_filter(
"#(?P<id>[0-9]+)", "https://github.com/zulip/zulip/issues/%(id)s"
)
request = {
"pattern": "#(?P<id>[0-9]+)",
"url_template": "https://github.com/zulip/zulip/issues/{id}",
}
result = client.call_endpoint("/realm/filters", method="POST", request=request)
# {code_example|end}
validate_against_openapi_schema(result, "/realm/filters", "post", "200")
@ -448,7 +453,7 @@ def update_realm_filter(client: Client) -> None:
filter_id = 1
request = {
"pattern": "#(?P<id>[0-9]+)",
"url_format_string": "https://github.com/zulip/zulip/issues/%(id)s",
"url_template": "https://github.com/zulip/zulip/issues/{id}",
}
result = client.call_endpoint(

94
zerver/openapi/zulip.yaml
View File

@ -3096,6 +3096,16 @@ paths:
Processing this event is important to doing Markdown local echo
correctly.
The client will not receive this event unless the event queue
is registered with `linkifier_url_template` client capability set to `true`.
See [`POST /register`](/api/register-queue#parameter-client_capabilities)
for how client capabilities can be specified.
**Changes**: Changed in Zulip 7.0 (feature level 176). This event no longer
gets sent unless the `linkifier_url_template` client capability is set to
`true`, due to the backwards-incompatible change from specifying the
URL from a `url_format` format string to `url_template`.
**Changes**: New in Zulip 4.0 (feature level 54), replacing the
previous `realm_filters` event type, which is still sent for
backwards compatibility reasons.
@ -3125,11 +3135,14 @@ paths:
description: |
The string regex pattern which represents the pattern that
should be linkified by this linkifier.
url_format:
url_template:
type: string
description: |
The URL format string to be used for linkifying matches.
The [RFC 6570](https://www.rfc-editor.org/rfc/rfc6570.html) compliant
URL template to be used for linkifying matches.
**Changes**: New in Zulip 7.0 (feature level 176). This replaced `url_format`,
which contained a URL format string.
id:
type: integer
description: |
@ -3141,7 +3154,7 @@ paths:
[
{
"pattern": "#(?P<id>[123])",
"url_format": "https://realm.com/my_realm_filter/%(id)s",
"url_template": "https://realm.com/my_realm_filter/{id}",
"id": 1,
},
],
@ -3155,6 +3168,9 @@ paths:
when the set of configured [linkifiers](/help/add-a-custom-linkifier)
for the organization has changed.
**Changes**: In Zulip 7.0 (feature level 176), clients will no longer
receive this event.
**Changes**: Deprecated in Zulip 4.0 (feature level 54), replaced by
the `realm_linkifiers` event type, which has a clearer name and format,
instead.
@ -3184,14 +3200,7 @@ paths:
example:
{
"type": "realm_filters",
"realm_filters":
[
[
"#(?P<id>[123])",
"https://realm.com/my_realm_filter/%(id)s",
1,
],
],
"realm_filters": [],
"id": 0,
}
- type: object
@ -9939,10 +9948,14 @@ paths:
description: |
The string regex pattern which represents the pattern that
should be linkified by this linkifier.
url_format:
url_template:
type: string
description: |
The URL format string to be used for linkifying matches.
The [RFC 6570](https://www.rfc-editor.org/rfc/rfc6570.html) compliant
URL template to be used for linkifying matches.
**Changes**: New in Zulip 7.0 (feature level 176). This replaced `url_format`,
which contained a URL format string.
id:
type: integer
description: |
@ -9954,7 +9967,7 @@ paths:
[
{
"pattern": "#(?P<id>[0-9]+)",
"url_format": "https://github.com/zulip/zulip/issues/%(id)s",
"url_template": "https://github.com/zulip/zulip/issues/{id}",
"id": 1,
},
],
@ -9971,7 +9984,7 @@ paths:
appear in messages and topics.
parameters:
- $ref: "#/components/parameters/LinkifierPattern"
- $ref: "#/components/parameters/LinkifierURLFormatString"
- $ref: "#/components/parameters/LinkifierURLTemplate"
responses:
"200":
description: Success.
@ -10032,7 +10045,7 @@ paths:
example: 2
required: true
- $ref: "#/components/parameters/LinkifierPattern"
- $ref: "#/components/parameters/LinkifierURLFormatString"
- $ref: "#/components/parameters/LinkifierURLTemplate"
responses:
"200":
$ref: "#/components/responses/SimpleSuccess"
@ -10300,6 +10313,15 @@ paths:
any server version. Clients can then use the `zulip_feature_level` in the
`/register` response or the presence/absence of a `user_settings` key to
determine where to look for the data.
- `linkifier_url_template`: Boolean for whether the client accepts linkifiers
that uses [RFC 6570](https://www.rfc-editor.org/rfc/rfc6570.html) compliant URL
template for linkifying matches. If false or unset, then the `realm_linkifiers`
list in the `/register` response will be empty if present, and no `realm_linkifiers`
events will be sent.
**Changes**: New in Zulip 7.0 (feature level 176). This capability is for
backwards-compatibility.
content:
application/json:
schema:
@ -10654,6 +10676,16 @@ paths:
Array of objects where each object describes a single
[linkifier](/help/add-a-custom-linkifier).
If the client capability `linkifier_url_template` is false or unset, then
the `realm_linkifiers` list in the `/register` response will be empty if
present, and no `realm_linkifiers` events will be sent.
See [`POST /register`](/api/register-queue#parameter-client_capabilities).
for how client capabilities can be specified.
**Changes**: In Zulip 7.0 (feature level 176), clients will always receive
an empty list in the response unless they set the `linkifier_url_template`
client capability to `true`.
**Changes**: New in Zulip 4.0 (feature level 54). Clients can
access these data on older server versions via the previous
`realm_filters` key.
@ -10666,10 +10698,14 @@ paths:
description: |
The string regex pattern which represents the pattern that
should be linkified on matching.
url_format:
url_template:
type: string
description: |
The URL with which the pattern matching string should be linkified.
The [RFC 6570](https://www.rfc-editor.org/rfc/rfc6570.html) compliant URL
template with which the pattern matching string should be linkified.
**Changes**: New in Zulip 7.0 (feature level 176). This replaced `url_format`,
which contained a URL format string.
id:
type: integer
description: |
@ -10687,11 +10723,13 @@ paths:
Legacy property for linkifiers. Present if `realm_filters` is
present in `fetch_event_types`.
An array of tuples (fixed-length arrays) where each tuple describes
When present, this is always an empty array.
**Changes**: Became empty in Zulip 7.0 (feature level 176). In previous versions,
there is an array of tuples (fixed-length arrays) where each tuple describes
a single [linkifier](/help/add-a-custom-linkifier).
The first element of the tuple is a string regex pattern which represents
the pattern that should be linkified on matching.
The second element is the URL with which the
pattern matching string should be linkified with and the third element
is the ID of the realm filter.
@ -18260,16 +18298,22 @@ components:
type: string
example: "#(?P<id>[0-9]+)"
required: true
LinkifierURLFormatString:
name: url_format_string
LinkifierURLTemplate:
name: url_template
in: query
description: |
The URL used for the link. If you used named groups for the `pattern`,
The [RFC
6570](https://www.rfc-editor.org/rfc/rfc6570.html) compliant URL template
used for the link. If you used named groups for the `pattern`,
you can insert their content here with
`%(name_of_the_capturing_group)s`.
`{name_of_the_capturing_group}`.
**Changes**: New in Zulip 7.0 (feature level 176). This replaced
the parameter `url_format_string`, which was a format string in which
named groups' content could be inserted with `%(name_of_the_capturing_group)s`.
schema:
type: string
example: https://github.com/zulip/zulip/issues/%(id)s
example: https://github.com/zulip/zulip/issues/{id}
required: true
DirectMemberOnly:
name: direct_member_only

10
zerver/tests/test_audit_log.py
View File

@ -842,13 +842,13 @@ class TestRealmAuditLog(ZulipTestCase):
linkifier_id = do_add_linkifier(
user.realm,
pattern="#(?P<id>[123])",
url_format_string="https://realm.com/my_realm_filter/%(id)s",
url_template="https://realm.com/my_realm_filter/{id}",
acting_user=user,
)
added_linkfier = LinkifierDict(
pattern="#(?P<id>[123])",
url_format="https://realm.com/my_realm_filter/%(id)s",
url_template="https://realm.com/my_realm_filter/{id}",
id=linkifier_id,
)
expected_extra_data = {
@ -871,12 +871,12 @@ class TestRealmAuditLog(ZulipTestCase):
user.realm,
id=linkifier_id,
pattern="#(?P<id>[0-9]+)",
url_format_string="https://realm.com/my_realm_filter/issues/%(id)s",
url_template="https://realm.com/my_realm_filter/issues/{id}",
acting_user=user,
)
changed_linkifier = LinkifierDict(
pattern="#(?P<id>[0-9]+)",
url_format="https://realm.com/my_realm_filter/issues/%(id)s",
url_template="https://realm.com/my_realm_filter/issues/{id}",
id=linkifier_id,
)
expected_extra_data = {
@ -902,7 +902,7 @@ class TestRealmAuditLog(ZulipTestCase):
)
removed_linkifier = {
"pattern": "#(?P<id>[0-9]+)",
"url_format": "https://realm.com/my_realm_filter/issues/%(id)s",
"url_template": "https://realm.com/my_realm_filter/issues/{id}",
}
expected_extra_data = {
"realm_linkifiers": initial_linkifiers,

35
zerver/tests/test_event_system.py
View File

@ -744,6 +744,35 @@ class FetchInitialStateDataTest(ZulipTestCase):
self.assertIn(prop, result)
self.assertIn(prop, result["user_settings"])
def test_realm_linkifiers_based_on_client_capabilities(self) -> None:
user = self.example_user("iago")
self.login_user(user)
data = {
"pattern": "#(?P<id>[123])",
"url_template": "https://realm.com/my_realm_filter/{id}",
}
post_result = self.client_post("/json/realm/filters", info=data)
self.assert_json_success(post_result)
result = fetch_initial_state_data(
user_profile=user,
linkifier_url_template=True,
)
self.assertEqual(result["realm_filters"], [])
self.assertEqual(result["realm_linkifiers"][0]["pattern"], "#(?P<id>[123])")
self.assertEqual(
result["realm_linkifiers"][0]["url_template"],
"https://realm.com/my_realm_filter/{id}",
)
# The default behavior should be `linkifier_url_template=False`
result = fetch_initial_state_data(
user_profile=user,
)
self.assertEqual(result["realm_filters"], [])
self.assertEqual(result["realm_linkifiers"], [])
def test_pronouns_field_type_support(self) -> None:
hamlet = self.example_user("hamlet")
result = fetch_initial_state_data(
@ -1211,7 +1240,7 @@ class FetchQueriesTest(ZulipTestCase):
self.login_user(user)
flush_per_request_caches()
with self.assert_database_query_count(38):
with self.assert_database_query_count(37):
with mock.patch("zerver.lib.events.always_want") as want_mock:
fetch_initial_state_data(user)
@ -1232,8 +1261,8 @@ class FetchQueriesTest(ZulipTestCase):
realm_embedded_bots=0,
realm_incoming_webhook_bots=0,
realm_emoji=1,
realm_filters=1,
realm_linkifiers=1,
realm_filters=0,
realm_linkifiers=0,
realm_playgrounds=1,
realm_user=3,
realm_user_groups=3,

46
zerver/tests/test_events.py
View File

@ -146,7 +146,6 @@ from zerver.lib.event_schema import (
check_realm_domains_remove,
check_realm_emoji_update,
check_realm_export,
check_realm_filters,
check_realm_linkifiers,
check_realm_playgrounds,
check_realm_update,
@ -263,6 +262,7 @@ class BaseAction(ZulipTestCase):
stream_typing_notifications: bool = True,
user_settings_object: bool = False,
pronouns_field_type_supported: bool = True,
linkifier_url_template: bool = True,
) -> List[Dict[str, Any]]:
"""
Make sure we have a clean slate of client descriptors for these tests.
@ -291,6 +291,7 @@ class BaseAction(ZulipTestCase):
stream_typing_notifications=stream_typing_notifications,
user_settings_object=user_settings_object,
pronouns_field_type_supported=pronouns_field_type_supported,
linkifier_url_template=linkifier_url_template,
)
)
@ -305,6 +306,7 @@ class BaseAction(ZulipTestCase):
include_subscribers=include_subscribers,
include_streams=include_streams,
pronouns_field_type_supported=pronouns_field_type_supported,
linkifier_url_template=linkifier_url_template,
)
# We want even those `send_event` calls which have been hooked to
@ -335,6 +337,7 @@ class BaseAction(ZulipTestCase):
client_gravatar=client_gravatar,
slim_presence=slim_presence,
include_subscribers=include_subscribers,
linkifier_url_template=linkifier_url_template,
)
post_process_state(self.user_profile, hybrid_state, notification_settings_null)
after = orjson.dumps(hybrid_state)
@ -361,6 +364,7 @@ class BaseAction(ZulipTestCase):
include_subscribers=include_subscribers,
include_streams=include_streams,
pronouns_field_type_supported=pronouns_field_type_supported,
linkifier_url_template=linkifier_url_template,
)
post_process_state(self.user_profile, normal_state, notification_settings_null)
self.match_states(hybrid_state, normal_state, events)
@ -2010,14 +2014,13 @@ class NormalActionsTest(BaseAction):
def test_realm_filter_events(self) -> None:
regex = "#(?P<id>[123])"
url = "https://realm.com/my_realm_filter/%(id)s"
url = "https://realm.com/my_realm_filter/{id}"
events = self.verify_action(
lambda: do_add_linkifier(self.user_profile.realm, regex, url, acting_user=None),
num_events=2,
num_events=1,
)
check_realm_linkifiers("events[0]", events[0])
check_realm_filters("events[1]", events[1])
regex = "#(?P<id>[0-9]+)"
linkifier_id = events[0]["realm_linkifiers"][0]["id"]
@ -2025,17 +2028,44 @@ class NormalActionsTest(BaseAction):
lambda: do_update_linkifier(
self.user_profile.realm, linkifier_id, regex, url, acting_user=None
),
num_events=2,
num_events=1,
)
check_realm_linkifiers("events[0]", events[0])
check_realm_filters("events[1]", events[1])
events = self.verify_action(
lambda: do_remove_linkifier(self.user_profile.realm, regex, acting_user=None),
num_events=2,
num_events=1,
)
check_realm_linkifiers("events[0]", events[0])
check_realm_filters("events[1]", events[1])
# Redo the checks, but assume that the client does not support URL template.
# apply_event should drop the event, and no state change should occur.
regex = "#(?P<id>[123])"
events = self.verify_action(
lambda: do_add_linkifier(self.user_profile.realm, regex, url, acting_user=None),
num_events=1,
linkifier_url_template=False,
state_change_expected=False,
)
regex = "#(?P<id>[0-9]+)"
linkifier_id = events[0]["realm_linkifiers"][0]["id"]
events = self.verify_action(
lambda: do_update_linkifier(
self.user_profile.realm, linkifier_id, regex, url, acting_user=None
),
num_events=1,
linkifier_url_template=False,
state_change_expected=False,
)
events = self.verify_action(
lambda: do_remove_linkifier(self.user_profile.realm, regex, acting_user=None),
num_events=1,
linkifier_url_template=False,
state_change_expected=False,
)
def test_realm_domain_events(self) -> None:
events = self.verify_action(

77
zerver/tests/test_markdown.py
View File

@ -1349,10 +1349,10 @@ class MarkdownTest(ZulipTestCase):
RealmFilter(
realm=realm,
pattern=r"#(?P<id>[0-9]{2,8})",
url_format_string=r"https://trac.example.com/ticket/%(id)s",
url_template=r"https://trac.example.com/ticket/{id}",
)
],
["<RealmFilter: zulip: #(?P<id>[0-9]{2,8}) https://trac.example.com/ticket/%(id)s>"],
["<RealmFilter: zulip: #(?P<id>[0-9]{2,8}) https://trac.example.com/ticket/{id}>"],
)
msg = Message(sender=self.example_user("othello"))
@ -1411,7 +1411,7 @@ class MarkdownTest(ZulipTestCase):
RealmFilter(
realm=realm,
pattern=r"#(?P<id>[a-zA-Z]+-[0-9]+)",
url_format_string=r"https://trac.example.com/ticket/%(id)s",
url_template=r"https://trac.example.com/ticket/{id}",
).save()
msg = Message(sender=self.example_user("hamlet"))
@ -1459,7 +1459,7 @@ class MarkdownTest(ZulipTestCase):
RealmFilter(
realm=realm,
pattern=r"hello#(?P<id>[0-9]+)",
url_format_string=r"https://trac.example.com/hello/%(id)s",
url_template=r"https://trac.example.com/hello/{id}",
).save()
converted_topic = topic_links(realm.id, "hello#123 #234")
self.assertEqual(
@ -1484,18 +1484,19 @@ class MarkdownTest(ZulipTestCase):
RealmFilter(
realm=realm,
pattern=r"url-(?P<id>[0-9]+)",
url_format_string="https://example.com/A%20Test/%%%ba/%(id)s",
url_template="https://example.com/A%20Test/%%%ba/{id}",
).save()
msg = Message(sender=self.example_user("hamlet"))
content = "url-123 is well-escaped"
converted = markdown_convert(content, message_realm=realm, message=msg)
self.assertEqual(
converted.rendered_content,
'<p><a href="https://example.com/A%20Test/%%ba/123">url-123</a> is well-escaped</p>',
'<p><a href="https://example.com/A%20Test/%25%25%ba/123">url-123</a> is well-escaped</p>',
)
converted_topic = topic_links(realm.id, content)
self.assertEqual(
converted_topic, [{"url": "https://example.com/A%20Test/%%ba/123", "text": "url-123"}]
converted_topic,
[{"url": "https://example.com/A%20Test/%25%25%ba/123", "text": "url-123"}],
)
def test_multiple_matching_realm_patterns(self) -> None:
@ -1505,23 +1506,23 @@ class MarkdownTest(ZulipTestCase):
RealmFilter(
realm=realm,
pattern="(?P<id>ABC-[0-9]+)",
url_format_string="https://trac.example.com/ticket/%(id)s",
url_template="https://trac.example.com/ticket/{id}",
),
RealmFilter(
realm=realm,
pattern="(?P<id>[A-Z][A-Z0-9]*-[0-9]+)",
url_format_string="https://other-trac.example.com/ticket/%(id)s",
url_template="https://other-trac.example.com/ticket/{id}",
),
RealmFilter(
realm=realm,
pattern="(?P<id>[A-Z][A-Z0-9]+)",
url_format_string="https://yet-another-trac.example.com/ticket/%(id)s",
url_template="https://yet-another-trac.example.com/ticket/{id}",
),
],
[
"<RealmFilter: zulip: (?P<id>ABC-[0-9]+) https://trac.example.com/ticket/%(id)s>",
"<RealmFilter: zulip: (?P<id>[A-Z][A-Z0-9]*-[0-9]+) https://other-trac.example.com/ticket/%(id)s>",
"<RealmFilter: zulip: (?P<id>[A-Z][A-Z0-9]+) https://yet-another-trac.example.com/ticket/%(id)s>",
"<RealmFilter: zulip: (?P<id>ABC-[0-9]+) https://trac.example.com/ticket/{id}>",
"<RealmFilter: zulip: (?P<id>[A-Z][A-Z0-9]*-[0-9]+) https://other-trac.example.com/ticket/{id}>",
"<RealmFilter: zulip: (?P<id>[A-Z][A-Z0-9]+) https://yet-another-trac.example.com/ticket/{id}>",
],
)
@ -1572,17 +1573,17 @@ class MarkdownTest(ZulipTestCase):
RealmFilter(
realm=realm,
pattern="ABC-42",
url_format_string="https://google.com",
url_template="https://google.com",
),
RealmFilter(
realm=realm,
pattern=r"com.+(?P<id>ABC\-[0-9]+)",
url_format_string="https://trac.example.com/ticket/%(id)s",
url_template="https://trac.example.com/ticket/{id}",
),
],
[
"<RealmFilter: zulip: ABC-42 https://google.com>",
r"<RealmFilter: zulip: com.+(?P<id>ABC\-[0-9]+) https://trac.example.com/ticket/%(id)s>",
r"<RealmFilter: zulip: com.+(?P<id>ABC\-[0-9]+) https://trac.example.com/ticket/{id}>",
],
)
@ -1621,29 +1622,29 @@ class MarkdownTest(ZulipTestCase):
RealmFilter(
realm=realm,
pattern="http",
url_format_string="http://example.com/",
url_template="http://example.com/",
),
RealmFilter(
realm=realm,
pattern="b#(?P<id>[a-z]+)",
url_format_string="http://example.com/b/%(id)s",
url_template="http://example.com/b/{id}",
),
RealmFilter(
realm=realm,
pattern="a#(?P<aid>[a-z]+) b#(?P<bid>[a-z]+)",
url_format_string="http://example.com/a/%(aid)s/b/%(bid)s",
url_template="http://example.com/a/{aid}/b/{bid}",
),
RealmFilter(
realm=realm,
pattern="a#(?P<id>[a-z]+)",
url_format_string="http://example.com/a/%(id)s",
url_template="http://example.com/a/{id}",
),
],
[
"<RealmFilter: zulip: http http://example.com/>",
"<RealmFilter: zulip: b#(?P<id>[a-z]+) http://example.com/b/%(id)s>",
"<RealmFilter: zulip: a#(?P<aid>[a-z]+) b#(?P<bid>[a-z]+) http://example.com/a/%(aid)s/b/%(bid)s>",
"<RealmFilter: zulip: a#(?P<id>[a-z]+) http://example.com/a/%(id)s>",
"<RealmFilter: zulip: b#(?P<id>[a-z]+) http://example.com/b/{id}>",
"<RealmFilter: zulip: a#(?P<aid>[a-z]+) b#(?P<bid>[a-z]+) http://example.com/a/{aid}/b/{bid}>",
"<RealmFilter: zulip: a#(?P<id>[a-z]+) http://example.com/a/{id}>",
],
)
# There should be 5 link matches in the topic, if ordered from the most priortized to the least:
@ -1692,7 +1693,7 @@ class MarkdownTest(ZulipTestCase):
flush_linkifiers(sender=RealmFilter, instance=cast(RealmFilter, instance))
def save_new_linkifier() -> None:
linkifier = RealmFilter(realm=realm, pattern=r"whatever", url_format_string="whatever")
linkifier = RealmFilter(realm=realm, pattern=r"whatever", url_template="whatever")
linkifier.save()
# start fresh for our realm
@ -1722,7 +1723,7 @@ class MarkdownTest(ZulipTestCase):
id=cur_precedence,
realm=realm,
pattern=f"abc{cur_precedence}",
url_format_string="http://foo.com",
url_template="http://foo.com",
)
linkifier.save()
linkifiers = linkifiers_for_realm(realm.id)
@ -1734,7 +1735,7 @@ class MarkdownTest(ZulipTestCase):
RealmFilter(
realm=realm,
pattern=r"#(?P<id>[0-9]{2,8})",
url_format_string=r"https://trac.example.com/ticket/%(id)s",
url_template=r"https://trac.example.com/ticket/{id}",
).save()
boring_msg = Message(sender=self.example_user("othello"))
boring_msg.set_topic_name("no match here")
@ -2425,14 +2426,14 @@ class MarkdownTest(ZulipTestCase):
realm = get_realm("zulip")
msg = Message(sender=sender_user_profile, sending_client=get_client("test"))
# Create a linkifier.
url_format_string = r"https://trac.example.com/ticket/%(id)s"
url_template = r"https://trac.example.com/ticket/{id}"
linkifier = RealmFilter(
realm=realm, pattern=r"#(?P<id>[0-9]{2,8})", url_format_string=url_format_string
realm=realm, pattern=r"#(?P<id>[0-9]{2,8})", url_template=url_template
)
linkifier.save()
self.assertEqual(
repr(linkifier),
"<RealmFilter: zulip: #(?P<id>[0-9]{2,8}) https://trac.example.com/ticket/%(id)s>",
"<RealmFilter: zulip: #(?P<id>[0-9]{2,8}) https://trac.example.com/ticket/{id}>",
)
# Create a user that potentially interferes with the pattern.
test_user = create_user(
@ -2515,14 +2516,14 @@ class MarkdownTest(ZulipTestCase):
msg = Message(sender=sender_user_profile, sending_client=get_client("test"))
user_profile = self.example_user("hamlet")
# Create a linkifier.
url_format_string = r"https://trac.example.com/ticket/%(id)s"
url_template = r"https://trac.example.com/ticket/{id}"
linkifier = RealmFilter(
realm=realm, pattern=r"#(?P<id>[0-9]{2,8})", url_format_string=url_format_string
realm=realm, pattern=r"#(?P<id>[0-9]{2,8})", url_template=url_template
)
linkifier.save()
self.assertEqual(
repr(linkifier),
"<RealmFilter: zulip: #(?P<id>[0-9]{2,8}) https://trac.example.com/ticket/%(id)s>",
"<RealmFilter: zulip: #(?P<id>[0-9]{2,8}) https://trac.example.com/ticket/{id}>",
)
# Create a user-group that potentially interferes with the pattern.
user_id = user_profile.id
@ -2773,14 +2774,14 @@ class MarkdownTest(ZulipTestCase):
realm = get_realm("zulip")
# Create a linkifier.
sender_user_profile = self.example_user("othello")
url_format_string = r"https://trac.example.com/ticket/%(id)s"
url_template = r"https://trac.example.com/ticket/{id}"
linkifier = RealmFilter(
realm=realm, pattern=r"#(?P<id>[0-9]{2,8})", url_format_string=url_format_string
realm=realm, pattern=r"#(?P<id>[0-9]{2,8})", url_template=url_template
)
linkifier.save()
self.assertEqual(
repr(linkifier),
"<RealmFilter: zulip: #(?P<id>[0-9]{2,8}) https://trac.example.com/ticket/%(id)s>",
"<RealmFilter: zulip: #(?P<id>[0-9]{2,8}) https://trac.example.com/ticket/{id}>",
)
# Create a topic link that potentially interferes with the pattern.
denmark = get_stream("Denmark", realm)
@ -2842,14 +2843,14 @@ class MarkdownTest(ZulipTestCase):
realm = get_realm("zulip")
# Create a linkifier.
sender_user_profile = self.example_user("othello")
url_format_string = r"https://trac.example.com/ticket/%(id)s"
url_template = r"https://trac.example.com/ticket/{id}"
linkifier = RealmFilter(
realm=realm, pattern=r"#(?P<id>[0-9]{2,8})", url_format_string=url_format_string
realm=realm, pattern=r"#(?P<id>[0-9]{2,8})", url_template=url_template
)
linkifier.save()
self.assertEqual(
repr(linkifier),
"<RealmFilter: zulip: #(?P<id>[0-9]{2,8}) https://trac.example.com/ticket/%(id)s>",
"<RealmFilter: zulip: #(?P<id>[0-9]{2,8}) https://trac.example.com/ticket/{id}>",
)
# Create a stream that potentially interferes with the pattern.
stream = self.make_stream(stream_name="Stream #1234", realm=realm)

6
zerver/tests/test_message_dict.py
View File

@ -249,16 +249,16 @@ class MessageDictTest(ZulipTestCase):
# and not linkified when sent to a stream in 'lear'.
zulip_realm = get_realm("zulip")
lear_realm = get_realm("lear")
url_format_string = r"https://trac.example.com/ticket/%(id)s"
url_template = r"https://trac.example.com/ticket/{id}"
links = {"url": "https://trac.example.com/ticket/123", "text": "#123"}
topic_name = "test #123"
linkifier = RealmFilter(
realm=zulip_realm, pattern=r"#(?P<id>[0-9]{2,8})", url_format_string=url_format_string
realm=zulip_realm, pattern=r"#(?P<id>[0-9]{2,8})", url_template=url_template
)
self.assertEqual(
repr(linkifier),
"<RealmFilter: zulip: #(?P<id>[0-9]{2,8}) https://trac.example.com/ticket/%(id)s>",
"<RealmFilter: zulip: #(?P<id>[0-9]{2,8}) https://trac.example.com/ticket/{id}>",
)
def get_message(sender: UserProfile, realm: Realm) -> Message:

131
zerver/tests/test_realm_linkifiers.py
View File

@ -3,7 +3,7 @@ import re
from django.core.exceptions import ValidationError
from zerver.lib.test_classes import ZulipTestCase
from zerver.models import RealmFilter, filter_format_validator
from zerver.models import RealmFilter, url_template_validator
class RealmFilterTest(ZulipTestCase):
@ -11,7 +11,7 @@ class RealmFilterTest(ZulipTestCase):
self.login("iago")
data = {
"pattern": "#(?P<id>[123])",
"url_format_string": "https://realm.com/my_realm_filter/%(id)s",
"url_template": "https://realm.com/my_realm_filter/{id}",
}
result = self.client_post("/json/realm/filters", info=data)
self.assert_json_success(result)
@ -20,11 +20,11 @@ class RealmFilterTest(ZulipTestCase):
linkifiers = self.assert_json_success(result)["linkifiers"]
self.assert_length(linkifiers, 1)
self.assertEqual(linkifiers[0]["pattern"], "#(?P<id>[123])")
self.assertEqual(linkifiers[0]["url_format"], "https://realm.com/my_realm_filter/%(id)s")
self.assertEqual(linkifiers[0]["url_template"], "https://realm.com/my_realm_filter/{id}")
def test_create(self) -> None:
self.login("iago")
data = {"pattern": "", "url_format_string": "https://realm.com/my_realm_filter/%(id)s"}
data = {"pattern": "", "url_template": "https://realm.com/my_realm_filter/{id}"}
result = self.client_post("/json/realm/filters", info=data)
self.assert_json_error(result, "This field cannot be blank.")
@ -37,105 +37,109 @@ class RealmFilterTest(ZulipTestCase):
self.assert_json_error(result, "Bad regular expression: bad repetition operator: ????")
data["pattern"] = r"ZUL-(?P<id>\d+)"
data["url_format_string"] = "$fgfg"
result = self.client_post("/json/realm/filters", info=data)
self.assert_json_error(result, "Enter a valid URL.")
data["pattern"] = r"ZUL-(?P<id>\d+)"
data["url_format_string"] = "https://realm.com/my_realm_filter/"
data["url_template"] = "$fgfg"
result = self.client_post("/json/realm/filters", info=data)
self.assert_json_error(
result, "Group 'id' in linkifier pattern is not present in URL format string."
result, "Group 'id' in linkifier pattern is not present in URL template."
)
data["url_format_string"] = "https://realm.com/my_realm_filter/#hashtag/%(id)s"
data["pattern"] = r"ZUL-(?P<id>\d+)"
data["url_template"] = "https://realm.com/my_realm_filter/"
result = self.client_post("/json/realm/filters", info=data)
self.assert_json_error(
result, "Group 'id' in linkifier pattern is not present in URL template."
)
data["url_template"] = "https://realm.com/my_realm_filter/#hashtag/{id}"
result = self.client_post("/json/realm/filters", info=data)
self.assert_json_success(result)
self.assertIsNotNone(re.match(data["pattern"], "ZUL-15"))
data["pattern"] = r"ZUL2-(?P<id>\d+)"
data["url_format_string"] = "https://realm.com/my_realm_filter/?value=%(id)s"
data["url_template"] = "https://realm.com/my_realm_filter/?value={id}"
result = self.client_post("/json/realm/filters", info=data)
self.assert_json_success(result)
self.assertIsNotNone(re.match(data["pattern"], "ZUL2-15"))
data["pattern"] = r"_code=(?P<id>[0-9a-zA-Z]+)"
data["url_format_string"] = "https://example.com/product/%(id)s/details"
data["url_template"] = "https://example.com/product/{id}/details"
result = self.client_post("/json/realm/filters", info=data)
self.assert_json_success(result)
self.assertIsNotNone(re.match(data["pattern"], "_code=123abcdZ"))
data["pattern"] = r"PR (?P<id>[0-9]+)"
data[
"url_format_string"
] = "https://example.com/~user/web#view_type=type&model=model&action=12345&id=%(id)s"
"url_template"
] = "https://example.com/~user/web#view_type=type&model=model&action=12345&id={id}"
result = self.client_post("/json/realm/filters", info=data)
self.assert_json_success(result)
self.assertIsNotNone(re.match(data["pattern"], "PR 123"))
data["pattern"] = r"lp/(?P<id>[0-9]+)"
data["url_format_string"] = "https://realm.com/my_realm_filter/?value=%(id)s&sort=reverse"
data["url_template"] = "https://realm.com/my_realm_filter/?value={id}&sort=reverse"
result = self.client_post("/json/realm/filters", info=data)
self.assert_json_success(result)
self.assertIsNotNone(re.match(data["pattern"], "lp/123"))
data["pattern"] = r"lp:(?P<id>[0-9]+)"
data["url_format_string"] = "https://realm.com/my_realm_filter/?sort=reverse&value=%(id)s"
data["url_template"] = "https://realm.com/my_realm_filter/?sort=reverse&value={id}"
result = self.client_post("/json/realm/filters", info=data)
self.assert_json_success(result)
self.assertIsNotNone(re.match(data["pattern"], "lp:123"))
data["pattern"] = r"!(?P<id>[0-9]+)"
data[
"url_format_string"
] = "https://realm.com/index.pl?Action=AgentTicketZoom;TicketNumber=%(id)s"
data["url_template"] = "https://realm.com/index.pl?Action=AgentTicketZoom;TicketNumber={id}"
result = self.client_post("/json/realm/filters", info=data)
self.assert_json_success(result)
self.assertIsNotNone(re.match(data["pattern"], "!123"))
# This block of tests is for mismatches between field sets
data["pattern"] = r"ZUL-(?P<id>\d+)"
data["url_format_string"] = r"https://realm.com/my_realm_filter/%(hello)s"
data["url_template"] = r"https://realm.com/my_realm_filter/{hello}"
result = self.client_post("/json/realm/filters", info=data)
self.assert_json_error(
result, "Group 'hello' in URL format string is not present in linkifier pattern."
result, "Group 'hello' in URL template is not present in linkifier pattern."
)
data["pattern"] = r"ZUL-(?P<id>\d+)-(?P<hello>\d+)"
data["url_format_string"] = r"https://realm.com/my_realm_filter/%(hello)s"
data["url_template"] = r"https://realm.com/my_realm_filter/{hello}"
result = self.client_post("/json/realm/filters", info=data)
self.assert_json_error(
result, "Group 'id' in linkifier pattern is not present in URL format string."
result, "Group 'id' in linkifier pattern is not present in URL template."
)
data["pattern"] = r"ZULZ-(?P<hello>\d+)-(?P<world>\d+)"
data["url_format_string"] = r"https://realm.com/my_realm_filter/%(hello)s/%(world)s"
data["url_template"] = r"https://realm.com/my_realm_filter/{hello}/{world}"
result = self.client_post("/json/realm/filters", info=data)
self.assert_json_success(result)
data["pattern"] = r"ZUL-(?P<id>\d+)-(?P<hello>\d+)-(?P<world>\d+)"
data["url_format_string"] = r"https://realm.com/my_realm_filter/%(hello)s"
data["url_template"] = r"https://realm.com/my_realm_filter/{hello}"
result = self.client_post("/json/realm/filters", info=data)
self.assert_json_error(
result, "Group 'id' in linkifier pattern is not present in URL format string."
result, "Group 'id' in linkifier pattern is not present in URL template."
)
data["pattern"] = r"ZUL-ESCAPE-(?P<id>\d+)"
data["url_format_string"] = r"https://realm.com/my_realm_filter/%%(ignored)s/%(id)s"
result = self.client_post("/json/realm/filters", info=data)
self.assert_json_success(result)
data["pattern"] = r"ZUL-URL-(?P<id>\d+)"
data["url_format_string"] = "https://example.com/%ba/%(id)s"
data["url_template"] = "https://example.com/%ba/{id}"
result = self.client_post("/json/realm/filters", info=data)
self.assert_json_success(result)
data["pattern"] = r"(?P<org>[a-zA-Z0-9_-]+)/(?P<repo>[a-zA-Z0-9_-]+)#(?P<id>[0-9]+)"
data["url_format_string"] = "https://github.com/%(org)s/%(repo)s/issue/%(id)s"
data["url_template"] = "https://github.com/{org}/{repo}/issue/{id}"
result = self.client_post("/json/realm/filters", info=data)
self.assert_json_success(result)
self.assertIsNotNone(re.match(data["pattern"], "zulip/zulip#123"))
data[
"pattern"
] = r"FOO_(?P<id>[a-f]{5});(?P<zone>[a-f]);(?P<domain>[a-z]+);(?P<location>[a-z]+);(?P<name>[a-z]{2,8});(?P<chapter>[0-9]{2,3});(?P<fragment>[a-z]{2,8})"
data[
"url_template"
] = "https://zone_{zone}{.domain}.net/ticket{/location}{/id}{?name,chapter}{#fragment:5}"
result = self.client_post("/json/realm/filters", info=data)
self.assert_json_success(result)
def test_not_realm_admin(self) -> None:
self.login("hamlet")
result = self.client_post("/json/realm/filters")
@ -147,7 +151,7 @@ class RealmFilterTest(ZulipTestCase):
self.login("iago")
data = {
"pattern": "#(?P<id>[123])",
"url_format_string": "https://realm.com/my_realm_filter/%(id)s",
"url_template": "https://realm.com/my_realm_filter/{id}",
}
result = self.client_post("/json/realm/filters", info=data)
linkifier_id = self.assert_json_success(result)["id"]
@ -163,13 +167,13 @@ class RealmFilterTest(ZulipTestCase):
self.login("iago")
data = {
"pattern": "#(?P<id>[123])",
"url_format_string": "https://realm.com/my_realm_filter/%(id)s",
"url_template": "https://realm.com/my_realm_filter/{id}",
}
result = self.client_post("/json/realm/filters", info=data)
linkifier_id = self.assert_json_success(result)["id"]
data = {
"pattern": "#(?P<id>[0-9]+)",
"url_format_string": "https://realm.com/my_realm_filter/issues/%(id)s",
"url_template": "https://realm.com/my_realm_filter/issues/{id}",
}
result = self.client_patch(f"/json/realm/filters/{linkifier_id}", info=data)
self.assert_json_success(result)
@ -181,26 +185,33 @@ class RealmFilterTest(ZulipTestCase):
self.assert_length(linkifier, 1)
self.assertEqual(linkifier[0]["pattern"], "#(?P<id>[0-9]+)")
self.assertEqual(
linkifier[0]["url_format"], "https://realm.com/my_realm_filter/issues/%(id)s"
linkifier[0]["url_template"], "https://realm.com/my_realm_filter/issues/{id}"
)
data = {
"pattern": r"ZUL-(?P<id>\d????)",
"url_format_string": "https://realm.com/my_realm_filter/%(id)s",
"url_template": "https://realm.com/my_realm_filter/{id}",
}
result = self.client_patch(f"/json/realm/filters/{linkifier_id}", info=data)
self.assert_json_error(result, "Bad regular expression: bad repetition operator: ????")
data["pattern"] = r"ZUL-(?P<id>\d+)"
data["url_format_string"] = "$fgfg"
data["url_template"] = "$fgfg"
result = self.client_patch(f"/json/realm/filters/{linkifier_id}", info=data)
self.assert_json_error(result, "Enter a valid URL.")
self.assert_json_error(
result, "Group 'id' in linkifier pattern is not present in URL template."
)
data["pattern"] = r"#(?P<id>[123])"
data["url_format_string"] = "https://realm.com/my_realm_filter/%(id)s"
data["url_template"] = "https://realm.com/my_realm_filter/{id}"
result = self.client_patch(f"/json/realm/filters/{linkifier_id + 1}", info=data)
self.assert_json_error(result, "Linkifier not found.")
data["pattern"] = r"#(?P<id>[123])"
data["url_template"] = "{id"
result = self.client_patch(f"/json/realm/filters/{linkifier_id}", info=data)
self.assert_json_error(result, "Invalid URL template.")
def test_valid_urls(self) -> None:
valid_urls = [
"http://example.com/",
@ -210,15 +221,15 @@ class RealmFilterTest(ZulipTestCase):
"https://example.com/!path",
"https://example.com/foo.bar",
"https://example.com/foo[bar]",
"https://example.com/%(foo)s",
"https://example.com/%(foo)s%(bars)s",
"https://example.com/%(foo)s/and/%(bar)s",
"https://example.com/?foo=%(foo)s",
"https://example.com/{foo}",
"https://example.com/{foo}{bars}",
"https://example.com/{foo}/and/{bar}",
"https://example.com/?foo={foo}",
"https://example.com/%ab",
"https://example.com/%ba",
"https://example.com/%21",
"https://example.com/words%20with%20spaces",
"https://example.com/back%20to%20%(back)s",
"https://example.com/back%20to%20{back}",
"https://example.com/encoded%2fwith%2fletters",
"https://example.com/encoded%2Fwith%2Fupper%2Fcase%2Fletters",
"https://example.com/%%",
@ -227,19 +238,25 @@ class RealmFilterTest(ZulipTestCase):
"https://example.com/%%(foo",
"https://example.com/%%(foo)",
"https://example.com/%%(foo)s",
"https://example.com{/foo,bar,baz}",
"https://example.com/{?foo*}",
"https://example.com/{+foo,bar}",
"https://chat{.domain}.com/{#foo}",
"https://zone_{zone}{.domain}.net/ticket{/location}{/id}{?name,chapter}{#fragment:5}",
"$not_a_url$",
]
for url in valid_urls:
filter_format_validator(url)
url_template_validator(url)
# No need to test this extensively, because most of the invalid
# cases should be handled and tested in the uri_template library
# we used for validation.
invalid_urls = [
"file:///etc/passwd",
"data:text/plain;base64,SGVsbG8sIFdvcmxkIQ==",
"https://example.com/%(foo)",
"https://example.com/%()s",
"https://example.com/%4!",
"https://example.com/%(foo",
"https://example.com/%2(foo)s",
"https://example.com/{foo",
"https://example.com/{{}",
"https://example.com/{//foo}",
"https://example.com/{bar++}",
]
for url in invalid_urls:
with self.assertRaises(ValidationError):
filter_format_validator(url)
url_template_validator(url)

2
zerver/tornado/django_api.py
View File

@ -83,6 +83,7 @@ def request_event_queue(
stream_typing_notifications: bool = False,
user_settings_object: bool = False,
pronouns_field_type_supported: bool = True,
linkifier_url_template: bool = False,
) -> Optional[str]:
if not settings.USING_TORNADO:
return None
@ -104,6 +105,7 @@ def request_event_queue(
"stream_typing_notifications": orjson.dumps(stream_typing_notifications),
"user_settings_object": orjson.dumps(user_settings_object),
"pronouns_field_type_supported": orjson.dumps(pronouns_field_type_supported),
"linkifier_url_template": orjson.dumps(linkifier_url_template),
}
if event_types is not None:

4
zerver/tornado/event_queue.py
View File

@ -96,6 +96,7 @@ class ClientDescriptor:
stream_typing_notifications: bool = False,
user_settings_object: bool = False,
pronouns_field_type_supported: bool = True,
linkifier_url_template: bool = False,
) -> None:
# These objects are serialized on shutdown and restored on restart.
# If fields are added or semantics are changed, temporary code must be
@ -120,6 +121,7 @@ class ClientDescriptor:
self.stream_typing_notifications = stream_typing_notifications
self.user_settings_object = user_settings_object
self.pronouns_field_type_supported = pronouns_field_type_supported
self.linkifier_url_template = linkifier_url_template
# Default for lifespan_secs is DEFAULT_EVENT_QUEUE_TIMEOUT_SECS;
# but users can set it as high as MAX_QUEUE_TIMEOUT_SECS.
@ -148,6 +150,7 @@ class ClientDescriptor:
stream_typing_notifications=self.stream_typing_notifications,
user_settings_object=self.user_settings_object,
pronouns_field_type_supported=self.pronouns_field_type_supported,
linkifier_url_template=self.linkifier_url_template,
)
def __repr__(self) -> str:
@ -181,6 +184,7 @@ class ClientDescriptor:
d.get("stream_typing_notifications", False),
d.get("user_settings_object", False),
d.get("pronouns_field_type_supported", True),
d.get("linkifier_url_template", False),
)
ret.last_connection_time = d["last_connection_time"]
return ret

4
zerver/tornado/views.py
View File

@ -156,6 +156,9 @@ def get_events_backend(
pronouns_field_type_supported: bool = REQ(
default=True, json_validator=check_bool, intentionally_undocumented=True
),
linkifier_url_template: bool = REQ(
default=False, json_validator=check_bool, intentionally_undocumented=True
),
) -> HttpResponse:
if all_public_streams and not user_profile.can_access_public_streams():
raise JsonableError(_("User not authorized for this query"))
@ -188,6 +191,7 @@ def get_events_backend(
stream_typing_notifications=stream_typing_notifications,
user_settings_object=user_settings_object,
pronouns_field_type_supported=pronouns_field_type_supported,
linkifier_url_template=linkifier_url_template,
)
result = in_tornado_thread(fetch_events)(

1
zerver/views/events_register.py
View File

@ -61,6 +61,7 @@ def events_register_backend(
("user_avatar_url_field_optional", check_bool),
("stream_typing_notifications", check_bool),
("user_settings_object", check_bool),
("linkifier_url_template", check_bool),
],
value_validator=check_bool,
),

8
zerver/views/realm_linkifiers.py
View File

@ -26,13 +26,13 @@ def create_linkifier(
request: HttpRequest,
user_profile: UserProfile,
pattern: str = REQ(),
url_format_string: str = REQ(),
url_template: str = REQ(),
) -> HttpResponse:
try:
linkifier_id = do_add_linkifier(
realm=user_profile.realm,
pattern=pattern,
url_format_string=url_format_string,
url_template=url_template,
acting_user=user_profile,
)
return json_success(request, data={"id": linkifier_id})
@ -58,14 +58,14 @@ def update_linkifier(
user_profile: UserProfile,
filter_id: int,
pattern: str = REQ(),
url_format_string: str = REQ(),
url_template: str = REQ(),
) -> HttpResponse:
try:
do_update_linkifier(
realm=user_profile.realm,
id=filter_id,
pattern=pattern,
url_format_string=url_format_string,
url_template=url_template,
acting_user=user_profile,
)
return json_success(request)

4
zerver/webhooks/front/doc.md
View File

@ -14,8 +14,8 @@ constructed above.
1. Go to the **Settings** page of your Zulip organization. Click on the
**Linkifiers** tab, and add a new linkifier. Set the pattern to
`cnv_(?P<id>[0-9a-z]+)`. Set the URL format string to
`https://app.frontapp.com/open/cnv_%(id)s`. This step is necessary to map
`cnv_(?P<id>[0-9a-z]+)`. Set the URL template to
`https://app.frontapp.com/open/cnv_{id}`. This step is necessary to map
Front conversations to topics in Zulip.
{!congrats.md!}

4
zerver/webhooks/stripe/doc.md
View File

@ -13,8 +13,8 @@ Get Zulip notifications for Stripe events!
1. [Optional] In Zulip, add a
[linkification filter](/help/add-a-custom-linkifier) with
**Pattern** `(?P<id>cus_[0-9a-zA-Z]+)` and **URL format string**
`https://dashboard.stripe.com/customers/%(id)s`.
**Pattern** `(?P<id>cus_[0-9a-zA-Z]+)` and **URL template**
`https://dashboard.stripe.com/customers/{id}`.
Zulip currently supports Stripe events for Charges, Customers, Discounts,
Sources, Subscriptions, Files, Invoices and Invoice items.