Commit Graph

128 Commits

Author SHA1 Message Date
Lauryn Menard ece752014c narrow: Add backend support for `is:dm` narrow.
Adds backend support for `is` operator with the `dm` operand. This
will deprecate the `is` operator with the `private` operand, but we
keep support for backwards-compatibility.

Note that there is some clean up of references to private messages
in the updated backend test. In commit 43ec7ed, the documentation
for `build_narrow_filter` wasn't updated for the rename of
`BuildNarrowFilterTest` to `NarrowLibraryTest`, so that's also
corrected in these changes.

The general API changelog and documentation updates will be done
in a final commit in the series of commits that adds support for
the various new direct message narrows.
2023-04-25 09:07:08 -07:00
Daniil Fadeev 2f203f4de1 emails: Inline CSS in emails in build_email.
Previously, we had an architecture where CSS inlining for emails was
done at provision time in inline_email_css.py. This was necessary
because the library we were using for this, Premailer, was extremely
slow, and doing the inlining for every outgoing email would have been
prohibitively expensive.

Now that we've migrated to a more modern library that inlines the
small amount of CSS we have into emails nearly instantly, we are able
to remove the complex architecture built to work around Premailer
being slow and just do the CSS inlining as the final step in sending
each individual email.

This has several significant benefits:

* Removes a fiddly provisioning step that made the edit/refresh cycle
  for modifying email templates confusing; there's no longer a CSS
  inlining step that, if you forget to do it, results in your testing a
  stale variant of the email templates.
* Fixes internationalization problems related to translators working
  with pre-CSS-inlined emails, and then Django trying to apply the
  translators to the post-CSS-inlined version.
* Makes the send_custom_email pipeline simpler and easier to improve.

Signed-off-by: Daniil Fadeev <fadeevd@zulip.com>
2023-04-05 12:22:29 -07:00
Alex Vandiver 92c8c17190 import: Add the UTF-8 flag on file entries in zipfiles from Slack.
Fixes: #22533.
2023-01-31 16:07:48 -08:00
Anders Kaseorg bd884c88ed Fix typos caught by typos.
https://github.com/crate-ci/typos

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2023-01-03 11:09:50 -08:00
Lauryn Menard 0c947985bd tests: Update tests to use example profile picture.
So that we can stop using Tim's photo for tests, adds an open
license profile picture to use instead.

Updates tests that used `tim.png` to use the new example profile
picture, which is located in `static/images/test-images/avatars/`.
2022-10-31 14:36:54 -07:00
Lauryn Menard 89713bab76 tests: Use Zulip icon for markdown test cases instead of octopus.
Updates the markdown test case that used `zulip-octopus.png` to
instead use an zulip logo that's also referenced in a frontend
puppeteer test, `static/images/logo/zulip-icon-128x128.png`.
2022-10-31 14:36:54 -07:00
Anders Kaseorg 7d485aa58f docs: Correct more “webapp” to “web app”.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-09-08 20:23:28 -07:00
Mateusz Mandera 4c7a9816ff gitter: Soft deactivate appropriate imported users.
We want to use the long_term_idle_helper logic for gitter imports just
like we do for slack.
2022-08-29 11:03:27 -07:00
Alya Abbott 7e3388dd9e images: Move /team images into landing-page directory. 2022-07-13 15:14:19 -07:00
Alex Vandiver ede4a88b49 populate_db: Remove unnecessary pre-population of URL cache.
76deb30312 changed this to not just be the URL, but rather a
prefixed hash of the URL, but failed to update this location which
wrote to it.  This meant that this pre-population step was writing to
the wrong keys in the durable cache, and thus ineffective.

Then, da33b72848 switched the cache to be in-memory, making this
write to the wrong keys in an in-process memory store.  There is no
way to pre-fill this sort of cache, except at server start-up.

Finally, and most fundamentally, 8c0c9ca7a4 then disabled
`inline_url_embed_preview` by default, making the code entirely moot.

Remove the triply-unnecessary code.
2022-04-15 14:48:12 -07:00
Anders Kaseorg b0ce4f1bce docs: Fix many spelling mistakes.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2022-02-07 18:51:06 -08:00
akshatdalton f2c7e8e80f markdown: Ensure all markdown fixtures have unique names.
This commit ensures that all markdown fixtures have unique
test names by rewriting the names of some of them and adding
a test in `test_markdown.py`.

Earlier this was over-writing the value for same keys in
`load_markdown_tests` in `test_markdown.py`.
2021-11-10 12:36:47 -08:00
Priyansh Garg 42f231c85c data_import: Ignore Rocket.Chat livechat streams/messages.
This resolves the issues reported in #20108, major chunk of which were
due to the incomplete support for importing the livechat streams/messages
in the tool. So, it's best not to import any livechat streams/messages for
now until a complete support for importing the same is developed.
2021-11-07 09:50:55 -08:00
Anders Kaseorg 8df6eccd42 dependencies: Upgrade JavaScript dependencies.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-11-05 17:34:13 -07:00
Priyansh Garg 0db9b7287b data_import: Import Rocket.Chat messages from direct discussions.
This commit adds functionality to import messages from the
Discussions having direct channels as their parent. As we don't
have topics in the PMs, the messages are imported in interleaved
form in the imported direct channels/PMs.

This was completely unsupported earlier and would have resulted in
an error.
2021-11-01 17:09:11 -07:00
Mateusz Mandera 4105ccdb17 saml: Implement IdP-initated logout for Keycloak.
Fixes #13948.
2021-10-27 13:13:55 -07:00
Alex Vandiver 8c72959951 push_notifications: Handle empty rendered_messages.
This parallels fe25517295, but for mobile notifications.  It also
adds a test, which verifies that such content does not crash either
mobile or email notifications.
2021-09-15 09:45:38 -07:00
Priyansh Garg 54452fef6c data_import: Fix channel mentions in Rocket.Chat import.
While the STREAM_LINK_REGEX and STREAM_TOPIC_LINK_REGEX
identifies the stream and topic mentions in the content
correctly (tested by printing out the matches), the
stream/topic mentions are still not linked to the
corresponding streams/topics for imported messages, as
a `zulip_message` instance is required for linking these
mentions to actual streams/topics (see `StreamPattern`
class in `markdown/__init__.py`) which is not provided
while processing the markdown for imported messages.
2021-08-09 06:38:26 -07:00
Priyansh Garg aed4e48da7 data_import: Import attachments from Rocket.Chat. 2021-08-09 06:38:26 -07:00
Priyansh Garg 65e28907cb data_import: Import custom emoji from Rocket.Chat. 2021-08-09 06:38:26 -07:00
Anders Kaseorg 42fa62e563 Revert "time_widget: Make the generated time string more readable."
This reverts commit 1965584eec.

This syntax has a bad interaction with table syntax and needs to be
rethought.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-08-03 16:45:31 -07:00
Ganesh Pawar 1965584eec time_widget: Make the generated time string more readable.
Before: <time:2021-07-14T00:14:00-07:00>
After: <time:2021-07-14|00:14:00|UTC-07:00>

Fixes #19205
2021-08-02 23:17:01 -07:00
Priyansh Garg 044fe547d3 data_import: Add huddle import support for Rocket.Chat. 2021-07-28 15:45:54 -07:00
Priyansh Garg 24dd0ff96c data_import: Add rocket chat import tool.
This commit allows to import the following from rocketchat:
* All users
* All public/private channels
* All teams and its public/private channels
* All discussion rooms as topics in their parent channel
* All the messages in all the channels
* All private conversations
* Reactions on messages (except for custom emojis)
* Mentions in messages (except @all, @here mentions)
2021-07-28 15:28:56 -07:00
akshatdalton 6b5812082e markdown: Fix shebang line eliminating behaviour of Codehilite.
See the block comment explaining the motivation for this change, but
basically, the shebang feature of Python-Markdown's Codehilite
extension could be really confusing and is not part of the CommonMark
standard.

1. https://python-markdown.github.io/extensions/code_hilite/#shebang-no-path
2. eacff473a2/markdown/extensions/codehilite.py (L164-L180)

Fixes: #18591.
2021-07-15 15:18:33 -07:00
akshatdalton f5c4d51ed2 resolve topic: Add `is:resolved` search keyword/filtering support.
This commit adds the backend support for `is:resolved` search keyword.
In the next commit, I will add the frontend support for the same.
2021-07-13 23:18:41 -07:00
Priyansh Garg 5b2e21965c data_import: Add import attachments support for Mattermost.
Add support for importing message attachments from Mattermost.

Fixes: #18959
2021-07-02 11:19:45 -07:00
Mateusz Mandera cd88a6e6fb tests: Rename APPLE_JWK to a generic name for re-use in other tests.
This can be re-used for the JWK in OIDC implementation in the follow-up
commits.
2021-05-23 13:30:17 -07:00
Ganesh Pawar 529f72fa3f markdown: Add support for sms and tel links.
Fixes #18390
2021-05-10 15:15:34 -07:00
Anders Kaseorg 544bbd5398 docs: Fix capitalization mistakes.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-10 09:57:26 -07:00
Anders Kaseorg 995389b4c1 markdown: Don’t apply further Markdown processing to KaTeX output.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-01 15:43:35 -07:00
Anders Kaseorg 802c8de0e8 fenced_code: Wrap math blocks in the frontend like the backend.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-05-01 15:36:15 -07:00
Arun Sankar 146b32d63a test users: Add an escape char to a test username.
Changed the name of the test-user cordelia from `Cordelia Lear` to
`Cordelia, Lear's daughter`.

This change will enable us to test users with escape characters in
their names.

I also updated the Node, Puppeteer, Backend tests and Fixtures to
support this change.
2021-04-13 11:42:06 -07:00
Anders Kaseorg de014a30e0 dependencies: Upgrade JavaScript dependencies.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-04-07 21:05:01 -07:00
Anders Kaseorg ceb7e2d2bd Revert "markdown: Add support to shorten GitHub links."
This reverts commit 9c6d8d9d81 (#16916).

This feature has known bugs, and also wants some design changes to
make it customizable like linkifiers, so we’re retargeting this to
post-4.x.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-04-02 15:52:34 -07:00
akshatdalton 9c6d8d9d81 markdown: Add support to shorten GitHub links.
We add support to shorten links and test their shortening in
well-organized, clean manner that makes it trivial to extend the
GitHub approach for GitLab and perhaps other services.

We only shorten basic types of GitHub links (issue, PR, commit) that
fit a set of simple common patterns; the default behaviour of Autolink
is kept for everything else.

Logic added in frontend and backend Markdown Processor is identical.
This makes easy to extend the logic for other services like GitLab.

Fixes #11895.
2021-03-25 00:39:44 -07:00
Ganesh Pawar a42f7a67e1 populate_db: Add images in test data.
This isn't quite the right model, because we're not actually going
through the upload code path, but it does at least provide some inline
image previews in the data.

Fixes part of #14991.
2021-01-27 17:52:28 -08:00
Mateusz Mandera bf9e5e52ce dependencies: Upgrade to Django 3.0.
Adjustments made due to changes in Django 3.0:
(https://docs.djangoproject.com/en/3.0/releases/3.0/)

- test_signup: INTERNAL_RESET_URL_TOKEN was moved to
  PasswordResetConfirmView.reset_url_token
- test_message_fetch:
  "add_never_cache_headers() and never_cache() now add the private
  directive to Cache-Control headers."
- "django.utils.html.escape() now uses html.escape() to escape HTML.
  This converts ' to &#x27; instead of the previous equivalent decimal
  code &#39;." - this requires adjusting the expected decimal code
  in some of the string fixtures in tests.
2021-01-26 10:20:00 -08:00
akshatdalton 806c1a0b8b markdown: Fix flickering of embedded link inside Italic.
This commit fixes a bug in marked.js which caused it to double-escape
HTML when rendering messages of the form: *[text](url)*.

This fixes a bug introduced in
3bdc8bbaa5, where an unnecessary
escape() call was added for the <em> code path, likely just because it
was adjacent to the others that needed it in the file.

Fix this, and add tests to verify that things are still being escaped
once after removing this extra escape.

Fixes #14845.
2020-11-06 10:09:15 -08:00
akshatdalton 620e9cbf72 markdown: Fix merging of separate quotations.
Initally, when writing two or more quotes, having
a blank line in between them, merges those quotes.
This created confusion especially in "quote and reply".

This commit fixes such issues. Now two or more quotes
having a blank line in between them, will not get merged.

This change is correct both for usability and for improving our
compatibility with CommonMark.

Fixes #14379.
2020-10-30 15:21:15 -07:00
Anders Kaseorg 72d6ff3c3b docs: Fix more capitalization issues.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-23 11:46:55 -07:00
Anders Kaseorg d81a93cdf3 requirements: Upgrade markdown to 3.3.1.
Upstream has slightly changed the whitespace around stashes.  Take
this opportunity to clean up the extra blank lines we were outputting.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-19 11:54:14 -07:00
Sumanth V Rao c563cdba61 markdown: Add data-code-lang attribute for locally echoed messages.
This mimics the backend logic for adding the data-attribute -
to know what Pygments language was used to highlight the code
block - in locally echoed messages.

New test added checks our logic for canonicalizing pygments alias
(for both frontend and backend).

Other fixtures and tests amended.
2020-09-18 17:12:26 -07:00
Tim Abbott ae58ed5a74 markdown: Tweak data-code-language testing and comments.
This should make it clearer the precise decisions we've made about the
intended semantics of this feature.
2020-09-15 12:30:57 -07:00
Sumanth V Rao b0c9e0a295 markdown: Rename fenced code data-attribute to data-code-language. 2020-09-15 20:09:58 +05:30
Sumanth V Rao 033351609d markdown: Add data-codehilite-language attr for fenced code.
When converting fenced code markdown, we add the language (if specified)
in a data-attribute by tweaking the HTML generated. Doing so, allows the
frontend to make use of this attr to display view-in-playground option
for codeblocks.

We use pygments to get the lexer subclass name and use that instead of
directly using the language in the data-attribute. Doing so, helps us
map different language aliases (like `js` and `javascript`) into a common
variable (like `JavaScript`) - and avoids the client from dealing with
multiple tags corresponding to the same language.

The html structure for a message like this:

``` js
..content..
```

would now be:

<div class="codehilite" data-codehilite-language="JavaScript">
    <pre>..content..</pre>
</div>

Tests and fixtures amended.
2020-09-14 21:25:19 -07:00
Anders Kaseorg 60a25b2721 docs: Fix spelling errors caught by codespell.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-11 10:23:06 -07:00
Anders Kaseorg 0b12d38e4d fixtures: Fix invalid JSON in narrow.json.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-07 11:12:32 -07:00
Vishnu KS 9e0ff58a6d team: Rename contrib to contributors in page_params. 2020-07-23 10:22:28 -07:00
Gittenburg 45e19dd6b9 emoji: Rename :slight_smile: to 😄.
Zulip converts :) to the 1F642 Unicode emoji and promotes the same emoji
in the popular section of the emoji picker.

Previously Zulip has labeled 1F642 as "slight smile". While that name
conforms to the Unicode standard (which describes the code point as
SLIGHTLY SMILING FACE), it didn't match our use case of the emoji.

If a user types :) or selects the first smile in the emoji picker they
probably mean to express a regular "smile" and not a "slight smile",
which raises the question why they are only smiling slightly.

This commit relabels 1F642 as 😄 and our previous 😄 263A as
:smiling_face:. Note that 263A looks different in our three supported
emoji sets, so it is not suited to be our "default smile".

This change does not require a migration since our emoji system stores
both unicode points and names and handles name changes transparently.
2020-07-21 16:49:54 -07:00