This seems to have been unnecessary since aiohttp 2, and now causes a
type error.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit d67831ba4b)
Unfortunately, because this migration has already been run for many
installations, we need to ship another copy of the migration.
It should be a noop when repeated.
(cherry picked from commit 66537c8bf8)
Currently, Slack messages containing hyperlinks
(e.g.,<http://foo.com|Foo!>) are converted like
normal links. This commit reformats Slack
hyperlinks into Zulip-friendly markdown
(e.g., [Foo!](http://foo.com)).
Part of #32165.
(cherry picked from commit fc50736f4e)
This fixes backfill data not being inserted in recent view if
recent view was open when it received the backfill data.
Fixed by inserted rows in the sorted order instead of inserting
them randomly which caused rows to not be inserted if there were
no rows around the row being inserted.
(cherry picked from commit 6fe3ceb0ee)
It's not an accurate descripton for an image that's been used in
production for many years, and we fully support.
(cherry picked from commit 81476308f5)
For resizing the icon.png files, we use resize_avatar, not resize_logo.
This is pretty confusing - sure, for icons we use the same function as
for avatars, but we should have a proper name for the function called in
the icon context. So this commit also adds resize_realm_icon, and
changes the calls to resize_avatar in icon contexts to
resize_realm_icon.
(cherry picked from commit 420849ff6a)
This lets slack conversions be done on development hosts, which have a
trailing :9991 on their EXTERNAL_HOST; otherwise, we generate fake
emails like `imported-slack-bot@host.name:9991` which fail to
validate.
(cherry picked from commit 2c51824b7d)
A file which unpacks to more than 10x its original size is suspect,
particularly if that results in an uncompressed size > 1GB.
(cherry picked from commit e68096c907)
This is some minor protection against malicious zipfiles (e.g. many
very deep directories to chew up inodes), in addition to validation.
(cherry picked from commit 6f7c14c9ec)
Aside of what's generally explained in the code comment, this is
motivated by the specific situation of import of Slack Connect channels.
These channels contain users who are "external collaborators" and
limited to a single channel in Slack. We don't have more sophisticated
handling of their import, which would map this concept 1-to-1 in Zulip -
but we create them as inactive dummy users, meaning they have to go
through signup before their account is usable.
The issue is that their imported UserProfile.role is set to Member and
when they register, the UserProfile gets reactivated with that role
unchanged. However, if e.g. the user is signing up after they received
an invitation from the admin, they should get the role that was
configured on the invite. In particular important if the user is meant
to still be "limited" and thus the admin invites them as a guest - they
definitely don't want the user to get a full Member account because of
this weird interaction between import and registration.
(cherry picked from commit 1d7d3fae61)
If do_delete_messages (and friends) are called for a massive number of
messages, the giant list of message ids is passed to Postgres even
though chunk_size makes all but the first chunk_size of message ids
useless.
(cherry picked from commit 18fbb5d146)
Apparently, Outlook ignores height/width CSS rules, but does support
the attribute on the image element itself, so specify that instead.
I don't think there are likely to be image tag implementations that
don't support the attribute, given that's the only thing that works in
Outlook.
(cherry picked from commit 9b67164270)
Previously, errors were returned using Zulip's default format,
which did not match Slack's expected response structure.
This change ensures that errors in the Slack incoming webhook handler
return JSON responses in Slack's expected format: {ok: false, error:
"error string"}.
Fixes: #31878.
(cherry picked from commit d448b75176)
html2text mangles Unicode by default, with a --unicode-snob option to
disable it. If I have to get called a “snob” for wanting to correctly
support non-English languages, then uh, I’ll take one for the team.
https://github.com/Alir3z4/html2text/blob/2024.2.26/html2text/config.py#L111-L150
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 42e1517255)
We simply forgot to `add_to_narrow` locally echoed messages if
the current narrow changed before we received confirmation from
server.
(cherry picked from commit 3fe1e554a6)
Zulip instances without a database included, like the Docker image,
would not fail to use TLS properly, since `TLS_REQCERT` was not set in
`/etc/ldap/ldap.conf`. While there's a few other ways we could fix
this, just installing libldap-common on app frontend instances seems
like a good solution, and has no impact on other Zulip systems, and it
was already being installed through a "Recommends" tier apt dependency
indirectly from the PostgreSQL server package.
Fixeszulip/docker-zulip#454.
(cherry picked from commit 9d68d89d01)
We didn't have thumbnailing for images coming from data import and this
commit adds the functionality.
There are a few fundamental issues that the implementation needs to
solve.
1. The images come from an untrusted source and therefore we don't want
to just pass them through to thumbnailing without checking. For that
reason, we cannot just import ImageAttachment rows from the export
data, even for zulip=>zulip imports.
The right way to process images is to pass them to maybe_thumbail(),
which runs libvips_check_image() on them to verify we're okay with
thumbnailing, creates ImageAttachment rows for them and sends them
to the thumbnailing queue worker. This approach lets us handle both
zulip=>zulip and 3rd party=>zulip imports in the same way,
2. There is a somewhat circular dependency between the Message,
Attachment and ImageAttachment import process:
- ImageAttachments would ideally be created after importing
Attachments, but they need to already exist at the time of Message
import. Otherwise, the markdown processor doesn't know it has to add
HTML for image previews to messages that reference images. This would
mean that messages imported from 3rd party tools don't get image
previews.
- Attachments only get created after Message import however, due to the
many-to-many relationship between Message and Attachment.
This is solved by fixing up some data of Attachments pre-emptively, such
as the path_ids. This gives us the necessary information for creating
ImageAttachments before importing Messages.
While we generate ImageAttachment rows synchronously, the actual
thumbnailing job is sent to the queue worker. Theoretically, the worker
could be very backlogged and not process the thumbnails anytime soon.
This is fine - if the app is loaded and tries to display a message with
such a not-yet-generated thumbnail, the code in `serve_file` will
generate the thumbnails synchronously on the fly and the user will see
the image preview displayed normally. See:
1b47134d0d/zerver/views/upload.py (L333-L342)
This commit makes the third-party data converters check for invalid user
emails. If it finds any, it’ll raise an Exception and show an error
message with all the bad emails listed out.
Fixes: #31783
(cherry picked from commit 6289a551aa)
@extend can only be used in an element selector, not directly within
@media.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 2c8ad219b1)
Commit bca41fd29f (#23028) introduced
this for reload.is_stale_refresh_token, which had always returned true
because it was operating on the raw JSON string rather than the parsed
data.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 782fa2a803)
This is not the best factored version of this, but it saves effort
changing the tests, and importantly should make failures involving
metadata only take a couple seconds rather than first doing a giant
BSON read before learning about them.
(cherry picked from commit 6e4da50577)
This prevents a deadlock between the thumbnailing worker and message
sending, as follows:
1. A user uploads an image, making Attachment and ImageAttachment
rows, as well as enqueuing a job in the thumbnailing queue.
2. Message sending starts a transaction, creates the Message row,
and calls `do_claim_attachments`, which edits the Attachment row
of the upload (implicitly locking it).
3. The thumbnailing worker starts a transaction, locks the
ImageAttachment row for its image, thumbnails it, and then
attempts to `select_for_update()` the message objects (joined to
the Attachments table) to find the ones which link to the
attachment in question. This query blocks, since "a locking
clause without a table list affects all tables used in the
statement"[^1] and the message-send request already has a write
lock on the Attachments row in question.
4. The message-send request attempts to re-fetch the ImageAttachment
row inside the transaction, which tries to pull a lock on it.
5. Deadlock, because the message-send request has the Attachment
lock, and waits for the ImageAttachment lock; the thumbnailing
worker has the ImageAttachment lock, and waits for the Attachment
lock.
We break this deadlock by limiting the
`update_message_rendered_content` `select_for_update` to only take
the lock on the Message table, and not also the Attachments table --
no changes will be made to the Attachments, so no lock is necessary
there. This allows the thumbnailing worker to successfully pull the
empty list of messages (since the message-send request has not
commits its transaction, and thus the Message row is not visible
yet), and release its ImageAttachment lock so that the message-send
request can proceed.
[^1]: https://www.postgresql.org/docs/current/sql-select.html#SQL-FOR-UPDATE-SHARE
(cherry picked from commit 3cbbf2307b)
This better simulates the Slack API, which is important, since some
integrations check this response and decide whether the Slack endpoint
is working based on what they receive.
(cherry picked from commit 5cbe3203f5)
Previously [{operator: "topic", operand: "one\xa0two"}] would be
unparsed to "topic:one\xa0two" which parses as [{operator: "topic",
operand: "one"}, {operator: "search", operand: "two"}], leading to
exceptions in the search pill system.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
(cherry picked from commit 9c2da46966)
Tim Abbott
Anders Kaseorg
PieterCK
Aman Agrawal
Mateusz Mandera
Alex Vandiver
Aditya Kumar Kasaudhan
Karl Stolley
Johan Nilsson
Pierre Carru