This commit updates all third-party importer tools (Slack, Mattermost,
and Rocket Chat) in the `zerver/data_import` directory to also output a
migration_status.json file in their output tarball.
This is required because all importable tarball will be checked for
migration compatibility during import.
Fixes#28443.
When transferring a realm to a server that has a different set of
applied migrations (different Zulip versions), there is a chance that
the imported data formats appear to be compatible but data invariants
could still be violated.
This commit adds an assertion during the import process to verify
that both the exported realm and the importing server have matching
Zulip versions and have a compatible set of migrations.
This commit updates the export process to write the migration status of
the realm as a JSON file to be included in the export tarball.
This is a preparatory step for adding an assertion to ensure that the
importing and exporting realms have a compatible set of applied
migrations.
'compose_views' is used inside an outer db transaction created in
'update_user_group_backend'.
`transaction.atomic()` block in 'compose_views' resulted in
savepoint creation.
This commit adds `savepoint=False` to avoid that.
Separate `avatars/<email_or_id>/medium?` endpoints into distinct
endpoints for email-based and user ID-based access. This change aligns
avatar endpoints with Zulip’s existing API path conventions (e.g., the
`users/` endpoint).
This commit adds a --reset-level optional argument to
send_zulip_update_announcements management command to
reset all the active realms to a given level.
Links to zulip messages can now be written as
`#**channel_name > topic_name @ message_id**.`
The `message_id` is replaced with `💬` in the rendered
message.
Fixes part of #31920
This adds `normalize_body_for_import` to normalize messages from
third-party importers by removing NUL bytes and also updates import
test files data to test this.
Fixes#31930.
This commit adds 'durable=True' to the outermost transaction
in 'remote_server_post_analytics'.
It also adds 'savepoint=False' to inner transaction.atomic
decorator to avoid creating savepoint.
This is as a part of our plan to explicitly mark all the
transaction.atomic decorators with either 'savepoint=False' or
'durable=True' as required.
* 'savepoint=True' is used in special cases.
Currently, the GitHub webhook sends activity from both public and private
repositories, which could lead to unintended disclosure of sensitive
information from private repositories.
This commit introduces a ignore_private_repositories parameter to the
webhook URL. When set to true, the webhook ignore processing activity from
private repositories, ensuring that such activities are not posted to
Zulip streams. By default, if the parameter is omitted or set to false,
activities from both public and private repositories are processed
normally. This provides users with the flexibility to control the
visibility of private repository activities without altering the default
behavior.
More importantly, this introduces a cleaner mechanism for individual
incoming webhooks to declare support for settings not common to all
webhook integrations.
Fixes#31638.
During account creation when a user opted to import settings
from an existing account, the "Mark visibility_policy_banner as
read" step was raising integrity error.
It is because 'copy_onboarding_steps' is already executed earlier
in the 'do_create_user' codeflow. If the source profile had already
marked 'visibility_policy_banner' as read, we were facing integrity
error.
This commit fixes the bug.
html2text mangles Unicode by default, with a --unicode-snob option to
disable it. If I have to get called a “snob” for wanting to correctly
support non-English languages, then uh, I’ll take one for the team.
https://github.com/Alir3z4/html2text/blob/2024.2.26/html2text/config.py#L111-L150
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Currently, Slack messages containing hyperlinks
(e.g.,<http://foo.com|Foo!>) are converted like
normal links. This commit reformats Slack
hyperlinks into Zulip-friendly markdown
(e.g., [Foo!](http://foo.com)).
Part of #32165.
Removed `edit_topic_policy` property, as the permission
to move messages between topcis is now controlled by
`can_move_messages_between_topics_group` setting.
Streams should not be marked as private, and subscribers
of the deactivated stream should not be removed.
Update the confirmation message when archiving a stream.
`is_archived` field is added to the stream and types.
Include a new `archived_channeels` client capability, to allow clients
to access data on archived channels, without breaking
backwards-compatibility for existing clients that don't know how to
handle these.
Also, included `exclude_archived` parameter to `/get-streams`,
which defaults to `true` as basic clients may not be interested
in archived streams.
On the frontend, the selection is still a dropdown of system groups but
on the API level, we have started accepting anonymous groups similar to
other settings
We've kept require system groups true for now until we switch to group
picker on the frontend.
On the frontend, the selection is still a dropdown of system groups but
on the API level, we have started accepting anonymous groups similar to
other settings.
We've kept require system groups true for now until we switch to group
picker on the frontend.
We didn't have thumbnailing for images coming from data import and this
commit adds the functionality.
There are a few fundamental issues that the implementation needs to
solve.
1. The images come from an untrusted source and therefore we don't want
to just pass them through to thumbnailing without checking. For that
reason, we cannot just import ImageAttachment rows from the export
data, even for zulip=>zulip imports.
The right way to process images is to pass them to maybe_thumbail(),
which runs libvips_check_image() on them to verify we're okay with
thumbnailing, creates ImageAttachment rows for them and sends them
to the thumbnailing queue worker. This approach lets us handle both
zulip=>zulip and 3rd party=>zulip imports in the same way,
2. There is a somewhat circular dependency between the Message,
Attachment and ImageAttachment import process:
- ImageAttachments would ideally be created after importing
Attachments, but they need to already exist at the time of Message
import. Otherwise, the markdown processor doesn't know it has to add
HTML for image previews to messages that reference images. This would
mean that messages imported from 3rd party tools don't get image
previews.
- Attachments only get created after Message import however, due to the
many-to-many relationship between Message and Attachment.
This is solved by fixing up some data of Attachments pre-emptively, such
as the path_ids. This gives us the necessary information for creating
ImageAttachments before importing Messages.
While we generate ImageAttachment rows synchronously, the actual
thumbnailing job is sent to the queue worker. Theoretically, the worker
could be very backlogged and not process the thumbnails anytime soon.
This is fine - if the app is loaded and tries to display a message with
such a not-yet-generated thumbnail, the code in `serve_file` will
generate the thumbnails synchronously on the fly and the user will see
the image preview displayed normally. See:
1b47134d0d/zerver/views/upload.py (L333-L342)
This commit introduces an assertion to verify that the avatar file for
system bots exists and findable.
In development, it'll assert that the avatar file exists in the static
directory. This isn't done in production environment to avoid
unnecessary overhead. It helps verify that the protocol to fetch system
bot avatars still works when making changes during development.
In production it'll check if the avatar file exists in the STATIC_ROOT
and return a default avatar png if it doesn't.
Previously, requesting system bots URLs did not return any -medium.png
variants and SVG file was also used for notification bots' avatar, which
was problematic.
In this commit, the -medium.png variants is added for the avatars of
system bots and zulip-icon-square.svg is also converted into
notification-bot.png for the notification bot. The get_avatar_url method
has been updated to return the "medium" file variants for the system
bots.
Additionally, the system bots' avatar files is moved to a dedicated
directory to simplify the hashing logic for these files. Now, all files
in the "images/static_avatars/" directory will be hashed.
The old endpoint for updating a user worked only via user id. Now we add
a different entry to this functionality, fetching the user by
.delivery_email.
update_user_backend becomes the main function handling all the logic,
invoked by the two endpoints.
This adds a new special UserProfile flag can_change_user_emails(disabled
by default) and the ability for changing the email address of users in
the realm via update_user_backend. This is useful for allowing
organizations to update user emails without needing to set up a SCIM
integration, but since it gives the ability to hijack user accounts, it
needs to be behind this additional permission and can't be just given to
organization owners by default. Analogical to how the
create_user_backend endpoint works.
This makes a Zulip server more isolated than relying on gravatar, and
avoids complex logistics if in the future we move system bots to live
inside individual realms.
Co-authored-by: PieterCK <pieterceka123@gmail.com>
This commit adds support to add subgroups to a group while
creating it.
User can add the subgroups to group irrespective of permissions
like user can add members during creating it.
This commit updates code to allow users with permission
to add members to add subgroups as well. And only users
with permission to manage the group can remove subgroups.
Also updated tests to check permissions in separate tests
and removed them from the existing test.
The comment about non-admins and non-moderators who are not
member of the group cannot update subgroups of that group
is not correct since there is no such restriction now after
c9d527603. The test passes because the member user is not
part of can_manage_group or can_manage_all_groups.
Removed `move_messages_between_streams_policy` property, as the permission
to move messages between channels is now controlled by
`can_move_messages_between_channels_group` setting.
We want to allow the user, who can add others to group, to
join the group as well irrespective of can_join_group setting.
Previously, the permission to add others (or say anyone) was
controlled by can_manage_group setting, but now it is controlled
by can_add_members_group setting. This commit fixes the code to
use can_add_members_group setting to check permission for joining
the group.
This commit also improves the tests for checking permission to
join the group such that different settings are tested in isolation.
This commit makes the third-party data converters check for invalid user
emails. If it finds any, it’ll raise an Exception and show an error
message with all the bad emails listed out.
Fixes: #31783
The main change is redefining ALLOW_GROUP_VALUED_SETTINGS to not
control code, but instead to instead control the configuration for
whether settings that have not been converted to use our modern UI
patterns should require system groups.
Fundamentally, it's the same for the realm/stream group-valued
settings, which don't have the new UI patterns yet.
We remove the visual hiding of the "can manage group" setting, which
was hidden only due to transitions being incomplete.
PieterCK
Mateusz Mandera
Prakhar Pratyush
Kislay Udbhav Verma
Harsh
Aditya Kumar Kasaudhan
Anders Kaseorg
Vector73
sanchi-t
Shubham Padia
Sahil Batra
Tim Abbott