Commit Graph

19156 Commits

Author SHA1 Message Date
Harsh e468818d2b import: Remove skipping of too-long messages during import.
This commit eliminates the skipping of messages longer than 10K characters during the import process.
2024-11-07 16:04:14 -08:00
PieterCK 92437b4ab5 integrations: Refactor slack_incoming webhook.
This commit refactors `render_attachment` and `render_block`
out of slack_incoming.py to promote reusability.

The primary motivation for this refactor is to add support for
converting integration bots messages in Slack exports, which could use
the same functions.

Part of #31311.
2024-11-07 10:47:08 -08:00
swayam0322 6c583c9bfe integrations: Update GoCD integration.
Previously, the GoCD integration relied on GoCD's integration with bug
tracking and monitoring tools through the users' Config XML file [1].
However, this feature no longer works as expected, as it cannot send
HTTPS payloads to an external endpoint.

This commit updates our GoCD integration to use Sentry's WebHook
notifier GoCD plugin [2] to send webhook payloads from users GoCD server
to Zulip. We are using an older version of the plugin—v0.0.6—because the
newer version—v0.0.9—doesn't work at this time.

Additionally, this change reformats the notifications to include more
details on the GoCD event, such as lists of passed and failed jobs.

Fixes #21224.

Co-authored-by: Pieter CK <pieterceka123@gmail.com>

[1]:
https://docs.gocd.org/current/integration/#integration-with-bug-tracking-and-story-management-tools
[2]:
https://github.com/getsentry/gocd-webhook-notification-plugin/releases/tag/v0.0.6
2024-11-07 10:25:29 -08:00
Prakhar Pratyush 86a909e703 users: Add `savepoint=False` to avoid creating savepoints.
'do_deactivate_user' is used inside an outer db transaction
created in 'sync_user_from_ldap'.

`transaction.atomic()` block in 'do_deactivate_user' resulted
in savepoint creation.

This commit adds `savepoint=False` to avoid that.
2024-11-05 17:58:47 -08:00
Prakhar Pratyush 27eeb08459 streams: Add `savepoint=False` to avoid creating savepoints.
'bulk_remove_subscriptions' is used inside an outer db transaction
created in 'do_change_bot_owner'.

`transaction.atomic()` block in 'bulk_remove_subscriptions'
resulted in savepoint creation.

This commit adds `savepoint=False` to avoid that.
2024-11-05 17:58:47 -08:00
Prakhar Pratyush 8c8cc80183 create_user: Add `savepoint=False` to avoid creating savepoints.
'do_activate_mirror_dummy_user' is used inside an outer db
transaction created in 'registration_helper'.

`transaction.atomic()` block in 'do_activate_mirror_dummy_user'
resulted in savepoint creation.

This commit adds `savepoint=False` to avoid that.
2024-11-05 17:58:47 -08:00
Prakhar Pratyush ca4760a04c recipients: Add `savepoint=False` to avoid creating savepoints.
'get_or_create_direct_message_group' is used inside an outer
db transaction created in 'edit_scheduled_message'.

`transaction.atomic()` block in 'get_or_create_direct_message_group'
resulted in savepoint creation.

This commit adds `savepoint=False` to avoid that.
2024-11-05 17:58:47 -08:00
Prakhar Pratyush 0e67e4f1a1 compose_views: Add `savepoint=False` to avoid creating savepoints.
'compose_views' is used inside an outer db transaction created in
'update_user_group_backend'.

`transaction.atomic()` block in 'compose_views' resulted in
savepoint creation.

This commit adds `savepoint=False` to avoid that.
2024-11-05 17:58:47 -08:00
Prakhar Pratyush 9c9866461a transaction: Add `durable=True` to the outermost db transactions.
This commit adds `durable=True` to the outermost db transactions
created in the following:
* confirm_email_change
* handle_upload_pre_finish_hook
* deliver_scheduled_emails
* restore_data_from_archive
* do_change_realm_subdomain
* do_create_realm
* do_deactivate_realm
* do_reactivate_realm
* do_delete_user
* do_delete_user_preserving_messages
* create_stripe_customer
* process_initial_upgrade
* do_update_plan
* request_sponsorship
* upload_message_attachment
* register_remote_server
* do_soft_deactivate_users
* maybe_send_batched_emails

It helps to avoid creating unintended savepoints in the future.

This is as a part of our plan to explicitly mark all the
transaction.atomic calls with either 'savepoint=False' or
'durable=True' as required.

* 'savepoint=True' is used in special cases.
2024-11-05 17:58:47 -08:00
PieterCK 7fc9fc32d1 avatars: Split email-based and user ID-based avatar endpoints.
Separate `avatars/<email_or_id>/medium?` endpoints into distinct
endpoints for email-based and user ID-based access. This change aligns
avatar endpoints with Zulip’s existing API path conventions (e.g., the
`users/` endpoint).
2024-11-05 09:06:17 -08:00
Lauryn Menard 3ebc507ddb signups: Move logic for realm admin notification to corporate app.
As we would like to send similar notifications for other billing
state changes (for all BillingSession types), it makes sense to
move the logic for creating and sending these admin realm internal
messages to the BillingSession framework in the corporate app.

In the case that a channel with the specified name does not exist,
we now send direct messages to the admin realm administrators with
the channel, topic and message so that the information is not lost
and so that the channel for these messages can be created.
2024-11-05 08:56:24 -08:00
Prakhar Pratyush 9a40319bfc management: Add option to reset all the active realms to a level.
This commit adds a --reset-level optional argument to
send_zulip_update_announcements management command to
reset all the active realms to a given level.
2024-11-05 08:51:06 -08:00
Tim Abbott c073e5adb4 migrations: Avoid following realm foreign key unnecessarily.
I encountered at least one scenario where the previous logic would
fail due to migration state confusion, but this could also be a bit
faster.
2024-11-04 23:07:43 -08:00
Tim Abbott 5571fb6a3e migrations: Fix migration 0617 case sensitivity bug. 2024-11-04 22:10:37 -08:00
Alya Abbott eea2499e02 zulip updates: Add update about sidebars and file size limit. 2024-11-04 16:54:58 -08:00
Kislay Udbhav Verma 000cc7bcde markdown: Add support for a pretty syntax for message links.
Links to zulip messages can now be written as
`#**channel_name > topic_name @ message_id**.`
The `message_id` is replaced with `💬` in the rendered
message.

Fixes part of #31920
2024-11-04 13:44:40 -08:00
Kislay Udbhav Verma 65c9b249b7 markdown: Refactor classes handling stream topic links.
The classes StreamPattern and StreamTopicPattern both had a
separate copy of the function `find_stream_id` which did the same
thing. Since another Pattern will be added as a part of #31920, it
is a good idea to move that function into a superclass which is then
inherited by all the related patterns.

Fixes part of #31920
2024-11-04 13:44:40 -08:00
Harsh 858fdeee39 import: Add function to normalize messages to import.
This adds `normalize_body_for_import` to normalize messages from
third-party importers by removing NUL bytes and also updates import
test files data to test this.

Fixes #31930.
2024-11-04 12:22:30 -08:00
Prakhar Pratyush a34577f82e user_groups: Add savepoint=False to avoid creating savepoints.
add_subgroups_to_user_group and remove_subgroups_from_user_group
are already inside outer db transactions. This commit explicitly
adds 'savepoint=False' to avoid creating savepoints.
2024-11-01 16:41:15 -07:00
Prakhar Pratyush c4f74f470d remote_server_post_analytics: Add durable=True to outermost transaction.
This commit adds 'durable=True' to the outermost transaction
in 'remote_server_post_analytics'.

It also adds 'savepoint=False' to inner transaction.atomic
decorator to avoid creating savepoint.

This is as a part of our plan to explicitly mark all the
transaction.atomic decorators with either 'savepoint=False' or
'durable=True' as required.

* 'savepoint=True' is used in special cases.
2024-11-01 16:41:15 -07:00
Prakhar Pratyush 3d597bb9b0 delete_message_backend: Add `durable=True` to the outermost transaction.
This commit adds 'durable=True' to the outermost transaction
in 'delete_message_backend'.

It also adds 'savepoint=False' to inner transaction.atomic
decorator to avoid creating savepoint.

This is as a part of our plan to explicitly mark all the
transaction.atomic decorators with either 'savepoint=False' or
'durable=True' as required.

* 'savepoint=True' is used in special cases.
2024-11-01 16:41:15 -07:00
Prakhar Pratyush 0fb5657131 transaction: Add durable=True to outermost transaction.atomic decorator.
This commit adds 'durable=True' to the outermost transactions
of the following functions:
* do_create_multiuse_invite_link
* do_revoke_user_invite
* do_revoke_multi_use_invite
* sync_ldap_user_data
* do_reactivate_remote_server
* do_deactivate_remote_server
* bulk_handle_digest_email
* handle_customer_migration_from_server_to_realm
* add_reaction
* remove_reaction
* deactivate_user_group

It helps to avoid creating unintended savepoints in the future.

This is as a part of our plan to explicitly mark all the
transaction.atomic decorators with either 'savepoint=False' or
'durable=True' as required.

* 'savepoint=True' is used in special cases.
2024-11-01 16:41:15 -07:00
Prakhar Pratyush 9371bdb81d invites: Add `durable=True` to transaction in `do_invite_users`.
This commit adds 'durable=True' to the outermost transaction
in 'do_invite_users'.

It also adds 'savepoint=False' to inner transaction.atomic
decorators to avoid creating savepoints.
2024-11-01 16:41:15 -07:00
aniebietafia 257ce8bca2 integrations: Create incoming webhook for Airbyte.
Note about the documentation: There are currently two "Save changes"
buttons on the Airbyte "Notifications" settings page, so the
instructions specify which one to use for clarity.
2024-11-01 10:04:48 -07:00
Tim Abbott f54817d78d groups: Enable group-valued settings in production. 2024-10-31 12:04:51 -07:00
Aditya Kumar Kasaudhan d1ff871523 webhooks: Support filtering GitHub activity from private repositories.
Currently, the GitHub webhook sends activity from both public and private
repositories, which could lead to unintended disclosure of sensitive
information from private repositories.

This commit introduces a ignore_private_repositories parameter to the
webhook URL. When set to true, the webhook ignore processing activity from
private repositories, ensuring that such activities are not posted to
Zulip streams. By default, if the parameter is omitted or set to false,
activities from both public and private repositories are processed
normally. This provides users with the flexibility to control the
visibility of private repository activities without altering the default
behavior.

More importantly, this introduces a cleaner mechanism for individual
incoming webhooks to declare support for settings not common to all
webhook integrations.

Fixes #31638.
2024-10-31 10:40:28 -07:00
Prakhar Pratyush fdf90f7ad1 create_user: Handle integrity error when importing settings.
During account creation when a user opted to import settings
from an existing account, the "Mark visibility_policy_banner as
read" step was raising integrity error.

It is because 'copy_onboarding_steps' is already executed earlier
in the 'do_create_user' codeflow. If the source profile had already
marked 'visibility_policy_banner' as read, we were facing integrity
error.

This commit fixes the bug.
2024-10-31 09:55:01 -07:00
Prakhar Pratyush 555ac613ac create_user: Add 'durable=True' to the outermost transaction.
We need to specify savepoint=False explicitly in the one subroutine
that already specifies a transaction.
2024-10-31 09:55:01 -07:00
Aditya Kumar Kasaudhan d448b75176 slack_incoming: Add ok=false to JSON in case of error.
Previously, errors were returned using Zulip's default format,
which did not match Slack's expected response structure.

This change ensures that errors in the Slack incoming webhook handler
return JSON responses in Slack's expected format: {ok: false, error:
"error string"}.

Fixes: #31878.
2024-10-30 15:33:29 -07:00
Tim Abbott 66537c8bf8 migrations: Fix migration 0576 local backfill path and repeat.
Unfortunately, because this migration has already been run for many
installations, we need to ship another copy of the migration.

It should be a noop when repeated.
2024-10-30 12:06:33 -07:00
Anders Kaseorg 42e1517255 email_notifications: Prevent html2text from mangling Unicode.
html2text mangles Unicode by default, with a --unicode-snob option to
disable it.  If I have to get called a “snob” for wanting to correctly
support non-English languages, then uh, I’ll take one for the team.

https://github.com/Alir3z4/html2text/blob/2024.2.26/html2text/config.py#L111-L150

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-10-30 09:49:23 -07:00
PieterCK fc50736f4e slack_data_import: Fix incorrect hyperlink conversion.
Currently, Slack messages containing hyperlinks
(e.g.,<http://foo.com|Foo!>) are converted like
normal links. This commit reformats Slack
hyperlinks into Zulip-friendly markdown
(e.g., [Foo!](http://foo.com)).

Part of #32165.
2024-10-30 09:48:32 -07:00
Aditya Kumar Kasaudhan 18a8125dac user_groups: Include group_id in success response on group creation.
Previously, the `group_id` was not returned in the success
response of the user group creation API.

This commit updates the API to return a success response
containing the unique ID of the user group with the key
`group_id`. This enhancement allows clients to easily reference
the newly created user group.

Fixes: #29686
2024-10-30 09:46:38 -07:00
Vector73 1ba1408b01 settings: Remove `edit_topic_policy` setting.
Removed `edit_topic_policy` property, as the permission
to move messages between topcis is now controlled by
`can_move_messages_between_topics_group` setting.
2024-10-29 16:27:04 -07:00
Vector73 ed5638ec3c settings: Add `can_move_messages_between_topics_group` realm setting.
Added `can_move_messages_between_topics_group` realm setting to replace
`edit_topic_policy`.
2024-10-29 16:27:04 -07:00
Mateusz Mandera d708a1bea5 tests: Fix order of mock.patch in use_s3_backend.
This is a somewhat hacky and fragile fix. Due to the order in which
imports seem to happen, the original ordering breaks
RealmImportExportTest: if one of the `use_s3_backend` tests runs before
test_import_realm, the latter will fail while processing thumbnailing,
as S3UploadBackend ends up leaking and
zerver.worker.thumbnail.upload_backend is still set to S3.

By making that mock.patch the first one that gets entered, and thus the
last one to get cleaned up, we fix the leak and upload_backend is set
back to LocalUploadBackend as it should.
2024-10-29 10:43:10 -07:00
sanchi-t c73038edea streams: Restrict access to archived streams. 2024-10-25 16:06:43 -07:00
Sanchit Sharma 795b2ba14e do_deactivate_stream: Remove unnecessary mutations.
Streams should not be marked as private, and subscribers
of the deactivated stream should not be removed.

Update the confirmation message when archiving a stream.
2024-10-25 16:06:42 -07:00
sanchi-t 7e97358c35 home: Update `client_capabilities` for viewing of archived streams. 2024-10-25 16:06:42 -07:00
sanchi-t af7ebde9e4 subscription: Include archived channels in streams list.
`is_archived` field is added to the stream and types.

Include a new `archived_channeels` client capability, to allow clients
to access data on archived channels, without breaking
backwards-compatibility for existing clients that don't know how to
handle these.

Also, included `exclude_archived` parameter to `/get-streams`,
which defaults to `true` as basic clients may not be interested
in archived streams.
2024-10-25 16:06:40 -07:00
sanchi-t c6fc25e5df do_deactivate_stream: Do not rename streams during archiving.
Functions related to archived streams are also updated.
2024-10-25 15:56:04 -07:00
sanchi-t e25bbe1005 migration: Rename previously archived streams to their original names.
As several archived streams may have the same new name,
it is essential to verify whether any stream, regardless
of its current status (active or archived), already has that name
before executing any renaming operation.
2024-10-25 15:56:04 -07:00
Tim Abbott 71d81484ad storage: Simplify system bot avatar logic. 2024-10-25 10:59:57 -07:00
PieterCK c3017e55d2 storage: Rework static avatar files hashing logic.
Previously, the hashing logic for static avatar files hashed the default
and medium files separately, which didn’t match how user-uploaded
avatars work—where you just add the "-medium.png" suffix to get the
medium version. Since we don’t have clear documentation for avatars yet,
this caused some issues for the mobile apps.

This commit makes sure the default and its medium variation share the
same hash.
2024-10-25 09:36:52 -07:00
Shubham Padia fe9952eab5 groups: Rename test to state that it checks allow_system_group.
We also remove the default group check which is already being checked in
do_set_realm_permission_group_setting_to_anonymous_groups_test.
2024-10-24 14:42:21 -07:00
Shubham Padia 1f0906aef7 groups: Remove list specifying group settings with new API format.
All the groups are now following the new API format, making the list
redundant.
2024-10-24 14:42:21 -07:00
Shubham Padia 9f9d5b2f98 groups: Accept anonymous groups for can_access_all_users_group.
On the frontend, the selection is still a dropdown of system groups but
on the API level, we have started accepting anonymous groups similar to
other settings
We've kept require system groups true for now until we switch to group
picker on the frontend.
2024-10-24 14:42:19 -07:00
Shubham Padia ceb0197c1b groups: Accept anonymous groups for create_multiuse_invite_group.
On the frontend, the selection is still a dropdown of system groups but
on the API level, we have started accepting anonymous groups similar to
other settings.
We've kept require system groups true for now until we switch to group
picker on the frontend.
2024-10-24 14:41:09 -07:00
Tim Abbott 3310aed462 send_custom_email: Support specifying sender email address. 2024-10-24 14:36:48 -07:00
Mateusz Mandera 481ef24b53 auth: Tweak invalid credentials error message.
We decided this is the better string to use.
2024-10-24 14:35:31 -07:00