Commit Graph

8793 Commits

Author SHA1 Message Date
Sahil Batra f197d881ca streams: Fix handling empty groups when computing setting values.
Since "Nobody" and other user-defined empty groups has no
members or subgroups, we need a UserGroup query.

Since we are doing UserGroup query, updated the code to
check whether the setting is set to anonymous group or
not just after doing UserGroup query and not in membership
queries.

Also, added tests to check query count when setting is set
to anonymous group.
2024-11-22 10:48:41 -08:00
Prakhar Pratyush c5b3d2e434 custom_check: Add rule to avoid creating savepoints.
This commit adds a custom rule to check python files
and raise lint error if they have transaction.atomic used
without any argument or savepoint=True is used.

It helps to avoid creating unintended savepoints in the future.
2024-11-21 14:55:15 -08:00
Prakhar Pratyush 62b3e49075 transaction: Add 'savepoint=True' explicitly in tests, if used.
We create savepoint at a couple of places in backend tests
to avoid rollback due to error raised within test's transaction.

This commit explicitly adds 'savepoint=True' at those places.
2024-11-21 14:55:15 -08:00
Prakhar Pratyush 175104ea01 streams: Add 'ChannelEmailAddress' model.
This commit removes the 'email_token' field from the
'Stream' model, introduces a new model 'ChannelEmailAddress',
and backfills it.

This is a prep work towards our plan to generate unique
channel emails for different users, which can be used
to check post permissions at message send time.
2024-11-21 14:53:28 -08:00
Anders Kaseorg 801e267e00 integrations: Properly pass data-categories as a JSON array.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-11-21 14:41:36 -08:00
Sahil Batra b6f895f61c streams: Optimize DB queries for computing permission setting values.
This commit updates code to use union so that we can compute values
for permission settings using a single DB query.
2024-11-21 11:28:54 -08:00
Sahil Batra d1e76a9281 streams: Allow admins to unsubscribe others irrespective of setting. 2024-11-21 11:28:54 -08:00
Niloth P 8e17ef3507 integrations: Refine CircleCI example screenshots.
- Removed the extra screenshot.
- Updated the fixture used to generate the screenshot to use a more
suitable topic name and commit message description.
2024-11-21 11:09:40 -08:00
Niloth P ab7d5ceacc integrations: Refine Zendesk example screenshot. 2024-11-21 11:09:40 -08:00
Vector73 4e89b4a88c settings: Add `can_invite_users_group` realm setting.
Added `can_invite_users_group` realm setting to replace
`invite_to_realm_policy`.
2024-11-20 13:35:05 -08:00
Sahil Batra a19287d581 streams: Optimize fetching setting values for web public subs.
This commit updates code to fetch the group setting values
efficiently for web public subscription objects.
2024-11-19 13:26:42 -08:00
Sahil Batra f4c00ce053 streams: Fix setting values passed in never subbed dicts.
This commit fixes the code to pass AnonymousSettingGroupDict
value instead of group ID when the setting is set to anonymous
groups in never subscribred streams subscription dicts.
2024-11-19 13:26:42 -08:00
Anders Kaseorg 532aee926c requirements: Upgrade Python requirements.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-11-19 11:09:34 -08:00
jitendra-ky ca14366e38 import: Replace generic Exception with CommandError.
This change improves error handling in `do_import_realm` by replacing
the use of a generic Exception with CommandError. The updated approach
provides clearer, user-friendly error messages when there is a version
mismatch between the exported data and the Zulip server.

Fixes #32292.
2024-11-18 18:35:14 -08:00
Sahil Batra d5a391a56b streams: Optmize code to send events on creating stream.
This commit updates code to effectively compute the setting
values when creating stream object to be sent in stream creation
events.
2024-11-18 11:55:19 -08:00
Sahil Batra 7adc83d2a0 streams: Optimize code for computing stream objects.
This commit updates code to optimize code for computing stream
objects to be sent with stream creation event and in response
for 'GET /streams/{stream_id}' endpoint by optimizing number
of queries while computing values for group permission settings.

This change does not benefit much currently as we only have one
stream group permission setting, but is important before we add
more stream permission settings.

There are a couple of places left where we can still optimize
the code and that would be done in further commits.
2024-11-18 11:55:19 -08:00
Sahil Batra 2cc0f482e1 streams: Refactor code to compute setting group values.
This commit adds a new function to compute setting group
values for a list of streams, so we can avoid having duplicate
code for computing setting group IDs from streams.
2024-11-18 11:55:19 -08:00
Shubham Padia b6ebf143cc streams: Backend changes to support anonymous groups.
can_remove_subscribers_group setting can now be set to
anonymous user groups.

Co-authored-by: Sahil Batra <sahil@zulip.com>
2024-11-16 17:11:08 -08:00
Sahil Batra b78ca79ccf user_groups: Define AnonymousSettingGroupDict in types.py.
AnonymousSettingGroupDict is now defined in types.py instead
of user_groups.py to avoid import cycles in future commits.
2024-11-15 16:36:37 -08:00
Sahil Batra 10ae4ffea4 subscription: Refactor code for creating subscription dicts.
This commit refactors code to create subscription dicts so
that we can efficiently compute can_remove_subscribers_group
setting when we add support to set the setting to anonymous
groups.
2024-11-15 16:36:37 -08:00
whilstsomebody 8356a9d9e4
narrow_filter: Add negation support for in:home narrow filter.
The in:home narrow filter is used to filter messages that appear
in the home view, i.e., messages that are not muted. Conversely,
`-in:home` should filter messages that are not in the home view,
i.e., muted messages. However, `-in:home` did not work as expected
because this filter lacked negation support, unlike similar code
paths.

This commit adds negation support for the in:home filter.

For more information, see:
<https://chat.zulip.org/#narrow/channel/378-api-design/topic/mark.20muted-topic.20messages.20as.20read/near/1980534>.
2024-11-14 15:10:01 -08:00
Anders Kaseorg 687f1b1651 tests: Rename the Node tests to *.cjs.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-11-13 09:18:56 -08:00
Anders Kaseorg b3f18b805d ruff: Enable B008 function-call-in-default-argument.
https://docs.astral.sh/ruff/rules/function-call-in-default-argument/

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-11-10 22:43:55 -08:00
PieterCK a9838d8089 import: Verify exported realm's migration compatibility.
When transferring a realm to a server that has a different set of
applied migrations (different Zulip versions), there is a chance that
the imported data formats appear to be compatible but data invariants
could still be violated.

This commit adds an assertion during the import process to verify
that both the exported realm and the importing server have matching
Zulip versions and have a compatible set of migrations.
2024-11-08 15:52:45 -08:00
PieterCK 40bcb4b42b export: Add migration status file to export tarball.
This commit updates the export process to write the migration status of
the realm as a JSON file to be included in the export tarball.

This is a preparatory step for adding an assertion to ensure that the
importing and exporting realms have a compatible set of applied
migrations.
2024-11-08 15:52:45 -08:00
Giovanni Silva 70d559cafa integrations: Update New Relic integration.
Newrelic has introduced a new update. This commit
updates the exisiting integration by adding support
for new types of payloads and cleaning up legacy
codes and fixtures.

Fixes #29729.

Co-authored-by: Pieter CK <pieterceka123@gmail.com>
Co-authored-by: Lauryn Menard <lauryn@zulip.com>
2024-11-08 15:49:56 -08:00
Mateusz Mandera 420849ff6a slack: Call the correct resize_* function when importing realm icon.
For resizing the icon.png files, we use resize_avatar, not resize_logo.
This is pretty confusing - sure, for icons we use the same function as
for avatars, but we should have a proper name for the function called in
the icon context. So this commit also adds resize_realm_icon, and
changes the calls to resize_avatar in icon contexts to
resize_realm_icon.
2024-11-08 15:43:18 -08:00
Mateusz Mandera a669fece3a email_mirror: Check that email gateway bot can send to the channel.
Otherwise this leads to an uncaught exception when the check happens
later in `check_message`.
2024-11-08 13:13:15 -08:00
swayam0322 6c583c9bfe integrations: Update GoCD integration.
Previously, the GoCD integration relied on GoCD's integration with bug
tracking and monitoring tools through the users' Config XML file [1].
However, this feature no longer works as expected, as it cannot send
HTTPS payloads to an external endpoint.

This commit updates our GoCD integration to use Sentry's WebHook
notifier GoCD plugin [2] to send webhook payloads from users GoCD server
to Zulip. We are using an older version of the plugin—v0.0.6—because the
newer version—v0.0.9—doesn't work at this time.

Additionally, this change reformats the notifications to include more
details on the GoCD event, such as lists of passed and failed jobs.

Fixes #21224.

Co-authored-by: Pieter CK <pieterceka123@gmail.com>

[1]:
https://docs.gocd.org/current/integration/#integration-with-bug-tracking-and-story-management-tools
[2]:
https://github.com/getsentry/gocd-webhook-notification-plugin/releases/tag/v0.0.6
2024-11-07 10:25:29 -08:00
Prakhar Pratyush 9c9866461a transaction: Add `durable=True` to the outermost db transactions.
This commit adds `durable=True` to the outermost db transactions
created in the following:
* confirm_email_change
* handle_upload_pre_finish_hook
* deliver_scheduled_emails
* restore_data_from_archive
* do_change_realm_subdomain
* do_create_realm
* do_deactivate_realm
* do_reactivate_realm
* do_delete_user
* do_delete_user_preserving_messages
* create_stripe_customer
* process_initial_upgrade
* do_update_plan
* request_sponsorship
* upload_message_attachment
* register_remote_server
* do_soft_deactivate_users
* maybe_send_batched_emails

It helps to avoid creating unintended savepoints in the future.

This is as a part of our plan to explicitly mark all the
transaction.atomic calls with either 'savepoint=False' or
'durable=True' as required.

* 'savepoint=True' is used in special cases.
2024-11-05 17:58:47 -08:00
Lauryn Menard 3ebc507ddb signups: Move logic for realm admin notification to corporate app.
As we would like to send similar notifications for other billing
state changes (for all BillingSession types), it makes sense to
move the logic for creating and sending these admin realm internal
messages to the BillingSession framework in the corporate app.

In the case that a channel with the specified name does not exist,
we now send direct messages to the admin realm administrators with
the channel, topic and message so that the information is not lost
and so that the channel for these messages can be created.
2024-11-05 08:56:24 -08:00
Alya Abbott eea2499e02 zulip updates: Add update about sidebars and file size limit. 2024-11-04 16:54:58 -08:00
Kislay Udbhav Verma 000cc7bcde markdown: Add support for a pretty syntax for message links.
Links to zulip messages can now be written as
`#**channel_name > topic_name @ message_id**.`
The `message_id` is replaced with `💬` in the rendered
message.

Fixes part of #31920
2024-11-04 13:44:40 -08:00
Kislay Udbhav Verma 65c9b249b7 markdown: Refactor classes handling stream topic links.
The classes StreamPattern and StreamTopicPattern both had a
separate copy of the function `find_stream_id` which did the same
thing. Since another Pattern will be added as a part of #31920, it
is a good idea to move that function into a superclass which is then
inherited by all the related patterns.

Fixes part of #31920
2024-11-04 13:44:40 -08:00
Harsh 858fdeee39 import: Add function to normalize messages to import.
This adds `normalize_body_for_import` to normalize messages from
third-party importers by removing NUL bytes and also updates import
test files data to test this.

Fixes #31930.
2024-11-04 12:22:30 -08:00
Prakhar Pratyush 3d597bb9b0 delete_message_backend: Add `durable=True` to the outermost transaction.
This commit adds 'durable=True' to the outermost transaction
in 'delete_message_backend'.

It also adds 'savepoint=False' to inner transaction.atomic
decorator to avoid creating savepoint.

This is as a part of our plan to explicitly mark all the
transaction.atomic decorators with either 'savepoint=False' or
'durable=True' as required.

* 'savepoint=True' is used in special cases.
2024-11-01 16:41:15 -07:00
Prakhar Pratyush 0fb5657131 transaction: Add durable=True to outermost transaction.atomic decorator.
This commit adds 'durable=True' to the outermost transactions
of the following functions:
* do_create_multiuse_invite_link
* do_revoke_user_invite
* do_revoke_multi_use_invite
* sync_ldap_user_data
* do_reactivate_remote_server
* do_deactivate_remote_server
* bulk_handle_digest_email
* handle_customer_migration_from_server_to_realm
* add_reaction
* remove_reaction
* deactivate_user_group

It helps to avoid creating unintended savepoints in the future.

This is as a part of our plan to explicitly mark all the
transaction.atomic decorators with either 'savepoint=False' or
'durable=True' as required.

* 'savepoint=True' is used in special cases.
2024-11-01 16:41:15 -07:00
Prakhar Pratyush 9371bdb81d invites: Add `durable=True` to transaction in `do_invite_users`.
This commit adds 'durable=True' to the outermost transaction
in 'do_invite_users'.

It also adds 'savepoint=False' to inner transaction.atomic
decorators to avoid creating savepoints.
2024-11-01 16:41:15 -07:00
aniebietafia 257ce8bca2 integrations: Create incoming webhook for Airbyte.
Note about the documentation: There are currently two "Save changes"
buttons on the Airbyte "Notifications" settings page, so the
instructions specify which one to use for clarity.
2024-11-01 10:04:48 -07:00
Aditya Kumar Kasaudhan d1ff871523 webhooks: Support filtering GitHub activity from private repositories.
Currently, the GitHub webhook sends activity from both public and private
repositories, which could lead to unintended disclosure of sensitive
information from private repositories.

This commit introduces a ignore_private_repositories parameter to the
webhook URL. When set to true, the webhook ignore processing activity from
private repositories, ensuring that such activities are not posted to
Zulip streams. By default, if the parameter is omitted or set to false,
activities from both public and private repositories are processed
normally. This provides users with the flexibility to control the
visibility of private repository activities without altering the default
behavior.

More importantly, this introduces a cleaner mechanism for individual
incoming webhooks to declare support for settings not common to all
webhook integrations.

Fixes #31638.
2024-10-31 10:40:28 -07:00
Prakhar Pratyush 555ac613ac create_user: Add 'durable=True' to the outermost transaction.
We need to specify savepoint=False explicitly in the one subroutine
that already specifies a transaction.
2024-10-31 09:55:01 -07:00
Anders Kaseorg 42e1517255 email_notifications: Prevent html2text from mangling Unicode.
html2text mangles Unicode by default, with a --unicode-snob option to
disable it.  If I have to get called a “snob” for wanting to correctly
support non-English languages, then uh, I’ll take one for the team.

https://github.com/Alir3z4/html2text/blob/2024.2.26/html2text/config.py#L111-L150

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-10-30 09:49:23 -07:00
Vector73 1ba1408b01 settings: Remove `edit_topic_policy` setting.
Removed `edit_topic_policy` property, as the permission
to move messages between topcis is now controlled by
`can_move_messages_between_topics_group` setting.
2024-10-29 16:27:04 -07:00
Vector73 ed5638ec3c settings: Add `can_move_messages_between_topics_group` realm setting.
Added `can_move_messages_between_topics_group` realm setting to replace
`edit_topic_policy`.
2024-10-29 16:27:04 -07:00
Mateusz Mandera d708a1bea5 tests: Fix order of mock.patch in use_s3_backend.
This is a somewhat hacky and fragile fix. Due to the order in which
imports seem to happen, the original ordering breaks
RealmImportExportTest: if one of the `use_s3_backend` tests runs before
test_import_realm, the latter will fail while processing thumbnailing,
as S3UploadBackend ends up leaking and
zerver.worker.thumbnail.upload_backend is still set to S3.

By making that mock.patch the first one that gets entered, and thus the
last one to get cleaned up, we fix the leak and upload_backend is set
back to LocalUploadBackend as it should.
2024-10-29 10:43:10 -07:00
sanchi-t c73038edea streams: Restrict access to archived streams. 2024-10-25 16:06:43 -07:00
Sanchit Sharma 795b2ba14e do_deactivate_stream: Remove unnecessary mutations.
Streams should not be marked as private, and subscribers
of the deactivated stream should not be removed.

Update the confirmation message when archiving a stream.
2024-10-25 16:06:42 -07:00
sanchi-t 7e97358c35 home: Update `client_capabilities` for viewing of archived streams. 2024-10-25 16:06:42 -07:00
sanchi-t af7ebde9e4 subscription: Include archived channels in streams list.
`is_archived` field is added to the stream and types.

Include a new `archived_channeels` client capability, to allow clients
to access data on archived channels, without breaking
backwards-compatibility for existing clients that don't know how to
handle these.

Also, included `exclude_archived` parameter to `/get-streams`,
which defaults to `true` as basic clients may not be interested
in archived streams.
2024-10-25 16:06:40 -07:00
Tim Abbott 71d81484ad storage: Simplify system bot avatar logic. 2024-10-25 10:59:57 -07:00