Commit Graph

497 Commits

Author SHA1 Message Date
Aditya Kumar Kasaudhan d1ff871523 webhooks: Support filtering GitHub activity from private repositories.
Currently, the GitHub webhook sends activity from both public and private
repositories, which could lead to unintended disclosure of sensitive
information from private repositories.

This commit introduces a ignore_private_repositories parameter to the
webhook URL. When set to true, the webhook ignore processing activity from
private repositories, ensuring that such activities are not posted to
Zulip streams. By default, if the parameter is omitted or set to false,
activities from both public and private repositories are processed
normally. This provides users with the flexibility to control the
visibility of private repository activities without altering the default
behavior.

More importantly, this introduces a cleaner mechanism for individual
incoming webhooks to declare support for settings not common to all
webhook integrations.

Fixes #31638.
2024-10-31 10:40:28 -07:00
Vector73 1ba1408b01 settings: Remove `edit_topic_policy` setting.
Removed `edit_topic_policy` property, as the permission
to move messages between topcis is now controlled by
`can_move_messages_between_topics_group` setting.
2024-10-29 16:27:04 -07:00
sanchi-t af7ebde9e4 subscription: Include archived channels in streams list.
`is_archived` field is added to the stream and types.

Include a new `archived_channeels` client capability, to allow clients
to access data on archived channels, without breaking
backwards-compatibility for existing clients that don't know how to
handle these.

Also, included `exclude_archived` parameter to `/get-streams`,
which defaults to `true` as basic clients may not be interested
in archived streams.
2024-10-25 16:06:40 -07:00
Shubham Padia 1f0906aef7 groups: Remove list specifying group settings with new API format.
All the groups are now following the new API format, making the list
redundant.
2024-10-24 14:42:21 -07:00
Prakhar Pratyush eaee5763d6 realm_export: Add realm_export_consent feature to API.
Fixes part of #31201.
2024-10-18 14:08:20 -07:00
Prakhar Pratyush 3314c89288 realm: Add maximum file size upload restriction.
This commit adds a restriction to the maximum file size
that can be uploaded to a realm based on its plan_type.
2024-10-11 17:16:48 -07:00
Prakhar Pratyush 808acc9e47 events: Migrate plan_type & upload_quota to update_dict event format.
'realm_upload_quota_mib` is updated when `plan_type` changes.

Earlier, we were including 'upload_quota' to update
`realm_upload_quota_mib` in extra_data field of 'realm op: update'
event format when property='plan_type'.

This commit migrate those two parameters to `realm op: update_dict`
event format.

* None of the clients processes these fields, so no compatibility
  code required.
* Renamed `upload_quota` to `upload_quota_mib` as it better aligns
  with our goal to encode units in the client-facing API names.
  Also, it helps to avoid extra code to update 'realm_upload_quota_mib`
  in web client, web client simply aligns with
  'realm["realm_" + key] = value'.
2024-10-11 17:16:48 -07:00
Sahil Batra 0b58820294 user_groups: Do not include deactivated users in anonymous group settings.
This commit updates code to not include deactivated users in the
anonymous group settings data sent to clients, where the setting
value is sent as a dict containing members and subgroups of the
anonymous group.
2024-10-10 11:37:44 -07:00
Sahil Batra 9292ad8186 user_groups: Do not include deactivated users in members list.
This commit updates code to not include deactivated users in
members list in the user groups object sent in "/register"
and "GET /user_groups" response and also in the response
returned by endpoint like "GET /user_groups/{group_id}/members".

The events code is also update to handle this -
- We expect clients to update the members list on receiving
"realm_user/update" event on deactivation. But for guests
who cannot access the user, "user_group/remove_members"
event is sent to update the group members list on deactivation.
- "user_group/add_members" event is sent to all the users on
reactivating the user.
2024-10-10 11:37:44 -07:00
Sahil Batra a7c534b1b8 events: Fix updating subscribers list when deactivating user.
We previously did not update the subscribers list for unsubscribed
and never subscribed streams when a user is deactivated or a
guest user loses access to some user.
2024-10-10 11:37:44 -07:00
Vector73 9e4e85e140 saved_snippets: Add backend for saved snippets.
Part of #31227.
2024-09-24 15:27:58 -07:00
Sahil Batra 5c790aac72 user_groups: Rename allow_deactivated in 'GET /user_groups'.
This commit renames "allow_deactivated" parameter in
"GET /user_groups" endpoint to "include_deactivated_groups", so
that we can have consistent naming here and for client capability
used for deciding whether to send deactivated groups in register
response and how to handle the related events.
2024-09-18 13:41:13 -07:00
Sahil Batra fb63c47ea6 user_groups: Add client capability to handle deactivated groups.
This commit adds a client capability to not receive data about
deactivated groups.
2024-09-18 13:41:13 -07:00
Vector73 0cd51f2fea settings: Remove `delete_own_message_policy` realm property.
Removed `delete_own_message_policy` realm property as the permission
to delete own messages is controlled by `can_delete_own_message_group`
setting.
2024-09-12 09:36:02 -07:00
Sahil Batra 688c5ad0af groups: Allow excluding deactivated groups in 'GET /user_groups' response. 2024-09-11 09:43:56 -07:00
Mateusz Mandera a36f906d1a presence: Add history_limit_days param to the API.
This param allows clients to specify how much presence history they want
to fetch. Previously, the server always returned 14 days of history.
With the recent migration of the presence API to the much more efficient
system relying on incremental fetches via the last_update_id param added
in #29999, we can now afford to provide much more history to clients
that request it - as all that historical data will only be fetched once.

There are three endpoints involved:
- `/register` - this is the main useful endpoint for this, used by API
clients to fetch initial data and register an events queue. Clients can
pass the `presence_history_limit_days` param here.
- `/users/me/presence` - this endpoint is currently used by clients to
update their presence status and fetch incremental data, making the new
functionality not particularly useful here. However, we still add the
new `history_limit_days` param here, in case in the future clients
transition to using this also for the initial presence data fetch.
- `/` - used when opening the webapp. Naturally, params aren't passed
here, so the server just assumes a value from
`settings.PRESENCE_HISTORY_LIMIT_DAYS_FOR_WEB_APP` and returns
information about this default value in page_params.
2024-09-10 13:15:35 -07:00
Sahil Batra 6919448917 settings: Remove create_web_public_stream_policy setting.
This commit removes create_web_public_stream_policy setting
since web-public channel creation permissions are now
handled by group-based setting.

We still pass "realm_create_web_public_stream_policy" in
"/register" response though for older clients with its
value being set depending on the value of group based
setting. If we cannot set its value to an appropriate enum
corresponding to the group setting, then we set it to
"Admins and moderators" considering that server will not
allow the users without permissions to create web-public
channels but the client can make sure that UI is
available to the users who have permission.
2024-08-01 22:49:33 -07:00
Sahil Batra 09102c5780 streams: Check permission to create web-public streams based on group setting. 2024-08-01 22:49:33 -07:00
Kenneth Rodrigues 97f15d8811 events: Migrate to typed_endpoint.
Migrate `event_register.py` and `tornado` to typed_endpoint.
Modified the tests to work with the migrated endpoints.
2024-07-31 17:10:06 -07:00
Prakhar Pratyush f48e87cd3c fetch_initial_state: Avoid doing one db query per announcement stream.
In 'fetch_initial_state_data' we were doing one database query
per announcement stream.

This commit updates the logic to prefetch those streams using
select_related hence avoiding the extra db queries.

Fixes #28909.
2024-07-22 10:21:22 -07:00
Alex Vandiver 556b92810b thumbnail: Advertize the thumbnail formats at client registration. 2024-07-16 13:22:15 -07:00
Anders Kaseorg 0fa5e7f629 ruff: Fix UP035 Import from `collections.abc`, `typing` instead.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-13 22:28:22 -07:00
Anders Kaseorg 531b34cb4c ruff: Fix UP007 Use `X | Y` for type annotations.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-13 22:28:22 -07:00
Anders Kaseorg e08a24e47f ruff: Fix UP006 Use `list` instead of `List` for type annotation.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-07-13 22:28:22 -07:00
Sahil Batra b44d30add7 events: Optimize computing can_create_streams and related fields.
Currently, for computing fields like can_create_public_streams
and can_create_private_steams fields, is_user_in_group is called
to check whether the user is part of the group which has the
permission. This means that there will be one DB query for each
field.

To optimize this, we now first fetch all the groups that the
user is member of, including the anonymous groups which are
used for settings, such that we can then just check whether
the user is part of the group which has the permission meaning
we would need only one query to compute all the fields.

This would be helpful when settings for other similar fields
will also be migrated to groups framework.
2024-06-24 10:00:08 -07:00
Sahil Batra 6164a41864 realm: Remove create_private_stream_policy setting.
This commit removes create_private_stream_policy setting as
we now use new group based setting.

The "/register" response includes realm_create_private_stream_policy
field to return a value representing superset of users who have the
permission to create private channels, as older clients still expect
this field.
2024-06-20 15:48:14 -07:00
Sahil Batra 0bae59fa4b streams: Check permission to create channels based on group setting.
This commit updates code to use new group based setting when
checking permission to create private channels.
2024-06-20 15:48:14 -07:00
Sahil Batra 428d4284f9 users: Update has_permission to accept realm as argument.
This commit helps in using the realm object which has the
prefetched group settings so that we can avoid extra queries
when calculating fields like can_create_public_streams.
2024-06-20 15:48:14 -07:00
Sahil Batra 19373d61e8 events: Do not call "can_create_..." functions twice.
There is no need to call settings_user.can_create_public_streams
and similar functions for private and web-public streams twice,
once to compute the field for a single stream type and one to
compute can_create_streams.

The value for each stream type can be used to compute value of
can_create_streams field.
2024-06-20 15:48:14 -07:00
Sahil Batra e23af20079 realm: Prefetch group settings only for "/register" response.
This commit updates code to prefetch realm group settings like
"can_create_public_channel_group" only when computing settings
for "/register" response by refetching the realm object with
select_related instead of fetching those settings in UserProfile
query.

This change is done because we do not need to prefetch these
settings for every UserProfile object and for most of the cases
where these settings are actually accessed, we can afford extra
query like when checking permission to create streams. But we
cannot afford one query extra for each setting when computing
these settings for "/register" response, so we re-fetch the
realm object with select_related leading to only one extra
query.

The query count changes in tests are -
- Query count increases by 1 when calling fetch_initial_state_data
for computing can_create_public_streams because Realm object from
UserProfile does not have prefetched setting fields.

- Query count increases by one in test_subs where streams are
created which is as expected due to the setting not being prefetched.

- Query count increases by 2 in tests in test_home.py where one
query is to refetch the realm object and one for computing
can_create_public_streams as mentioned above.
2024-06-16 11:55:28 -07:00
Sahil Batra dea62a3fd9 events: Pass realm to fetch_initial_state_data always.
This commit makes passing realm mandatory to fetch_initial_state_data.

This is a prep commit for refetching the realm object with
select_related for group setting fields so that extra queries
can be avoided when computing "/registe" response.
2024-06-16 11:55:28 -07:00
Sahil Batra 222995b1be settings: Remove create_public_stream_policy.
This commit removes create_public_stream_policy setting
since public channel creation permissions are now handled
by group-based setting.

We still pass "realm_create_public_stream_policy" in
"/register" response though for older clients with its
value being set depending on the value of group based
setting. If we cannot set its value to an appropriate
enum corresponding to the group setting, then we set
it to "Members only" considering that server will not
allow the users without permissions to create public
channels but the client can make sure that UI is
available to the users who have permission.
2024-06-10 12:24:45 -07:00
Sahil Batra 9365604c03 streams: Check permission to create streams based on group setting. 2024-06-10 12:24:45 -07:00
Sahil Batra 3bda17949e realm: Add new group setting for who can create public streams. 2024-06-10 12:24:44 -07:00
Mateusz Mandera 512f4d1476 presence: Backend implementation of the last_update_id API.
This builds on top of 016880f54d which
maintains correct .last_update_id for UserPresence objects; now we add
the related API changes to utilize it.
2024-06-06 17:25:54 -07:00
Vector73 d6672c57ff user_status: Rename `get_user_status_dict` function.
Renames `get_user_status_dict` function to `get_all_users_status_dict`
to distinguish it from new `get_user_status` function.
2024-05-22 18:07:22 -07:00
Sahil Batra d1bcac0822 realm: Use enums for accessing setting values.
This commit updates code, majorly in tests, to use
setting values from enums instead of directly using
the constants defined in Realm.

We still have those constants defined Realm as they
are used in a couple of places where the same code
is used for different settings. These will be
handled later.
2024-05-22 17:20:37 -07:00
Prakhar Pratyush 643998fcfa onboarding_steps: Rename 'hotspots' files to 'onboarding_steps'. 2024-05-10 12:30:22 -07:00
Vector73 ac4dde24ae realm: Add an alias `realm_url` and deprecate `realm_uri` in the API.
The naming `uri` is deprecated while `url` should be used in order to
satisfy URL standards. For this reason, four endpoints are affected:

* The response content of three endpoints `/server_settings`,
`/register` and `/realm` that contain a field `realm_uri` is
changed to `realm_url`.

* In one of the common fields for all mobile push notifications payloads,
`realm_url` field is now added as an alias to `realm_uri`.

For backwards compatibility, we keep the field `realm_uri` and add
an alias `realm_url`.

Co-authored-by: Junyao Chen <junyao.chen@socitydao.org>
2024-05-08 17:39:15 -07:00
Vector73 8ab526a25a models: Replace realm.uri with realm.url.
In #23380, we are changing all occurrences of uri with url in order to
follow the latest URL standard. Previous PRs #25038 and #25045 has
replaced the occurences of uri that has no direct relation with realm.

This commit changes just the model property, which has no API
compatibility concerns.
2024-05-08 11:12:43 -07:00
Alex Vandiver 823f732132 narrow: Move python-only implemenation into its own file.
Tornado uses this, and none of the heavy-weight SQLAlchemy code; so it
just adds to the startup time of the Tornado server.
2024-04-16 09:48:11 -07:00
Mateusz Mandera 540d419ef7 events: Fix realm_upload_quota_mib value to be in MiB.
This was bytes until now.
2024-04-15 15:08:56 -07:00
Mateusz Mandera 1ede8da46c events: Change format of authentication_methods realm_update_dict event.
Makes the event in line with state_data.realm_authentication_methods.
2024-03-05 11:48:58 -08:00
Mateusz Mandera da9e4e6e54 backends: Implementation of restricting certain backends by plan.
Only affects zulipchat, by being based on the BILLING_ENABLED setting.

The restricted backends in this commit are
- AzureAD - restricted to Standard plan
- SAML - restricted to Plus plan, although it was already practically
  restricted due to requiring server-side configuration to be done by us

This restriction is placed upon **enabling** a backend - so
organizations that already have a backend enabled, will continue to be
able to use it. This allows us to make exceptions and enable a backend
for an org manually via the shell, and to grandfather organizations into
keeping the backend they have been relying on.
2024-03-05 11:48:58 -08:00
Anders Kaseorg 570f3dd447 python: Reformat with Ruff formatter.
https://docs.astral.sh/ruff/formatter/

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2024-02-29 17:07:16 -08:00
Alex Vandiver c1aadbe52e events: Do nothing in apply_event for restart events.
These signal that the Tornado process restarted, which in itself is
not notable for apply_events.
2024-02-27 10:18:11 -08:00
Prakhar Pratyush fe1a20ebb3 settings: Add realm-level setting 'zulip_update_announcements_stream'.
This commit adds a realm-level setting named
'zulip_update_announcements_stream' that configures the
stream to which zulip updates should be posted.

Fixes part of #28604.
2024-02-25 09:33:00 -08:00
Prakhar Pratyush ee612dafac settings: Rename signup_notifications_stream realm setting.
This commit renames the realm-level setting
'signup_notifications_stream' to 'signup_announcements_stream'.

The new name reflects better what the setting does.
2024-02-21 09:04:23 -08:00
Prakhar Pratyush ab453fbe20 settings: Rename notifications_stream to new_stream_announce..._stream.
This commit renames the realm-level setting 'notifications_stream'
to 'new_stream_announcements_stream'.

The new name reflects better what the setting does.
2024-02-21 09:04:23 -08:00
Prakhar Pratyush 0c83bca81e events: Use 'get_notifications_stream' realm method.
This commit updates the code in 'fetch_initial_state_data'
function to use the 'get_notifications_stream' method
instead of re-writing the logic.
2024-02-21 09:04:23 -08:00