Commit Graph

13252 Commits

Author SHA1 Message Date
Tim Abbott 3810eaaa25 api docs: Document that servers ignore unexpected event types.
Clients like the mobile and terminal apps will want to rely on this
detail of our API, so we should write it down.
2021-04-19 14:14:14 -07:00
Sumanth V Rao f9b79999ed playgrounds: Avoid json encoding each field in POST /realm/playgrounds.
This prevents us from having to json encode every field in the POST
request to /realm/playgrounds, and keeps the client logic simpler
when adding a playground.
2021-04-19 11:23:09 -07:00
Riken Shah 1d9417f9a1 refactor: Rename `dev_panel.html` for more clarity.
This commit renames `dev_panel.html` to `integrations_dev_panel.html`.
2021-04-19 10:25:05 -07:00
Riken Shah bd9e6ae97e refactor: Move dev-only templates to `templates/zerver/development`.
This a prep commit for adding a check to lint to ensure
only development-related templates entrypoint name
starts with `dev`.
2021-04-19 10:25:05 -07:00
Mateusz Mandera ccfcc186ad subs: Fix subscriber_..._history_access to not exclude subbed guests.
Guests are supposed to have stream history access to public streams
they're subscribed to.
2021-04-19 10:10:51 -07:00
Mateusz Mandera 68d1f2d7ef streams: Add realm check in can_access_stream_history.
The caller is supposed validate the stream and user realm match, but
since this is a security-sensitive function, we should have this
defensive code to protect against some validation bugs in the caller
leading to this being called incorrectly and returning True.
2021-04-19 10:10:51 -07:00
Mateusz Mandera f5c4005f8a actions: Fix some lists incorrectly named "subscribers".
These contain subscriptions, not subscribers.
2021-04-19 10:10:51 -07:00
Mateusz Mandera 4e26a9e9d6 subs: Fix codepaths incorrectly fetching subs of deactivated users.
Fixes #17922.

These two places fetch subscriptions for the sake of getting user ids to
send events to. Clearly deactivated users should be excluded from that.
2021-04-19 10:10:51 -07:00
Mateusz Mandera 50bfbb588e subs: Allow filtering by is_user_active in get_active_subscriptions.
get_active_subscriptions_for_stream_id should allow specifying whether
subscriptions of deactivated users should be included in the result.
Active subs of deactivated users are  a subtlety that's easy to miss
when writing relevant code, so we make include_deactivated_users a
mandatory kwarg - this will force callers to definitely give thought to
whether such subs should be included or not.

This commit is just a refactoring, we keep original behavior everywhere
- there are places where subs of deactivates users should probably be
excluded but aren't - we don't fix that here, it'll be addressed in
follow-up commits.
2021-04-19 10:10:51 -07:00
sahil839 4ac3fabadd models: Add new helper can_move_messages_between_streams.
This commit adds new helper can_move_messages_between_streams
which will be used to check whether a user is allowed to move
messages from one stream to another according to value of
'move_messages_between_streams_policy'.
2021-04-16 15:16:08 -07:00
sahil839 2dc99aa90f settings: Add new setting for controlling who can move msgs to stream.
This commit adds a new setting 'move_messages_between_streams_policy`
for controlling who can move messages from one stream to other.
2021-04-16 15:10:39 -07:00
Vishnu KS e72dea1de6 models: Create get_first_human_user function in Realm. 2021-04-16 13:22:01 -07:00
Mateusz Mandera 2983a7e799 test_home: Fix user_activity queue event format in a test.
Current production code uses client_id in the event dict and this test
should be updated to reflect that. Old format event can still be
consumed by the worker, but that is already tested by
WorkerTest.test_UserActivityWorker.
2021-04-16 11:02:48 -07:00
Tim Abbott 48d99886fd event_queue: Update rule for when we can remove compatibility code.
Since c3a8a15bae removed the last
instance of code using the dictionary code path, we actually need to
wait until one can no longer upgrade directly from 4.x to master in
order to avoid breakage should we remove this compatibility code,
since only today did we stop generating the old event format.
2021-04-16 09:58:21 -07:00
Tim Abbott 260861426c queue_processors: Document when can remove compatibility code. 2021-04-16 09:55:14 -07:00
Mateusz Mandera c3a8a15bae delete_messages: Pass a list of user ids in the event in all cases.
The bulk deletion codepath was using dicts instead of user ids in the
event, as opposed to the other codepath which was adjusted to pass just
user ids before. We make the bulk codepath consistent with the other
one. Due to the dict-type events happening in 3.*, we move the goal for
deleting the compat code in process_notification to 5.0.
2021-04-16 09:54:14 -07:00
Anders Kaseorg f59f2ca165 requirements: Re-drop direct dependency on mock.
This was dropped in commit 840cf4b885
(#15091), but commit 1432067959
(#17047) mistakenly reintroduced it.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-04-15 21:47:33 -07:00
Anders Kaseorg bdb20a8002 integrations: Convert deprecated Django url to path.
django.conf.urls.url is actually a deprecated alias of
django.urls.re_path, but we want path instead of re_path.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-04-15 18:01:34 -07:00
Anders Kaseorg 2939d29b6d python: Convert deprecated Django smart_text alias to smart_str.
django.utils.encoding.smart_text is a deprecated alias of
django.utils.encoding.smart_str as of Django 3.0, and will be removed
in Django 4.0.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-04-15 18:01:34 -07:00
Anders Kaseorg dcdb00a5e6 python: Convert deprecated Django is_safe_url.
django.utils.http.is_safe_url is a deprecated alias of
django.utils.http.url_has_allowed_host_and_scheme as of Django 3.0,
and will be removed in Django 4.0.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-04-15 18:01:34 -07:00
Anders Kaseorg e7ed907cf6 python: Convert deprecated Django ugettext alias to gettext.
django.utils.translation.ugettext is a deprecated alias of
django.utils.translation.gettext as of Django 3.0, and will be removed
in Django 4.0.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-04-15 18:01:34 -07:00
Adam Birds 545cd961f4 integrations: Add docs for GitHub Actions integration.
I have added a documentation page for the GitHub Actions integration to
`/integrations/doc/github-actions` with a link to the Zulip GitHub
Actions repository.

Tweaked by tabbott to add cross-links with the main GitHub integration.
2021-04-15 16:42:31 -07:00
Tim Abbott 208721b3d7 left sidebar: Fix misleading "search streams" label.
This widget only filters the user's subscription -- it's only suggest
public streams that the user is not subscribed to.  "Filter" is the
correct label for a widget with this use case.
2021-04-15 15:36:05 -07:00
Mateusz Mandera b4542cc059 message_edit: Verify the message is in a stream in move message API.
This wasn't being validated before. There wasn't any possibility to
actually succeed in moving a private message, because the codepath would
fail at assert message.is_stream_message() in do_update_message - but we
should have proper error handling for that case instead of internal
server errors.
2021-04-14 12:37:34 -07:00
Mateusz Mandera 0c0e83eaff message_edit: Verify user has access to old stream when moving message.
Otherwise an admin can move a topic from a private stream they're no
longer a part of - including the newest messages in the topic, that
they're not supposed to have access to.
2021-04-14 12:37:34 -07:00
Tim Abbott 9d852870ee streams: Delete risky helper get_stream_by_id. 2021-04-14 12:37:34 -07:00
Tim Abbott 7856098bdb archive: Use access_web_public_stream helper.
This is just a refactor to delete potentially security-sensitive
duplicated code.
2021-04-14 12:37:34 -07:00
Mateusz Mandera 3ba8348c51 CVE-2021-30487: Prevent admins from moving topics to disallowed streams.
A bug in the implementation of the topic moving API resulted in
organization administrators being able to move messages to streams they
shouldn't be allowed to - private streams they weren't subscribed to and
streams in other organization hosted by the same Zulip installation.

In our current model realm admins can't send messages to private streams
they're not subscribed to - and being able move messages to a
stream effectively allows to send messages to that stream and thus the
two need to be consistent.
2021-04-14 12:37:34 -07:00
Mateusz Mandera 140655d69e CVE-2021-30479: Prevent guest user access to all_public_streams API.
A bug in the implementation of the all_public_streams API feature
resulted in guest users being able to receive message traffic to public
streams that should have been only accessible to members of the
organization.
2021-04-14 12:37:34 -07:00
Mateusz Mandera 6e11754642 CVE-2021-30478: Prevent API super users from forging messages to other organizations.
A bug in the implementation of the can_forge_sender permission
(previously is_api_super_user) resulted in users with this permission
being able to send messages appearing as if sent by a system bots,
including to other organizations hosted by the same Zulip installation.

- The send message API had a bug allowing an api super user to
  use forging to send messages to other realms' streams, as a
  cross-realm bot. We fix this most directly by eliminating the
  realm_str parameter - it is not necessary for any valid current use
  case. The email gateway doesn't use this API despite the comment in
  that block suggesting otherwise.
- The conditionals inside access_stream_for_send_message are changed up
  to improve security. They were generally not ordered very well,
  allowing the function to successfully return due to very weak
  acceptance conditions - skipping the higher importance checks that
  should lead to raising an error.
- The query count in test_subs is decreased because
  access_stream_for_send_message returns earlier when doing its check
  for a cross-realm bot sender - some subscription checking queries are
  skipped.
- A linkifier test in test_message_dict needs to be changed. It didn't
  make much sense in the first place, because it was creating a message
  by a normal user, to a stream outside of the user's realm. That
  shouldn't even be allowed.
2021-04-14 12:37:34 -07:00
Mateusz Mandera 4235be759d CVE-2021-30477: Prevent outgoing webhook bots from sending arbitrary messages to any stream.
A bug in the implementation of replies to messages sent by outgoing
webhooks to private streams meant that an outgoing webhook bot could be
used to send messages to private streams that the user was not intended
to be able to send messages to.

Completely skipping stream access check in check_message whenever the
sender is an outgoing webhook bot is insecure, as it might allow someone
with access to the bot's API key to send arbitrary messages to all
streams in the organization. The check is only meant to be bypassed in
send_response_message, where the stream message is only being sent
because someone mentioned the bot in that stream (and thus the bot
posting there is the desired outcome). We get much better control over
what's going by passing an explicit argument to check_message when
skipping the access check is desirable.
2021-04-14 12:37:34 -07:00
Aman Agrawal 802c450b3f realm: Add setting to configure GIPHY rating.
Organization admins can use this setting to restrict the maximum
rating of GIFs that will be retrieved from GIPHY. Also, there
is option to disable GIPHY too.
2021-04-14 10:29:39 -07:00
Tim Abbott c30a4c5466 settings: Standardize on "video call provider" (not "video chat").
We had a mix of the two names, and "video call provider" both feels
more professional and more clear about precisely what it does.

We don't change the API fields, since it doesn't seem worth an API
migration.
2021-04-14 08:48:21 -07:00
m-e-l-u-h-a-n dd308528c2 docs(integrations): Document zoom video provider in /integrations.
Moves documentation about using zoom as video call provider
to /integrations. This documentation was earlier present
at /help/start-a-call and is moved as asked in issue #17588.
2021-04-14 08:44:00 -07:00
m-e-l-u-h-a-n 4077673da7 docs(integrations): Add Big Blue Button video provider on /integrations.
Moves documentation about using Big Blue Button as video call
provider to /integrations. This documentation was earlier
present at /help/start-a-call and is moved as asked in issue #17588.
2021-04-14 08:44:00 -07:00
m-e-l-u-h-a-n 13e43917db docs(integrations): Document jitsi video provider in /integrations.
Moves documentation about using jitsi as video call provider
to /integrations. This documentation was earlier present
at /help/start-a-call and is moved as asked in issue #17588.
2021-04-14 08:44:00 -07:00
akshatdalton 53f123df72 linkifiers: Extract shared parameter in `zulip.yaml`.
In zulip.yaml, `pattern` and `url_format_string` are extracted
as a shared parameters.

This is done as a preparatory commit for `Add settings UI to edit
linkifiers`.

Related issue: #10830.
2021-04-14 01:37:00 -07:00
Tim Abbott 9f57961e5f stream_subscription: Remove opaque reference to guest role. 2021-04-13 21:49:57 -07:00
Anders Kaseorg 5fed442bb1 templates: Remove context argument from {{#tr}} block helper.
It only had one nontrivial use, and it’s easily replaced using the
builtin {{#with}} block helper.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-04-13 18:59:46 -07:00
100RABHpy 2ead0fa824 openapi: Refactor the way we find uncalled curl test functions.
In "test_curl_examples.py" we find the functions that registered but
never called. To improve readablity, now we have the full
implementation in curl_param_value_generators, rather than inspecting
its fields from another module.
2021-04-13 18:21:38 -07:00
sahil839 e40ebe1a5d test_users: Refactor user role change tests to use a single helper.
Currently, there are separate tests for testing change of one role
to other, precisely 8, with most of them having similar structure
of code. This commit adds a helper function check_user_role_change
which contains all the code for testing and the tests for different
role just use this helper function to avoid duplication of code.

This refactor is helpful considering we would want to add tests
for moderators also, which would contain multiple tests for
testing changing different user roles to moderator and vice versa.

Tweaked by timabbott to make the code more readable by checking for
every user role flag instead of just checking the certain flags and
using conditionals.

Co-authored-by: Tim Abbott
2021-04-13 17:49:36 -07:00
sahil839 d8c58527cc models: Remove unused function can_access_all_realm_members.
This commit removes can_access_all_realm_members function as
it is not used anywhere in code other than tests.

This function was originally added in 4483e33102 and was
only used in digest.py other than the tests, but its use
in diget.py was removed in 735b6cb761 and the function
itself was not removed from models.py.
2021-04-13 17:49:36 -07:00
sahil839 685fbffd91 tests: Refactor check_has_permission_policies to check for all user roles.
We refactor check_has_permission_policies to check for all user roles for
each value of policy. This will help in handle a case where a guest is
allowed to do something but moderator isn't.

We need to do user_profile.refresh_from_db() in validation_func because
the realm object from user_profile is used in has_permission and we need
updated realm instance after changing the policy.

This is a follow-up commit to 9a4c58cb.
2021-04-13 17:48:23 -07:00
Anders Kaseorg d8a36d0702 i18n: Remove i18next.
Fixes #17890.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-04-13 17:41:10 -07:00
Anders Kaseorg 2c3787196d makemessages: Remove custom _plural handling.
Plurals are handled natively by the ICU MessageFormat syntax, so I
think we don’t have to do anything here.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-04-13 17:41:10 -07:00
Anders Kaseorg ef4c07356f makemessages: Adjust regex for new {{#tr}} syntax.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-04-13 17:41:10 -07:00
Anders Kaseorg 810ba7fce1 makemessages: Call ‘formatjs extract’ for JavaScript extraction.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-04-13 17:41:10 -07:00
Anders Kaseorg d0c13736c7 makemessages: Annihilate opposing \\n kludges.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-04-13 17:41:10 -07:00
Abhijeet Prasad Bodas 3947b0c80a linkifiers: Update API to send data using dictionaries.
* This introduces a new event type `realm_linkifiers` and
a new key for the initial data fetch of the same name.
Newer clients will be expected to use these.

* Backwards compatibility is ensured by changing neither
the current event nor the /register key. The data which
these hold is the same as before, but internally, it is
generated by processing the `realm_linkifiers` data.
We send both the old and the new event types to clients
whenever the linkifiers are changed.
Older clients will simply ignore the new event type, and
vice versa.

* The `realm/filters:GET` endpoint (which returns tuples)
is currently used by none of the official Zulip clients.
This commit replaces it with `realm/linkifiers:GET` which
returns data in the new dictionary format.
TODO: Update the `get_realm_filters` method in the API
bindings, to hit this new URL instead of the old one.

* This also updates the webapp frontend to use the newer
events and keys.
2021-04-13 12:16:07 -07:00
Arun Sankar 146b32d63a test users: Add an escape char to a test username.
Changed the name of the test-user cordelia from `Cordelia Lear` to
`Cordelia, Lear's daughter`.

This change will enable us to test users with escape characters in
their names.

I also updated the Node, Puppeteer, Backend tests and Fixtures to
support this change.
2021-04-13 11:42:06 -07:00