Commit Graph

13230 Commits

Author SHA1 Message Date
Tim Abbott 208721b3d7 left sidebar: Fix misleading "search streams" label.
This widget only filters the user's subscription -- it's only suggest
public streams that the user is not subscribed to.  "Filter" is the
correct label for a widget with this use case.
2021-04-15 15:36:05 -07:00
Mateusz Mandera b4542cc059 message_edit: Verify the message is in a stream in move message API.
This wasn't being validated before. There wasn't any possibility to
actually succeed in moving a private message, because the codepath would
fail at assert message.is_stream_message() in do_update_message - but we
should have proper error handling for that case instead of internal
server errors.
2021-04-14 12:37:34 -07:00
Mateusz Mandera 0c0e83eaff message_edit: Verify user has access to old stream when moving message.
Otherwise an admin can move a topic from a private stream they're no
longer a part of - including the newest messages in the topic, that
they're not supposed to have access to.
2021-04-14 12:37:34 -07:00
Tim Abbott 9d852870ee streams: Delete risky helper get_stream_by_id. 2021-04-14 12:37:34 -07:00
Tim Abbott 7856098bdb archive: Use access_web_public_stream helper.
This is just a refactor to delete potentially security-sensitive
duplicated code.
2021-04-14 12:37:34 -07:00
Mateusz Mandera 3ba8348c51 CVE-2021-30487: Prevent admins from moving topics to disallowed streams.
A bug in the implementation of the topic moving API resulted in
organization administrators being able to move messages to streams they
shouldn't be allowed to - private streams they weren't subscribed to and
streams in other organization hosted by the same Zulip installation.

In our current model realm admins can't send messages to private streams
they're not subscribed to - and being able move messages to a
stream effectively allows to send messages to that stream and thus the
two need to be consistent.
2021-04-14 12:37:34 -07:00
Mateusz Mandera 140655d69e CVE-2021-30479: Prevent guest user access to all_public_streams API.
A bug in the implementation of the all_public_streams API feature
resulted in guest users being able to receive message traffic to public
streams that should have been only accessible to members of the
organization.
2021-04-14 12:37:34 -07:00
Mateusz Mandera 6e11754642 CVE-2021-30478: Prevent API super users from forging messages to other organizations.
A bug in the implementation of the can_forge_sender permission
(previously is_api_super_user) resulted in users with this permission
being able to send messages appearing as if sent by a system bots,
including to other organizations hosted by the same Zulip installation.

- The send message API had a bug allowing an api super user to
  use forging to send messages to other realms' streams, as a
  cross-realm bot. We fix this most directly by eliminating the
  realm_str parameter - it is not necessary for any valid current use
  case. The email gateway doesn't use this API despite the comment in
  that block suggesting otherwise.
- The conditionals inside access_stream_for_send_message are changed up
  to improve security. They were generally not ordered very well,
  allowing the function to successfully return due to very weak
  acceptance conditions - skipping the higher importance checks that
  should lead to raising an error.
- The query count in test_subs is decreased because
  access_stream_for_send_message returns earlier when doing its check
  for a cross-realm bot sender - some subscription checking queries are
  skipped.
- A linkifier test in test_message_dict needs to be changed. It didn't
  make much sense in the first place, because it was creating a message
  by a normal user, to a stream outside of the user's realm. That
  shouldn't even be allowed.
2021-04-14 12:37:34 -07:00
Mateusz Mandera 4235be759d CVE-2021-30477: Prevent outgoing webhook bots from sending arbitrary messages to any stream.
A bug in the implementation of replies to messages sent by outgoing
webhooks to private streams meant that an outgoing webhook bot could be
used to send messages to private streams that the user was not intended
to be able to send messages to.

Completely skipping stream access check in check_message whenever the
sender is an outgoing webhook bot is insecure, as it might allow someone
with access to the bot's API key to send arbitrary messages to all
streams in the organization. The check is only meant to be bypassed in
send_response_message, where the stream message is only being sent
because someone mentioned the bot in that stream (and thus the bot
posting there is the desired outcome). We get much better control over
what's going by passing an explicit argument to check_message when
skipping the access check is desirable.
2021-04-14 12:37:34 -07:00
Aman Agrawal 802c450b3f realm: Add setting to configure GIPHY rating.
Organization admins can use this setting to restrict the maximum
rating of GIFs that will be retrieved from GIPHY. Also, there
is option to disable GIPHY too.
2021-04-14 10:29:39 -07:00
Tim Abbott c30a4c5466 settings: Standardize on "video call provider" (not "video chat").
We had a mix of the two names, and "video call provider" both feels
more professional and more clear about precisely what it does.

We don't change the API fields, since it doesn't seem worth an API
migration.
2021-04-14 08:48:21 -07:00
m-e-l-u-h-a-n dd308528c2 docs(integrations): Document zoom video provider in /integrations.
Moves documentation about using zoom as video call provider
to /integrations. This documentation was earlier present
at /help/start-a-call and is moved as asked in issue #17588.
2021-04-14 08:44:00 -07:00
m-e-l-u-h-a-n 4077673da7 docs(integrations): Add Big Blue Button video provider on /integrations.
Moves documentation about using Big Blue Button as video call
provider to /integrations. This documentation was earlier
present at /help/start-a-call and is moved as asked in issue #17588.
2021-04-14 08:44:00 -07:00
m-e-l-u-h-a-n 13e43917db docs(integrations): Document jitsi video provider in /integrations.
Moves documentation about using jitsi as video call provider
to /integrations. This documentation was earlier present
at /help/start-a-call and is moved as asked in issue #17588.
2021-04-14 08:44:00 -07:00
akshatdalton 53f123df72 linkifiers: Extract shared parameter in `zulip.yaml`.
In zulip.yaml, `pattern` and `url_format_string` are extracted
as a shared parameters.

This is done as a preparatory commit for `Add settings UI to edit
linkifiers`.

Related issue: #10830.
2021-04-14 01:37:00 -07:00
Tim Abbott 9f57961e5f stream_subscription: Remove opaque reference to guest role. 2021-04-13 21:49:57 -07:00
Anders Kaseorg 5fed442bb1 templates: Remove context argument from {{#tr}} block helper.
It only had one nontrivial use, and it’s easily replaced using the
builtin {{#with}} block helper.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-04-13 18:59:46 -07:00
100RABHpy 2ead0fa824 openapi: Refactor the way we find uncalled curl test functions.
In "test_curl_examples.py" we find the functions that registered but
never called. To improve readablity, now we have the full
implementation in curl_param_value_generators, rather than inspecting
its fields from another module.
2021-04-13 18:21:38 -07:00
sahil839 e40ebe1a5d test_users: Refactor user role change tests to use a single helper.
Currently, there are separate tests for testing change of one role
to other, precisely 8, with most of them having similar structure
of code. This commit adds a helper function check_user_role_change
which contains all the code for testing and the tests for different
role just use this helper function to avoid duplication of code.

This refactor is helpful considering we would want to add tests
for moderators also, which would contain multiple tests for
testing changing different user roles to moderator and vice versa.

Tweaked by timabbott to make the code more readable by checking for
every user role flag instead of just checking the certain flags and
using conditionals.

Co-authored-by: Tim Abbott
2021-04-13 17:49:36 -07:00
sahil839 d8c58527cc models: Remove unused function can_access_all_realm_members.
This commit removes can_access_all_realm_members function as
it is not used anywhere in code other than tests.

This function was originally added in 4483e33102 and was
only used in digest.py other than the tests, but its use
in diget.py was removed in 735b6cb761 and the function
itself was not removed from models.py.
2021-04-13 17:49:36 -07:00
sahil839 685fbffd91 tests: Refactor check_has_permission_policies to check for all user roles.
We refactor check_has_permission_policies to check for all user roles for
each value of policy. This will help in handle a case where a guest is
allowed to do something but moderator isn't.

We need to do user_profile.refresh_from_db() in validation_func because
the realm object from user_profile is used in has_permission and we need
updated realm instance after changing the policy.

This is a follow-up commit to 9a4c58cb.
2021-04-13 17:48:23 -07:00
Anders Kaseorg d8a36d0702 i18n: Remove i18next.
Fixes #17890.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-04-13 17:41:10 -07:00
Anders Kaseorg 2c3787196d makemessages: Remove custom _plural handling.
Plurals are handled natively by the ICU MessageFormat syntax, so I
think we don’t have to do anything here.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-04-13 17:41:10 -07:00
Anders Kaseorg ef4c07356f makemessages: Adjust regex for new {{#tr}} syntax.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-04-13 17:41:10 -07:00
Anders Kaseorg 810ba7fce1 makemessages: Call ‘formatjs extract’ for JavaScript extraction.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-04-13 17:41:10 -07:00
Anders Kaseorg d0c13736c7 makemessages: Annihilate opposing \\n kludges.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-04-13 17:41:10 -07:00
Abhijeet Prasad Bodas 3947b0c80a linkifiers: Update API to send data using dictionaries.
* This introduces a new event type `realm_linkifiers` and
a new key for the initial data fetch of the same name.
Newer clients will be expected to use these.

* Backwards compatibility is ensured by changing neither
the current event nor the /register key. The data which
these hold is the same as before, but internally, it is
generated by processing the `realm_linkifiers` data.
We send both the old and the new event types to clients
whenever the linkifiers are changed.
Older clients will simply ignore the new event type, and
vice versa.

* The `realm/filters:GET` endpoint (which returns tuples)
is currently used by none of the official Zulip clients.
This commit replaces it with `realm/linkifiers:GET` which
returns data in the new dictionary format.
TODO: Update the `get_realm_filters` method in the API
bindings, to hit this new URL instead of the old one.

* This also updates the webapp frontend to use the newer
events and keys.
2021-04-13 12:16:07 -07:00
Arun Sankar 146b32d63a test users: Add an escape char to a test username.
Changed the name of the test-user cordelia from `Cordelia Lear` to
`Cordelia, Lear's daughter`.

This change will enable us to test users with escape characters in
their names.

I also updated the Node, Puppeteer, Backend tests and Fixtures to
support this change.
2021-04-13 11:42:06 -07:00
Tim Abbott 4e8a5f4da8 test_signup: Fix test making DNS query to public Internet.
This appears to be the result of our having copied parts of the
previous test without copying its DNS query decorator.
2021-04-13 10:47:35 -07:00
Anders Kaseorg b01d43f339 mypy: Fix strict_equality violations.
puppet/zulip/files/nagios_plugins/zulip_postgresql/check_postgresql_replication_lag:98: error: Non-overlapping equality check (left operand type: "List[List[str]]", right operand type: "Literal[0]")  [comparison-overlap]
zerver/tests/test_realm.py:650: error: Non-overlapping container check (element type: "Dict[str, Any]", container item type: "str")  [comparison-overlap]

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-04-13 09:18:18 -07:00
Tim Abbott 2e928a0853 markdown: Remove logic for creating markdown engines for all realms.
This logic likely never ran due to a combination of bugs.

* Running `maybe_update_markdown_engines` unconditionally meant that
  `if md_engine_key in md_engines` was likely always true.
* Introduced in 65838bb: DEFAULT_MARKDOWN_KEY could never be in
  md_engines, so should we have ever reached that code path, we'd have
  tried to rebuild all markdown engines every time.

And it also wasn't clearly helpful -- because we fetch all linkifiers
for a realm on every request anyway, we don't really save database
queries by doing a bulk fetch on startup, and doing so would likely
result in a material regression to Zulip's overall startup time that
we were creating markdown engines for large numbers of realms in bulk
during process startup.
2021-04-13 09:18:18 -07:00
Tim Abbott c6a50499f7 management: Use better fetch query in edit_linkifiers. 2021-04-13 09:18:18 -07:00
Abhijeet Prasad Bodas 2b9f2cc8ff mute user: Add some comments on message fetch.
These explain why we don't consider user mutes
in message fetching/unread data.
2021-04-13 09:15:49 -07:00
Abhijeet Prasad Bodas 8b098b95bb mute user: Mark as read old messages immediately.
When a user is muted, in the same request,
we mark any existing unreads from that user
as read.

This is done for all types of messages
(PM/huddle/stream) and regardless of whether
the user was mentioned in them.

This will not break the unread count logic
of the web frontend, because that algorithm
decides which messages to mark as read based
only on the pointer location and the whitespace
at the bottom, not on what messages have already
been marked as read.
2021-04-13 09:08:47 -07:00
Abhijeet Prasad Bodas 2f56f8d0ed mute user: Mark as read new messages.
Messages sent by muted users are marked as read
as soon as they are sent (or, more accurately,
while creating the database entries itself), regardless
of type (stream/huddle/PM).

ede73ee4cd, makes it easy to
pass a list to `do_send_messages` containing user-ids for
whom the message should be marked as read.
We add the contents of this list to the set of muter IDs,
and then pass it on to `create_user_messages`.

This benefits from the caching behaviour of `get_muting_users`
and should not cause performance issues long term.

The consequence is that messages sent by muted users will
not contribute to unread counts and notifications.

This commit does not affect the unread messages
(if any) present just before muting, but only handles
subsequent messages. Old unreads will be handled in
further commits.
2021-04-13 09:08:47 -07:00
Abhijeet Prasad Bodas b140c17441 mute user: Cache list of muter IDs.
This commit defines a new function `get_muting_users`
which will return a list of IDs of users who have muted
a given user.
Whenever someone mutes/unmutes  a user, the cache will be
flushed, and subsequently when that user sends a message,
the cache will be populated with the list of people who
have muted them (maybe empty).

This data is a good candidate for caching because-

1. The function will later be called from the message send
codepath, and we try to minimize database queries there.

2. The entries will be pretty tiny.

3. The entries won't churn too much. An average user will
send messages much more frequently than get muted/unmuted,
and the first time penalty of hitting the db and populating
the cache should ideally get amortized by avoiding several
DB lookups on subsequent message sends.

The actual code to call this function will be written in
further commits.
2021-04-13 09:08:47 -07:00
Abhijeet Prasad Bodas 9602aa1467 mute user: Record entries in RealmAuditLog.
This makes it so that RealmAuditLog entries are
created when a user mutes/unmutes someone.

We don't really need to store the time, but we
do so anyways, because the `event_time` field
is currently a non-nullable one in the `RealmAuditLog`
model, and making it nullable would risk allowing
not specifying the time in other more important
code which also creates `RealmAuditLog` entries.

This also fixes an incorrect test of successfully
unmuting with the API. Earlier it did not mock
the time in the `views/muting.py` code to return
`mute_time`.
2021-04-13 09:08:47 -07:00
Anders Kaseorg 1fe29aad42 queue_processors: Simplify unnecessary use of Optional.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-04-13 08:54:26 -07:00
Anders Kaseorg 9a2140d60e storage: Fix get_storage and remove_storage with empty keys list.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-04-13 08:53:03 -07:00
Zeeshan Equbal 2da4443cc5
api: Add max_message_length field to API data.
Commit 4a3ad0d introduced some extra stream-level parameters
to the `realm` object. This commit extends that to add a
max_message_length paramter too in the same server_level.
2021-04-12 16:03:31 -07:00
Tim Abbott 4a3ad0da06 api: Improve encoding of stream/topic max field lengths.
Previously, you had to request the `stream` event type in order to get
the stream-level parameters; this was a bad design in part because the
`subscription` event type has similar data and is preferred by most
clients.

So we move these to the `realm` object.  We also add the maximum topic
length, as an adjacent parameter.

While changing this, we also fix these to better match the names of
similar API parameters.
2021-04-10 10:07:57 -07:00
Tim Abbott cdbcb43706 api: Fix encoding of strings in realm endpoint.
* Don't require strings to be unnecessarily JSON-encoded.
* Use check_capped_string rather than custom code for length checks.
* Update frontend to pass the right parameters.

With a much simplified populate_data_for_request design suggested by
Anders; we only support a handful of data types, all of which are
correctly encoded automatically by jQuery.

Fixes part of #18035.
2021-04-09 16:27:30 -07:00
Abhijeet Prasad Bodas 224a8e0408 mute user: Add tests for realm import.
This covers the code added in 3bfcaa3968.
2021-04-08 23:04:28 -07:00
Abhijeet Prasad Bodas 152508e346 mute user: Reduce two database fetches when unmuting to one.
Previously, when unmuting a user, we used to make
two database fetches - one to verify that the user
is has been muted before, and one while actually
unmuting the user.

This reduces that to one, by passing around the
`MutedUser` object fetched in the first round.

Since the new function returns `Optional[MutedUser]`,
we need to use a hack for events tests, because
mypy does not yet use the type inferred from
`assert foo is not None` in nested functions like lambdas.
See python/mypy@8780d45507.
2021-04-08 23:04:28 -07:00
Abhijeet Prasad Bodas 32ab9872b1 refactor: Use API instead of functions in test_muting_users.py.
Instead of using internal functions for data setup,
we use the API so that these tests are more
end-to-end.

This commit also removes a now unnecessary
`if date_muted is None` check.
2021-04-08 23:04:28 -07:00
Abhijeet Prasad Bodas b500ff39c1 refactor: Respect example user conventions in test_muting_users.py.
This makes it consistent with the rest of the codebase.
Hamlet and Cordelia are usually the main users which do
things in automated tests.
2021-04-08 23:04:28 -07:00
Abhijeet Prasad Bodas e912bee6b6 refactor: Use variables for dates in test_muting_users.py.
This cleans up some code added in 3bfcaa3968.

Also fixes some indentation to be more readable:
- `mock.patch` is in a single line.
- Dictionaries are one field per line.
2021-04-08 23:04:28 -07:00
shanukun e65e39764a refactor: Make acting_user a mandatory kwarg for do_remove_realm_domain. 2021-04-08 17:50:10 -07:00
shanukun dd0a8738a3 refactor: Make acting_user a mandatory kwarg for do_change_notification_settings. 2021-04-08 17:50:10 -07:00
shanukun 0a21476004 refactor: Make acting_user a mandatory kwarg for do_change_default_all_public_streams. 2021-04-08 17:50:10 -07:00