Commit Graph

13594 Commits

Author SHA1 Message Date
akshatdalton 3cc6ff57db test_link_embed: Use responses module to mock HTTP responses. 2021-06-09 12:17:26 -07:00
akshatdalton bef7120390 test_slack_importer: Use responses module to mock HTTP responses. 2021-06-09 12:17:26 -07:00
akshatdalton 1259f5a70e minor: Properly unpack `kwargs`. 2021-06-09 12:17:26 -07:00
Alex Vandiver 721546dfc0 subdomains: Extend "static" to include resources hosted on S3.
This causes avatars and emoji which are hosted by Zulip in S3 (or
compatible) servers to no longer go through camo.  Routing these
requests through camo does not add any privacy benefit (as the request
logs there go to the Zulip admins regardless), and may break emoji
imported from Slack before 1bf385e35f,
which have `application/octet-stream` as their stored Content-Type.
2021-06-08 15:28:10 -07:00
Abhijeet Prasad Bodas 58da384da3 muting: Don't enqueue notifications for messages from muted senders.
Earlier, the notification-blocking for messages from muted senders
was a side-effect of we never sending notifications for messages
with the "read" flag.

This commit decouples these two things, as a prep for having new
settings which will allow users to **always** receive email
notifications, including when/if they read the message during the
time the notifications is in the queue.

We still mark muted-sender messages as "read" when they are sent,
because that's desirable anyways.
2021-06-08 14:58:14 -07:00
Abhijeet Prasad Bodas 006b92ed6d message send: Plumb sender muters through `get_recipient_info`.
This will make it possible to share this code with the message
update codepath.
2021-06-08 14:58:14 -07:00
Mateusz Mandera c54b48452d saml: Support syncing custom profile fields with SAML attributes.
Fixes #17277.

The main limitation of this implementation is that the sync happens if
the user authing already exists. This means that a new user going
through the sign up flow will not have their custom fields synced upon
finishing it. The fields will get synced on their consecutive log in via
SAML in the future. This can be addressed in the future by moving the
syncing code further down the codepaths to login_or_register_remote_user
and plumbing the data through to the user creation process.

We detail that limitation in the documentation.
2021-06-08 14:47:33 -07:00
Mateusz Mandera 00c7ac15df saml: Define a TypedDict for the type of IdP settings dict.
The old type in default_settings wasn't right - limit_to_subdomains is a
List[str]. We define a TypeDict for capturing the typing of the settings
dict more correctly and to allow future addition of configurable
attributes of other non-str types.
2021-06-08 14:30:21 -07:00
Abhijeet Prasad Bodas 30c993451d test_message_edit_notifications: Cleanup irrelevant parameters.
This is in the same spirit as the previous commit.
2021-06-08 11:10:18 -07:00
Abhijeet Prasad Bodas 2bbdd42e1a test_event_queue: Fill-up default data in missedmessage_hook tests.
This allows us to skip sending parameters which are irrelevant
to what we are testing, and only send the specific changed data.
2021-06-08 11:10:18 -07:00
Abhijeet Prasad Bodas d9395e7b52 test_event_queue: Introduce helper to fill-up default values.
This allows us to only mention the values that are relevant
to the behavior being tested by the `check_notify` function
in the current assertion.
2021-06-08 11:10:18 -07:00
Abhijeet Prasad Bodas 8eec7b4718 test_event_queue: Extract common data setup in `check_will_notify`.
The `message_id` and `user_profile_id` values don't really matter for
our testing here, so we might as well set these dummy values in the
main function.
2021-06-08 11:10:18 -07:00
Anders Kaseorg ad7466a2bf dependencies: Upgrade JavaScript dependencies.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-06-08 10:15:52 -07:00
Tim Abbott 3eee7c067c test_example: Add discussion of other common pitfalls.
The new test_example file is great; here I document more pitfalls that
I've seen very often with tests contributed by folks new to the
project.
2021-06-07 16:02:12 -07:00
Steve Howell 47da28429b tests: Add test_example.py.
The idea here is to quickly learn basic testing
idioms with code that's executed (unlike docs
which are prone to bitrot).
2021-06-07 15:29:49 -07:00
Abhijeet Prasad Bodas b7fcb0275c cache: Use `id`s instead of `UserProfile`s for get_muting_users.
This will make it easier to call this function in the message
send codepath.
2021-06-07 13:41:37 -07:00
Tim Abbott d84d61f718 mypy: Fix user_id typo in maybe_enqueue_notifications_for_message_update.
This also lets us remove importing UserProfile at all in this module.
2021-06-07 13:31:54 -07:00
Abhijeet Prasad Bodas 5f5323be7d event_queue: Pass `bool`s to `maybe_enqueue_for_message_update`.
This simplifies the logic in the function itself and makes this function
consistent with how we pass data to `maybe_enqueue_notifications`.
2021-06-07 13:29:37 -07:00
Abhijeet Prasad Bodas 22d19375f2 event_queue: Reorder `maybe_enqueue_for_update` parameters.
This will make `maybe_enqueue_notifications_for_message_update`'s
arguments more consistent with `maybe_enqueue_notifications`.
2021-06-07 19:54:04 +05:30
Abhijeet Prasad Bodas 99b181629d actions: Create dataclass for message read events.
This allows us to deduplicate the common fields.
2021-06-07 19:51:45 +05:30
Abhijeet Prasad Bodas e33c4105b4 actions: Use `custom_profile_field` as variable name, not `field`.
This is a prep change for importing (and using) `dataclasses.field`
elsewhere in the same file, because pyflakes would throw "Import
module shodowed" errors otherwise.
2021-06-07 19:50:36 +05:30
Gaurav Pandey 9b696cf212 api: Expose event_queue_longpoll_timeout_seconds in /register.
Rename poll_timeout to event_queue_longpoll_timeout_seconds
and change its value from 90000 ms to 90 sec. Expose its
value in register api response when realm data is fetched.
Bump API_FEATURE_LEVEL to 74.
2021-06-05 07:37:19 -07:00
Gaurav Pandey 18ef0c3bc1 api: Expose server_needs_upgrade in register response.
Expose the boolean value server_needs_upgrade in the
responses for register api so that it can be used
by mobile and terminal clients as well.
Highlighted in api changelog as part of
feature level 74 in commit fb93c96
(next commit).
2021-06-05 07:33:25 -07:00
Gaurav Pandey f82aba5a3d compatibility: Shift functions to new module.
Shift functions used for compatibility from
zerver.lib.home (is_outdated_server) and
zerver.view.compatibility (pop_numerals,
version_lt, find_mobile_os,
is_outdated_desktop_app, is_unsupported_browser)
to zerver.lib.compatibility module.
2021-06-05 07:33:25 -07:00
Suyash Vardhan Mathur 44c25619d7 curl_examples: Add testing for template endpoints.
This commit adds support for testing of
those endpoints whose .md files would
be deleted in favour of their pages
to be generated automatically by the template.

curl examples for such endpoints must exist
in accordance to the pattern of template, which
can be used to run the tests for them.
2021-06-04 09:43:24 -07:00
Suyash Vardhan Mathur 93cbe6f175 curl_examples: Refactor testing code.
The list curl_commands_to_test gets filled
already and so, the code to test each command
in the list can be moved out of the block that
opens the file. The only change in this commit
is reducing an indentation for the entire block.

This has been done to reuse the whole block
in case the file does not exist.
2021-06-04 09:43:24 -07:00
Suyash Vardhan Mathur 2359ddd402 curl_examples: Refactor testing code.
This commit refactors the code to find the
lines to be first, and then test all the
lines that contained commands.

This was done to avoid duplication of
the code for the other case, where the
.md file won't exist, as those are soon
to be deleted in favour of a common template.
2021-06-04 09:43:24 -07:00
Suyash Vardhan Mathur 64df949f7a curl_examples: Use endpoint list from rest-endpoints.
Soon, the .md files of each endpoint would
be removed and be auto-generated from OpenAPI
data.
So, instead of using the files directly,
we should check from the list of endpoints and
open the files from there.

A follow-up to change logic when the .md files
get deleted should be done.
2021-06-04 09:43:24 -07:00
Abhijeet Prasad Bodas 86d6872a80 onboarding: Select Message FOR UPDATE when adding reaction.
Although race conditions are super-unlikely for these onboarding
messages, it's better to write correct code since it might be copied.
2021-06-04 08:18:17 -07:00
Abhijeet Prasad Bodas 90b6fa7881 reactions: Select Message FOR UPDATE when adding/removing.
This locks the message row while a reaction is being added/removed,
which will handle race conditions caused by deleting the message
at the same time.

We make sure that events work happens outside the transaction,
so that in case there's some problem with the queue processor, the
locks aren't held for too long.

As a nice side-effect, we also handle race conditions from double
adding reactions, because once the message is locked, a duplicate
request will wait till the earlier transaction commits, and hence
will not throw `IntegrityErrors`s (rather, will be handled in our
safety check in the /views code itself), which earlier had to be
handled explicitly.
2021-06-04 08:18:17 -07:00
Abhijeet Prasad Bodas efc2f49e3c submessage: Select Message FOR UPDATE when creating.
This locks the message while creating a submessage, which
will handle race conditions caused by deleting the message
simultaneously.

We make sure that events work happens outside the transaction,
so that in case there's a problem with the queue processor,
the locks aren't held for too long.
2021-06-04 08:18:17 -07:00
Abhijeet Prasad Bodas 5f4113cf60 message delete: Select Message FOR UPDATE when archiving.
Further commits will start locking the message rows while
adding related fields like reactions or submessages,
to handle races caused by deleting the message itself at the
same time.

The message locking implemented then will create a possibility
of deadlocks, where the related field transaction holds a lock
on the message row, and the message-delete transaction holds a
lock on the database row of the related field (which will also
need to be deleted when the message is deleted), and both
transactions wait for each other.

To prevent such a deadlock, we lock the message itself while
it is being deleted, so that the message-delete transaction
will have to wait till the other transaction (which is about
to delete the related field, and also holds a lock on the
message row) commits.

https://chat.zulip.org/#narrow/near/1185943 has more details.
2021-06-04 08:18:17 -07:00
Abhijeet Prasad Bodas 1a9f385e17 access_message: Allow selecting message row FOR UPDATE.
This is a prep change to start using `SELECT FOR UPDATE` queries
when there is a chance of race conditions.
2021-06-04 08:18:16 -07:00
Abhijeet Prasad Bodas 51f5bbcd57 test_events: Run on_commit send_event hooks immediately.
Further commits will hook some `send_event` calls to `on_commit`.
With those changes, these will never be executed in tests, because
transactions never get commited with `TestCase`, which the
`ZulipTestCase` is a subclass of.

We want to make sure that these events are actually sent for testing
purposes, hence this change.

There's no need to actually capture the callbacks, because the
events are already thoroughly tested.
2021-06-04 18:12:39 +05:30
Abhijeet Prasad Bodas 60464a4cea tests: Run on_commit hooks immediately in tornado_redirected_to_list.
Further commits will hook `send_event` calls to `on_commit`
in some cases. This change will make it easier to test such
situations.

We don't need to actually capture the callbacks, because the
events sent are already tested via the list in which they are
captured by `tornado_redirected_to_list`.
2021-06-04 18:12:39 +05:30
Abhijeet Prasad Bodas 8b14704d35 test_classes: Fix misplaced `tornado_redirected_to_list` comment. 2021-06-04 18:12:21 +05:30
Tim Abbott b920929182 lint: Fix line-wrapping for a recently merged test change. 2021-06-03 18:52:39 -07:00
sahil839 bf9c17e8a8 message: Fix moving messages between streams for non-admins.
This commit fixes a bug where moving messages between streams was
not allowed for non-admins when allow_community_topic_editing was
set to false and move_messages_between_streams_policy was set to
Realm.POLICY_MEMBERS_ONLY.

The bug is fixed by calling can_edit_content_or_topic only when
topic or content edit is there and not in the case where only
message is moved from one stream to another.
2021-06-03 17:53:59 -07:00
sahil839 db294cb386 message: Extract logic to check message edit permissions in a function.
This commit extracts the logic of checking the message edit permissions,
like whether the sender is same as user, whether it is a (no topic)
message or whether community topic editing is allowed, into a separate
function.

This is a prep commit for fixing a bug where permission to move messages
between streams is affected by permission of editing topics.
2021-06-03 17:53:55 -07:00
sahil839 6be232aa42 message_edit: Check 'topic_name is not None' and not 'content is None'.
Previously when enforcing the check to do not allow editing topics
after a certain time, we were checking whether 'content is None' and
considering it as that if content is None then there must be topic
edit.

But after adding support for moving messages between streams it can be
the case that we are neither changing topic nor content and just moving
streams, and the original code raises error if this is done after the
time limit of editing topics, which is wrong.

This commit fixes this by actually checking 'topic_name is not None'.
2021-06-03 17:13:32 -07:00
sahil839 e2835d3f4f message_edit: Modify the error message for topic edit deadline.
This commit modifies the error message shown when topic edit
deadline is passed to make it more clear that the limit is for
editing message's topic.
2021-06-03 17:13:32 -07:00
Suyash Vardhan Mathur 94a22f9323 apidocs: Modify tests to reflect new templates.
Soon, each endpoint won't necessarily have a .md
file, but would generate API doc directly from
OpenAPI data using a template.

So, the lists of endpoints to be tested should not
be taken from the .md files, but from the REST
endpoints available in the sidebar.

This commit also adds a missing test for an invalid
article being accessed in the URL of an API page.
2021-06-03 10:52:22 -07:00
Suyash Vardhan Mathur d1ccf15cd3 apidocs: Fix invalid API page bug.
The current logic to get API pages' title using
OperationID should be used when the first line
of the file explicitly mentions so.

In cases where the files didn't begin with `#` but also
didn't need to get title from OpenAPI summary,
this logic fails and causes Server error.
This particularly happens when the article is invalid,
and the `missing.md` file doesn't need title to be
generated, but doesn't start with `#` either.

This commit fixes the logic of using the generated title and covers the bug.
2021-06-03 10:47:47 -07:00
Vishnu KS 5db53029a5 api: Include is_billing_admin as an attribute in user response.
This is sufficiently useful that it should be made available to clients.
2021-06-03 10:27:07 -07:00
Vishnu KS 3d5ee69b21 stripe: Create make_user_billing_admin helper function. 2021-06-03 10:13:59 -07:00
Wesley Aptekar-Cassels d5ba94082a markdown: Increase max rendered message length to 1MB.
This should help with #17425, where messages with lots of LaTeX are
lost, due to the large expansion factor.

This isn't a total fix for this - large messages with lots of LaTeX
can still end up larger than 1MB, and rendering could timeout, but
this fix should help significantly.

1MB is still small enough that I don't expect we'll run into any DOS
problems - my testing didn't show any problems rendering messages that
contain ~1MB of LaTeX.
2021-06-03 10:10:35 -07:00
seiwailai b584790541 validator: Add generic check_or.
Fixes part of #17914. Added generic check_or function and tests.
2021-06-03 09:49:50 -07:00
Alex Vandiver 4525543ec6 delete_realm: Disallow deletion of realms we've had billing plans with. 2021-06-03 09:38:12 -07:00
akshatdalton 7df62ebbaf settings: Make `MAX_MESSAGE_LENGTH` a server-level setting.
This will offer users who are self-hosting to adjust
this value. Moreover, this will help to reduce the
overall time taken to test `test_markdown.py` (since
this can be now overridden with `override_settings`
Django decorator).

This is done as a prep commit for #18641.
2021-06-03 09:26:28 -07:00
akshatdalton 6143cb6e73 test_markdown: Use assertTrue/assertFalse instead of assertEqual. 2021-06-02 17:20:45 -07:00