Commit Graph

12338 Commits

Author SHA1 Message Date
Tim Abbott 992c6126a8 docs: Update many references to Casper. 2020-08-30 17:16:02 -07:00
sahil839 fbae1685d6 openapi: Fix examples in response for deactivating user endpoints.
This commit fixes examples in "400" response for deactivating user
endpoints to have msg as "Cannot deactivate the last organization
owner" instead of "Cannot deactivate the last organization
administrator".

We had already removed the restriction on deactivating last admin
and added it for last owner, while adding owner role.
2020-08-30 17:12:57 -07:00
Tim Abbott c81b9cb516 api docs: Expand details on typing notifications API.
This provides proper documentation on exactly what is expected of
clients doing typing notifications.
2020-08-30 16:43:44 -07:00
orientor 148c375e5b events: Add documentation and tests for `typing: stop` event.
The `typing: stop` event did not have any tests in test_events
hence its documentation wasn't added. So add tests and relevant
documentation for the typing stop event. Also edit the documentation
of `typing: start` to include the fact that servers should use
their own timeout incase `stop` event event isn't received.

Fixes #16122.
2020-08-30 16:43:44 -07:00
Alex Vandiver 4d0ad41307 export: Fix emoji mentioned in tests to be correct. 2020-08-30 15:26:50 -07:00
Alex Vandiver b23a55e669 export: Add a --deactivate flag which happens just prior to export.
This removes a manual step during export.
2020-08-30 15:26:50 -07:00
Alex Vandiver 15d7e7a6fd export: Prompt for confirmation if a consent message is involved.
We display the text of the consent message, and then continue with the
export, which will scroll the content off the screen.  Allow the
administrator time to examine the contents of the message, and decide
whether to proceed based on that and the fraction of users that have
responded so far.
2020-08-30 15:26:50 -07:00
Alex Vandiver e3ba79b9df export: Show total number of users in the realm as well.
Knowing 10 users opted in isn't useful unless one knows if the realm
has 10 or 100 total users.
2020-08-30 15:26:50 -07:00
Alex Vandiver ea65b75875 export: Move tarball and directory creation as late as possible.
This stops creating empty tarballs and directories if the option
validation fails for some reason.
2020-08-30 15:26:50 -07:00
Alex Vandiver 1e714d8a0f lint: Catch mock asserts whose leading "assert_" is missing. 2020-08-30 15:24:01 -07:00
Tim Abbott 5548ab8b99 rest: Simplify authentication error handling.
This pure refactor removes a now unnecessarily nested else clause,
helping keep this key piece of code readable.
2020-08-30 14:51:50 -07:00
Aman fd5423a8f9 exceptions: Extract json_unauths into MissingAuthenticationError.
We raise two types of json_unauthorized when
MissingAuthenticationError is raised. Raising the one
with www_authenticate let's the client know that user needs
to be logged in to access the requested content.

Sending `www_authenticate='session'` header with the response
also stops modern web-browsers from showing a login form to the
user and let's the client handle it completely.

Structurally, this moves the handling of common authentication errors
to a single shared middleware exception handler.
2020-08-30 14:51:50 -07:00
Alex Vandiver 81893c9dbb actions: Invalid flag operation is a user error. 2020-08-29 11:38:59 -04:00
orientor 12efa41ed6 api_docs: Add response details to outgoing webhooks documentation.
Improve OpenAPI documentation of /zulip-outgoing-webhook by moving
data and making appropriate additions from its couterpart in the
/outgoing-webhook docs. Then remove the redundant documentation
from the doc and add command to render OpenAPI documetation. Also
add a test to outgoing_webhooks_interface.py to ensure that OpenAPI
documentation is correct.

Fixes #16203.
2020-08-28 16:47:10 -07:00
Anders Kaseorg 468c5b9a58 tests: Make tests pass with zilencer disabled.
This lets the backend tests pass if zilencer has been (manually)
removed from EXTRA_INSTALLED_APPS, by skipping the tests that require
it.  test-backend complains that some URLs are untested in this case:

ERROR: Some URLs are untested!  Here's the list of untested URLs:
   api/v1/users/me/android_gcm_reg_id
   api/v1/users/me/apns_device_token
   team/

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-28 11:34:09 -07:00
Anders Kaseorg 51f993e084 python: Remove unittest.mock.Mock uses from production code.
It’s somewhat expensive to import and confuses mypy.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-28 11:34:09 -07:00
Tim Abbott b31ff487c9 decorator: Avoid accessing mock RemoteZulipServer.
It's never safe to access the mock RemoteZulipServer object; this
caused exceptions on every request in production for any server with
ZILENCER_ENABLED=False.
2020-08-27 12:53:29 -07:00
Mateusz Mandera d247db37a5 rate_limit: Handle the case of request.user being a RemoteZulipServer.
For now we can just skip rate limiting for this case and rate limit by
the server uuid or simply by IP in a follow-up.
2020-08-27 11:40:35 -07:00
Mohit Gupta 3390a70bcd tests: Add extra console output detection in test-backend output.
This commit adds automatic detection of extra output (other than
printed by testing library or tools) in stderr and stdout by code under
test test-backend when it is run with flag --ban-console-output.
It also prints the test that produced the extra console output.

Fixes: #1587.
2020-08-27 11:39:53 -07:00
Tim Abbott c6a2f121ca test_openapi: Remove now-unnecessary regex hack.
This hack became dead code in
4f0f734810, causing coverage to fail.
2020-08-26 16:21:06 -07:00
Tim Abbott 4f0f734810 urls: Simplify URL patterns for presence.
Extracting a section for presence endpoints and using path() rather
than re_path() results in a much cleaner implementation of this
concept.

This eliminates the last case where test_openapi couldn't correctly
match an endpoint documentation with the OpenAPI definitions for it.
2020-08-26 15:55:55 -07:00
Kartik Srivastava 0b77525814 api docs: Document POST /user_groups/{group_id}/members endpoint. 2020-08-26 15:40:19 -07:00
Kartik Srivastava 7677ba2d2b api docs: Rename 'group_id' to 'user_group_id'.
This renames 'group_id' to 'user_group_id' in the api docs to remove
the naming mismatch between the url config and the docs and eventually
remove the 'user_groups' endpoints from 'pending_endpoints' in
test_openapi.py.
2020-08-26 15:40:19 -07:00
Kartik Srivastava b9b9a79382 user_groups: Use 'path_only=True' in REQ for 'user_group_id' in views.
'user_groups' endpoints are currently under 'pending_endpoints' in
test_openapi.py (even after being documented except one), due to the
'user_group_id' and 'group_id' parameter name mismatch in the
url config and the view functions.

This commit includes 'path_only=True' for 'user_group_id' parameter in
views to avoid the failure of 'test_openapi_arguments', in
test_openapi.py, which excludes the path parameters. This is a prep
commit for renaming 'group_id' to 'user_group_id' in the documentation
and removing the 'user_groups' endpoints from 'pending_endpoints'.
2020-08-26 15:40:19 -07:00
Tim Abbott 2095ed6ae7 api docs: Edit custom profile field descriptions. 2020-08-26 15:35:10 -07:00
Kartik Srivastava 190701a062 api docs: Document POST /realm/profile_fields api endpoint. 2020-08-26 12:48:16 -07:00
Kartik Srivastava b74bf64c9d api docs: Document PATCH /realm/profile_fields endpoint. 2020-08-26 12:48:16 -07:00
Kartik Srivastava f8d6b9755a api docs: Document GET /realm/profile_fields endpoint. 2020-08-26 12:48:16 -07:00
Kartik Srivastava 94a29f5870 openapi: Add OpenAPI data for /realm/profile_fields api endpoint. 2020-08-26 12:48:16 -07:00
Mateusz Mandera 06151672ba
queue: Use locking to avoid race conditions in missedmessage_emails.
This queue had a race condition with creation of another Timer while
maybe_send_batched_emails is still doing its work, which may cause
two or more threads to be running maybe_send_batched_emails
at the same time, mutating the shared data simultaneously.

Another less likely potential race condition was that
maybe_send_batched_emails after sending out its email, can call
ensure_timer(). If the consume function is run simultaneously
in the main thread, it will call ensure_timer() too, which,
given unfortunate timings, might lead to both calls setting a new Timer.

We add locking to the queue to avoid such race conditions.

Tested manually, by print debugging with the following setup:
1. Making handle_missedmessage_emails sleep 2 seconds for each email,
   and changed BATCH_DURATION to 1s to make the queue start working
   right after launching.
2. Putting a bunch of events in the queue.
3. ./manage.py process_queue --queue_name missedmessage_emails
4. Once maybe_send_batched_emails is called and while it's processing
the events, I pushed more events to the queue. That triggers the
consume() function and ensure_timer().

Before implementing the locking mechanism, this causes two threads
to run maybe_send_batched_emails at the same time, mutating each other's
shared data, causing a traceback such as

Exception in thread Thread-3:
Traceback (most recent call last):
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.6/threading.py", line 1182, in run
    self.function(*self.args, **self.kwargs)
  File "/srv/zulip/zerver/worker/queue_processors.py", line 507, in maybe_send_batched_emails
    del self.events_by_recipient[user_profile_id]
KeyError: '5'

With the locking mechanism, things get handled as expected, and
ensure_timer() exits if it can't obtain the lock due to
maybe_send_batched_emails still working.

Co-authored-by: Tim Abbott <tabbott@zulip.com>
2020-08-26 12:40:59 -07:00
Tim Abbott 1fddf16b73 Revert "exceptions: Extract json_unauths into MissingAuthenticationError."
This reverts commit c355f6b8d8.
2020-08-25 17:42:07 -07:00
Tim Abbott 5dceee5bd8 Revert "exceptions: Remove dead code msg_format method."
This reverts commit d6ab7ea8ff.
2020-08-25 17:42:07 -07:00
Tim Abbott fd3a43b51e Revert "rest: Simplify authentication error handling."
This reverts commit 1f90a31fa3.
2020-08-25 17:42:07 -07:00
Tim Abbott d6ab7ea8ff exceptions: Remove dead code msg_format method. 2020-08-25 17:17:15 -07:00
Tim Abbott 3b2a262b6f topic: Reorder topic history functions. 2020-08-25 17:03:48 -07:00
Tim Abbott 88a28d5470 topic: Refactor get_topic_history_for_stream.
This now uses get_topic_history_for_public_stream as a subroutine, to
avoid duplicating that large section of SQL.
2020-08-25 17:03:13 -07:00
Aman c3a8492697 topic: Rename get_topic_history_for_web_public_stream. 2020-08-25 17:01:12 -07:00
Tim Abbott 1f90a31fa3 rest: Simplify authentication error handling.
This pure refactor removes a now unnecessarily nested else clause,
helping keep this key piece of code readable.
2020-08-25 16:59:05 -07:00
Aman c355f6b8d8 exceptions: Extract json_unauths into MissingAuthenticationError.
We raise two types of json_unauthorized when
MissingAuthenticationError is raised. Raising the one
with www_authenticate let's the client know that user needs
to be logged in to access the requested content.

Sending `www_authenticate='session'` header with the response
also stops modern web-browsers from showing a login form to the
user and let's the client handle it completely.

Structurally, this moves the handling of common authentication errors
to a single shared middleware exception handler.
2020-08-25 16:52:21 -07:00
Hemanth V. Alluri 31a5e0b441 openapi: Fix DELETE /users/me status 200 response description.
If you look at line number 1121 (new) of commit 14c0a387cf,
I seem to have accidently set the description for a status
200 response to "Bad Request" instead of "Success" which
is what it really is. It's basically an ugly typo (maybe
due to hastily copy-pasting the template).

Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
2020-08-24 16:30:38 -07:00
Mateusz Mandera 4e8b087c82 rate_limiter: Fix flakiness of tests in test_external.
I noticed RateLimitTests.test_hit_ratelimits fails when run as an
individual test, but never when run after other tests. That's due to the
first API request in a run of tests taking a long time, as detailed in
the comment on the change to the setUp method.
2020-08-24 16:22:04 -07:00
Mateusz Mandera 934bdb9651 rate_limit: Improve dummy request objects in RateLimitTestCase.
Django always sets request.user to a UserProfile or AnonymousUser
instance, so it's better to mimic that in the tests where we pass a
dummy request objects for rate limiter testing purposes.
2020-08-24 16:22:04 -07:00
Mateusz Mandera 699c4e8549 rate_limit: Remove inaccurate comment in rate_limit decorator.
The data is now stored in memory if things are happening inside tornado.
That aside, there is no reason for a comment on a rate_limit_user call
to talk about low level implementation details of that function.
2020-08-24 16:22:04 -07:00
Mateusz Mandera c00aab8ede rate_limit: Delete code handling impossible cases with request.user.
I can find no evidence of it being possible to get an Exception when
accessing request.user or for it to be falsy. Django should always set
request.user to either a UserProfile (if logged in) or AnonymousUser
instance. Thus, this seems to be dead code that's handling cases that
can't happen.
2020-08-24 16:22:04 -07:00
Alex Vandiver b4c2ae9cae settings: Adjust documentation and comment references to settings.py.
`zproject/settings.py` itself is mostly-empty now.  Adjust the
references which should now point to `zproject/computed_settings.py`
or `zproject/default_settings.py`.
2020-08-24 13:13:16 -07:00
orientor 372e010dbb events: Add `op` field to `update_message_flags` events.
`update_message_flags` events used `operation` instead of `op`, the
latter being the standard field used in other events. So add `op`
field to `update_message_flags` and mark `operation` as deprecated,
so that it can be removed later.
2020-08-24 12:42:03 -07:00
Steve Howell f429df3401 github tests: Extract verify_post_is_ignored. 2020-08-24 12:34:46 -07:00
Steve Howell dfe6960584 minor: Format IGNORED_EVENTS for github. 2020-08-24 12:34:46 -07:00
Steve Howell 0863874050 jira: Ignore worklog_updated event. 2020-08-24 12:34:46 -07:00
Steve Howell 7b344044b4 jira: Ignore version_released event. 2020-08-24 12:34:46 -07:00