Commit Graph

12524 Commits

Author SHA1 Message Date
Anders Kaseorg 468c5b9a58 tests: Make tests pass with zilencer disabled.
This lets the backend tests pass if zilencer has been (manually)
removed from EXTRA_INSTALLED_APPS, by skipping the tests that require
it.  test-backend complains that some URLs are untested in this case:

ERROR: Some URLs are untested!  Here's the list of untested URLs:
   api/v1/users/me/android_gcm_reg_id
   api/v1/users/me/apns_device_token
   team/

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-28 11:34:09 -07:00
Anders Kaseorg 51f993e084 python: Remove unittest.mock.Mock uses from production code.
It’s somewhat expensive to import and confuses mypy.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-28 11:34:09 -07:00
Tim Abbott b31ff487c9 decorator: Avoid accessing mock RemoteZulipServer.
It's never safe to access the mock RemoteZulipServer object; this
caused exceptions on every request in production for any server with
ZILENCER_ENABLED=False.
2020-08-27 12:53:29 -07:00
Mateusz Mandera d247db37a5 rate_limit: Handle the case of request.user being a RemoteZulipServer.
For now we can just skip rate limiting for this case and rate limit by
the server uuid or simply by IP in a follow-up.
2020-08-27 11:40:35 -07:00
Mohit Gupta 3390a70bcd tests: Add extra console output detection in test-backend output.
This commit adds automatic detection of extra output (other than
printed by testing library or tools) in stderr and stdout by code under
test test-backend when it is run with flag --ban-console-output.
It also prints the test that produced the extra console output.

Fixes: #1587.
2020-08-27 11:39:53 -07:00
Tim Abbott c6a2f121ca test_openapi: Remove now-unnecessary regex hack.
This hack became dead code in
4f0f734810, causing coverage to fail.
2020-08-26 16:21:06 -07:00
Tim Abbott 4f0f734810 urls: Simplify URL patterns for presence.
Extracting a section for presence endpoints and using path() rather
than re_path() results in a much cleaner implementation of this
concept.

This eliminates the last case where test_openapi couldn't correctly
match an endpoint documentation with the OpenAPI definitions for it.
2020-08-26 15:55:55 -07:00
Kartik Srivastava 0b77525814 api docs: Document POST /user_groups/{group_id}/members endpoint. 2020-08-26 15:40:19 -07:00
Kartik Srivastava 7677ba2d2b api docs: Rename 'group_id' to 'user_group_id'.
This renames 'group_id' to 'user_group_id' in the api docs to remove
the naming mismatch between the url config and the docs and eventually
remove the 'user_groups' endpoints from 'pending_endpoints' in
test_openapi.py.
2020-08-26 15:40:19 -07:00
Kartik Srivastava b9b9a79382 user_groups: Use 'path_only=True' in REQ for 'user_group_id' in views.
'user_groups' endpoints are currently under 'pending_endpoints' in
test_openapi.py (even after being documented except one), due to the
'user_group_id' and 'group_id' parameter name mismatch in the
url config and the view functions.

This commit includes 'path_only=True' for 'user_group_id' parameter in
views to avoid the failure of 'test_openapi_arguments', in
test_openapi.py, which excludes the path parameters. This is a prep
commit for renaming 'group_id' to 'user_group_id' in the documentation
and removing the 'user_groups' endpoints from 'pending_endpoints'.
2020-08-26 15:40:19 -07:00
Tim Abbott 2095ed6ae7 api docs: Edit custom profile field descriptions. 2020-08-26 15:35:10 -07:00
Kartik Srivastava 190701a062 api docs: Document POST /realm/profile_fields api endpoint. 2020-08-26 12:48:16 -07:00
Kartik Srivastava b74bf64c9d api docs: Document PATCH /realm/profile_fields endpoint. 2020-08-26 12:48:16 -07:00
Kartik Srivastava f8d6b9755a api docs: Document GET /realm/profile_fields endpoint. 2020-08-26 12:48:16 -07:00
Kartik Srivastava 94a29f5870 openapi: Add OpenAPI data for /realm/profile_fields api endpoint. 2020-08-26 12:48:16 -07:00
Mateusz Mandera 06151672ba
queue: Use locking to avoid race conditions in missedmessage_emails.
This queue had a race condition with creation of another Timer while
maybe_send_batched_emails is still doing its work, which may cause
two or more threads to be running maybe_send_batched_emails
at the same time, mutating the shared data simultaneously.

Another less likely potential race condition was that
maybe_send_batched_emails after sending out its email, can call
ensure_timer(). If the consume function is run simultaneously
in the main thread, it will call ensure_timer() too, which,
given unfortunate timings, might lead to both calls setting a new Timer.

We add locking to the queue to avoid such race conditions.

Tested manually, by print debugging with the following setup:
1. Making handle_missedmessage_emails sleep 2 seconds for each email,
   and changed BATCH_DURATION to 1s to make the queue start working
   right after launching.
2. Putting a bunch of events in the queue.
3. ./manage.py process_queue --queue_name missedmessage_emails
4. Once maybe_send_batched_emails is called and while it's processing
the events, I pushed more events to the queue. That triggers the
consume() function and ensure_timer().

Before implementing the locking mechanism, this causes two threads
to run maybe_send_batched_emails at the same time, mutating each other's
shared data, causing a traceback such as

Exception in thread Thread-3:
Traceback (most recent call last):
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.6/threading.py", line 1182, in run
    self.function(*self.args, **self.kwargs)
  File "/srv/zulip/zerver/worker/queue_processors.py", line 507, in maybe_send_batched_emails
    del self.events_by_recipient[user_profile_id]
KeyError: '5'

With the locking mechanism, things get handled as expected, and
ensure_timer() exits if it can't obtain the lock due to
maybe_send_batched_emails still working.

Co-authored-by: Tim Abbott <tabbott@zulip.com>
2020-08-26 12:40:59 -07:00
Tim Abbott 1fddf16b73 Revert "exceptions: Extract json_unauths into MissingAuthenticationError."
This reverts commit c355f6b8d8.
2020-08-25 17:42:07 -07:00
Tim Abbott 5dceee5bd8 Revert "exceptions: Remove dead code msg_format method."
This reverts commit d6ab7ea8ff.
2020-08-25 17:42:07 -07:00
Tim Abbott fd3a43b51e Revert "rest: Simplify authentication error handling."
This reverts commit 1f90a31fa3.
2020-08-25 17:42:07 -07:00
Tim Abbott d6ab7ea8ff exceptions: Remove dead code msg_format method. 2020-08-25 17:17:15 -07:00
Tim Abbott 3b2a262b6f topic: Reorder topic history functions. 2020-08-25 17:03:48 -07:00
Tim Abbott 88a28d5470 topic: Refactor get_topic_history_for_stream.
This now uses get_topic_history_for_public_stream as a subroutine, to
avoid duplicating that large section of SQL.
2020-08-25 17:03:13 -07:00
Aman c3a8492697 topic: Rename get_topic_history_for_web_public_stream. 2020-08-25 17:01:12 -07:00
Tim Abbott 1f90a31fa3 rest: Simplify authentication error handling.
This pure refactor removes a now unnecessarily nested else clause,
helping keep this key piece of code readable.
2020-08-25 16:59:05 -07:00
Aman c355f6b8d8 exceptions: Extract json_unauths into MissingAuthenticationError.
We raise two types of json_unauthorized when
MissingAuthenticationError is raised. Raising the one
with www_authenticate let's the client know that user needs
to be logged in to access the requested content.

Sending `www_authenticate='session'` header with the response
also stops modern web-browsers from showing a login form to the
user and let's the client handle it completely.

Structurally, this moves the handling of common authentication errors
to a single shared middleware exception handler.
2020-08-25 16:52:21 -07:00
Hemanth V. Alluri 31a5e0b441 openapi: Fix DELETE /users/me status 200 response description.
If you look at line number 1121 (new) of commit 14c0a387cf,
I seem to have accidently set the description for a status
200 response to "Bad Request" instead of "Success" which
is what it really is. It's basically an ugly typo (maybe
due to hastily copy-pasting the template).

Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
2020-08-24 16:30:38 -07:00
Mateusz Mandera 4e8b087c82 rate_limiter: Fix flakiness of tests in test_external.
I noticed RateLimitTests.test_hit_ratelimits fails when run as an
individual test, but never when run after other tests. That's due to the
first API request in a run of tests taking a long time, as detailed in
the comment on the change to the setUp method.
2020-08-24 16:22:04 -07:00
Mateusz Mandera 934bdb9651 rate_limit: Improve dummy request objects in RateLimitTestCase.
Django always sets request.user to a UserProfile or AnonymousUser
instance, so it's better to mimic that in the tests where we pass a
dummy request objects for rate limiter testing purposes.
2020-08-24 16:22:04 -07:00
Mateusz Mandera 699c4e8549 rate_limit: Remove inaccurate comment in rate_limit decorator.
The data is now stored in memory if things are happening inside tornado.
That aside, there is no reason for a comment on a rate_limit_user call
to talk about low level implementation details of that function.
2020-08-24 16:22:04 -07:00
Mateusz Mandera c00aab8ede rate_limit: Delete code handling impossible cases with request.user.
I can find no evidence of it being possible to get an Exception when
accessing request.user or for it to be falsy. Django should always set
request.user to either a UserProfile (if logged in) or AnonymousUser
instance. Thus, this seems to be dead code that's handling cases that
can't happen.
2020-08-24 16:22:04 -07:00
Alex Vandiver b4c2ae9cae settings: Adjust documentation and comment references to settings.py.
`zproject/settings.py` itself is mostly-empty now.  Adjust the
references which should now point to `zproject/computed_settings.py`
or `zproject/default_settings.py`.
2020-08-24 13:13:16 -07:00
orientor 372e010dbb events: Add `op` field to `update_message_flags` events.
`update_message_flags` events used `operation` instead of `op`, the
latter being the standard field used in other events. So add `op`
field to `update_message_flags` and mark `operation` as deprecated,
so that it can be removed later.
2020-08-24 12:42:03 -07:00
Steve Howell f429df3401 github tests: Extract verify_post_is_ignored. 2020-08-24 12:34:46 -07:00
Steve Howell dfe6960584 minor: Format IGNORED_EVENTS for github. 2020-08-24 12:34:46 -07:00
Steve Howell 0863874050 jira: Ignore worklog_updated event. 2020-08-24 12:34:46 -07:00
Steve Howell 7b344044b4 jira: Ignore version_released event. 2020-08-24 12:34:46 -07:00
Steve Howell 52c17fc731 minor: Format IGNORED_EVENTS for jira. 2020-08-24 12:34:46 -07:00
Steve Howell 56bfae3fac jira tests: Test ignored events. 2020-08-24 12:34:46 -07:00
Steve Howell 1a011b2adb trello: Ignore dueComplete field.
It's possible that this is a new name for the "due"
field, but it's not totally clear.

In the exception we saw in the field:

    payload['action']['data']['old']['dueComplete'] = False
    payload['action']['data']['card']['dueComplete'] = True
2020-08-24 12:34:46 -07:00
Steve Howell 1bbf5491a7 trello: Ignore changes to card "cover".
These events happen when somebody changes the
background image for a Trello card, or something
similar.
2020-08-24 12:34:46 -07:00
Steve Howell 43cc4a2c50 trello tests: Test that we ignore "pos" data.
We will extend this test soon, once we begin
ignoring other fields.
2020-08-24 12:34:46 -07:00
Steve Howell bfe04ff6cf trello: Ignore updateCheckItem/updateList actions. 2020-08-24 12:34:46 -07:00
Steve Howell 37159fc235 trello tests: Generalize test for createCheckItem.
We remove the fixture for create_check_item, which
has been bit-rotting for as long as we have ignored
this type of card data.

Our new test is more powerful, in the sense that it
shows we successfully ignore all fixtures of this
type.

If we want to handle this, we'll just need to get
new, representative fixture data from trello.
2020-08-24 12:34:46 -07:00
Steve Howell b193c22e4d trello tests: Extract verify_post_is_ignored. 2020-08-24 12:34:46 -07:00
Steve Howell 95fe690e8d webhook tests: Avoid lots of unnecessary json dumping.
Commit c4254497b2
curiously had get_body() round tripping its data
through json load and dump.

I have seen this done for pretty-printing reasons,
but it doesn't apply here.

And if you're doing it for validation reasons,
you only need to do half the work, as my commit
here demonstrates.

We arguably don't even need the fail-fast code
here, since our fixtures are linted to be proper
json, I believe, plus downstream code probably
gives reasonably easy-to-diagnose symptoms.
2020-08-24 12:34:46 -07:00
Steve Howell 6d65af1790 webhook tests: Remove needless get_body functions.
These all did essentially what the parent implementation
does, except for checking that it's valid json.
2020-08-24 12:34:46 -07:00
Steve Howell 7fbe08f515 webhook tests: Introduce get_payload.
We introduce get_payload for the relatively
exceptional cases where webhooks return payloads
as dicts.

Having a simple "str" type for get_body will
allow us to extract test helpers that use
payloads from get_body() without the ugly
`Union[str, Dict[str, str]]` annotations.

I also tightened up annotations in a few places
where we now call get_payload (using Dict[str, str]
instead of Dict[str, Any]).

In the zendesk test I explicitly stringify
one of the parameters to satisfy mypy.
2020-08-24 12:34:46 -07:00
Steve Howell f7e4cc28eb mypy: Use str for get_body in bitbucket.
We don't need to claim a union type here.
2020-08-24 12:34:46 -07:00
Steve Howell 025d1fa180 webhook tests: Require args for check_webhook.
We tighten up the mypy types here.  And then
once we know that expected_message and expected_topic
are never None, we don't have call the do_test_message
and do_test_topic helpers any more, so we eliminate
them, too.

Finally, we don't return a message, since no tests
use the message currently.
2020-08-24 12:34:46 -07:00
Steve Howell fae3535e79 webhook tests: Avoid check_webhook for multi messages.
If we're not passing in expected_topic or expected_message
to check_webhook, it's better to just call send_webhook_payload,
since we'll want to explicitly check our messages
anyway.

This preps us to always require those fields for
check_webhook, which can prevent insidious testing no-ops.
2020-08-24 12:34:46 -07:00
Steve Howell f74aa29a1c webhook tests: Extract assert_stream_message.
This forces us to be a bit more explicit about testing
the three key values in any stream message, and it
also de-clutters the code a bit.  I eventually want
to phase out do_test_topic and friends, since they
have the pitfall that you can call them and have them
do nothing, because they don't actually require
values to be be passed in.

I also clean up the code a bit for the tests that
have two new messages arriving.
2020-08-24 12:34:46 -07:00
Steve Howell 3a710ab996 webhook tests: Remove stream_name parameter.
Having an optional stream_name parameter makes
it confusing to read the code if you know your
webhook is sending private messages.

And then the other two callers are already
checking topics, so they might as well check
stream names, too.

We also have the two stream-oriented callers
make their own call to "subscribe".  And we
future-proof this by making sure the exception
for no-message-being-sent calls out that gotcha.

Somewhat in passing, we now assert that
self.STREAM_NAME is not None in the main
helper.  This is partly to satisfy mypy, but
it's also a good sanity check.

This also sets the stage for the next commit,
where I'll add an assert_stream_message helper.
2020-08-24 12:34:46 -07:00
Steve Howell 2fdf966d9f webhook tests: Rename helper to send_webhook_payload.
Not all webhook payloads are json, so send_json_payload was a
bit misleading.

In passing I also remove "bytes" from the Union type for
"payload" parameter.
2020-08-24 12:34:46 -07:00
Steve Howell 388053db6b webhook tests: Rename main helper to check_webhook.
Almost all webhook tests use this helper, except a few
webhooks that write to private streams.

Being concise is important here, and the name
`self.send_and_test_stream_message` always confused
me, since it sounds you're sending a stream message,
and it leaves out the webhook piece.

We should consider renaming `send_and_test_private_message`
to something like `check_webhook_private`, but I couldn't
decide on a great name, and it's very rarely used.  So
for now I just made sure the docstrings of the two
sibling functions reference each other.
2020-08-24 12:34:46 -07:00
Steve Howell 00001a396b webhook tests: Shorten topic constants.
The "EXPECTED_" prefix and "_EVENTS" suffix
usually provided more noise than signal.

We also use module constants to avoid the "self."
noise.  It also makes it a bit more clear which
constants actually have to be in the class (e.g.
"FIXTURE_DIR_NAME") to do their job.
2020-08-24 12:34:46 -07:00
Steve Howell 0f970c6971 minor: Avoid do_test_message.
This function is a bad idea, as it leads to a possible situation
where you aren't actually testing anything:

    def do_test_message(self, msg: Message, expected_message: Optional[str]) -> None:
        if expected_message is not None:
            self.assertEqual(msg.content, expected_message)

Unfortunately, it's called deep in the stack in some places, but
we can safely replace it with assertEqual here.
2020-08-24 12:34:46 -07:00
Steve Howell fbef6e3c03 zendesk tests: Fix anti-patterns.
We had optional parameters for expected_topic and
expected_message, which are trivial to eliminate,
since the integration is really simple.

And we were doing strange things trying to reset
class variables at the end of tests.  Now we just
set them explicitly in the tests.
2020-08-24 12:34:46 -07:00
Steve Howell 3de2e4f034 dialogflow test: Fix nonsensical tests.
The test helper here was taking an "expected_topic"
parameter that it just ignored, and then the
dialogflow tests were passing in expected messages
in that slot, so the actual "expected_message" var
was "None" and was ignored.  So the tests weren't
testing anything.

Now we eliminate the crufty expected_topic parameter
and require an actual value for "expected_message".

I also clean up the mypy type for content_type,
and I remove the `content_type is None` check,
since all callers either pass in a str content
type or default to "application/json".
2020-08-24 12:34:46 -07:00
Alex Vandiver ad8943a64a url_preview: Only extract img tags with an `src`.
Some `<img>` tags do not have an SRC, if they are rewritten using JS
to have one later.  Attempting to access `first_image['src']` on these
will raise an exception, as they have no such attribute.

Only look for images which have a defined `src` attribute on them.  We
could instead check if `first_image.has_attr('src')`, but this seems
only likely to produce fewer valid images.
2020-08-18 14:26:21 -04:00
Alex Vandiver 5b74de7be7 markdown: Add another twitter code to retry-later.
Error code 131 is documented to be an arbitrary server error on
Twitter's side; add it to the retry list.
2020-08-18 10:32:24 -07:00
Alex Vandiver 092ed87ae3 markdown: Cache Twitter 403 responses that are semi-permanent.
03ca3afbc2 added more codes that are equivalent to 404's; this adds to
the list of cache-as-None codes a couple which are equivalent to
403's.  It does not comprise _all_ possible 403-like codes -- many of
them are "the client is not OK," which is relevant to log as an error
still.
2020-08-18 10:32:24 -07:00
Alex Vandiver aa5f98907c sentry: Deal more gracefully with missing stacktraces.
Not all languages produce exceptions with context or filenames.  See
https://develop.sentry.dev/sdk/event-payloads/stacktrace/
2020-08-18 11:17:42 -04:00
Alex Vandiver 0db311ddad sentry: Stacks are returned most-recent last.
Per [1], the sentry API returns frames sorted from oldest to newest.
As such, matching against the first filename that matches is most
likely not the right frame.

Match against the last frame with the guilty filename.

[1] https://develop.sentry.dev/sdk/event-payloads/stacktrace/
2020-08-18 11:17:42 -04:00
Alex Vandiver 0078f16f06 tests: Add an explicit order_by to prevent flakes.
Without an order_by, this can fail spuriously.
2020-08-18 11:15:35 -04:00
Steve Howell 33d7a22685 test_events: Fix loop structure in settings test.
The original commit was broken here:
b553507412

The intention was to run the same loop for all
settings, but instead, we did a funny loop of
just resetting schema_checker, and then we only
actually tested the last value of the loop.
2020-08-17 13:00:47 -04:00
Steve Howell f33a314a4d openapi: Add enum for stream role. 2020-08-16 08:35:18 -04:00
sahil839 f046c9c58a streams: Add role field to Subscription objects passed to clients.
This commit adds "role" field to the Subscription objects passed to
clients.  This is important preparation for being able to work on the
frontend for this feature.
2020-08-14 16:33:11 -07:00
Tim Abbott f94a8adf9e export: Remove duplicate 'analytics' zerver_realm object.
This fixes a harmless duplication of data in the Zulip data export
format.
2020-08-14 15:45:11 -07:00
arpit551 7568f6f9a8 export: Renamed zerver_analytics to zerver_realm.
While exporting analytics data we were using wrong table name
'zerver_analytics' in analytics config. Renamed it with
correct table name 'zerver_realm'.
2020-08-14 15:45:11 -07:00
arpit551 0d6047840b decorator: Updated user_passes_test function from Django 2.2.
Since bug https://bugs.python.org/issue3445 was resolved in Python
3.3, we can avoid the use of assigned=available_attrs(view_func) in
wraps decorator (which we were only using because we'd copied code
that handled that from Django).

Also available_attrs is now depreciated from Django 3.0 onwards.
2020-08-14 11:40:13 -07:00
arpit551 af3a34fbca cache: Used lru_cache from functools instead of django.utils.lru_cache.
Django 3.0 removed private Python 2 compatibility APIs
so used lru_cache() directly from functools.

We cast lru_cache to Any to avoid attr-defined error in mypy since we
are adding extra field, 'key_prefix', to this object later.
2020-08-14 11:34:04 -07:00
Alex Vandiver 2b6989a40f queue: Remove a no-longer-correct comment.
This comment stopped being true in 5686821150, and very much stopped
being relevant in dd40649e04 when the middleware entirely stopped
publishing to a queue.
2020-08-14 11:30:13 -07:00
Tim Abbott f2c9ee8000 migrations: Fix migration 0283 with orjson.
This function now matches the copy in zerver/lib/actions.py.

This is the same migration as
b250e42f61c525029bd2b3bbb8f4ea93ece62072; orjson enforces that we
don't use integers as keys in JSON dictionaries.
2020-08-14 11:22:03 -07:00
Tim Abbott b494b16791 meta tags: Use open graph titles/descriptions.
Apparently, we were incorrectly using constants for title/description
rather than the nice non-constant values from og:title and
og:description in our meta tags.
2020-08-14 10:58:18 -07:00
Steve Howell 9b6da158ad openapi: Add enum values to role node in specs.
We will eventually want to improve descriptions here
too.
2020-08-14 10:40:29 -04:00
Steve Howell 5931ebffd2 openapi: Use enum for message_type. 2020-08-14 10:40:29 -04:00
Clara Dantas 05bf72a75c attachments: Add is_web_public field.
This commit adds the is_web_public field in the AbstractAttachment
class. This is useful when validating user access to the attachment,
as otherwise we would have to make a query in the db to check if
that attachment was sent in a message in a web-public stream or not.
2020-08-12 17:26:03 -07:00
sahil839 ca1a8ac78f streams: Allow stream admin to update and deactivate streams.
The new Stream administrator role is allowed to manage a stream they
administer, including:
* Setting properties like name, description, privacy and post-policy.
* Removing subscribers
* Deactivating the stream

The access_stream_for_delete_or_update is modified and is used only
to get objects from database and further checks for administrative
rights is done by check_stream_access_for_delete_or_update.

We have also added a new exception class StreamAdministratorRequired.
2020-08-12 17:02:01 -07:00
sahil839 78da9fd3ab subscription: Add role field to Subscription class.
This commit adds role field to the Subscription class. Currently,
there are two option of roles - STREAM_ADMINISTRATOR and MEMBER.

We also add a property 'is_stream_admin' for checking whether the
user is stream admin or not.
2020-08-12 16:47:04 -07:00
Aman Agrawal 9f9daeea5b message_fetch: Allow access to web-public msgs for unauth users.
Via API, users can now access messages which are in web-public
streams without any authentication.

If the user is not authenticated, we assume it is a web-public
query and add `streams:web-public` narrow if not already present
to the narrow. web-public streams are also directly accessible.

Any malformed narrow which is not allowed in a web-public query
results in a 400 or 401. See test_message_fetch for the allowed
queries.
2020-08-12 16:39:02 -07:00
Tim Abbott 28b43b4edc message_fetch: Refactor get_base_query_for_search. 2020-08-12 16:33:59 -07:00
Tim Abbott 221cd496c6 NarrowBuilder: self.user_profile.realm -> self.realm. 2020-08-12 16:33:59 -07:00
Tim Abbott 35bd14574d message_fetch: Refactor public-streams query logic. 2020-08-12 16:33:59 -07:00
Tim Abbott d2e191b94e NarrowBuilder: Rename self.user_realm to self.realm.
The previous name will be very confusing in upcoming commits where the
UserProfile becomes Optional but Realm is not (E.g. for web-public
queries).
2020-08-12 16:33:59 -07:00
Anders Kaseorg a9539972f4 openapi: Fix validate_schema recursion structure.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-12 16:11:29 -07:00
Anders Kaseorg f1a9c87897 openapi: Add missing object types.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-12 16:11:29 -07:00
Anders Kaseorg ff46de305a openapi: Use reasonable variable names.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-12 16:11:29 -07:00
Anders Kaseorg 1d1149903b openapi: Remove unused document_events member.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-12 16:11:29 -07:00
Anders Kaseorg 4990e6d479 openapi: Deduplicate last modified check.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-12 16:11:29 -07:00
Alex Vandiver 596cf2580b sentry: Ignore all SuspiciousOperation loggers.
django.security.DisallowedHost is only one of a set of exceptions that
are "SuspiciousOperation" exceptions; all return a 400 to the user
when they bubble up[1]; all of them are uninteresting to Sentry.
While they may, in bulk, show a mis-configuration of some sort of the
application, such a failure should be detected via the increase in
400's, not via these, which are uninteresting individually.

While all of these are subclasses of SuspiciousOperation, we enumerate
them explicitly for a number of reasons:

 - There is no one logger we can ignore that captures all of them.
   Each of the errors uses its own logger, and django does not supply
   a `django.security` logger that all of them feed into.

 - Nor can we catch this by examining the exception object.  The
   SuspiciousOperation exception is raised too early in the stack for
   us to catch the exception by way of middleware and check
   `isinstance`.  But at the Sentry level, in `add_context`, it is no
   longer an exception but a log entry, and as such we have no
   `isinstance` that can be applied; we only know the logger name.

 - Finally, there is the semantic argument that while we have decided
   to ignore this set of security warnings, we _may_ wish to log new
   ones that may be added at some point in the future.  It is better
   to opt into those ignores than to blanket ignore all messages from
   the security logger.

This moves the DisallowedHost `ignore_logger` to be adjacent to its
kin, and not on the middleware that may trigger it.  Consistency is
more important than locality in this case.

Of these, the DisallowedHost logger if left as the only one that is
explicitly ignored in the LOGGING configuration in
`computed_settings.py`; it is by far the most frequent, and the least
likely to be malicious or impactful (unlike, say, RequestDataTooBig).

[1] https://docs.djangoproject.com/en/3.0/ref/exceptions/#suspiciousoperation
2020-08-12 16:08:38 -07:00
Anders Kaseorg d0f4af5f8c python: Catch JSONDecodeError instead of ValueError when decoding JSON.
These weren’t wrong since orjson.JSONDecodeError subclasses
json.JSONDecodeError which subclasses ValueError, but the more
specific ones express the intention more clearly.

(ujson raised ValueError directly, as did json in Python 2.)

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-12 11:59:59 -07:00
Alex Vandiver 153f16ee6a links: Flatten the set into a list before serializing into the queue.
orjson does not transparently do this set-to-list translation, unlike
ujson.
2020-08-12 11:42:24 -07:00
Kartik Srivastava 63173d5554 api: Return 'user_id' in 'POST /users' response.
This adds 'user_id' to the simple success response for 'POST /users'
api endpoint, to make it convenient for API clients to get details
about users they just created.  Appropriate changes have been made in
the docs and test_users.py.

Fixes #16072.
2020-08-11 16:40:12 -07:00
palash 8393f64120 test_auth_backends: Remove mock.patch('logging.warning').
Removed mock.patch('logging.warning') as no logs were being generated
in these blocks.
2020-08-11 16:24:51 -07:00
Anders Kaseorg 2061bd95f1 zulip.yaml: Don’t redundantly escape slashes.
These escapes are valid YAML 1.2 (for JSON compatibility) but not
valid YAML 1.1, which means they don’t work with the faster
yaml.CSafeLoader that we’d like to transition to.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-11 15:20:34 -07:00
Aman Agrawal 94839c9492 NarrowBuilder: Directly use denormalized 'recipient_id' of streams.
'recipient_id' was is now directly accessible and we can now use
it instead of doing an extra query to Recipient model.
2020-08-11 11:22:22 -07:00
Anders Kaseorg 61d0417e75 python: Replace ujson with orjson.
Fixes #6507.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-11 10:55:12 -07:00
Anders Kaseorg 123790a72d webhooks: Fix invalid JSON in Hello World test fixtures.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-11 10:47:13 -07:00
Anders Kaseorg 5043f6039b webhooks: Fix invalid JSON in Freshdesk test fixtures.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-11 10:47:13 -07:00
Anders Kaseorg ec6856bc3b webhooks: Fix invalid JSON in Dialogflow test fixtures.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-11 10:47:13 -07:00
Anders Kaseorg 114b0a2982 message_edit: Output a list, not a reversed iterator.
A list_reverseiterator object is not JSON serializable, and orjson
enforces this.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-11 10:47:13 -07:00
Anders Kaseorg 03d2540899 export: Post-process authentication_methods BitHandler field to list.
A BitHandler object is not JSON serializable, and orjson enforces
this.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-11 10:47:13 -07:00
Anders Kaseorg 2cf2547b27 export: Add missing datetime fields for post-processing.
datetime objects are not ordinarily JSON serializable.  While both
ujson and orjson have special cases to serialize datetime objects,
they do it in different ways.  So we want to fix the post-processing
code to do its job.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-11 10:47:13 -07:00
Alex Vandiver 28c627452f sentry: Ignore DisallowedHost messages.
This is a misconfiguration of the client, not the server.
2020-08-11 10:38:14 -07:00
Alex Vandiver f00ff1ef62 middleware: Make HostDomain into a process_request, not process_response.
It is more suited for `process_request`, since it should stop
execution of the request if the domain is invalid.  This code was
likely added as a process_response (in ea39fb2556) because there was
already a process_response at the time (added 7e786d5426, and no
longer necessary since dce6b4a40f).

It quiets an unnecessary warning when logging in at a non-existent
realm.

This stops performing unnecessary work when we are going to throw it
away and return a 404.  The edge case to this is if the request
_creates_ a realm, and is made using the URL of the new realm; this
change would prevent the request before it occurs. While this does
arise in tests, the tests do not reflect reality -- real requests to
/accounts/register/ are made via POST to the same (default) realm,
redirected there from `confirm-preregistrationuser`.  The tests are
adjusted to reflect real behavior.

Tweaked by tabbott to add a block comment in HostDomainMiddleware.
2020-08-11 10:37:55 -07:00
Alex Vandiver 816f91ae27 auth: Remove redirect on invalid realm during password reset.
This redirect was never effective -- because of the
HostDomainMiddleware, all requests to invalid domains have their
actual results thrown away, and replaced by an "Invalid realm" 404.
These lines are nonetheless _covered_ by coverage, because they do
run; the redirect is simply ineffective.  This can be seen by the test
that was added with them, in c8edbae21c, actually testing the contents
for the invalid realm wording, not the "find your accounts" wording.
2020-08-11 10:27:29 -07:00
Anders Kaseorg 6dea085187 docs: Correct “login” as a verb to “log in”.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-11 10:25:53 -07:00
Anders Kaseorg fb2e56e3c9 docs: Fix capitalization of keyboard keys.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-11 10:25:53 -07:00
Anders Kaseorg c155403884 docs: Fix various capitalization errors.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-11 10:25:52 -07:00
Anders Kaseorg 768f9f93cd docs: Capitalize Markdown consistently.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-11 10:23:06 -07:00
Anders Kaseorg 60a25b2721 docs: Fix spelling errors caught by codespell.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-11 10:23:06 -07:00
Alex Vandiver 2928bbc8bd logging: Report stack_info on logging.exception calls.
The exception trace only goes from where the exception was thrown up
to where the `logging.exception` call is; any context as to where
_that_ was called from is lost, unless `stack_info` is passed as well.
Having the stack is particularly useful for Sentry exceptions, which
gain the full stack trace.

Add `stack_info=True` on all `logging.exception` calls with a
non-trivial stack; we omit `wsgi.py`.  Adjusts tests to match.
2020-08-11 10:16:54 -07:00
Alex Vandiver e3a093b73f sentry: Add two more languages seen. 2020-08-11 10:15:02 -07:00
Alex Vandiver 34250d76a2 sentry: Don't drop (and server error) on languages we don't know.
It is better to show the stacktrace without highlighting than to drop
it entirely.
2020-08-11 10:15:02 -07:00
Alex Vandiver 569de55b82 sentry: Rename mapping variable to be more explicitly-named. 2020-08-11 10:15:02 -07:00
Alex Vandiver 712affd4df sentry: Document default topic. 2020-08-11 10:15:02 -07:00
Mateusz Mandera a7039c815e queue_processors: Fix UnboundLocalError in QueueProcessingWorker.
consume_time_seconds wasn't properly defined at the beginning, so when
a BaseException that isn't a subclass of Exception is thrown, the
finally: block could be entered with it still undefined.
2020-08-11 10:09:42 -07:00
Alex Vandiver 8fc6d740cd tornado: Set a half-second default timeout on requests from django.
By defaults, `requests` has no timeout on requests, which can lead to
waiting indefinitely.  Add a half-second timeout on these; this is
applied _inside_ each retry, not overall -- that is, with retries any
of these functions may take a total of 1.5s.
2020-08-10 16:55:56 -07:00
Alex Vandiver 89cd31e07c tornado: Switch trust_env = False out for no_proxy.
Use the `no_proxy` proxy, which explicitly disables proxy usage for
particular hosts.  This is a slightly cleaner solution than ignoring
all of the environment, as removing proxies is specifically what we
are attempting to accomplish.
2020-08-10 16:55:56 -07:00
Alex Vandiver 5bea344fde tornado: Factor out common error-handling pattern into an HTTP adapter.
The change in #2764 provided a better error message on one of the
three calls into Tornado, but left the other two with the old error
message.  `raise_for_status` was used on two out of three.

Use a custom HTTPAdapter to apply this pattern to all requests from
Django to Tornado.
2020-08-10 16:55:56 -07:00
Alex Vandiver bb754e0902 tornado: Add a retry with backoff to django-to-tornado requests.
This better hides errors from users during the moments when Tornado is
being restarted.
2020-08-10 16:55:56 -07:00
Alex Vandiver e87aecf61c tornado: Make requests_client a singleton, not a global. 2020-08-10 16:55:56 -07:00
Alex Vandiver 6918556648 tornado: Swap to early-abort, for fewer long hanging blocks. 2020-08-10 16:55:56 -07:00
Alex Vandiver 39368cad3a tornado: Extract functions called from django into one module.
This makes clearer the separation of concerns.
2020-08-10 16:55:56 -07:00
Tim Abbott e12225ace7 realm_audit_log: Fix malformed RealmAuditLog entries.
In f8bcf39014, we fixed buggy
marshalling of Streams and similar data structures where we were
including the Stream object rather than its ID in dictionaries passed
to ujson, and ujson happily wrote that large object dump into the
RealmAuditLog.extra_data field.

This commit includes a migration to fix those corrupted RealmAuditLog
entries, and because the migration loop is the same, also fixes the
format of similar RealmAuditLog entries to be in a more natural format
that doesn't weirdly nest and duplicate the "property" field.

Fixes #16066.
2020-08-09 16:51:29 -07:00
Steve Howell 6f97e9dfa9 mypy: Use object, not Any, in event_schema.
This requires a few redundant runtime isinstance
checks, but the extra assertions arguably make
the code more readable, and isinstance checks
are extremely negligible.
2020-08-08 11:30:46 -04:00
Tim Abbott ba34e2a61c help: Mark Zoom integration as stable.
We've had plenty of happy production usage at this point.
2020-08-07 16:45:30 -07:00
Anders Kaseorg f166f4c28c webhooks: Fix invalid JSON in Papertrail test fixtures.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-07 15:07:27 -07:00
Anders Kaseorg ae3af9cd56 webhooks: Fix invalid JSON in Jira test fixtures.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-07 15:06:05 -07:00
Anders Kaseorg c523657d48 test_subs: Remove incorrect encoding before JSON serialization.
bytes is not JSON serializable, and orjson enforces this.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-07 11:14:37 -07:00
Anders Kaseorg 0b12d38e4d fixtures: Fix invalid JSON in narrow.json.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-07 11:12:32 -07:00
Anders Kaseorg 23b815bb50 queue: Fix types to reflect that Pika channels receive bytes.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-07 11:12:32 -07:00
Anders Kaseorg 0d1cc8c171 test_subs: Remove absurd bot_owner parameter from request.
Seriously now.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-07 11:12:32 -07:00
Anders Kaseorg c9d1f547bd test_auth_backends: Remove absurd user_profile parameter from requests.
What even did you think this was supposed to do.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-07 11:12:32 -07:00
Anders Kaseorg 36bae2e86c message_fetch: Use strings as dict keys.
JSON keys must be strings, and orjson enforces this.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-07 11:12:32 -07:00
Anders Kaseorg 3e7503a607 html_diff: Really return str, not bytes.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-07 11:12:32 -07:00
Anders Kaseorg 80c0e60edc integrations: Decode webhook responses to str.
bytes is not JSON serializable, and orjson enforces this.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-07 11:12:32 -07:00
Anders Kaseorg 25c20a21bd event_queue: Convert users argument of send_event to list.
set is not JSON serializable, and orjson enforces this.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-07 11:12:32 -07:00
Anders Kaseorg fe5c524890 user_status: Use strings as dict keys.
JSON keys must be strings, and orjson enforces this.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-07 10:53:06 -07:00
Anders Kaseorg a329b538d2 test_helpers: Fix instrument_url decorator type.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-07 10:53:06 -07:00
Anders Kaseorg 28143c0ee2 drafts: Use strings as dict keys.
JSON keys must be strings, and orjson enforces this.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-07 10:53:06 -07:00
Anders Kaseorg b35e4428a9 test_custom_profile_data: Convert order QuerySet to list.
A QuerySet object is not JSON serializable, and orjson enforces this.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-07 10:53:06 -07:00
Anders Kaseorg 44af001045 custom_profile_fields: Use strings as dict keys.
JSON keys must be strings, and orjson enforces this.  Mypy didn’t
catch the mismatched type of profiles_by_user_id because it doesn’t
understand CustomProfileFieldValue.field_id.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-07 10:53:06 -07:00
Anders Kaseorg b250e42f61 realm_user_count_by_role: Use strings as dict keys.
JSON keys must be strings, and orjson enforces this.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-07 10:53:06 -07:00
Anders Kaseorg f8bcf39014 actions: Do not attempt JSON serialization of a Stream object.
It doesn’t end well.  Or sometimes it doesn’t end (OverflowError:
Maximum recursion level reached).

Introduced by commits ccdf52fef6 and
94d2de8b4a (#15601).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-07 10:53:06 -07:00
Sumanth V Rao a003d592b3 models: Fix typo in comments for get_users_by_delivery_email. 2020-08-06 15:46:56 -07:00
Steve Howell 15ffd2b666 event_schema: Extract check_stream_delete. 2020-08-06 12:29:43 -07:00
Steve Howell 059e0bb81e event_schema: Extract check_default_streams. 2020-08-06 12:29:43 -07:00
Steve Howell 36ec1571d4 event_schema: Extract check_default_stream_groups. 2020-08-06 12:29:43 -07:00
orientor fae3f1ca53 openapi: Improve Bots documentation by dividing bots.
Firstly divide the Bot schema into Bot and BasicBot for ease
of reusability. Also separate bot remove and bot delete into
two separate events.
2020-08-06 12:29:43 -07:00
orientor 1d88c9e12e openapi: Document /register and add tests for it.
We'll want to do more iteration on the details here, but this is a
huge milestone.

Fixes #14188.
2020-08-05 17:57:24 -07:00
Mateusz Mandera 15752067dc auth: Treat emails case-insensitively in ExternalAuthResult.
Our intent throughout the codebase is to treat email
case-insensitively.
The only codepath affected by this bug is remote_user_sso, as that's the
only one that currently passes potentially both a user_profile and
ExternalAuthDataDict when creating the ExternalAuthResult. That's why we
add a test specifically for that codepath.
2020-08-05 11:40:35 -07:00
orientor 63af93ebe7 openapi: Add documentation for bot `services`. 2020-08-04 17:15:27 -07:00
orientor 1a6aeb710e openapi: Move frequently used data to schemas section without any edits.
To increase code reusability and reduce code redundancy, we move data
structures which occur multiple times in the OpenAPI documentation to
the `schemas` section. Note that this a pure data movement commit
without any changes to the data beyond removing over-specific
descriptions (E.g. that suggest the user group was just created).

(Future commits will use these)
2020-08-04 17:15:27 -07:00
orientor 8108acbdfd openapi_py: Make `/events` checking strict.
Previously there was a documented_events set which provided for partial
OpenAPI documentation while documentation was still going on. But since
the documentation is complete now, remove it.
2020-08-03 18:07:35 -07:00
orientor e7c9c55664 api: Complete get_events response format documetation.
This giant commit completes basic OpenAPI documentation for all events
in Zulip's real-time events API.

Further work will be required in the near future to make
/api/get_events usable.

With many edits by tabbott for wording and correctness (especially
around which clients receive events, and their purpose).
2020-08-03 18:07:09 -07:00
Tim Abbott 00fd9afad5 embed: Remove useless 'sender' field.
The variant `update_message` events have this extra sender field not
present in normal update_message events; this field has no purpose, so
we remove it.
2020-08-03 18:04:38 -07:00
Tim Abbott 99a54ba67e tornado: Fix ID lists leaked to the events API.
Apparently, `update_message` events unexpectedly contained what were
intended to be internal data structures about which users were
mentioned in a given message.

The bug has been present and accumulating new data structures for
years.

Fixing this should improve the performance of handling update_message
events as well as cleaning up this API's interface.

This was discovered by our automated API documentation schema checking
tooling detecting these unexpected elements in these event
definitions; that same logic should prevent future bugs like this from
being introduced in the future.
2020-08-03 17:52:39 -07:00
orientor d036a01516 test_events: Convert tuples to arrays for OpenAPI testing.
Use `ujson.loads(ujson.dumps())` wrapper on events sent for OpenAPI
testing so that all tuples are converted into arrays as tuples aren't
valid in JSON.
2020-08-03 16:55:36 -07:00
Clara Dantas f26d0affef bulk_update: Do updates in bulk where applicable. 2020-08-03 16:16:42 -07:00
Clara Dantas be6b2b248f Error reporting emails: Indicate user role.
Adds user role to the user's information in the error reporting
emails, as some bugs are role-dependent.

Fixes: #15344
2020-08-03 13:35:51 -07:00
Clara Dantas b4dd118aa1 refactor: Create a user object in report.
To make it easier to check if there is user information to be used
in the error report emails, we create a user object inside report.
Now, to check if we have the user's full name, email, etc, we just
need to do report['user']['user_full_name'] rather than check
each information one by one, because if the value of one key in
the report is different than None, all the others will be as well.
2020-08-03 13:35:51 -07:00
Mateusz Mandera bb46b21123 find_account: Fix the email search query.
The search should be case-insensitive.
2020-08-02 12:37:20 -07:00
Anders Kaseorg 159641bab8 timestamp: Remove datetime_to_precise_timestamp for datetime.timestamp.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-07-31 22:10:14 -07:00
Anders Kaseorg 6ec808b8df js: Add "use strict" directive to CommonJS files.
ES and TypeScript modules are strict by default and don’t need this
directive.  ESLint will remind us to add it to new CommonJS files and
remove it from ES and TypeScript modules.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-07-31 22:09:46 -07:00
Alex Vandiver 90cdda9836 markdown: Link the twitter response code docs inline. 2020-07-31 10:35:41 -07:00
Alex Vandiver 03ca3afbc2 markdown: Treat more twitter codes as also permanent failures.
Per the API documentation[1], the following codes all correspond to
HTTP 404:

 - `34`: **Sorry, that page does not exist.**  The specified resource
   was not found.
 - `144`: **No status found with that ID.**  The requested Tweet ID is
   not found (if it existed, it was probably deleted)
 - `421`: **This Tweet is no longer available.**  The Tweet cannot be
   retrieved. This may be for a number of reasons.
 - `422`: **This Tweet is no longer available because it violated the
   Twitter Rules.**  The Tweet is not available in the API.

Treat all of these identically.

[1] https://developer.twitter.com/en/docs/basics/response-codes
2020-07-31 10:35:41 -07:00
Alex Vandiver fc141af30e markdown: Factor out twitter error code handling. 2020-07-31 10:35:41 -07:00
Hemanth V. Alluri 4e1024da5c webhooks/gitlab: Add an option to exclude MR title from topics.
Since the title of a merge request can often change, it shouldn't be a
part of the topic that we send the message to. Otherwise things would
get messy and confusing.

But at the same time we don't want to make this mandatory. So we add
a new boolean GET parameter that can toggle whether or not the topic
should include the MR title (`use_merge_request_title`).

Fixes #15951.

Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
2020-07-31 10:33:52 -07:00
Clara Dantas ca2123fec1 test_subs: Improve readability of URLs that have request params. 2020-07-30 16:59:51 -07:00
Clara Dantas 6b011b241b actions: Remove duplicated conditional.
In the gather_subscriptions_helper functions, there was a repeated
conditional inside one another.
2020-07-30 16:59:51 -07:00
orientor 314c8ce52c events_documentation: Add more documentation. 2020-07-30 16:39:28 -07:00
Mohit Gupta 8c04c03408 test_logging_handlers: Use assertLogs to verify error logs.
This avoids spam in test-backend output.
2020-07-30 13:38:26 -07:00
Tim Abbott f8e2c1f693 test_management_commands: Fix export test.
This was broken in the last commit.
2020-07-30 13:26:06 -07:00
Tim Abbott 6130a61be0 export: Only print .s with percent_callback to console.
The S3 data export tool's upload code path uses this nice boto
callback feature for showing a progress bar, which is nice for the
management command.  It's spammy/broken in production and the backend
tests, so we change percent_callback to be a parameter passed in so
that it can only be used in the contexts where it makes sense.
2020-07-30 13:14:53 -07:00
Anders Kaseorg 022c4fbfc7 Revert "digest: Support digest of web public streams for guest users."
This reverts commit c3779338c6 (part
of #14638), which incorrectly depended on commits from the future,
with the effect of either halting the flow of entropic time in an
irresolvable temporal paradox, summoning extradimensional beings to
rain destruction on the galaxy, or failing CI.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-07-29 21:05:59 -07:00
Clara Dantas c3779338c6 digest: Support digest of web public streams for guest users. 2020-07-29 17:52:36 -07:00
Clara Dantas a9af80d7a2 streams: Make /streams endpoint return also web-public streams.
This commit modifies the /streams endpoint so that the web-public
streams are included in the default list of streams that users
have access to.

This is part of PR #14638 that aims to allow guest users to
browse and subscribe themselves to web public streams.
2020-07-29 17:52:36 -07:00
Clara Dantas 0994b029d6 streams: Grant authorization to guest users to subscribe.
Modifies filter_stream_authorization so that web-public streams are
added in the list of authorized streams that a guest user can
subscribe.

This commit is part of PR #14638 that aims to allow guest users
to browse and subscribe to web-public streams.
2020-07-29 17:52:36 -07:00
Clara Dantas e22e12fe60 streams: Grant guest users access to web-public streams.
In this commit, we grant guest users access to stream history,
send message and common stream data of web-public streams.

This is part of PR #14638 that aims to allow guest users to
browse and subscribe to web-public streams.
2020-07-29 17:52:36 -07:00
Clara Dantas 523bb30f33 actions: Modify validate_user_access_to_subscribers.
This modification allows guest users to have access to web-public
streams subscribers, even if they aren't subscribed or never
subscribed to that stream.

This commit is part of PR #14638 that aims to allow guest users to
browser and subscribe to web-public streams.
2020-07-29 17:52:36 -07:00
Clara Dantas 6a1d9480bc actions: Modify gather_subscriptions() to also gather web-public streams.
Now, gather_subscriptions include web-public streams in the 3 sets
of streams that it returns, subscribed, unsubscribed and never
subscribed.

This is part of PR #14638 that aims to allow guest users to browse and
subscribe to web-public streams.
2020-07-29 17:52:36 -07:00
Clara Dantas fda94a49c1 streams: Change flow in filter_stream_authorization.
This change makes the flow more coherent by instead of checking,
in the last condition, if the user isn't authorized to access that
stream, check if they are, as it is done in the other checks. Only
if all the conditions are false, which means that the user doesn't
have access to that stream, the stream is added to the
unauthorized_streams list.
2020-07-29 17:52:36 -07:00
Gittenburg 0706de2305 docs: Make tabbed sections accessible from keyboard.
Part of #15948.
2020-07-29 11:40:39 -07:00
davidscherer 0e90442d8b
integrations: Don't call prettify_date on Trello card descriptions.
Card descriptions aren't dates, and calling prettify_date on them results in removing upper case T characters, replacing uppercase Z characters with " UTC", etc. in descriptions when they appear in Zulip.

This was pretty clearly just a copy/paste mistake (these functions are very closely parallel to the *_due_date_* functions above, which do work on dates and call prettify_date).
2020-07-28 17:58:19 -07:00
Hemanth V. Alluri 21c9adbb3c drafts: Add an API endpoint for fetching drafts.
This endpoint will allow a user to fetch their drafts.

Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
2020-07-28 17:18:35 -07:00
Hemanth V. Alluri 630fd10be1 drafts: Add an API endpoint for deleting drafts.
This endpoint will allow a user to delete a single draft.

Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
2020-07-28 17:18:35 -07:00
Hemanth V. Alluri 2d307c760e drafts: Add an API endpoint for editing drafts.
This endpoint will allow a user to edit a single draft.

Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
2020-07-28 17:18:35 -07:00
Hemanth V. Alluri a0f71b7458 drafts: Add an API endpoint for creating drafts.
This endpoint will allow a user to create drafts in bulk.

Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
2020-07-28 17:18:35 -07:00
Hemanth V. Alluri 0e893b9045 models/drafts: Add a model for storing Draft messages.
Also add a Draft object-to-dictionary conversion method.
The following commits will provide an API around this
model using which our clients can sync drafts across each
other (if they so wish too). As of making this commit, we
haven't finalized exactly how our clients will use this.

See https://chat.zulip.org/#narrow/stream/2-general/topic/drafts
For some of the discussion around this model and in general,
around this feature.

Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
2020-07-28 17:18:35 -07:00
Hemanth V. Alluri d5f42e2722 timestamp: Add function to convert a datetime to precise Unix timestamp.
Unlike the other Python datetime to Unix timestamp conversion
function (`datetime_to_timestamp`), `datetime_to_precise_timestamp`
won't drop the microseconds.

Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
2020-07-28 17:18:35 -07:00
Dinesh 9583554d44 auth: Add check_config for apple auth.
Apple has some other obligatory settings other than key and secret.
To handle that this commit adds a function check_config() similar
to that of SAML.
2020-07-28 17:12:49 -07:00
Dinesh 4afce5d94d apple_auth: Change BUNDLE_ID setting to APP_ID everywhere.
The apple developer webapp consistently refers this App ID. So,
this clears any confusion that can occur.

Since python social auth only requires us to include App ID in
_AUDIENCE(a list), we do that in computed settings making it easier for
server admin and we make it much clear by having it set to
APP_ID instead of BUNDLE_ID.
2020-07-28 17:12:49 -07:00
Dinesh c15d7e3202 requirements: Update social-auth-core to latest version.
Uses git release as this version 3.4.0 is not released to pypi.
This is required for removing some overriden functions of
apple auth backend class AppleAuthBackend.

With the update we also make following changes:

* Fix full name being populated as "None None".
c5c74f27dd that's included in update assigns first_name and last_name
to None when no name is provided by apple. Due to this our
code is filling return_data['full_name'] to 'None None'.
This commit fixes it by making first and last name strings empty.

* Remove decode_id_token override.
Python social auth merged the PR we sent including the changes
we made to decode_id_token function. So, now there is no
necessity for the override.

* Add _AUDIENCE setting in computed_settings.py.
`decode_id_token` is dependent on this setting.
2020-07-28 17:12:49 -07:00
Gittenburg 87d8a54b91 help: Stop collapsing sidebar sections.
Three reasons:

1. The sliding was disorienting.
2. The collapsing disallowed searching for other pages with Ctrl+F.
3. The collapsing mechanism wasn't accessible (not usable with the
   keyboard / no ARIA tags).

Tweaked by tabbott to center the left sidebar on the selected page.

Part of #15948.
2020-07-28 16:28:58 -07:00
orientor f266b52469 events_documentation: Document all events of type stream and some others.
Document all events of `type`=stream i.e all `op`s. Also document some other
events.

Tweaked by tabbott to clarify some documentation details (especially
around who receives events).
2020-07-28 16:00:12 -07:00
Tim Abbott c94f725db6 models: Improve exception for invalid FAKE_EMAIL_DOMAIN. 2020-07-28 11:41:06 -07:00
Hemanth V. Alluri eb2809effe webhooks/freshdesk: Remove key checking step.
The idea behind doing this is that we would rather let the code error
out rather than add to the logs. It's webhook code usually never uses
the logging module so this section of legacy code needed to be changed
or removed.

Assists PR #15942.

Signed-off-by: Hemanth V. Alluri <hdrive1999@gmail.com>
2020-07-28 10:55:11 -07:00
Clara Dantas 831298d387 test_classes: Modify functions to deal with web-public streams.
Modify common_subscribe_to_streams to perform subscription in
web-public streams as well, and make_stream function to create
web-public streams.
2020-07-27 18:30:06 -07:00
Tim Abbott dbde901684 events: Document unintentionally exposed API fields.
It's a bug that these are sent in event payloads; I'll open an issue
for resolving that issue.  For now we document them mainly to make our
tests pass.
2020-07-27 18:08:41 -07:00
orientor 227c90a4ae openapi: Document `message` event. 2020-07-27 18:08:41 -07:00
orientor ec40a5dda4 openapi_py: Add validation for `/events`.
Edit the function `validate_against_openapi_schema` and add some
helper functions to allow for validation of documented events.
Also add OpenAPI response validation in `verify_action` as it is
called in a large number of `/events` tests.
2020-07-27 18:08:41 -07:00
orientor ed8d1925fb openapi: Edit `/events` OpenAPI specification for validation.
Some events in `/events` were incorrectly or incompletely documented.
Rectify them.
2020-07-27 18:08:41 -07:00
orientor 86f7b8f7d9 openapi_markdown: Add parser for events documentation.
Add parser so that documentation of the various events returned by
`/events` are rendered to the docs directly from their OpenAPI
specification.
2020-07-27 18:08:41 -07:00
orientor cdacd3223b openapi: Document various events. 2020-07-27 18:08:41 -07:00
Tim Abbott 3252dfaa72 api: Document API change for Stream.date_created. 2020-07-27 17:10:59 -07:00
Alex Vandiver 9266315a1f middleware: Stop shadowing top-level logger definition on line 33. 2020-07-27 16:46:13 -07:00
Tim Abbott 3d1a1e0d20 test_logging_handlers: Avoid printing to console.
This lets us test the recursion bug behavior of this logging handler
without resulting in `logging.error` output being printed to the
console in the event that the test passes.
2020-07-27 16:33:36 -07:00
Sumanth V Rao 9b6de63afe stream/docs: Add date_created to Stream.API_FIELDS.
The parameter Stream.date_created is now sent down to the clients
for both:

    - client.get_streams()
    - client.list_subscriptions()

API docs updated for stream and subscriptions.

Fixes #15410
2020-07-27 16:33:36 -07:00
Alex Vandiver 1b2d0271af sentry: Prevent double-logging of JSON-formatted errors.
Capture and report the initial exception, not the formatted text-only
message traceback.
2020-07-27 11:07:55 -07:00
Mohit Gupta e9203d92bb test_upload: Use assertLogs in upload tests to verify logs.
This will avoid spam in test-backend output.
2020-07-27 11:02:17 -07:00
Mohit Gupta c9136586d9 test_signup: Use assertLogs in signup tests to verify logging.
This will avoid logs to spam test-backend output.
2020-07-27 11:02:17 -07:00
Mohit Gupta 1668c7395e test_settings: Verify logs of domain mismatch in password change test.
Use assertLogs for verifying debug log produced due to LDAP domain
mismatch while changing password. This will avoid spam in test-backend
output.
2020-07-27 11:02:17 -07:00
Mohit Gupta dbbacdd3d9 test_sessions: Use assertLogs to avoid spam in test-backend output. 2020-07-27 11:02:17 -07:00
Mohit Gupta 4192c6f959 test_report: Use assertLogs to avoid spam in test-backend output. 2020-07-27 11:02:17 -07:00
Mohit Gupta 8b9deb9c71 test_realm: Use assertLogs to avoid spam in test-backend output. 2020-07-27 11:02:17 -07:00
Mohit Gupta 82646e27c2 test_realm_export: Use assertLogs in endpoint tests to verify info log.
This is to avoid spam in test-backend output.
2020-07-27 11:02:17 -07:00
Mohit Gupta 3eda7586f3 test_queue_worker: Use assertlogs to verify logs while testing workers.
This will avoid spam in output of test-backend.
2020-07-27 11:02:17 -07:00
Mohit Gupta 732f1fa44d test_queue: Use assertLogs to verify warning in test_queue_error_json.
This will avoid spam in test-backend output.
2020-07-27 11:02:17 -07:00
Mohit Gupta 68b9f0b3cf tests: User assertLogs to verify info logs while soft deactivation.
This will avoid spamming of test-backend output.
2020-07-27 11:02:17 -07:00
orientor 7bb5fac8e0 test_message_send: Mark tests using intentiionally undocumented features.
Some parameters such as `to` and `topic` have been intentionally
undocumentecd hence fail request validation. So mark tests which
fail due to this accordingly.
2020-07-26 16:26:57 -07:00
orientor bdf9d912e3 test_classes: Use OpenAPI request validator for validating requests.
Expand the `validate_api_response_openapi` function to also validate
requests to ensure better testing.
2020-07-26 16:26:57 -07:00
orientor c91c106cfb openapi_py: Change condition for invalid requests.
Change the condition for allowing failed validation to the condition
that `if the test fails, response status code begins with 4`. Also
add `intentionally_undocumented` argument in `validate_request` for
allowing passing of tests which return `200` responses but fail
validation due to some intentionally undocumented feature in
OpenAPI specification.
2020-07-26 16:26:56 -07:00
Mohit Gupta 8b8cfb2e73 test_link_embed: Add assertLogs to prevent spam in test-backend. 2020-07-26 16:14:17 -07:00
Mohit Gupta 0ca3c1d904 test_home: Use assertLogs to test info logs by do_soft_deactivate_users.
This will verify logs without spamming test-backend output.
2020-07-26 16:14:17 -07:00
Mohit Gupta 85fcd0ad99 test_events: Use assertLogs to avoid spam in test-backend output. 2020-07-26 16:14:17 -07:00
Mohit Gupta 4928b55996 test_cache: Use assertLogs to avoid spam in test-backend output. 2020-07-26 16:14:17 -07:00
Mohit Gupta 133a5f2a7c tests: Add assertLogs for test_auth_backends.
This commit tests logging of warning log using assertLogs instead of
printing it in test output hence avoiding spam in test output.
2020-07-26 16:14:17 -07:00
Mohit Gupta 67a67fcc02 tests: Avoid "import logging" and setting root logger level in tests.
Use assertLogs to avoid spamming the test-backend output.
2020-07-26 16:14:17 -07:00
Mohit Gupta c4fe91af74 test_push_notifications: Add assertLogs to verify logging in tests.
This will prevent spam in test-backend output and test logging of logs
by the code being tested.
2020-07-26 16:14:17 -07:00
palash aba53551ec test: Add order_by in query for RealmAuditLog.
Added order_by("id") clause in query for RealmAuditLog
for consistent output.
It was causing zerver.tests.test_audit_log.TestRealmAuditLog
to fail due to order mismatch.
2020-07-26 15:47:16 -07:00
Vishnu KS 5b0b1efb15 support: Add functionality to approve sponsorship requests.
This should make it much easier to process these requests.
2020-07-24 17:55:38 -07:00
Steve Howell 21dfe76295 tests: Remove dead print-delay code. 2020-07-24 15:30:17 -07:00
Tim Abbott 63991a0a3c test_message_dict: Remove clock time assertion.
Clock time checks lead to tests that nondeterministically fail when
the CI container is super slow, and there's no good reason this test
in particular needs to do that sort of test in addition to our
standard database query count check (which is already does).
2020-07-24 13:29:26 -07:00
Steve Howell ce77a733e1 outgoing webhook tests: Make setup explicit.
Now when you are reading a single test, you can
explicitly see that the event and service handler
are tied to your bot, which is our test bot
for outgoing webhooks.
2020-07-24 13:13:34 -07:00
Steve Howell fb2a69ec78 outgoing webhook tests: Remove dead code.
The first assignment is redundant, and the
second assignment is ignored.
2020-07-24 13:13:34 -07:00
Steve Howell 36027d495a outgoing webhook tests: Avoid mock decorator.
Decorating an entire test with a mock makes it
hard to ascertain where the actual mock behavior
is expected to happen, plus it clutters up
the parameter list.

In fact, we remove a dubious re-assertion here that
a mock was called.  The assertion that a mock was
called was true, but it was misleading to think
the code right before it had invoked the mock.
2020-07-24 13:13:34 -07:00
Steve Howell 063e27ab52 mypy: Move accidental mock annotation for self.
Curiously, the obviously broken annotation for
self obscured the lack of assert here, so I
fixed that too.
2020-07-24 13:13:34 -07:00
Steve Howell 924292ba65 outgoing webhook tests: Move code out of with blocks. 2020-07-24 13:13:34 -07:00
majordwarf ad23c6e808 emoji: Switch to `user_profile.realm` for cross-realm reactions.
A temporary fix to enable cross-realm emoji reactions till `Message.realm`
is denormalized.

Fixes #15866.
2020-07-24 13:11:27 -07:00
Tim Abbott 29c66cf7c2 actions: Remove log_event and its legacy settings.
Now that we've finally converted these to use RealmAuditLog, we can
remove this ultra-legacy bit of code.
2020-07-24 12:13:16 -07:00
arpit551 0d79b55b2e audit_log: Record RealmAuditLog in do_change_notification_settings.
Removed logging with log_event and used RealmAuditLog instead.
Added tests in test_audit_log for the same.
2020-07-24 12:04:40 -07:00
arpit551 54df9290b9 audit_log: Record RealmAuditLog in do_rename_stream.
Removed logging with  log_event and used RealmAuditLog instead.
Added tests in test_audit_log for the same.
2020-07-24 12:00:31 -07:00
arpit551 94d2de8b4a audit_log: Record RealmAuditLog while changing default streams.
Removed logging with log_event and used RealmAuditLog instead.
Added tests in test_audit_log for the same.
2020-07-24 12:00:31 -07:00
arpit551 753713fde7 audit_log: Record RealmAuditLog in do_change_subscription_property.
Removed logging with log_subscription_property_change
and used RealmAuditLog instead.
Added tests in test_audit_log for the same.
2020-07-24 12:00:29 -07:00
arpit551 e6f54a3278 audit_log: Record RealmAuditLog in do_change_icon_source.
Removed logging with  log_event and used RealmAuditLog instead.
Added tests in test_audit_log for the same.
2020-07-24 11:56:40 -07:00
arpit551 12a19b0ff1 actions.py: Removed unnecessary logging in notify_subscription_added/removed.
We already record RealmAuditLog in bulk_add/remove_subscription
so there is no need to log while notifying.
2020-07-24 11:51:47 -07:00
arpit551 ccdf52fef6 audit_log: Log RealmAuditLog for realm notification properties.
Log RealmAuditLog for do_set_realm_notifications_stream and
do_set_realm_signup_notifications_stream function.
Added tests for the same.
2020-07-24 11:51:47 -07:00
arpit551 8f6a1c3f40 actions.py: Removed do_set_realm_message_deleting function.
Used do_set_realm_property function instead of
do_set_realm_message_deleting and removed it.
2020-07-24 11:51:47 -07:00
arpit551 422fa0ff7d audit_log: Log RealmAuditLog for do_set_realm_message_editing.
Log RealmAuditLog for do_set_realm_message_editing. Added tests for
same.
2020-07-24 11:51:46 -07:00