Commit Graph

4973 Commits

Author SHA1 Message Date
Alex Vandiver ad8943a64a url_preview: Only extract img tags with an `src`.
Some `<img>` tags do not have an SRC, if they are rewritten using JS
to have one later.  Attempting to access `first_image['src']` on these
will raise an exception, as they have no such attribute.

Only look for images which have a defined `src` attribute on them.  We
could instead check if `first_image.has_attr('src')`, but this seems
only likely to produce fewer valid images.
2020-08-18 14:26:21 -04:00
Alex Vandiver 0078f16f06 tests: Add an explicit order_by to prevent flakes.
Without an order_by, this can fail spuriously.
2020-08-18 11:15:35 -04:00
Steve Howell 33d7a22685 test_events: Fix loop structure in settings test.
The original commit was broken here:
b553507412

The intention was to run the same loop for all
settings, but instead, we did a funny loop of
just resetting schema_checker, and then we only
actually tested the last value of the loop.
2020-08-17 13:00:47 -04:00
Tim Abbott b494b16791 meta tags: Use open graph titles/descriptions.
Apparently, we were incorrectly using constants for title/description
rather than the nice non-constant values from og:title and
og:description in our meta tags.
2020-08-14 10:58:18 -07:00
Clara Dantas 05bf72a75c attachments: Add is_web_public field.
This commit adds the is_web_public field in the AbstractAttachment
class. This is useful when validating user access to the attachment,
as otherwise we would have to make a query in the db to check if
that attachment was sent in a message in a web-public stream or not.
2020-08-12 17:26:03 -07:00
sahil839 ca1a8ac78f streams: Allow stream admin to update and deactivate streams.
The new Stream administrator role is allowed to manage a stream they
administer, including:
* Setting properties like name, description, privacy and post-policy.
* Removing subscribers
* Deactivating the stream

The access_stream_for_delete_or_update is modified and is used only
to get objects from database and further checks for administrative
rights is done by check_stream_access_for_delete_or_update.

We have also added a new exception class StreamAdministratorRequired.
2020-08-12 17:02:01 -07:00
Aman Agrawal 9f9daeea5b message_fetch: Allow access to web-public msgs for unauth users.
Via API, users can now access messages which are in web-public
streams without any authentication.

If the user is not authenticated, we assume it is a web-public
query and add `streams:web-public` narrow if not already present
to the narrow. web-public streams are also directly accessible.

Any malformed narrow which is not allowed in a web-public query
results in a 400 or 401. See test_message_fetch for the allowed
queries.
2020-08-12 16:39:02 -07:00
Anders Kaseorg ff46de305a openapi: Use reasonable variable names.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-12 16:11:29 -07:00
Anders Kaseorg 4990e6d479 openapi: Deduplicate last modified check.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-12 16:11:29 -07:00
Kartik Srivastava 63173d5554 api: Return 'user_id' in 'POST /users' response.
This adds 'user_id' to the simple success response for 'POST /users'
api endpoint, to make it convenient for API clients to get details
about users they just created.  Appropriate changes have been made in
the docs and test_users.py.

Fixes #16072.
2020-08-11 16:40:12 -07:00
palash 8393f64120 test_auth_backends: Remove mock.patch('logging.warning').
Removed mock.patch('logging.warning') as no logs were being generated
in these blocks.
2020-08-11 16:24:51 -07:00
Aman Agrawal 94839c9492 NarrowBuilder: Directly use denormalized 'recipient_id' of streams.
'recipient_id' was is now directly accessible and we can now use
it instead of doing an extra query to Recipient model.
2020-08-11 11:22:22 -07:00
Anders Kaseorg 61d0417e75 python: Replace ujson with orjson.
Fixes #6507.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-11 10:55:12 -07:00
Alex Vandiver f00ff1ef62 middleware: Make HostDomain into a process_request, not process_response.
It is more suited for `process_request`, since it should stop
execution of the request if the domain is invalid.  This code was
likely added as a process_response (in ea39fb2556) because there was
already a process_response at the time (added 7e786d5426, and no
longer necessary since dce6b4a40f).

It quiets an unnecessary warning when logging in at a non-existent
realm.

This stops performing unnecessary work when we are going to throw it
away and return a 404.  The edge case to this is if the request
_creates_ a realm, and is made using the URL of the new realm; this
change would prevent the request before it occurs. While this does
arise in tests, the tests do not reflect reality -- real requests to
/accounts/register/ are made via POST to the same (default) realm,
redirected there from `confirm-preregistrationuser`.  The tests are
adjusted to reflect real behavior.

Tweaked by tabbott to add a block comment in HostDomainMiddleware.
2020-08-11 10:37:55 -07:00
Anders Kaseorg 6dea085187 docs: Correct “login” as a verb to “log in”.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-11 10:25:53 -07:00
Anders Kaseorg 768f9f93cd docs: Capitalize Markdown consistently.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-11 10:23:06 -07:00
Anders Kaseorg 60a25b2721 docs: Fix spelling errors caught by codespell.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-11 10:23:06 -07:00
Alex Vandiver 2928bbc8bd logging: Report stack_info on logging.exception calls.
The exception trace only goes from where the exception was thrown up
to where the `logging.exception` call is; any context as to where
_that_ was called from is lost, unless `stack_info` is passed as well.
Having the stack is particularly useful for Sentry exceptions, which
gain the full stack trace.

Add `stack_info=True` on all `logging.exception` calls with a
non-trivial stack; we omit `wsgi.py`.  Adjusts tests to match.
2020-08-11 10:16:54 -07:00
Alex Vandiver 39368cad3a tornado: Extract functions called from django into one module.
This makes clearer the separation of concerns.
2020-08-10 16:55:56 -07:00
Tim Abbott e12225ace7 realm_audit_log: Fix malformed RealmAuditLog entries.
In f8bcf39014, we fixed buggy
marshalling of Streams and similar data structures where we were
including the Stream object rather than its ID in dictionaries passed
to ujson, and ujson happily wrote that large object dump into the
RealmAuditLog.extra_data field.

This commit includes a migration to fix those corrupted RealmAuditLog
entries, and because the migration loop is the same, also fixes the
format of similar RealmAuditLog entries to be in a more natural format
that doesn't weirdly nest and duplicate the "property" field.

Fixes #16066.
2020-08-09 16:51:29 -07:00
Anders Kaseorg c523657d48 test_subs: Remove incorrect encoding before JSON serialization.
bytes is not JSON serializable, and orjson enforces this.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-07 11:14:37 -07:00
Anders Kaseorg 0b12d38e4d fixtures: Fix invalid JSON in narrow.json.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-07 11:12:32 -07:00
Anders Kaseorg 0d1cc8c171 test_subs: Remove absurd bot_owner parameter from request.
Seriously now.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-07 11:12:32 -07:00
Anders Kaseorg c9d1f547bd test_auth_backends: Remove absurd user_profile parameter from requests.
What even did you think this was supposed to do.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-07 11:12:32 -07:00
Anders Kaseorg 25c20a21bd event_queue: Convert users argument of send_event to list.
set is not JSON serializable, and orjson enforces this.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-07 11:12:32 -07:00
Anders Kaseorg fe5c524890 user_status: Use strings as dict keys.
JSON keys must be strings, and orjson enforces this.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-07 10:53:06 -07:00
Anders Kaseorg a329b538d2 test_helpers: Fix instrument_url decorator type.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-07 10:53:06 -07:00
Anders Kaseorg b35e4428a9 test_custom_profile_data: Convert order QuerySet to list.
A QuerySet object is not JSON serializable, and orjson enforces this.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-07 10:53:06 -07:00
Anders Kaseorg f8bcf39014 actions: Do not attempt JSON serialization of a Stream object.
It doesn’t end well.  Or sometimes it doesn’t end (OverflowError:
Maximum recursion level reached).

Introduced by commits ccdf52fef6 and
94d2de8b4a (#15601).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-07 10:53:06 -07:00
Steve Howell 15ffd2b666 event_schema: Extract check_stream_delete. 2020-08-06 12:29:43 -07:00
Steve Howell 059e0bb81e event_schema: Extract check_default_streams. 2020-08-06 12:29:43 -07:00
Steve Howell 36ec1571d4 event_schema: Extract check_default_stream_groups. 2020-08-06 12:29:43 -07:00
Mateusz Mandera 15752067dc auth: Treat emails case-insensitively in ExternalAuthResult.
Our intent throughout the codebase is to treat email
case-insensitively.
The only codepath affected by this bug is remote_user_sso, as that's the
only one that currently passes potentially both a user_profile and
ExternalAuthDataDict when creating the ExternalAuthResult. That's why we
add a test specifically for that codepath.
2020-08-05 11:40:35 -07:00
orientor d036a01516 test_events: Convert tuples to arrays for OpenAPI testing.
Use `ujson.loads(ujson.dumps())` wrapper on events sent for OpenAPI
testing so that all tuples are converted into arrays as tuples aren't
valid in JSON.
2020-08-03 16:55:36 -07:00
Clara Dantas be6b2b248f Error reporting emails: Indicate user role.
Adds user role to the user's information in the error reporting
emails, as some bugs are role-dependent.

Fixes: #15344
2020-08-03 13:35:51 -07:00
Clara Dantas b4dd118aa1 refactor: Create a user object in report.
To make it easier to check if there is user information to be used
in the error report emails, we create a user object inside report.
Now, to check if we have the user's full name, email, etc, we just
need to do report['user']['user_full_name'] rather than check
each information one by one, because if the value of one key in
the report is different than None, all the others will be as well.
2020-08-03 13:35:51 -07:00
Mateusz Mandera bb46b21123 find_account: Fix the email search query.
The search should be case-insensitive.
2020-08-02 12:37:20 -07:00
Anders Kaseorg 159641bab8 timestamp: Remove datetime_to_precise_timestamp for datetime.timestamp.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-07-31 22:10:14 -07:00
Clara Dantas ca2123fec1 test_subs: Improve readability of URLs that have request params. 2020-07-30 16:59:51 -07:00
Mohit Gupta 8c04c03408 test_logging_handlers: Use assertLogs to verify error logs.
This avoids spam in test-backend output.
2020-07-30 13:38:26 -07:00
Tim Abbott f8e2c1f693 test_management_commands: Fix export test.
This was broken in the last commit.
2020-07-30 13:26:06 -07:00
Tim Abbott 6130a61be0 export: Only print .s with percent_callback to console.
The S3 data export tool's upload code path uses this nice boto
callback feature for showing a progress bar, which is nice for the
management command.  It's spammy/broken in production and the backend
tests, so we change percent_callback to be a parameter passed in so
that it can only be used in the contexts where it makes sense.
2020-07-30 13:14:53 -07:00
Anders Kaseorg 022c4fbfc7 Revert "digest: Support digest of web public streams for guest users."
This reverts commit c3779338c6 (part
of #14638), which incorrectly depended on commits from the future,
with the effect of either halting the flow of entropic time in an
irresolvable temporal paradox, summoning extradimensional beings to
rain destruction on the galaxy, or failing CI.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-07-29 21:05:59 -07:00
Clara Dantas c3779338c6 digest: Support digest of web public streams for guest users. 2020-07-29 17:52:36 -07:00
Clara Dantas a9af80d7a2 streams: Make /streams endpoint return also web-public streams.
This commit modifies the /streams endpoint so that the web-public
streams are included in the default list of streams that users
have access to.

This is part of PR #14638 that aims to allow guest users to
browse and subscribe themselves to web public streams.
2020-07-29 17:52:36 -07:00
Clara Dantas 0994b029d6 streams: Grant authorization to guest users to subscribe.
Modifies filter_stream_authorization so that web-public streams are
added in the list of authorized streams that a guest user can
subscribe.

This commit is part of PR #14638 that aims to allow guest users
to browse and subscribe to web-public streams.
2020-07-29 17:52:36 -07:00
Clara Dantas e22e12fe60 streams: Grant guest users access to web-public streams.
In this commit, we grant guest users access to stream history,
send message and common stream data of web-public streams.

This is part of PR #14638 that aims to allow guest users to
browse and subscribe to web-public streams.
2020-07-29 17:52:36 -07:00
Clara Dantas 523bb30f33 actions: Modify validate_user_access_to_subscribers.
This modification allows guest users to have access to web-public
streams subscribers, even if they aren't subscribed or never
subscribed to that stream.

This commit is part of PR #14638 that aims to allow guest users to
browser and subscribe to web-public streams.
2020-07-29 17:52:36 -07:00
Clara Dantas 6a1d9480bc actions: Modify gather_subscriptions() to also gather web-public streams.
Now, gather_subscriptions include web-public streams in the 3 sets
of streams that it returns, subscribed, unsubscribed and never
subscribed.

This is part of PR #14638 that aims to allow guest users to browse and
subscribe to web-public streams.
2020-07-29 17:52:36 -07:00
Gittenburg 0706de2305 docs: Make tabbed sections accessible from keyboard.
Part of #15948.
2020-07-29 11:40:39 -07:00