Commit Graph

11678 Commits

Author SHA1 Message Date
Anders Kaseorg 7e9db327b3 request: Improve validator type so mypy can check it against REQ.
Old: a validator returns None on success and returns an error string
on error.

New: a validator returns the validated value on success and raises
ValidationError on error.

This allows mypy to catch mismatches between the annotated type of a
REQ parameter and the type that the validator actually validates.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-20 22:29:15 -07:00
Anders Kaseorg 596bfb47c4 papertrail: Replace custom validator with check_dict.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-20 22:29:15 -07:00
Anders Kaseorg 114f859d3a api_travis_webhook: Fix message type annotation.
Not all values of this dict are strings, just the ones we use.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-20 22:29:15 -07:00
Anders Kaseorg c0fa42dd8c complete_zoom_user: Validate state type correctly.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-20 22:29:15 -07:00
Anders Kaseorg 64038163e7 update_user_backend: Validate profile_data type correctly.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-20 22:29:15 -07:00
Anders Kaseorg c584462b32 update_storage: Validate storage type correctly.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-20 22:29:15 -07:00
Anders Kaseorg 717e349834 events_register_backend: Validate client_capabilities type correctly.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-20 22:29:15 -07:00
Anders Kaseorg 475fc5910f update_user_custom_profile_data: Validate data type correctly.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-20 22:29:15 -07:00
Anders Kaseorg 6e33b9c6f2 get_events_backend: Validate narrow type correctly.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-20 22:29:15 -07:00
Tim Abbott d503549f0b retention: Add some more system documentation. 2020-06-20 17:35:07 -07:00
Steve Howell fff2a81536 mypy: Fix payload_url. 2020-06-20 17:09:15 -07:00
Steve Howell 82036741b9 mypy: Fix service_interface 2020-06-20 17:09:15 -07:00
Steve Howell 704c84502b mypy: Fix authentication_methods.
We know the key of the dictionary is a string
here, since we get it via json.
2020-06-20 17:09:15 -07:00
orientor fbf647283b openapi: Fix validate_against_openapi_schema nested object validation.
We had a bug in `validate_against_openapi_schema` that prevented it
from correctly inspecting nested arrays.

Fix the bug and address all the exceptions, either via
EXCLUDE_PROPERTIES or fixing them when simple.  Also add a test case
for nested verification.
2020-06-20 13:23:39 -07:00
Tim Abbott f98d244ed6 models: Change attachment.size to not be nullable.
Attachment objects in production are only created in one place, which
passses a size.  Additionally, I verified in multiple production
environments with old data that this never actually happens (or has
happened).

So we should make the data model correctly reflect the possibilities here.
2020-06-20 13:15:02 -07:00
Steve Howell 22ead64d54 tests: Extract USING_TYPE_STRUCTURE. 2020-06-20 05:22:35 -04:00
Steve Howell cc3d87b078 refactor: Rename check_variable_type.
Rename the validator to check_union, to conform
more to Python typing nomenclature.

And we rename one of the test helpers to the
simpler `check_types`.  (The test helper
was using "variable" in the "var" sense.)
2020-06-20 05:22:35 -04:00
Steve Howell fbe45fa889 test helpers: Tweak common_subscribe_to_streams.
We assert that the post was successful, to give
more immediate feedback for tests that don't
bother to check the return value and may be
implicitly assuming this method just works in
all cases.

And we also make it more convenient for tests
that are happy-path tests--they don't have to
do the assertion themselves.  (And they're still
free to do deeper checks on the json.)

We opt out with allow_fail=True.  We probably want
a more direct API eventually for tests that are
clearly trying to test the failure path for
subscribing to streams.

It's possible that a couple tests here that I added
allow_fail=True to just have flawed data setup--
I don't have time to investigate all cases, but
hopefully they will at least stand out more.
2020-06-19 22:03:12 -07:00
Steve Howell 2e9f689add mypy: Fix annotation for principals. 2020-06-19 21:23:23 -07:00
Tim Abbott a4fdc6f826 api docs: Fix a few misformatted elements. 2020-06-19 17:22:30 -07:00
Tim Abbott 1b620c7e9f openapi: Fix duplicated declarations for GET /messages response.
There were 2 copies of this block.
2020-06-19 15:28:10 -07:00
akashaviator 03325bd615 api docs: Document GET /messages/matches_narrow endpoint.
With extensive tweaks from tabbott to provide clear explanations for
the features.
2020-06-19 15:28:10 -07:00
Tim Abbott 14c0a9a228 messages: Deduplicate non-search code path in messages_in_narrow.
The non-search code path here was simulating the response and escaping
logic from get_search_fields by duplicating what it would do with an
empty set of highlight locations.

We can produce much more readable code by just passing an empty list
of locations in this case.
2020-06-19 15:09:57 -07:00
Steve Howell 00e60c0c91 events: Use stream_id for peer_add/peer_remove.
Two things were broken here:
    * we were using name(s) instead of id(s)
    * we were always sending lists that only
      had one element

Now we just send "stream_id" instead of "subscriptions".

If anything, we should start sending a list of users
instead of a list of streams.  For example, see
the code below:

    if peer_user_ids:
        for new_user_id in new_user_ids:
            event = dict(type="subscription", op="peer_add",
                         stream_id=stream.id,
                         user_id=new_user_id)
            send_event(realm, event, peer_user_ids)

Note that this only affects the webapp, as mobile/ZT
don't use this.
2020-06-19 14:48:42 -07:00
orientor 58b93c3e83 openapi: Show API argument type in docs.
Currently the API docs do not specify whether a given API parameter
is to be specified in `query` or in `path`. Edit the docs so as
to show the type of argument right beside argument name.
2020-06-19 14:42:01 -07:00
orientor 1505f77ca3 openapi: Parse {!api-admin-only.md!} in OpenAPI markdown extension.
Currently, the OpenAPI extension for rendering description in docs
cannot parse {!api-admin-only.md!}. Edit order of markdown extensions
in app_filters.py so that rendering of OpenAPI elements takes place
before substitution of files using `include`.
2020-06-19 11:04:46 -07:00
Tim Abbott a66a8419b9 bankruptcy: Remove broken push notifications loop.
The loop I added here in 5b49839b08 was
ill-conceived.  The critical issue was that despite its name,
do_clear_mobile_push_notifications_for_ids does not immediately clear
push notifications (Except in our test suite, where `send_event`
immediately calls into the queue worker code!).

Instead, it queues work to clear those push notifications.  Which
means that the first user to declare bankruptcy with a large number of
unreads will fill the queue, and then this will just be an infinite
loop adding more work to the queue.
2020-06-19 10:54:14 -07:00
arpit551 c7d0192755 reaction: Fix missing unique constraint on Reactions model.
This fixes a missing unique constraint on the Reactions data model
state when using multiple aliases for an emoji code.  As with any
missing unique constraints, we first need to apply a migration that
eliminates violations of the rule; in this case, deleting the
duplicates is correct.

Added unique constraint for "user_profile", "message",
"reaction_type", "emoji_code".

Fixes #15347.
2020-06-19 10:07:42 -07:00
Steve Howell 7c6ddf90ae mypy: Fix default_language annotation. 2020-06-19 12:35:47 -04:00
Steve Howell 9cde534cf4 mypy: Fix event_types annotation. 2020-06-19 12:35:47 -04:00
Tim Abbott bfc30e4214 api docs: Improve headings and organization for endpoints.
Mostly, this is a change in ordering to make more sense, but we also
fix several names that were clearly confusing.

We restore the convention that each endpoint has the same title at the
top of the page as what we have in the sidebar menu, which appears to
have been violated in many recent updates to API documentation.
2020-06-18 22:38:05 -07:00
orientor d8ef853545 openapi: Make api-docs filename match with operationIds.
api docs filenames are basically the operationId of their endpoint
in zulip.yaml with `_` replaced by `-`. But some operationIds have
changed, so change the affected filenames. Make changes in other
files accordingly.
2020-06-18 22:38:05 -07:00
Hashir Sarwar 5200598a31 events: Don't send avatar URLs of long term idle users.
This adds a new client_capability that clients such as the mobile apps
can use to avoid unreasonable network bandwidth consumed sending
avatar URLs in organizations with 10,000s of users.

Clients don't strictly need this data, as they can always use the
/avatar/{user_id} endpoint to fetch the avatar if desired.

This will be more efficient especially for realms with
10,000+ users because the avatar URLs would increase the
payload size significantly and cost us more bandwidth.

Fixes #15287.
2020-06-18 21:35:16 -07:00
Hashir Sarwar 9911ec3e6d cache: Add `long_term_idle` in `realm_user_dict_fields`.
We need this field to avoid O(N) database operations
while fetching realm user data for clients with
`user_avatar_url_field_optional` flag enabled.

Part of #15287.
2020-06-18 20:23:10 -07:00
Karim Tayie 7233608d4e tests: Extend test_get_accounts_for_email.
This extends get_accounts_for_email test by adding a deactivated
user and assert that get_accounts_for_email doesn't return any accounts
for that deactivated user.

Fixes #14807.
2020-06-18 20:19:08 -07:00
Tim Abbott 0b6ebb4fbb upload: Remove unused get_realm_for_filename. 2020-06-18 17:55:13 -07:00
Tim Abbott 5962d1ea14 upload: Avoid fetching bucket objects repeatedly.
This takes of advantage of saving the bucket object on the
UploadBackend class to deduplicate a bunch of redundant code getting
buckets.
2020-06-18 17:55:13 -07:00
Wyatt Hoodes 2ef791fc21 upload.py: Support using non S3-providers.
With #14378, we regressed back to the state of that
prior to 7e0ea61b00.

We fix this by getting our avatar bucket on
object initialization, and use the appropriate means
of gathering the network location for the urls.

Fixes #14484.
2020-06-18 17:55:13 -07:00
Vishnu KS eb829e6dce realm_logo: Return default logo for realms on LIMITED plan.
This completes our effort to avoid destroying realm logo state while
providing a path to dowgrade from paid plans to LIMITED.
2020-06-18 17:46:26 -07:00
Vishnu KS d0220fc5fc import tests: Pass realm to _setup_export_files.
_setup_export_files modifies the zulip realm. We used to
call realm.refresh_from_db in tests after _setup_export_files was
called to make sure that the change is reflected. But sometimes
calling refresh_from_db was missed out here and there.

This commit makes calling refresh_from_db after _setup_export_files
unnecessary.
2020-06-18 17:45:35 -07:00
Vishnu KS 0a9e114014 import tests: Run test_import_files in billing disabled mode.
We don't really care about the plan specific details
in these tests. It's taken care in test_plan_type.
2020-06-18 17:45:35 -07:00
Tim Abbott dbe9f09dae lint: Fix extra blank lines. 2020-06-18 17:45:35 -07:00
Vishnu KS cde4486f8c billing: Support switching from monthly to annual plan. 2020-06-18 17:34:03 -07:00
Siddharth Varshney bbb07aed38 portico: Add hint text for invite only realm on /login.
Fixes: #10542

This will make obvious for users what to do when they are not able to login.

Test added by hackerkid.
2020-06-18 17:23:11 -07:00
sahil839 c488a35f10 streams: Add API for changing stream-level message_retention_days.
This commit adds backend support for setting message_retention_days
while creating streams and updating it for an existing stream. We only
allow organization owners to set/update it for a stream.

'message_retention_days' field for a stream existed previously also, but
there was no way to set it while creating streams or update it for an
exisiting streams using any endpoint.
2020-06-18 17:00:15 -07:00
Anders Kaseorg 3916ea23a9 python: Combine some split import groups.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-18 15:54:11 -07:00
Anders Kaseorg b2cb992d9f bugdown: Use standard library datetime.timezone.utc.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-18 15:47:20 -07:00
Tim Abbott 3ec068f10c bugdown: Import datetime our normal way. 2020-06-18 14:14:53 -07:00
Rohitt Vashishtha 6ea3816fa6 markdown: Use html5 <time> tag for timestamps.
Previously, we had implemented:
    <span class="timestamp" data-timestamp="unix time">Original text</span>
The new syntax is:
    <time timestamp="ISO 8601 string">Original text</time>
    <span class="timestamp-error">Invalid time format: Original text</span>

Since python and JS interpretations of the ISO format are very
slightly different, we force both of them to drop milliseconds
and use 'Z' instead of '+00:00' to represent that the string is
in UTC. The resultant strings look like: 2011-04-11T10:20:30Z.

Fixes #15431.
2020-06-18 14:11:33 -07:00
Dinesh 0445311430 auth: Make apple log in and sign up buttons consistent with others. 2020-06-18 13:06:10 -07:00