Commit Graph

65 Commits

Author SHA1 Message Date
Anders Kaseorg fead14951c python: Convert assignment type annotations to Python 3.6 style.
This commit was split by tabbott; this piece covers the vast majority
of files in Zulip, but excludes scripts/, tools/, and puppet/ to help
ensure we at least show the right error messages for Xenial systems.

We can likely further refine the remaining pieces with some testing.

Generated by com2ann, with whitespace fixes and various manual fixes
for runtime issues:

-    invoiced_through: Optional[LicenseLedger] = models.ForeignKey(
+    invoiced_through: Optional["LicenseLedger"] = models.ForeignKey(

-_apns_client: Optional[APNsClient] = None
+_apns_client: Optional["APNsClient"] = None

-    notifications_stream: Optional[Stream] = models.ForeignKey('Stream', related_name='+', null=True, blank=True, on_delete=CASCADE)
-    signup_notifications_stream: Optional[Stream] = models.ForeignKey('Stream', related_name='+', null=True, blank=True, on_delete=CASCADE)
+    notifications_stream: Optional["Stream"] = models.ForeignKey('Stream', related_name='+', null=True, blank=True, on_delete=CASCADE)
+    signup_notifications_stream: Optional["Stream"] = models.ForeignKey('Stream', related_name='+', null=True, blank=True, on_delete=CASCADE)

-    author: Optional[UserProfile] = models.ForeignKey('UserProfile', blank=True, null=True, on_delete=CASCADE)
+    author: Optional["UserProfile"] = models.ForeignKey('UserProfile', blank=True, null=True, on_delete=CASCADE)

-    bot_owner: Optional[UserProfile] = models.ForeignKey('self', null=True, on_delete=models.SET_NULL)
+    bot_owner: Optional["UserProfile"] = models.ForeignKey('self', null=True, on_delete=models.SET_NULL)

-    default_sending_stream: Optional[Stream] = models.ForeignKey('zerver.Stream', null=True, related_name='+', on_delete=CASCADE)
-    default_events_register_stream: Optional[Stream] = models.ForeignKey('zerver.Stream', null=True, related_name='+', on_delete=CASCADE)
+    default_sending_stream: Optional["Stream"] = models.ForeignKey('zerver.Stream', null=True, related_name='+', on_delete=CASCADE)
+    default_events_register_stream: Optional["Stream"] = models.ForeignKey('zerver.Stream', null=True, related_name='+', on_delete=CASCADE)

-descriptors_by_handler_id: Dict[int, ClientDescriptor] = {}
+descriptors_by_handler_id: Dict[int, "ClientDescriptor"] = {}

-worker_classes: Dict[str, Type[QueueProcessingWorker]] = {}
-queues: Dict[str, Dict[str, Type[QueueProcessingWorker]]] = {}
+worker_classes: Dict[str, Type["QueueProcessingWorker"]] = {}
+queues: Dict[str, Dict[str, Type["QueueProcessingWorker"]]] = {}

-AUTH_LDAP_REVERSE_EMAIL_SEARCH: Optional[LDAPSearch] = None
+AUTH_LDAP_REVERSE_EMAIL_SEARCH: Optional["LDAPSearch"] = None

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-22 11:02:32 -07:00
Anders Kaseorg f8c95cda51 mypy: Add specific codes to type: ignore annotations.
https://mypy.readthedocs.io/en/stable/error_codes.html

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-22 10:46:33 -07:00
Anders Kaseorg 088f7ee5d6 python: Convert type checks to isinstance checks.
Generated by autopep8 --aggressive, with the setup.cfg configuration
from #14532.  In general, an isinstance check may not be equivalent to
a type check because it includes subtypes; however, that’s usually
what you want.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-21 17:58:09 -07:00
Anders Kaseorg 1cf63eb5bf python: Whitespace fixes from autopep8.
Generated by autopep8, with the setup.cfg configuration from #14532.
I’m not sure why pycodestyle didn’t already flag these.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-21 17:58:09 -07:00
shubhamgupta2956 7c3169a1d9 api_docs: Add "MessageId" common component.
To facilitate re-use of the same parameters in other paths, this commit
store the content of the parameter "message_id" in components.
2020-04-20 17:52:49 -07:00
orientor cfa7724bcc openapi: Use "description: |" for multiline paragraphs.
"description: |" supports markdown and is overall better for
writing multiline paragraphs. So use it in multiline paragraphs
and line-wrap the newly formed paragraphs accordingly.

Edited by tabbott to change most single-line descriptions to use this
format as well.
2020-04-16 20:02:02 -07:00
Anders Kaseorg c734bbd95d python: Modernize legacy Python 2 syntax with pyupgrade.
Generated by `pyupgrade --py3-plus --keep-percent-format` on all our
Python code except `zthumbor` and `zulip-ec2-configure-interfaces`,
followed by manual indentation fixes.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-09 16:43:22 -07:00
shubhamgupta2956 793c3f25e7 api_docs: Migrate POST /zulip-outgoing-webhook.
This commit migrates zulip outging webhook payload to
/zulip-outgoing-webhook:post in OpenAPI.

Since this migrates the last payloads from api/fixtures.json to
OpenAPI, this commit removes api/fixtures.json file and the functions
accessing the file.

Tweaked by tabbott to further remove an unnecessary conditional.
2020-04-02 14:55:32 -07:00
shubhamgupta2956 6ebf408fc4 api_docs: Migrate POST /rest-error-handling.
This commit migrates payload of errors which are common to many endpoints
to /rest-error-handling:post in OpenAPI.
2020-04-02 14:48:29 -07:00
shubhamgupta2956 12474a3deb api docs: Migrate REAL-TIME /events to OpenAPI.
Migrate "call_on_each_event" from api/arguments.json to
/events:real-time in OpenAPI.

This is a bit of a hack, but it lets us eliminate this secondary
arguments.json file, which is probably worth it.

Tweaked by tabbott to fix various formatting issues in the original
documentation while I was looking at it.
2020-03-27 17:43:35 -07:00
Anders Kaseorg 56aadf6503 tests: Fix CI apocalypse perpetrated by previous commit.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-03-26 21:25:25 -07:00
Steve Howell b26f2dcd4b typing: Deprecate emails in typing endpoint.
The only clients that should use the typing
indicators endpoint are our internal clients,
and they should send a JSON-formatted list
of user_ids.

Unfortunately, we still have some older versions
of mobile that still send emails.

In this commit we fix non-user-facing things
like docs and tests to promote the user_ids
interface that has existed since about version
2.0 of the server.

One annoyance is that we documented the
typing endpoint with emails, instead of the
more modern user_ids, which may have delayed
mobile converting to user_ids (and which
certainly caused confusion).  It's trivial
to update the docs, but we need to short
circuit one assertion in the openapi tests.

We also clean up the test structure for the
typing tests:

    TypingHappyPathTest.test_start_to_another_user
    TypingHappyPathTest.test_start_to_multiple_recipients
    TypingHappyPathTest.test_start_to_self
    TypingHappyPathTest.test_start_to_single_recipient
    TypingHappyPathTest.test_stop_to_another_user
    TypingHappyPathTest.test_stop_to_self

    TypingValidateOperatorTest.test_invalid_parameter
    TypingValidateOperatorTest.test_missing_parameter

    TypingValidateUsersTest.test_argument_to_is_not_valid_json
    TypingValidateUsersTest.test_bogus_user_id
    TypingValidateUsersTest.test_empty_array
    TypingValidateUsersTest.test_missing_recipient

    TypingValidationHelpersTest.test_recipient_for_user_ids
    TypingValidationHelpersTest.test_recipient_for_user_ids_non_existent_id

    TypingLegacyMobileSupportTest.test_legacy_email_interface
2020-02-28 12:39:36 -08:00
harshavardhanpb cac4feb263 openapi: Move openapi.py into zerver/openapi.py.
Fixes #14006
2020-02-24 12:21:26 -05:00
akashaviator 7bc470c699 openapi: Add OpenAPI format data for /users/{user_id} endpoint.
This adds the OpenAPI format data for /users/{user_id} endpoint
and also removes 'users/{user_id}' from 'pending_endpoints' in
zerver/tests/test_openapi.py .
2020-02-15 23:08:10 -08:00
rht 41e3db81be dependencies: Upgrade to Django 2.2.10.
Django 2.2.x is the next LTS release after Django 1.11.x; I expect
we'll be on it for a while, as Django 3.x won't have an LTS release
series out for a while.

Because of upstream API changes in Django, this commit includes
several changes beyond requirements and:

* urls: django.urls.resolvers.RegexURLPattern has been replaced by
  django.urls.resolvers.URLPattern; affects OpenAPI code and related
  features which re-parse Django's internals.
  https://code.djangoproject.com/ticket/28593
* test_runner: Change number to suffix. Django changed the name in this
  ticket: https://code.djangoproject.com/ticket/28578
* Delete now-unnecessary SameSite cookie code (it's now the default).
* forms: urlsafe_base64_encode returns string in Django 2.2.
  https://docs.djangoproject.com/en/2.2/ref/utils/#django.utils.http.urlsafe_base64_encode
* upload: Django's File.size property replaces _get_size().
  https://docs.djangoproject.com/en/2.2/_modules/django/core/files/base/
* process_queue: Migrate to new autoreload API.
* test_messages: Add an extra query caused by .refresh_from_db() losing
  the .select_related() on the Realm object.
* session: Sync SessionHostDomainMiddleware with Django 2.2.

There's a lot more we can do to take advantage of the new release;
this is tracked in #11341.

Many changes by Tim Abbott, Umair Waheed, and Mateusz Mandera squashed
are squashed into this commit.

Fixes #10835.
2020-02-13 16:27:26 -08:00
Chris Heald bddb370750 tests: Reorder python version logic to be more clear. 2020-02-11 17:34:56 -08:00
Chris Heald 3236483d0e tests: Fix type reflection for Python 3.7.
In python 3.5-3.6, generic types had an __origin__ attribute which
indicated which generic they originated from; the code was reflecting on
that value to check types against the openapi spec. In python3.7, this
changed, and there's no longer an immediately simple way to get this
information in all cases. __origin__ appears to be the implementing
class now, returning `list` or `collections.abc.Iterator` rather than
`typing.List` and `typing.Iterator`. This adds a sloppy-but-effective
mechanism for inferring if a type maps to the List/Dict/Iterator/Mapping
types and gets the test suite passing again.
2020-02-11 17:34:56 -08:00
Tim Abbott cfe3ba31f5 test_openapi: Add prioritization for list of undocumented endpoints. 2020-01-31 13:52:36 -08:00
Tim Abbott c0712431df openapi: Add hacky support for oneOf parameter types.
This is required for the upcoming type behavior of the "anchor"
parameter.

This change is the minimal work required to have our OpenAPI code not
fail when checking a union-type value of this form.  We'll likely want
to, in the future, do something nicer, but it'd require more extensive
infrastructure for parsing of OpenAPI data that it's worth with our
current approach (we may want to switch to using a library).
2020-01-29 11:24:58 -08:00
Vishnu KS c8ede33fc3 openapi: Specify securityScheme for the API in root level.
We used to specify the securityScheme for each REST operation seperately.
This is unecessary as the securityScheme can be specified in root level
and would be automatically applied to all operations. This also prevents
us accidentally not specifying the securityScheme for some operations and
was the case for /users/me/subscriptions PATCH endpoint. The root level
securityScheme can be also overriden in the operational level when
necessary.

swagger.io/docs/specification/authentication/#security
2019-12-06 11:19:08 -08:00
Anders Kaseorg 40f4ead738 mypy: Upgrade from 0.720 to 0.730.
Fixes #13269.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-11-13 12:38:45 -08:00
Gloria Elston f8855ca179 api: Remove legacy emoji reactions endpoint.
The original/legacy emoji reactions endpoints made use of HTTP PUT and
didn't have an API that could correctly handle situations where the
emoji names change over time.  We stopped using the legacy endpoints
some time ago, so we can remove them now.

This requires straightforward updates to older tests that were still
written against the legacy API.

Fixes #12940.
2019-11-12 13:07:06 -08:00
Mateusz Mandera 29314f3195 api: Remove unused /get_auth_backends endpoint.
This legacy endpoint was designed for the original native Zulip mobile
apps, which were deprecated years ago in favor of the React Native
app.

It was replaced by /server_settings for active use years ago, so it's
safe to remove it now.
2019-11-01 12:30:59 -07:00
Tim Abbott b12d3d54c6 events: Fix documentation testing for /events.
Most of the failures were due to parameters that are not intended to
be used by third-party code, so the correct fix for those was the set
intentionally_undocumented=True.

Fixes #12969.
2019-10-21 16:50:10 -07:00
Tim Abbott c35472e025 test_openapi: Fix confused error messages for missing parameters.
These conditionals were backwards.
2019-10-21 16:26:08 -07:00
Vishnu Ks cb64fc8732 bugdown: Show example value for path params in curl example. 2019-10-15 15:40:44 -07:00
Vishnu KS cd06e0ab79 tests: Remove /messages/{message_id} from buggy endpoints. 2019-10-07 11:45:11 -07:00
Vishnu KS bfc9089124 tests: Remove /users/me/subscriptions/muted_topics from buggy endpoints.
Even though required attribute of stream and stream_id params is marked
false in openapi specification, the API expects atleast one of the
params to be set. There is no way to specify relationships like this
openapi and they dont seem to have any plan to implement this in future.

https://github.com/OAI/OpenAPI-Specification/issues/256
2019-10-07 11:42:52 -07:00
Vishnu KS e7419f815a tests: Remove /settings/notifications from buggy endpoints. 2019-10-07 11:42:52 -07:00
Vishnu Ks 30ad650afb bugdown: Don't show path params as data in curl example.
With test added by tabbott.
2019-10-05 18:17:36 -07:00
Laurel Michaels a67769091d docs: Add OpenAPI data for `/messages/{message_id}/reactions`.
This change adds the OpenAPI data needed to document the POST and
DELETE methods associated with this endpoint.

Descriptions edited slightly by tabbott.
2019-09-24 15:57:51 -07:00
Tim Abbott 3781a0dae2 urls: Reformat how we handle API-only mobile endpoints.
This restructures the API endpoints that we currently have implemented
more or less for exclusive use by the mobile and desktop apps (things
like checking what authentication methods are supported) to use a
system that can be effectively parsed by our test_openapi
documentation.

This brings us close to being able to eliminate
`buggy_documentation_endpoints` as a persistently nonempty list.
2019-08-19 15:34:40 -07:00
Tim Abbott 27bfed7896 openapi: Remove presence from untestable endpoints.
This add some regular expression manipulation hacks to make it
possible for us to validate the documentation for the presence
endpoint with a slightly more complex regular expression capture
group.
2019-08-19 15:17:28 -07:00
Tim Abbott 44db59bbe0 openapi: Fix handling of parameters passed via the URL/path.
Previously, our OpenAPI documentation validation was failing for some
endpoints because it didn't account for the `in: path` type of
parameter, resulting in a mismatch between what was declared via REQ
and what was declared in the OpenAPI docs.

We fix this by excluding the path type parameters in both places from
what's considered by documentation using the `path_only` flag.

I doubt this is the correct long-term fix; in particular, I don't
think we're actually running the validators for these path-only
parameters.  The examples that exist today are all IDs with validators
for being non-negative numbers, but longer-term I think we'll want to
do something different (possibly at the REQ layer, see the TODO).
2019-08-19 15:06:08 -07:00
Tim Abbott 26ac3ebd3e openapi: Suggest the -sS options to curl.
These options prevent curl from doing the downloading progress bar
(which is clutter).
2019-08-17 11:35:08 -07:00
Tim Abbott 2366490ffc openapi: Pass api_url to curl example generation.
Our new curl example generation logic was broken, in that it hardcoded
localhost:9991 (without an HTTP method or anything) as the API URL.

It requires a bit of plumbing to make this possible.
2019-08-17 11:35:08 -07:00
Wyatt Hoodes 896886b412 realm_export: Add endpoint for export deletion. 2019-08-12 17:51:46 -07:00
Anders Kaseorg 1ffbc0009d test_openapi: Clean up type ignores.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-09 17:42:33 -07:00
Anders Kaseorg 7e92ff9d0a request: Delete request.pyi and make request.py type check.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-07 15:37:21 -07:00
Hemanth V. Alluri ac69717604 openapi: Migrate /messages to use the new cURL example system.
Now that we can create cURL examples based on the OpenAPI
documentation. We can begin using simple one line tags in
the documentation instead of manually creating cURL examples.

Fixes part of #12878.
2019-08-05 21:27:24 -07:00
Hemanth V. Alluri 5af753d940 api_code_examples: Parse kwargs/mods for example code generation.
Now we can also include extra keyword arguments to specify
modifications in how the example code should be generated
in the generate_code_example template tag.

E.g. generate_code_example(curl, exclude=["param1", "param2"])
2019-08-05 21:27:23 -07:00
Hemanth V. Alluri b20cf095e7 curl: Add code to auto generate cURL examples from OpenAPI docs.
This commit extends api_code_examples.py to support automatically
generating cURL examples from the OpenAPI documentation. This way
work won't have to be repeated and we can also drastically reduce
the chance of introducing faulty cURL examples (via. an automated
test which can now be easily created).
2019-08-05 21:13:14 -07:00
Hemanth V. Alluri 0682f4f12a openapi: Extend type checking to check the elements of lists.
Now that the type of the elements of each array/list are also
specified in the OpenAPI documentation, we can extend our type
checking even further.
2019-08-05 21:06:19 -07:00
Hemanth V. Alluri 14c0a387cf openapi: Remove /users/me from pending endpoints.
This commit progress our efforts to reduce pending_endpoints
as well as to migrate away from templates/zerver/api/fixtures
and towards our OpenAPI documentation.

Similar to commit d62b75fc.
2019-08-05 21:06:19 -07:00
Hemanth V. Alluri 3528c454c0 openapi: Update tests to ensure that request parameter types match.
This commit add an pretty elaborate extension to the existing
openapi documentation validation test: test_openapi_arguments.

This does a metacode analysis, comparing the openapi documentation
with the appropriate function's declaration, default values etc.

While it has some limitations, it is able to catch various common
classes of mistakes in the types declared for our OpenAPI
documentation.
2019-07-26 12:49:59 -07:00
Hemanth V. Alluri 1b8009837b openapi: Rename variable function to function_name in validation test.
This is a prep commit for adding validation of the request variable
types since then we would need to actually analyze the code of the
actual function itself and we would need a variable storing the
function itself.
2019-07-21 18:46:48 -07:00
Hemanth V. Alluri d62b75fc4c openapi: Remove /users/me/subscriptions from pending_endpoints.
In commit 7c71e98, we added a special exception for the
/users/me/subscriptions endpoint in the automatic validation test.
By adding some extra documentation, we now remove this extra code,
as well as the endpoint from the list of pending endpoints.
2019-07-21 18:39:53 -07:00
Hemanth V. Alluri ff73c45fa3 openapi: Improve exception message for pending_endpoint with docs.
In the validation test, we now use a different message for when there
is an endpoint in pending_endpoints with some documentation already.
This change is a bit hackish, but it's okay since we'll be removing it
once we've resolved all pending endpoints (which is bound to happen).
2019-07-21 18:39:53 -07:00
Tim Abbott c91f659a1d openapi: Fix test output for undocumented parameters.
The output had the error messages swapped.
2019-07-15 13:39:30 -07:00
Tim Abbott 7c71e987ac openapi: Fix endpoints incorrectly tagged as documentation_pending. 2019-07-15 13:39:29 -07:00