Mirror
/
zulip
mirror of https://github.com/zulip/zulip.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity

api-tests: Reduce error output for `/register` openapi validation. 

For descriptive endpoints, such as `/register`, that might raise
Schema Validation errors via `validate_against_openapi_schema`,
omits the OpenAPI schema definition in the error output.

Also omits the error instance definition in the error output
when it is a jsonschema object with over 100 properties. This
means that the test instance for objects, like user settings,
will be printed in the error output, but the test instance for
the entire endpoint will not be printed to the console.

The omitted output can be thousands of lines long making it
difficult to find the initial console output that actually helps
the contributor with debugging.

Adds a section in "Documenting REST API endpoints" about
debugging and understanding these errors that is linked to
in the error console output.
This commit is contained in:
Lauryn Menard 2022-04-21 17:55:05 +02:00 committed by Tim Abbott
parent e80e3b07c4
commit c2bcfb52aa
3 changed files with 200 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

183
docs/documentation/api.md
View File

@ -335,3 +335,186 @@ it? There's several major benefits to this system:
Using the standard OpenAPI format gives us flexibility, though; if we
later choose to migrate to third-party tools, we don't need to redo
the actual documentation work in order to migrate tools.
## Debugging schema validation errors
A common function used to validate and test Zulip's REST API is
`validate_against_openapi_schema`. It is used to verify that every
successful API response returned in the backend and documentation test
suites are a documented possibility in the API documentation.
Therefore, when you add a new feature or setting to Zulip, you will most
likely need to update the API documentation (`zerver/openapi/zulip.yaml`)
in order to pass existing tests that use this function. Additionally, if
you're writing documentation for a new or undocumented REST API endpoint,
you'll want to use this function to validate and test your changes in
`zerver/openapi/python_examples.py`.
Below are some examples to help you when debugging the schema validation
errors produced by `validate_against_openapi_schema`. Before reading
through the examples, we recommend reviewing the
[OpenAPI configuration](openapi.md) documentation if you're unfamiliar
with the format.
If you use Visual Studio Code, an OpenAPI extension can be very helpful in
navigating Zulip's large and detailed OpenAPI file; see
`.vscode/extensions.json`.
### Deconstructing the error output
To start with a clear example, let's imagine that we are writing the
documentation for the REST API endpoint for uploading a file,
[POST /api/v1/user_uploads](https://zulip.com/api/upload-file).
There are no parameters for this endpoint, and only one return value
specific to this endpoint, `uri`, which is the URL of the uploaded file.
If we comment out that return value and example from the existing API
documentation in `zerver/openapi/zulip.yaml`, e.g.:
```yaml
/user_uploads:
post:
operationId: upload-file
...
responses:
"200":
description: Success.
content:
application/json:
schema:
allOf:
- $ref: "#/components/schemas/JsonSuccessBase"
- $ref: "#/components/schemas/SuccessDescription"
- additionalProperties: false
properties:
result: {}
msg: {}
# uri:
# type: string
# description: |
# The URI of the uploaded file.
example:
{
"msg": "",
"result": "success",
# "uri": "/user_uploads/1/4e/m2A3MSqFnWRLUf9SaPzQ0Up_/zulip.txt",
}
```
We will now get an error when we run the API documentation test suite
in the development environment (`tools/test-api`):
```console
Running API tests...
2022-12-19 15:05:42.347 WARN [django.server] "POST /api/v1/users HTTP/1.1" 400 88
Waiting for children to stop...
Traceback (most recent call last):
File "tools/test-api", line 93, in <module>
test_the_api(client, nonadmin_client, owner_client)
File "/srv/zulip/zerver/openapi/python_examples.py", line 1636, in test_the_api
test_users(client, owner_client)
File "/srv/zulip/zerver/openapi/python_examples.py", line 1555, in test_users
upload_file(client)
File "/srv/zulip/zerver/openapi/python_examples.py", line 52, in _record_calls_wrapper
return test_func(*args, **kwargs)
File "/srv/zulip/zerver/openapi/python_examples.py", line 1284, in upload_file
validate_against_openapi_schema(result, "/user_uploads", "post", "200")
File "/srv/zulip/zerver/openapi/openapi.py", line 489, in validate_against_openapi_schema
raise SchemaError(message) from None
zerver.openapi.openapi.SchemaError: 1 response validation error(s) at post /api/v1/user_uploads (200):
ValidationError: Additional properties are not allowed ('uri' was unexpected)
Failed validating 'additionalProperties' in schema['allOf'][2]:
{'additionalProperties': False,
'example': {'msg': '',
'result': 'success',
'properties': {'msg': {}, 'result': {}}}
On instance:
{'msg': '',
'result': 'success',
'uri': '/user_uploads/2/85/XoqF0K7XEOLVGylgdpof80RB/img.jpg'}
```
We can see in the traceback that a `SchemaError` was raised in
`validate_against_openapi_schema`:
```console
File "/srv/zulip/zerver/openapi/openapi.py", line 478, in validate_against_openapi_schema
raise SchemaError(message) from None
```
The next line in the output, let's us know how many errors were found
and for what endpoint.
```console
zerver.openapi.openapi.SchemaError: 1 response validation error(s) at post /api/v1/user_uploads (200):
```
As expected from commenting out the code above, there was one validation
error for the `POST /api/v1/user_uploads` endpoint. The next line gives
more information about that error.
```console
ValidationError: Additional properties are not allowed ('uri' was unexpected)
```
We see that there was a `uri` value returned by the endpoint that hasn't
been documented. The next few lines of output, show us what return values
are documented (again due to our changes) for this endpoint.
```console
Failed validating 'additionalProperties' in schema['allOf'][2]:
{'additionalProperties': False,
'example': {'msg': '',
'result': 'success',
'properties': {'msg': {}, 'result': {}}}
```
And finally, we see the test instance that did not match our current
documentation, which includes the `uri` return value.
```console
On instance:
{'msg': '',
'result': 'success',
'uri': '/user_uploads/2/85/XoqF0K7XEOLVGylgdpof80RB/img.jpg'}
```
This is a useful example because the endpoint's documentation is short
and straightforward, helping to easily identify the parts of the
error output that are useful in debugging these errors when testing the
API documentation.
### Adding a realm setting
Building on [the new feature tutorial](../tutorials/new-feature-tutorial.md)
example, if the realm setting for `mandatory_topics` was not documented
in the `POST /api/v1/register` endpoint, running `tools/test-api` in the
development environment would result in this error:
```console
...
zerver.openapi.openapi.SchemaError: 1 response validation error(s) at post /api/v1/register (200):
ValidationError: Additional properties are not allowed ('realm_mandatory_topics' was unexpected)
Failed validating 'additionalProperties' in schema['allOf'][2]:
'OpenAPI schema omitted due to length of output.'
On instance:
'Error instance omitted due to length of output.'
```
Because this endpoint is very long and descriptive, we do not print the
entire documentation schema (or test instance, in this case) to the
console. Doing so would print thousands of lines of output that are not
useful for debugging what is missing from the API documentation.
The key information for debugging this endpoint is in the line beginning
with `ValidationError`. There we can see that the documentation does not
include the new `realm_mandatory_topics` boolean that we added in the
example feature tutorial, and we can look at other similar realm settings
to add the documentation for that new feature.

6
docs/tutorials/new-feature-tutorial.md
View File

@ -517,6 +517,12 @@ the setting enabled).
Visit Zulip's [Django testing](../testing/testing-with-django.md)
documentation to learn more about the backend testing framework.
Also note that you may already need to update the API documentation for
your new feature to pass new or existing backend tests at this point.
The tutorial for [writing REST API endpoints](../documentation/api.md)
can be a helpful resource, especially the section on [debugging schema
validation errors](../documentation//api.md#debugging-schema-validation-errors).
### Update the frontend
After completing the process of adding a new feature on the backend,

11
zerver/openapi/openapi.py
View File

@ -474,7 +474,18 @@ def validate_against_openapi_schema(
validator_value=brief_error_validator_value,
cause=error.cause,
)
# Some endpoints have long, descriptive OpenAPI schemas
# which, when printed to the console, do not assist with
# debugging, so we omit some of the error information.
if path in ["/register"] and isinstance(error, JsonSchemaValidationError):
error.schema = "OpenAPI schema omitted due to length of output."
if len(error.instance) > 100:
error.instance = "Error instance omitted due to length of output."
message += f"\n\n{type(error).__name__}: {error}"
message += (
"\n\nFor help debugging these errors see: "
"https://zulip.readthedocs.io/en/latest/documentation/api.html#debugging-schema-validation-errors"
)
raise SchemaError(message) from None
return True