We had a bug in `validate_against_openapi_schema` that prevented it
from correctly inspecting nested arrays.
Fix the bug and address all the exceptions, either via
EXCLUDE_PROPERTIES or fixing them when simple. Also add a test case
for nested verification.
Currently the API docs do not specify whether a given API parameter
is to be specified in `query` or in `path`. Edit the docs so as
to show the type of argument right beside argument name.
Currently, the OpenAPI extension for rendering description in docs
cannot parse {!api-admin-only.md!}. Edit order of markdown extensions
in app_filters.py so that rendering of OpenAPI elements takes place
before substitution of files using `include`.
Mostly, this is a change in ordering to make more sense, but we also
fix several names that were clearly confusing.
We restore the convention that each endpoint has the same title at the
top of the page as what we have in the sidebar menu, which appears to
have been violated in many recent updates to API documentation.
api docs filenames are basically the operationId of their endpoint
in zulip.yaml with `_` replaced by `-`. But some operationIds have
changed, so change the affected filenames. Make changes in other
files accordingly.
This adds a new client_capability that clients such as the mobile apps
can use to avoid unreasonable network bandwidth consumed sending
avatar URLs in organizations with 10,000s of users.
Clients don't strictly need this data, as they can always use the
/avatar/{user_id} endpoint to fetch the avatar if desired.
This will be more efficient especially for realms with
10,000+ users because the avatar URLs would increase the
payload size significantly and cost us more bandwidth.
Fixes#15287.
This commit adds backend support for setting message_retention_days
while creating streams and updating it for an existing stream. We only
allow organization owners to set/update it for a stream.
'message_retention_days' field for a stream existed previously also, but
there was no way to set it while creating streams or update it for an
exisiting streams using any endpoint.
The term `parameter` is a better word than `argument` for data passed
to an API endpoint; this is why OpenAPI uses in their terminology.
Replace `argument` with `parameter` in the API docs to improve their
readability.
Fixes#15435.
This commit removes is_old_stream property from the stream objects
returned by the API. This property was unnecessary and is essentially
equivalent to 'stream_weekly_traffic != null'.
We compute sub.is_old_stream in stream_data.update_calculated_fields
in frontend code and it is used to check whether we have a non-null
stream_weekly_traffic or not.
Fixes#15181.
This is designed to have no user-facing change unless the client
declares bulk_message_deletion in its client_capabilities.
Clients that do so will receive a single bulk event for bulk deletions
of messages within a single conversation (topic or PM thread).
Backend implementation of #15285.
This adds a powerful end-to-end test for Zulip's API documentation:
For every documented API endpoint (with a few declared exceptions that
we hope to remove), we verify that every API response received by our
extensive backend test suite matches the declared schema.
This is a critical step towards being able to have complete, high
quality API documentation.
Fixes#15340.
Generated by pyupgrade --py36-plus --keep-percent-format.
Now including %d, %i, %u, and multi-line strings.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
The narrow parameter was incorrectly documented as a one-level array
rather than an array of arrays, and the tests incorrectly expected
this due to a combination of design and implementation bugs.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Use read-only types (List ↦ Sequence, Dict ↦ Mapping, Set ↦
AbstractSet) to guard against accidental mutation of the default
value.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
There seems to have been a confusion between two different uses of the
word “optional”:
• An optional parameter may be omitted and replaced with a default
value.
• An Optional type has None as a possible value.
Sometimes an optional parameter has a default value of None, or None
is otherwise a meaningful value to provide, in which case it makes
sense for the optional parameter to have an Optional type. But in
other cases, optional parameters should not have Optional type. Fix
them.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Fixes#2665.
Regenerated by tabbott with `lint --fix` after a rebase and change in
parameters.
Note from tabbott: In a few cases, this converts technical debt in the
form of unsorted imports into different technical debt in the form of
our largest files having very long, ugly import sequences at the
start. I expect this change will increase pressure for us to split
those files, which isn't a bad thing.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Automatically generated by the following script, based on the output
of lint with flake8-comma:
import re
import sys
last_filename = None
last_row = None
lines = []
for msg in sys.stdin:
m = re.match(
r"\x1b\[35mflake8 \|\x1b\[0m \x1b\[1;31m(.+):(\d+):(\d+): (\w+)", msg
)
if m:
filename, row_str, col_str, err = m.groups()
row, col = int(row_str), int(col_str)
if filename == last_filename:
assert last_row != row
else:
if last_filename is not None:
with open(last_filename, "w") as f:
f.writelines(lines)
with open(filename) as f:
lines = f.readlines()
last_filename = filename
last_row = row
line = lines[row - 1]
if err in ["C812", "C815"]:
lines[row - 1] = line[: col - 1] + "," + line[col - 1 :]
elif err in ["C819"]:
assert line[col - 2] == ","
lines[row - 1] = line[: col - 2] + line[col - 1 :].lstrip(" ")
if last_filename is not None:
with open(last_filename, "w") as f:
f.writelines(lines)
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
This new endpoint returns a 'user' dictionary which, as of now,
contains a single key 'is_subscribed' with a boolean value that
represents whether the user with the given 'user_id' is subscribed
to the stream with the given 'stream_id'.
Fixes#14966.
The only clients that should use the typing
indicators endpoint are our internal clients,
and they should send a JSON-formatted list
of user_ids.
We now enforce this, which removes some
complexity surrounding legacy ways of sending
users, such as emails and comma-delimited
strings of user_ids.
There may be a very tiny number of mobile
clients that still use the old emails API.
This won't have any user-facing effect on
the mobile users themselves, but if you type
a message to your friend on an old mobile
app, the friend will no longer see typing
indicators.
Also, the mobile team may see some errors
in their Sentry logs from the server rejecting
posts from the old mobile clients.
The error messages we report here are a bit
more generic, since we now just use REQ
to do validation with this code:
validator=check_list(check_int)
This also allows us to remove a test hack
related to the API documentation. (We changed
the docs to reflect the modern API in an
earlier commit, but the tests couldn't be
fixed while we still had the more complex
semantics for the "to" parameter.)
Currently there are no checks in validate_against_openapi_schema
to check whether the `content` it received actually matched with
the `response code`. For example during testing if a certain endpoint
was returning 400 but it was expected to return 200, then it would
pass schema validation as it would only have `msg` and `result` keys.
Add this validation and fix the wrong response returning points.
Add test to validate example responses in zulip.yaml. Also change
zulip.yaml for some wrong examples or for cases which were not
covered by `test-api`. Also enhance `validate_against_openapi_schema`.
This implementation overrides some of PSA's internal backend
functions to handle `state` value with redis as the standard
way doesn't work because of apple sending required details
in the form of POST request.
Includes a mixin test class that'll be useful for testing
Native auth flow.
Thanks to Mateusz Mandera for the idea of using redis and
other important work on this.
Documentation rewritten by tabbott.
Co-authored-by: Mateusz Mandera <mateusz.mandera@zulip.com>
We're migrating to using the cleaner zulip.com domain, which involves
changing all of our links from ReadTheDocs and other places to point
to the cleaner URL.
This commit removes short_name and client_id fields from the user
objects returned by get_profile_backend because neither of them
had a purpose.
* short_name hasn't been present anywhere else in the Zulip API for
several years, and isn't set through any coherent algorithm.
* client_id was a forgotten 2013-era predecessor to the queue_id field
returned by the register_event_queue process.
The combination of these changes gets us close to having `get_profile`
have the exact same format as other endpoints fetching a user object.
This commit changes get_profile_backend to be based on format_user_row
such that it's a superset of the fields for our other endpoints for
getting data on a user.
To be clear, this does not removes any of the exisiting fields, that
were returned by this endpoint.
This change adds some fields to the User object returned by the
endpoint. API docs are updated accordingly for the added fields.
Generated by pyupgrade --py36-plus --keep-percent-format, but with the
NamedTuple changes reverted (see commit
ba7906a3c6, #15132).
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Option to disable breadcrumb messages were given in both message edit
form and topic edit stream popover.
User now has the option to select which stream to send the notification
of stream edit of a topic via checkboxes in the UI.
Objects whose properties are not described were validated by the
current validator. Edit it so that objects with no `properties`
or `additionalProperties` attribute i.e. opaque objects get
invalidated.
Also make changes in zulip.yaml to fix any opaque objects (tweaked by
tabbott to edit the documentation for better clarity).
Currently, `validate_against_openapi_schema` checks only the top
level of the response dictionary. Improve it so that it can
validate objects and arrays at all levels. Also edit zulip.yaml
accordingly. And for new response keys which were not defined
before add VERY basic documentation.
This commit modifies the backend to accept user ids when subscribing
users to streams.
It also migrates all existing tests to use this API, aside from a
small set of tests for the legacy API.
There's no reason to send data beyond the user `id` of the uploader,
and reason not to, as the previous model was both awkward when
`author=None` and resulted in unecessary parsing complexity for
clients.
Modified by tabbott to add the frontend changes and API documentation.
Fixes#15115.
Currently response return values have to be written twice, once in
the docs and once in zulip.yaml. Create a markdown extension so
that the return values in api docs are rendered using content from
zulip.yaml
There are several places where the return values description is better
in the api docs than in the zulip.yaml file. Import such changes to
zulip.yaml before we replace the current hacky API docs with the
zulip.yaml content.
This commit changes the update user API endpoint to accept role
as parameter instead of the bool parameters is_guest and is_admin.
User role dropdown in user info modal is also modified to use
"dropdown_options_widget".
Modified by tabbott to document the API change.
The swagger validator is a basic tool to check whether our
openapi specification file follows the basic syntax. But to ensure
that our zulip.yaml file is not only syntactically compatible but
also describes our API well, we need to add custom tests. This
commit currently checks whether each endpoint has an `operationId`
and a valid tag. It also makes it easier to check for custom rules
in the future.
This refactors render_javascript_code_example to avoid shelling out to
node and parse the javascript file with python instead, to get example
code snippets.
This commit adds python code to call javascript_examples.js in its
two supported modes. tools/test-api asserts that the example output
is as expected, whereas the API markdown extension is used to render
these examples in the docs.
This refactors `extract_code_example` to return a nested list
of code snippets between '{code_example|start/end}' instead of
returing a list of all the lines between '{code_example|start/end}'
markers in the code examples.
Appropriate changes have been made to render_python_code_example.
This refactors `ExamplesHandler` to avoid running examples in a loop
and add result objects to `response_data` array one by one with
`generate_validation_data`.
This file will act as the container for all JS API examples to use
in our documentation, similar to our python and curl API testing
and examples generation code.
This module has two modes of operation:
- node javascript_examples.js generate-responses
This mode runs all the examples against a server and prints the JSON
output of all the examples we ran.
- node javascript_examples.js generate-example <endpoint>
This mode prints example code for endpoints like: /users:post. We then
want to render this full example code in our docs.
'tags' attribute is helpful in differentiating and grouping the
endpoints on basis of their usage. For example tags like 'messages'
help in grouping all endpoints related to messages and thus make the
api specification more user-friendly. So give tags to the endpoints
on the basis of what heading they are under in the API docs.
'operationId' helps code generators in naming functions and other purposes.
So name operationId of endpoints as their function names in python-zulip-api
if it exists else use most appropriate function name.
Part of #14100 .
Zulip's openapi specification in zulip.yaml has various examples
for various schemas. Validate the example with their respective
schemas to ensure that all the examples are schematically correct.
Part of #14100.
Some examples mentioned in zulip.yaml did not match their schema.
Change either the schema or the example so that all examples are
valid with respect to their schemas.
Previously api_description and api_code_examples were two independent
markdown extensions for displaying OpenAPI content used in the same
places. We combine them into a single markdown extension (with two
processors) and move them to the openapi folder to make the codebase
more readable and better group the openapi code in the same place.
To facilitate re-use of the same parameters in other paths, this commit
store the content of the parameter "include_custom_profile_fields" in
components.
To facilitate re-use of the same parameters in other paths, this commit
store the content of the parameter "history_public_to_subscribers" in
components.
For privacy-minded folks who don't want to leak the
information of whether they're online, this adds an
option to disable sending presence updates to other
users.
The new settings lies in the "Other notification
settings" section of the "Notification settings"
page, under a "Presence" subheading.
Closes#14798.
I imagine this can be improved in various ways, but I've initialized
this with all the **Changes** entries recorded in either zulip.yaml or
the rest of the API documentation, and I expect we'll be able to
iterate on this effectively.
It'll also be useful as a record of changes that we should remember to
document the API documentation as we document more endpoints that
currently don't discuss these issues.
While working on this, I fixed various issues where feature levels
could be mentioned or endpoints didn't properly document changes.
Firstly, change endpoint descriptions in zulip.yaml so that they
match their counterpart in the api docs. Then edit the api docs
so that they use api description markdown extension for displaying
endpoint description.
Add function in openapi.py to access endpoint descriptions written
in zulip.yaml. Use this function for creating a markdown extension
for rendering endpoint descriptions written in zulip.yaml.
We use this extension for a single endpoint to get test coverage.
Includes this change:
* openapi/python_examples: Update get_single_user.
This updates get_single_user to pass keyword arguments to
get_user_by_id instead of passing a dictionary.
Which is required for CI to pass, as we indeed fixed the API of that
function (which had only been present with the wrong API for one release).
The purpose is to provide a way for (non-webapp) clients,
like the mobile and terminal apps, to tell whether the
server it's talking to is new enough to support a given
API feature -- in particular a way that
* is finer-grained than release numbers, so that for
features developed after e.g. 2.1.0 we can use them
immediately on servers deployed from master (like
chat.zulip.org and zulipchat.com) without waiting the
months until a 2.2 release;
* is reliable, unlike e.g. looking at the number of
commits since a release;
* doesn't lead to a growing bag of named feature flags
which the server has to go on sending forever.
Tweaked by tabbott to extend the documentation.
Closes#14618.
Openapi had descriptive response codes for endpoints with multiple
responses for same response code. But this does not fall in line
with openapi specifications. So change descriptive response codes
like "400_auth" and "400_anauth" to "400_0" and "400_1" for all
such endpoints. Also make the necessary changes in openapi.py so
as to be able to read the schema in such cases and generate example
in such cases.
zulip.yaml is not in compliance with openapi specifications file.
Edit it so that it passes verification as an openapi specification
file.
Fixes#14582 .
This is be useful for the mobile and desktop apps to hand an uploaded
file off to the system browser so that it can render PDFs (Etc.).
The S3 backend implementation is simple; for the local upload backend,
we use Django's signing feature to simulate the same sort of 60-second
lifetime token.
Co-Author-By: Mateusz Mandera <mateusz.mandera@protonmail.com>
"description: |" supports markdown and is overall better for
writing multiline paragraphs. So use it in multiline paragraphs
and line-wrap the newly formed paragraphs accordingly.
Edited by tabbott to change most single-line descriptions to use this
format as well.
This commit moves /rest-error-handling examples to components section so
that they can be re-used in individual endpoints where it's example can
be highlighted more easiy.
Generated by `pyupgrade --py3-plus --keep-percent-format` on all our
Python code except `zthumbor` and `zulip-ec2-configure-interfaces`,
followed by manual indentation fixes.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
This commit reuses the existing infrastructure for moving a topic
within a stream to add support for moving topics from one stream to
another.
Split from the original full-feature commit so that we can merge just
the backend, which is finished, at this time.
This is a large part of #6427.
The feature is incomplete, in that we don't have real-time update of
the frontend to handle the event, documentation, etc., but this commit
is a good mergable checkpoint that we can do further work on top of.
We also still ideally would have a test_events test for the backend,
but I'm willing to leave that for follow-up work.
This appears to have switched to tabbott as the author during commit
squashing sometime ago, but this commit is certainly:
Co-Authored-By: Wbert Adrián Castro Vera <wbertc@gmail.com>
This commit migrates zulip outging webhook payload to
/zulip-outgoing-webhook:post in OpenAPI.
Since this migrates the last payloads from api/fixtures.json to
OpenAPI, this commit removes api/fixtures.json file and the functions
accessing the file.
Tweaked by tabbott to further remove an unnecessary conditional.
In zulip.yaml simple json success response which only contains 'msg'
and 'result' properties has been described repeatedly in multiple
endpoints. Instead, use SimpleSuccess template for such responses
to increase code modularity and reusablility.
Migrate "call_on_each_event" from api/arguments.json to
/events:real-time in OpenAPI.
This is a bit of a hack, but it lets us eliminate this secondary
arguments.json file, which is probably worth it.
Tweaked by tabbott to fix various formatting issues in the original
documentation while I was looking at it.
Most part of "/message/{message_id}" is migrated to OpenAPI. This commit
migrated the remaning payload "update-message-edit-permission-error"
from "api/fixtures.json" to OpenAPI. This commit also fixes an error
schema in "zulip.yaml" for this payload.
This refactors get_single_user to use get_user_by_id instead of
call_endpoint. Doing so is only possible now that we've upgraded
python-zulip-api to a version with the new function.
The previous system for documenting arguments was very ugly if any of
the examples or descriptions were wrong. After thinking about this
for a while, I concluded the core problem was that a table was the
wrong design element to use for API parameters, and we'd be much
better off with individual card-type widgets instead.
This rewrites the API arguments documentation implementation to use a
basic sort of card-like system with some basic styling; I think the
result is a lot more readable, and it's a lot more clear how we would
add additional OpenAPI details (like parameter types) to the
documentation.
This refactors remove_reaction in python_examples.py to validate the
result with validate_against_openapi_schema. Minor changes and some
additions have been made to the OpenAPI format data for
/messages/{message_id}/reactions endpoint.
This refactors add_reaction in python_examples.py to use the
openapi_test_function decorator and validate result with
validate_against_openapi_schema. Minor changes have been made to the
OpenAPI format data for /messages/{message_id}/reactions endpoint.
This also adds add-emoji.md to templates/zerver/api and adds
add-emoji to rest-endpoints.md (templates/zerver/help/include).
This refactors get_members_backend to return user data of a single
user in the form of a dictionary (earlier being a list with a single
dictionary).
This also refactors it to return the data with an appropriate key
(inside a dictionary), "user" or "members", according to the type of
data being returned.
Tweaked by tabbott to use somewhat less opaque code and simple OpenAPI
descriptions.
This commit mostly makes our tests less
noisy, since emails are no longer an important
detail of sending messages (they're not even
really used in the API).
It also sets us up to have more scrutiny
on delivery_email/email in the future
for things that actually matter. (This is
a prep commit for something along those
lines, kind of hard to explain the full
plan.)
The only clients that should use the typing
indicators endpoint are our internal clients,
and they should send a JSON-formatted list
of user_ids.
Unfortunately, we still have some older versions
of mobile that still send emails.
In this commit we fix non-user-facing things
like docs and tests to promote the user_ids
interface that has existed since about version
2.0 of the server.
One annoyance is that we documented the
typing endpoint with emails, instead of the
more modern user_ids, which may have delayed
mobile converting to user_ids (and which
certainly caused confusion). It's trivial
to update the docs, but we need to short
circuit one assertion in the openapi tests.
We also clean up the test structure for the
typing tests:
TypingHappyPathTest.test_start_to_another_user
TypingHappyPathTest.test_start_to_multiple_recipients
TypingHappyPathTest.test_start_to_self
TypingHappyPathTest.test_start_to_single_recipient
TypingHappyPathTest.test_stop_to_another_user
TypingHappyPathTest.test_stop_to_self
TypingValidateOperatorTest.test_invalid_parameter
TypingValidateOperatorTest.test_missing_parameter
TypingValidateUsersTest.test_argument_to_is_not_valid_json
TypingValidateUsersTest.test_bogus_user_id
TypingValidateUsersTest.test_empty_array
TypingValidateUsersTest.test_missing_recipient
TypingValidationHelpersTest.test_recipient_for_user_ids
TypingValidationHelpersTest.test_recipient_for_user_ids_non_existent_id
TypingLegacyMobileSupportTest.test_legacy_email_interface
This adds update_user to python_examples.py in zerver/openapi.
This also adds update-user.md to templates/zerver/api and adds
update-user to rest-endpoints.md (templates/zerver/help/include).
This adds deactivate_user to python_examples.py in zerver/openapi.
This also adds delete-user.md to templates/zerver/api and adds
delete-user to rest-endpoints.md (templates/zerver/help/include).
This adds get_single_user to python_examples.py in zerver/openapi.
This also adds get-single-user.md to templates/zerver/api and adds
get-single-user to rest-endpoints.md (templates/zerver/help/include).
This adds the OpenAPI format data for /users/{user_id} endpoint
and also removes 'users/{user_id}' from 'pending_endpoints' in
zerver/tests/test_openapi.py .
This commit includes a new `stream_post_policy` setting,
by replacing the `is_announcement_only` field from the Stream model,
which is done by mirroring the structure of the existing
`create_stream_policy`.
It includes the necessary schema and database migrations to migrate
the is_announcement_only boolean field to stream_post_policy,
a smallPositiveInteger field similar to many other settings.
This change is done to allow organization administrators to restrict
new members from creating and posting to a stream. However, this does
not affect admins who are new members.
With many tweaks by tabbott to documentation under /help, etc.
Fixes#13616.
This flag affects page_params and the
payload you get back from POSTs to this
url:
users/me/presence
The flag does not yet affect the
presence events that get sent to a
client.
A wart that has long been present inin Zulip's get_messages API is how
to request "the latest messages" in the API. Previously, the
recommendation was basically to pass anchor=10000000000000000 (for an
appropriately huge number). An accident of the server's implementation
meant that specific number of 0s was actually important to avoid a
buggy (or at least wasteful) value of found_newest=False if the query
had specified num_after=0 (since we didn't check).
This was the cause of the mobile issue
https://github.com/zulip/zulip-mobile/issues/3654.
The solution is to allow passing a special value of anchor='newest',
basically a special string-type value that the server can interpret as
meaning the user precisely just wants the most recent messages. We
also add an analogous anchor='oldest' or similar to avoid folks
needing to write a somewhat ugly anchor=0 for fetching the very first
messages.
We may want to also replace the use_first_unread_anchor argument to be
a "first_unread" value for the anchor parameter.
While it's not always ideal to make a value have a variable type like
this, in this case it seems like a really clean way to express the
idea of what the user is asking for in the API.
orientor
Tim Abbott
akashaviator
Hashir Sarwar
sahil839
Aman Agrawal
Anders Kaseorg
Anders Kaseorg
Steve Howell
Dinesh
sahil9001
Rohitt Vashishtha
Tim Abbott
shubhamgupta2956
Eeshan Garg
Stefan Weil
Vishnu KS
harshavardhanpb
Ryan Rehman