openapi: Add markdown extension for rendering return values in API docs.

Currently response return values have to be written twice, once in
the docs and once in zulip.yaml. Create a markdown extension so
that the return values in api docs are rendered using content from
zulip.yaml

templates/zerver/api/add-linkifiers.md

@@ -23,7 +23,7 @@
 
 #### Return values
 
-* `id`: The numeric ID assigned to this filter.
+{generate_return_values_table|zulip.yaml|/realm/filters:post}
 
 #### Example response
 

templates/zerver/api/add-subscriptions.md

@@ -63,16 +63,7 @@ the `principals` argument, like so:
 
 #### Return values
 
-* `subscribed`: A dictionary where the key is the email address of
+{generate_return_values_table|zulip.yaml|/users/me/subscriptions:post}
-  the user/bot and the value is a list of the names of the streams
-  that were subscribed to as a result of the query.
-
-* `already_subscribed`: A dictionary where the key is the email address of
-  the user/bot and the value is a list of the names of the streams
-  that the user/bot is already subscribed to.
-
-* `unauthorized`: A list of names of streams that the requesting user/bot
-  was not authorized to subscribe to.
 
 #### Example response
 

templates/zerver/api/dev-fetch-api-key.md

@@ -19,9 +19,7 @@
 
 #### Return values
 
-* `api_key`: The API key that can be used to authenticate as the requested
+{generate_return_values_table|zulip.yaml|/dev_fetch_api_key:post}
-    user.
-* `email`: The email address of the user who owns the API key.
 
 #### Example response
 

templates/zerver/api/get-all-streams.md

@@ -48,11 +48,7 @@ as URL query parameters, like so:
 
 #### Return values
 
-* `stream_id`: The unique ID of a stream.
+{generate_return_values_table|zulip.yaml|/streams:get}
-* `name`: The name of a stream.
-* `description`: A short description of a stream.
-* `invite-only`: Specifies whether a stream is private or not.
-  Only people who have been invited can access a private stream.
 
 #### Example response
 

templates/zerver/api/get-all-users.md

@@ -51,26 +51,7 @@ You may pass the `client_gravatar` query parameter as follows:
 
 #### Return values
 
-* `members`: A list of dictionaries, each containing the details for
+{generate_return_values_table|zulip.yaml|/users:get}
-  one of the users in the organization.
-    * `email`: The email address of the user or bot.
-    * `is_bot`: A boolean specifying whether the user is a bot or not.
-    * `avatar_url`: URL to the user's gravatar. `None` if the `client_gravatar`
-      query parameter was set to `True`.
-    * `full_name`: Full name of the user or bot.
-    * `is_admin`: A boolean specifying whether the user is an admin or not.
-    * `bot_type`: `None` if the user isn't a bot. `1` for a `Generic` bot.
-      `2` for an `Incoming webhook` bot. `3` for an `Outgoing webhook` bot.
-      `4` for an `Embedded` bot.
-    * `user_id`: The ID of the user.
-    * `bot_owner_id`: If the user is a bot (i.e. `is_bot` is `True`),
-      `bot_owner` is the user ID of the bot's owner (usually, whoever
-      created the bot).  **Changes**: New in Zulip 2.2 (feature level
-      1).  In previous versions, there was a `bot_owner` field
-      containing the email address of the bot's owner.
-    * `is_active`: A boolean specifying whether the user is active or not.
-    * `is_guest`: A boolean specifying whether the user is a guest user or not.
-    * `timezone`: The time zone of the user.
 
 #### Example response
 

templates/zerver/api/get-events-from-queue.md

@@ -77,9 +77,7 @@ endpoint and a queue would be registered in the absence of a `queue_id`.
 
 #### Return values
 
-* `events`: An array (possibly zero-length if `dont_block` is set) of events
+{generate_return_values_table|zulip.yaml|/events:get}
-  with IDs newer than `last_event_id`. Event IDs are guaranteed to be increasing,
-  but they are not guaranteed to be consecutive.
 
 #### Example response
 

templates/zerver/api/get-message-history.md

@@ -23,18 +23,7 @@
 
 #### Return values
 
-* `message_history`: a chronologically sorted array of `snapshot` objects,
+{generate_return_values_table|zulip.yaml|/messages/{message_id}/history:get}
-    containing the modified state of the message before and after the edit:
-    * `topic`: the topic for the message.
-    * `content`: the body of the message.
-    * `rendered_content`: the already rendered, HTML version of `content`.
-    * `prev_content`: the body of the message before being edited.
-    * `prev_rendered_content`: the already rendered, HTML version of
-        `prev_content`.
-    * `user_id`: the ID of the user that made the edit.
-    * `content_html_diff`: an HTML diff between this version of the message
-        and the previous one.
-    * `timestamp`: the UNIX timestamp for this editi.
 
 Please note that the original message's snapshot only contains the fields
 `topic`, `content`, `rendered_content`, `timestamp` and `user_id`. This

templates/zerver/api/get-messages.md

@@ -52,68 +52,7 @@ When a request is successful, this endpoint returns a dictionary
 containing the following (in addition to the `msg` and `result` keys
 present in all Zulip API responses).
 
-* `anchor`: the same `anchor` specified in the request.
+{generate_return_values_table|zulip.yaml|/messages:get}
-* `found_newest`: whether the `messages` list includes the latest message in
-    the narrow.
-* `found_oldest`: whether the `messages` list includes the oldest message in
-    the narrow.
-* `found_anchor`: whether it was possible to fetch the requested anchor, or
-    the closest in the narrow has been used.
-* `messages`: an array of `message` objects, each containing the following
-    fields:
-    * `avatar_url`: The URL of the user's avatar.
-    * `client`: A Zulip "client" string, describing what Zulip client
-      sent the message.
-    * `content`: The content/body of the message.
-    * `content_type`: The HTTP `content_type` for the message content.  This
-      will be `text/html` or `text/x-markdown`, depending on
-      whether `apply_markdown` was set.
-    * `display_recipient`: Data on the recipient of the message;
-      either the name of a stream or a dictionary containing data on
-      the users who received the message.
-    * `flags`: The user's [message flags][message-flags] for the message.
-    * `id`: The unique message ID.  Messages should always be
-      displayed sorted by ID.
-    * `is_me_message`: Whether the message is a [/me status message][status-messages]
-    * `reactions`: Data on any reactions to the message.
-        * `emoji_code`: An encoded version of the emoji's unicode codepoint.
-        * `emoji_name`: Name of the emoji.
-        * `reaction_type`: If the reaction uses a [custom emoji](/help/add-custom-emoji),
-          `reaction_type` will be set to `realm_emoji`.
-        * `user_id`: The ID of the user who added the reaction.
-           **Changes**: New in Zulip 2.2 (feature level 2). The `user`
-           object is deprecated and will be removed in the future.
-        * `user`: Dictionary with data on the user who added the reaction, including
-           the user ID as the `id` field.  **Note**: In the [events
-           API](/api/get-events-from-queue), this `user` dictionary
-           confusing had the user ID in a field called `user_id`
-           instead.  We recommend ignoring fields other than the user
-           ID.  **Deprecated** and to be removed in a future release
-           once core clients have migrated to use the `user_id` field.
-    * `recipient_id`: A unique ID for the set of users receiving the
-      message (either a stream or group of users).  Useful primarily
-      for hashing.
-    * `sender_email`: The email address of the message's sender.
-    * `sender_full_name`: The full name of the message's sender.
-    * `sender_id`: The user ID of the message's sender.
-    * `sender_realm_str`: A string identifier for the realm the sender
-      is in.
-    * `sender_short_name`: Reserved for future use.
-    * `stream_id`: Only present for stream messages; the ID of the stream.
-    * `subject`: The `topic` of the message (only present for stream
-      messages).  The name is a legacy holdover from when topics were
-      called "subjects".
-    * `topic_links`: Data on any links to be included in the `topic`
-      line (these are generated by [custom linkification
-      filters][linkification-filters] that match content in the
-      message's topic.)  **Changes**: New in Zulip 2.2 (feature level
-      1).  Previously, this field was called `subject_links`; clients
-      are recommended to rename `subject_links` to `topic_links` if
-      present for compatibility with older Zulip servers.
-    * `submessages`: Data used for certain experimental Zulip integrations.
-    * `timestamp`: The UNIX timestamp for when the message was sent,
-      in UTC seconds.
-    * `type`: The type of the message: `stream` or `private`.
 
 #### Example response
 

templates/zerver/api/get-org-emoji.md

@@ -39,19 +39,7 @@ zulip(config).then((client) => {
 
 #### Return values
 
-* `emoji`: An object that contains `emoji` objects, each identified with their
+{generate_return_values_table|zulip.yaml|/realm/emoji:get}
-    emoji ID as the key, and containing the following properties:
-    * `id`: The ID for this emoji, same as the object's key.
-    * `name`: The user-friendly name for this emoji. Users in the organization
-        can use this emoji by writing this name between colons (`:name:`).
-    * `source_url`: The path relative to the organization's URL where the
-        emoji's image can be found.
-    * `deactivated`: Whether the emoji has been deactivated or not.
-    * `author`: An object describing the user who created the custom emoji,
-        with the following fields:
-        * `id`: The creator's user ID.
-        * `email`: The creator's email address.
-        * `full_name`: The creator's full name.
 
 
 #### Example response

templates/zerver/api/get-presence.md

@@ -23,23 +23,7 @@
 
 #### Return values
 
-* `presence`: An object containing the presence details for every type
+{generate_return_values_table|zulip.yaml|/users/{email}/presence:get}
-  of client the user has ever logged into.
-    * `{client_name}` or `aggregated`: the keys for these objects are
-      the names of the different clients where this user is logged in,
-      like `website`, `ZulipDesktop`, `ZulipTerminal`, or
-      `ZulipMobile`. There is also an `aggregated` key, which matches
-      the contents of the object that has been updated most
-      recently. For most applications, you'll just want to look at the
-      `aggregated` key.
-        * `timestamp`: when this update was received; if the timestamp
-          is more than a few minutes in the past, the user is offline.
-        * `status`: either `active` or `idle`: whether the user had
-          recently interacted with Zulip at the time in the timestamp
-          (this distinguishes orange vs. green dots in the Zulip web
-          UI; orange/idle means we don't know whether the user is
-          actually at their computer or just left the Zulip app open
-          on their desktop).
 
 #### Example response
 

templates/zerver/api/get-profile.md

@@ -41,11 +41,7 @@ This endpoint takes no arguments.
 
 #### Return values
 
-* `pointer`: The integer ID of the message that the pointer is currently on.
+{generate_return_values_table|zulip.yaml|/users/me:get}
-* `max_message_id`: The integer ID of the last message by the user/bot with
-  the given profile.
-
-The rest of the return values are quite self-descriptive.
 
 #### Example response
 

templates/zerver/api/get-raw-message.md

@@ -23,7 +23,7 @@
 
 #### Return values
 
-* `raw_content`: The raw content of the message.
+{generate_return_values_table|zulip.yaml|/messages/{message_id}:get}
 
 #### Example response
 

templates/zerver/api/get-stream-id.md

@@ -42,7 +42,7 @@ zulip(config).then((client) => {
 
 #### Return values
 
-* `stream_id`: The ID of the given stream.
+{generate_return_values_table|zulip.yaml|/get_stream_id:get}
 
 #### Example response
 

templates/zerver/api/get-stream-topics.md

@@ -41,9 +41,7 @@ zulip(config).then((client) => {
 
 #### Return values
 
-* `topics`: An array of `topic` objects, which contain:
+{generate_return_values_table|zulip.yaml|/users/me/{stream_id}/topics:get}
-    * `name`: The name of the topic.
-    * `max_id`: The message ID of the last message sent to this topic.
 
 #### Example response
 

templates/zerver/api/get-subscribed-streams.md

@@ -46,28 +46,7 @@ You may pass the `include_subscribers` query parameter as follows:
 
 #### Return values
 
-* `subscriptions`: A list of dictionaries where each dictionary contains
+{generate_return_values_table|zulip.yaml|/users/me/subscriptions:get}
-  information about one of the subscribed streams.
-    * `stream_id`: The unique ID of a stream.
-    * `name`: The name of a stream.
-    * `description`: A short description of a stream.
-    * `invite-only`: Specifies whether a stream is private or not.
-      Only people who have been invited can access a private stream.
-    * `subscribers`: A list of email addresses of users who are also subscribed
-      to a given stream. Included only if `include_subscribers` is `true`.
-    * `desktop_notifications`: A boolean specifying whether desktop notifications
-      are enabled for the given stream.
-    * `push_notifications`: A boolean specifying whether push notifications
-      are enabled for the given stream.
-    * `audible_notifications`: A boolean specifying whether audible notifications
-      are enabled for the given stream.
-    * `pin_to_top`: A boolean specifying whether the given stream has been pinned
-      to the top.
-    * `email_address`: Email address of the given stream.
-    * `is_muted`: Whether the given stream is muted or not. Muted streams do
-      not count towards your total unread count and thus, do not show up in
-      `All messages` view (previously known as `Home` view).
-    * `color`: Stream color.
 
 #### Example response
 

templates/zerver/api/get-user-groups.md

@@ -25,12 +25,7 @@
 
 #### Return values
 
-* `user_groups`: A list of dictionaries, where each dictionary contains information
+{generate_return_values_table|zulip.yaml|/user_groups:get}
-  about a user group.
-    * `description`: The human-readable description of the user group.
-    * `id`: The user group's integer id.
-    * `members`: The integer User IDs of the user group members.
-    * `name`: User group name.
 
 #### Example response
 

templates/zerver/api/get-user.md

@@ -29,22 +29,7 @@ You may pass the `client_gravatar` or `include_custom_profile_fields` query para
 
 #### Return values
 
-* `user`: A dictionary that contains the requested user's details.
+{generate_return_values_table|zulip.yaml}|/users/{user_id}:get}
-    * `email`: The email address of the user or bot.
-    * `is_bot`: A boolean specifying whether the user is a bot or not.
-    * `avatar_url`: URL to the user's gravatar. `None` if the `client_gravatar`
-      query parameter was set to `True`.
-    * `full_name`: Full name of the user or bot.
-    * `is_admin`: A boolean specifying whether the user is an admin or not.
-    * `bot_type`: `None` if the user isn't a bot. `1` for a `Generic` bot.
-      `2` for an `Incoming webhook` bot. `3` for an `Outgoing webhook` bot.
-      `4` for an `Embedded` bot.
-    * `user_id`: The ID of the user.
-    * `bot_owner_id`: If the user is a bot (i.e. `is_bot` is `True`), `bot_owner_id`
-      is user ID of the user who owns the bot (usually the creator).
-    * `is_active`: A boolean specifying whether the user is active or not.
-    * `is_guest`: A boolean specifying whether the user is a guest user or not.
-    * `timezone`: The time zone of the user.
 
 #### Example response
 

templates/zerver/api/list-linkifiers.md

@@ -23,11 +23,7 @@
 
 #### Return values
 
-* `filters`: An array of tuples, each representing one of the
+{generate_return_values_table|zulip.yaml|/realm/filters:get}
-  linkifiers set up in the organization. Each of these tuples contain the
-  pattern, the formatted URL and the filter's ID, in that order. See
-  the [Create linkifiers](/api/add-linkifiers) article for details on what
-  each field means.
 
 #### Example response
 

templates/zerver/api/register-queue.md

@@ -87,9 +87,7 @@ zulip(config).then((client) => {
 
 #### Return values
 
-* `queue_id`: The ID of the queue that has been allocated for your client.
+{generate_return_values_table|zulip.yaml|/register:post}
-* `last_event_id`: The initial value of `last_event_id` to pass to
-  `GET /api/v1/events`.
 
 #### Example response
 

templates/zerver/api/remove-subscriptions.md

@@ -55,11 +55,7 @@ administrative privileges.
 
 #### Return values
 
-* `removed`: A list of the names of streams which were unsubscribed from as
+{generate_return_values_table|zulip.yaml|/users/me/subscriptions:delete}
-  a result of the query.
-
-* `not_removed`: A list of the names of streams that the user is already
-  unsubscribed from, and hence doesn't need to be unsubscribed.
 
 #### Example response
 

templates/zerver/api/render-message.md

@@ -44,7 +44,7 @@ zulip(config).then((client) => {
 
 #### Return values
 
-* `rendered`: The rendered HTML.
+{generate_return_values_table|zulip.yaml|/messages/render:post}
 
 #### Example response
 

templates/zerver/api/send-message.md

@@ -75,7 +75,7 @@ file.
 
 #### Return values
 
-* `id`: The ID of the newly created message
+{generate_return_values_table|zulip.yaml|/messages:post}
 
 #### Example response
 A typical successful JSON response may look like:

templates/zerver/api/server-settings.md

@@ -23,49 +23,7 @@
 
 #### Return values
 
-* `authentication_methods`: object in which each key-value pair in the object
+{generate_return_values_table|zulip.yaml|/server_settings:get}
-  indicates whether the authentication method is enabled on this server.
-* `zulip_version`: the version of Zulip running in the server.
-* `zulip_feature_level`: an integer indicating what features are
-    available on the server. The feature level increases monotonically;
-    a value of N means the server supports all API features introduced
-    before feature level N.  This is designed to provide a simple way
-    for client apps to decide whether the server supports a given
-    feature or API change.  See the [changelog](/api/changelog) for
-    details on what each feature level means.
-
-    **Changes**.  New in Zulip 2.2.  We recommend using an implied value
-    of 0 for Zulip servers that do not send this field.
-
-* `push_notifications_enabled`: whether mobile/push notifications are enabled.
-* `is_incompatible`: whether the Zulip client that has sent a request to
-  this endpoint is deemed incompatible with the server.
-* `email_auth_enabled`: setting for allowing users authenticate with an
-  email-password combination.
-* `require_email_format_usernames`: whether usernames should have an
-  email address format. This is important for clients to know whether
-  the validate email address format in a login prompt; this value will
-  be false if the server has
-  [LDAP authentication][ldap-auth]
-  enabled with a username and password combination.
-* `realm_uri`: the organization's canonical URI.
-* `realm_name`: the organization's name (for display purposes).
-* `realm_icon`: the URI of the organization's logo as a square image,
-  used for identifying the organization in small locations in the
-  mobile and desktop apps.
-* `realm_description`: HTML description of the organization, as configured by
-  the [organization profile](/help/create-your-organization-profile).
-* `external_authentication_methods`: list of dictionaries describing
-  the available external authentication methods (such as
-  google/github/SAML) enabled for this organization. Each dictionary
-  specifies the name and icon that should be displayed on the login
-  buttons (`display_name` and `display_icon`, where `display_icon` can
-  be `null`, if no icon is to be displayed), the URLs that
-  should be accessed to initiate login/signup using the method
-  (`login_url` and `signup_url`) and `name`, which is a unique,
-  stable, machine-readable name for the authentication method.  The
-  list is sorted in the order in which these authentication methods
-  should be displayed.
 
 [ldap-auth]: https://zulip.readthedocs.io/en/latest/production/authentication-methods.html#ldap-including-active-directory
 

templates/zerver/api/update-message-flags.md

@@ -117,7 +117,7 @@ zulip(config).then((client) => {
 
 #### Return values
 
-* `messages`: An array with the IDs of the modified messages.
+{generate_return_values_table|zulip.yaml|/messages/flags:post}
 
 #### Example response
 

templates/zerver/api/upload-file.md

@@ -34,7 +34,7 @@ to 25MB.
 
 #### Return values
 
-* `uri`: The URI of the uploaded file.
+{generate_return_values_table|zulip.yaml|/user_uploads:post}
 
 #### Example response
 

zerver/lib/bugdown/api_return_values_table_generator.py

@@ -0,0 +1,78 @@
+import re
+
+from markdown.extensions import Extension
+from markdown.preprocessors import Preprocessor
+from zerver.openapi.openapi import get_openapi_return_values
+from typing import Any, Dict, Optional, List
+import markdown
+
+REGEXP = re.compile(r'\{generate_return_values_table\|\s*(.+?)\s*\|\s*(.+)\s*\}')
+
+
+class MarkdownReturnValuesTableGenerator(Extension):
+    def __init__(self, configs: Optional[Dict[str, Any]]=None) -> None:
+        self.config: Dict[str, Any] = {}
+
+    def extendMarkdown(self, md: markdown.Markdown, md_globals: Dict[str, Any]) -> None:
+        md.preprocessors.add(
+            'generate_return_values', APIReturnValuesTablePreprocessor(md, self.getConfigs()), '_begin'
+        )
+
+
+class APIReturnValuesTablePreprocessor(Preprocessor):
+    def __init__(self, md: markdown.Markdown, config: Dict[str, Any]) -> None:
+        super().__init__(md)
+
+    def run(self, lines: List[str]) -> List[str]:
+        done = False
+        while not done:
+            for line in lines:
+                loc = lines.index(line)
+                match = REGEXP.search(line)
+
+                if not match:
+                    continue
+
+                doc_name = match.group(2)
+                endpoint, method = doc_name.rsplit(':', 1)
+                return_values: Dict[str, Any] = {}
+                return_values = get_openapi_return_values(endpoint, method)
+                text = self.render_table(return_values, 0)
+                line_split = REGEXP.split(line, maxsplit=0)
+                preceding = line_split[0]
+                following = line_split[-1]
+                text = [preceding] + text + [following]
+                lines = lines[:loc] + text + lines[loc+1:]
+                break
+            else:
+                done = True
+        return lines
+
+    def render_desc(self, description: str, spacing: int, return_value: Optional[str]=None) -> str:
+        description = description.replace('\n', '\n' + ((spacing + 4) * ' '))
+        if return_value is None:
+            return (spacing * " ") + "* " + description
+        return (spacing * " ") + "* `" + return_value + "`: " + description
+
+    def render_table(self, return_values: Dict[str, Any], spacing: int) -> List[str]:
+        IGNORE = ["result", "msg"]
+        ans = []
+        for return_value in return_values:
+            if return_value in IGNORE:
+                continue
+            description = return_values[return_value]['description']
+            ans.append(self.render_desc(description, spacing, return_value))
+            if 'properties' in return_values[return_value]:
+                ans += self.render_table(return_values[return_value]['properties'], spacing + 4)
+            if 'additionalProperties' in return_values[return_value]:
+                ans.append(self.render_desc(return_values[return_value]['additionalProperties']
+                                            ['description'], spacing + 4))
+                ans += self.render_table(return_values[return_value]['additionalProperties']['properties'],
+                                         spacing + 8)
+            if ('items' in return_values[return_value] and
+               'properties' in return_values[return_value]['items']):
+                ans += self.render_table(return_values[return_value]['items']['properties'], spacing + 4)
+        return ans
+
+def makeExtension(*args: Any, **kwargs: str) -> MarkdownReturnValuesTableGenerator:
+    return MarkdownReturnValuesTableGenerator(kwargs)

zerver/openapi/openapi.py

@@ -101,6 +101,17 @@ def get_openapi_parameters(endpoint: str, method: str,
                       parameter['in'] != 'path']
     return parameters
 
+def get_openapi_return_values(endpoint: str, method: str,
+                              include_url_parameters: bool=True) -> List[Dict[str, Any]]:
+    openapi_endpoint = openapi_spec.spec()['paths'][endpoint][method.lower()]
+    response = openapi_endpoint['responses']['200']['content']['application/json']['schema']
+    # In cases where we have used oneOf, the schemas only differ in examples
+    # So we can choose any.
+    if 'oneOf' in response:
+        response = response['oneOf'][0]
+    response = response['properties']
+    return response
+
 def validate_against_openapi_schema(content: Dict[str, Any], endpoint: str,
                                     method: str, response: str) -> None:
     """Compare a "content" dict with the defined schema for a specific method

zerver/templatetags/app_filters.py

@@ -11,6 +11,7 @@ from jinja2.exceptions import TemplateNotFound
 
 import zerver.lib.bugdown.fenced_code
 import zerver.lib.bugdown.api_arguments_table_generator
+import zerver.lib.bugdown.api_return_values_table_generator
 import zerver.openapi.markdown_extension
 import zerver.lib.bugdown.nested_code_blocks
 import zerver.lib.bugdown.tabbed_sections
@@ -107,6 +108,8 @@ def render_markdown_path(markdown_file_path: str,
             ),
             zerver.lib.bugdown.api_arguments_table_generator.makeExtension(
                 base_path='templates/zerver/api/'),
+            zerver.lib.bugdown.api_return_values_table_generator.makeExtension(
+                base_path='templates/zerver/api/'),
             zerver.lib.bugdown.nested_code_blocks.makeExtension(),
             zerver.lib.bugdown.tabbed_sections.makeExtension(),
             zerver.lib.bugdown.help_settings_links.makeExtension(),