api-docs: Move markdown files to top level directory.

- Updates `.prettierignore` for the new directory.
- Updates any reference to the API documentation directory for
  markdown files to be `api_docs/` instead of `zerver/api/`.
- Removes a reference link from `docs/documentation/api.md` that
  hasn't referenced anything in the text since commit 0542c60.
- Update rendering of API documentation for new directory.

.prettierignore

@@ -1,3 +1,4 @@
+/api_docs/**/*.md
 /corporate/tests/stripe_fixtures
 /help/**/*.md
 /locale

docs/documentation/api.md

@@ -33,11 +33,10 @@ Our API documentation is defined by a few sets of files:
   our [help center docs](helpcenter.md), with some special
   extensions for rendering nice code blocks and example
   responses. Most API endpoints share a common template,
-  `templates/zerver/api/api-doc-template.md`, which renders the
+  `api_docs/api-doc-template.md`, which renders the
   OpenAPI description of the API endpoint. A handful of endpoints that
   require special content, as well as pages that document general API
-  details rather than specific endpoints, live at
-  `templates/zerver/api/*.md`.
+  details rather than specific endpoints, live at `api_docs/*.md`.
 - We have an extensive set of tests designed to validate that the data
   in the OpenAPI file matching the implementation. Specifically,
   `zerver/tests/test_openapi.py` compares every endpoint's accepted
@@ -57,8 +56,8 @@ Our API documentation is defined by a few sets of files:
 - The cURL examples are generated and tested using
   `zerver/openapi/curl_param_value_generators.py`.
 - The REST API index
-  (`templates/zerver/api/include/rest-endpoints.md`) in the broader
-  /api left sidebar (`templates/zerver/api/sidebar_index.md`).
+  (`api_docs/include/rest-endpoints.md`) in the broader
+  /api left sidebar (`api_docs/sidebar_index.md`).
 
 This first section is focused on explaining how the API documentation
 system is put together; when actually documenting an endpoint, you'll
@@ -67,7 +66,7 @@ want to also read the [Step by step guide](#step-by-step-guide).
 ## How it works
 
 To understand how this documentation system works, start by reading an
-existing doc file (`templates/zerver/api/render-message.md` is a good
+existing doc file (`api_docs/render-message.md` is a good
 example; accessible live
 [here](https://zulip.com/api/render-message) or in the development
 environment at `http://localhost:9991/api/render-message`).
@@ -180,7 +179,7 @@ wherever that string appears in the API documentation.
 
 We have a separate Markdown extension to document the parameters that
 an API endpoint supports. You'll see this in files like
-`templates/zerver/api/render-message.md` via the following Markdown
+`api_docs/render-message.md` via the following Markdown
 directive (implemented in
 `zerver/lib/markdown/api_arguments_table_generator.py`):
 
@@ -290,13 +289,13 @@ above.
 
 1. Finally, if the API docs page of the endpoint doesn't follow the
    common API docs template in
-   `templates/zerver/api/api-docs-template.md`, then add its custom
-   Markdown file under `templates/zerver/api/`. However, it is a goal
+   `api_docs/api-docs-template.md`, then add its custom
+   Markdown file under `api_docs/`. However, it is a goal
    to minimize the number of files that diverse from the common
    template, so only do this if there's a good reason.
 
 1. Add the endpoint to the index in
-   `templates/zerver/api/include/rest-endpoints.md`. The URL should
+   `api_docs/include/rest-endpoints.md`. The URL should
    match the `operationId` for the endpoint, and the link text should
    match the title of the endpoint from the OpenAPI `summary` field.
 
@@ -307,14 +306,12 @@ above.
    make sure that copy-pasting the code in your examples works, and
    post an example of the output in the pull request.
 
-1. Document the new API in `templates/zerver/api/changelog.md` and
+1. Document the new API in `api_docs/changelog.md` and
    bump the `API_FEATURE_LEVEL` in `version.py`. Also, make sure to
    add a `**Changes**` entry in the description of the new API/event
    in `zerver/openapi/zulip.yaml`, which mentions the API feature level
    at which they were added.
 
-[javascript-examples]: https://github.com/zulip/zulip-js/tree/main/examples
-
 ## Why a custom system?
 
 Given that our documentation is written in large part using the

docs/tutorials/new-feature-tutorial.md

@@ -67,7 +67,7 @@ organization in Zulip). The following files are involved in the process:
 **Documentation**
 
 - `zerver/openapi/zulip.yaml`: OpenAPI definitions for the Zulip REST API.
-- `templates/zerver/api/changelog.md`: documentation listing all changes to the Zulip Server API.
+- `api_docs/changelog.md`: documentation listing all changes to the Zulip Server API.
 - `help/...`: end user facing documentation (Help Center) for the application.
 
 ### Adding a field to the database
@@ -164,7 +164,7 @@ provides more detailed information about writing and editing feature
 
 **API documentation:** A new feature will probably impact the REST API
 documentation as well, which will mean updating `zerver/openapi/zulip.yaml`
-and modifying `templates/zerver/api/changelog.md` for a new feature
+and modifying `api_docs/changelog.md` for a new feature
 level. [Documenting REST API endpoints](../documentation/api.md)
 explains Zulip's API documentation system and provides a step by step
 guide to adding or updating documentation for an API endpoint.
@@ -714,7 +714,7 @@ documentation is to read more about Zulip's
 and [OpenAPI configuration](../documentation/openapi.md).
 
 In particular, if there is an API change, **make sure** you document
-your new feature in `templates/zerver/api/changelog.md` and bump the
+your new feature in `api_docs/changelog.md` and bump the
 `API_FEATURE_LEVEL` in `version.py`. The API feature level allows the
 developers of mobile clients and other tools using the Zulip API to
 programmatically determine whether the Zulip server they are

templates/zerver/documentation_main.html

@@ -20,7 +20,7 @@
             {% elif page_is_help_center %}
             {{ render_markdown_path(sidebar_index, pure_markdown=True) }}
             {% else %}
-            {{ render_markdown_path(sidebar_index, api_uri_context) }}
+            {{ render_markdown_path(sidebar_index, context=api_uri_context, pure_markdown=True) }}
             {% endif %}
 
             {% if not page_is_policy_center %}
@@ -40,7 +40,7 @@
             {% elif page_is_help_center %}
             {{ render_markdown_path(article, context=api_uri_context, help_center=True, pure_markdown=True) }}
             {% else %}
-            {{ render_markdown_path(article, api_uri_context) }}
+            {{ render_markdown_path(article, context=api_uri_context, pure_markdown=True) }}
             {% endif %}
 
             <div id="footer" class="documentation-footer">

tools/linter_lib/custom_check.py

@@ -826,6 +826,7 @@ markdown_rules = RuleList(
         {
             "pattern": "https://zulip.readthedocs.io/en/latest/[a-zA-Z0-9]",
             "exclude": {
+                "api_docs/",
                 "docs/contributing/contributing.md",
                 "docs/overview/readme.md",
                 "docs/README.md",
@@ -857,7 +858,6 @@ markdown_rules = RuleList(
             "description": "Don't link directly to line numbers",
         },
     ],
-    exclude_files_in="help/",
 )
 
 help_markdown_rules = RuleList(

tools/release

@@ -98,8 +98,8 @@ if [ -z "$is_prerelease" ]; then
         changed_api_level=$(git diff-tree -G API_FEATURE_LEVEL HEAD -- version.py)
         [ -n "$changed_api_level" ] || fail "$version did not adjust API_FEATURE_LEVEL in version.py"
         feature_level="$(extract_version "API_FEATURE_LEVEL")"
-        grep -q -F -x "**Feature level $feature_level**" templates/zerver/api/changelog.md \
-            || fail "Feature level $feature_level is not documented in templates/zerver/api/changelog.md"
+        grep -q -F -x "**Feature level $feature_level**" api_docs/changelog.md \
+            || fail "Feature level $feature_level is not documented in api_docs/changelog.md"
     fi
 fi
 

version.py

@@ -31,8 +31,8 @@ DESKTOP_WARNING_VERSION = "5.4.3"
 # use the new feature/API until the bump.
 #
 # Changes should be accompanied by documentation explaining what the
-# new level means in templates/zerver/api/changelog.md, as well as
-# "**Changes**" entries in the endpoint's documentation in `zulip.yaml`.
+# new level means in api_docs/changelog.md, as well as "**Changes**"
+# entries in the endpoint's documentation in `zulip.yaml`.
 API_FEATURE_LEVEL = 159
 
 # Bump the minor PROVISION_VERSION to indicate that folks should provision

zerver/lib/templates.py

@@ -151,7 +151,7 @@ def render_markdown_path(
         md_macro_extension = zerver.lib.markdown.include.makeExtension(base_path="help/include/")
     else:
         md_macro_extension = zerver.lib.markdown.include.makeExtension(
-            base_path="templates/zerver/api/include/"
+            base_path="api_docs/include/"
         )
     if not any(doc in markdown_file_path for doc in docs_without_macros):
         extensions = [md_macro_extension, *extensions]

zerver/openapi/test_curl_examples.py

@@ -38,9 +38,7 @@ def test_generated_curl_examples_for_success(client: Client) -> None:
     # on "add" tests coming before "remove" tests in some cases.  We
     # should try to either avoid ordering dependencies or make them
     # very explicit.
-    rest_endpoints_path = os.path.join(
-        settings.DEPLOY_ROOT, "templates/zerver/api/include/rest-endpoints.md"
-    )
+    rest_endpoints_path = os.path.join(settings.DEPLOY_ROOT, "api_docs/include/rest-endpoints.md")
     with open(rest_endpoints_path) as f:
         rest_endpoints_raw = f.read()
     ENDPOINT_REGEXP = re.compile(r"/api/\s*(.*?)\)")
@@ -48,7 +46,7 @@ def test_generated_curl_examples_for_success(client: Client) -> None:
 
     for endpoint in endpoint_list:
         article_name = endpoint + ".md"
-        file_name = os.path.join(settings.DEPLOY_ROOT, "templates/zerver/api/", article_name)
+        file_name = os.path.join(settings.DEPLOY_ROOT, "api_docs/", article_name)
         curl_commands_to_test = []
 
         if os.path.exists(file_name):

zerver/views/documentation.py

@@ -69,6 +69,7 @@ class MarkdownDirectoryView(ApiURLView):
     path_template = ""
     policies_view = False
     help_view = False
+    api_doc_view = False
 
     def get_path(self, article: str) -> DocumentationArticle:
         http_status = 200
@@ -91,27 +92,33 @@ class MarkdownDirectoryView(ApiURLView):
         endpoint_name = None
         endpoint_method = None
 
-        # Absolute path cases
-        if (self.policies_view or self.help_view) and self.path_template.startswith("/"):
-            if not os.path.exists(path):
-                article = "missing"
-                http_status = 404
-                path = self.path_template % (article,)
+        if not self.path_template.startswith("/"):
+            # Relative paths only used for policies documentation
+            # when it is not configured or in the dev environment
+            assert self.policies_view
 
-            return DocumentationArticle(
-                article_path=path,
-                article_http_status=http_status,
-                endpoint_path=None,
-                endpoint_method=None,
-            )
+            try:
+                loader.get_template(path)
+                return DocumentationArticle(
+                    article_path=path,
+                    article_http_status=http_status,
+                    endpoint_path=endpoint_name,
+                    endpoint_method=endpoint_method,
+                )
+            except loader.TemplateDoesNotExist:
+                return DocumentationArticle(
+                    article_path=self.path_template % ("missing",),
+                    article_http_status=404,
+                    endpoint_path=None,
+                    endpoint_method=None,
+                )
 
-        if self.path_template == "/zerver/api/%s.md":
-            # Hack: `self.path_template` has a leading `/`, so we use + to add directories.
-            api_documentation_path = os.path.join(settings.DEPLOY_ROOT, "templates") + path
-            if not os.path.exists(api_documentation_path):
+        if not os.path.exists(path):
+            if self.api_doc_view:
                 try:
+                    # API endpoints documented in zerver/openapi/zulip.yaml
                     endpoint_name, endpoint_method = get_endpoint_from_operationid(article)
-                    path = "/zerver/api/api-doc-template.md"
+                    path = self.path_template % ("api-doc-template",)
                 except AssertionError:
                     return DocumentationArticle(
                         article_path=self.path_template % ("missing",),
@@ -119,22 +126,19 @@ class MarkdownDirectoryView(ApiURLView):
                         endpoint_path=None,
                         endpoint_method=None,
                     )
+            elif self.help_view or self.policies_view:
+                article = "missing"
+                http_status = 404
+                path = self.path_template % (article,)
+            else:
+                raise AssertionError("Invalid documentation view type")
 
-        try:
-            loader.get_template(path)
-            return DocumentationArticle(
-                article_path=path,
-                article_http_status=http_status,
-                endpoint_path=endpoint_name,
-                endpoint_method=endpoint_method,
-            )
-        except loader.TemplateDoesNotExist:
-            return DocumentationArticle(
-                article_path=self.path_template % ("missing",),
-                article_http_status=404,
-                endpoint_path=None,
-                endpoint_method=None,
-            )
+        return DocumentationArticle(
+            article_path=path,
+            article_http_status=http_status,
+            endpoint_path=endpoint_name,
+            endpoint_method=endpoint_method,
+        )
 
     def get_context_data(self, **kwargs: Any) -> Dict[str, Any]:
         article = kwargs["article"]
@@ -149,12 +153,8 @@ class MarkdownDirectoryView(ApiURLView):
         ):
             # Absolute path case
             article_absolute_path = documentation_article.article_path
-        elif documentation_article.article_path.startswith("/"):
-            # Hack: `context["article"] has a leading `/`, so we use + to add directories.
-            article_absolute_path = (
-                os.path.join(settings.DEPLOY_ROOT, "templates") + documentation_article.article_path
-            )
         else:
+            # Relative path case
             article_absolute_path = os.path.join(
                 settings.DEPLOY_ROOT, "templates", documentation_article.article_path
             )
@@ -173,13 +173,15 @@ class MarkdownDirectoryView(ApiURLView):
             sidebar_article = self.get_path("sidebar_index")
             sidebar_index = sidebar_article.article_path
             title_base = "Zulip terms and policies"
-        else:
+        elif self.api_doc_view:
             context["page_is_api_center"] = True
             context["doc_root"] = "/api/"
             context["doc_root_title"] = "API documentation"
             sidebar_article = self.get_path("sidebar_index")
             sidebar_index = sidebar_article.article_path
             title_base = "Zulip API documentation"
+        else:
+            raise AssertionError("Invalid documentation view type")
 
         # The following is a somewhat hacky approach to extract titles from articles.
         endpoint_name = None
@@ -187,7 +189,7 @@ class MarkdownDirectoryView(ApiURLView):
         if os.path.exists(article_absolute_path):
             with open(article_absolute_path) as article_file:
                 first_line = article_file.readlines()[0]
-            if context["article"] == "/zerver/api/api-doc-template.md":
+            if self.api_doc_view and context["article"].endswith("api-doc-template.md"):
                 endpoint_name, endpoint_method = (
                     documentation_article.endpoint_path,
                     documentation_article.endpoint_method,
@@ -195,9 +197,7 @@ class MarkdownDirectoryView(ApiURLView):
                 assert endpoint_name is not None
                 assert endpoint_method is not None
                 article_title = get_openapi_summary(endpoint_name, endpoint_method)
-            elif (
-                self.path_template == "/zerver/api/%s.md" and "{generate_api_header(" in first_line
-            ):
+            elif self.api_doc_view and "{generate_api_header(" in first_line:
                 api_operation = context["PAGE_METADATA_URL"].split("/api/")[1]
                 endpoint_name, endpoint_method = get_endpoint_from_operationid(api_operation)
                 article_title = get_openapi_summary(endpoint_name, endpoint_method)

zproject/urls.py

@@ -786,7 +786,9 @@ help_documentation_view = MarkdownDirectoryView.as_view(
     help_view=True,
 )
 api_documentation_view = MarkdownDirectoryView.as_view(
-    template_name="zerver/documentation_main.html", path_template="/zerver/api/%s.md"
+    template_name="zerver/documentation_main.html",
+    path_template=f"{settings.DEPLOY_ROOT}/api_docs/%s.md",
+    api_doc_view=True,
 )
 policy_documentation_view = MarkdownDirectoryView.as_view(
     template_name="zerver/documentation_main.html",