documentation_crawler: Exclude communities page.

Since /communities/#all was failing the spider test due to `#all`
not being present as an `id` on the page, we exclude it from
this check saying that it is not really a documentation page.
This commit is contained in:
Aman Agrawal 2022-10-20 20:08:27 +00:00 committed by Tim Abbott
parent 6266954229
commit 587f52e351
1 changed files with 5 additions and 0 deletions

View File

@ -146,6 +146,11 @@ class BaseDocumentationSpider(scrapy.Spider):
):
return
# This page has some invisible to the user anchor links like #all
# that are currently invisible, and thus would otherwise fail this test.
if url.startswith("http://localhost:9981/communities"):
return
callback: Callable[[Response], Optional[Iterator[Request]]] = self.parse
dont_filter = False
method = "GET"