The crawler used to be called directly for checking external links.
Now the scrapy command calls the crawl_with_status wrapper.
Crawl_with_status has been modified to pass the external parameter in
the previous commit, so we can now use this simpler approach.
This option causes test-documentation to only verify internal links
that we control and are important to be correct. This prevents
test-documentation from flaking in CI due to issues with the dozens of
third-party blocks that we link to from various parts of our
documentation.
Tweaked by tabbott for comment clarity, and to also include
github.com/zulip links.
Fixes#10942.
In tools/test-documentation line 6:
echo -e "\e[${color_code}m${message}\e[0m" >&2
^-- SC1117: Backslash is literal in "\e". Prefer explicit escaping: "\\e".
^-- SC1117: Backslash is literal in "\e". Prefer explicit escaping: "\\e".
In tools/test-documentation line 41:
scrapy crawl_with_status documentation_crawler $loglevel
^-- SC2086: Double quote to prevent globbing and word splitting.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
This is fairly often -- though not always! -- failing, with a nasty
failure mode where it takes like 6 minutes to time out. See
discussion on #7748 (search for "bad link").
Actually, after seeing it happen just now when running
test-documentation on my laptop, on some other link, it occurs to me
that I've seen this before -- it's fairly common in Travis, too. It's
just that it doesn't actually cause the build to fail :-/, and on
Travis we haven't been paying as close attention to slow builds as we
are on Circle right now.
Generally stderr is the conventional place for this sort of running
commentary, and it's better set up for it: by default stdout may have
a buffer inside the process so that things written to it don't reach
the outside until later, while stderr is always by default unbuffered,
so messages are printed immediately.
Here, until the previous commit, because our color-reset sequence was
being printed without a following newline (with `echo -n`), it was
getting buffered; and then error messages from `scrapy` to stderr were
being erroneously painted with the color intended for the message
"Testing links in documentation...".
We already do this by default in tools/build-docs, but since we
migrated test-documentation to not run that directly (to disable
collapsing), we need to add the recent parallelism fix here too.
It saves about 5-10s when running this test suite for me, which is
good, but definitely leaves me feeling like there could be more
improvement.
This commit helps reduce clutter on the navigation sidebar.
Creates new directories and moves relevant files into them.
Modifies index.rst, symlinks, and image paths accordingly.
This commit also enables expandable/collapsible navigation items,
renames files in docs/development and docs/production,
modifies /tools/test-documentation so that it overrides a theme setting,
Also updates links to other docs, file paths in the codebase that point
to developer documents, and files that should be excluded from lint tests.
Note that this commit does not update direct links to
zulip.readthedocs.io in the codebase; those will be resolved in an
upcoming follow-up commit (it'll be easier to verify all the links
once this is merged and ReadTheDocs is updated).
Fixes#5265.
- Add script to compile documentation build and start crawler
to check documentation.
- Add documentation test script to backend travis test case.
- Add log level argument to test-documentation script.
Fixes#1492
Anders Kaseorg
Sumanth V Rao
Anders Kaseorg
Greg Price
Tim Abbott
David Rosa Tamsen
Cory Lynch
K.Kanakhin