This checks both that all images under static/images/help/ are used in
the help documentation, and also that none of the image tags are broken.
- Improve documentation spiders and crawler with spider error state.
Fixes#3070.
- Add spider for documentation crawler, which processes user help
documentation on test development server.
- Add tool script for help documentation checking launching.
Fixes#2639
Sometimes remote servers are unavailable or on maintance
what is the cause of broken back-end tests. Such response errors
is excluded from raised exception cases and just notify about
wrong link in test log.
- Add scrapy project with rewrited 'crawl' command and spider to check documentation.
Command was rewrited due to return exit status by the exception existing,
it returns exit code 0 if exception happens inside spider in standard behavour.
- Add scrapy requirements to for dev environment. It was added to twisted requirements list.
Fixes#1492
Tim Abbott
Rishi Gupta
K.Kanakhin