- Add scrapy project with rewrited 'crawl' command and spider to check documentation.
Command was rewrited due to return exit status by the exception existing,
it returns exit code 0 if exception happens inside spider in standard behavour.
- Add scrapy requirements to for dev environment. It was added to twisted requirements list.
Fixes#1492
Twisted is not python 3 compatible. So for now create a python2
venv and install twisted in it when running provision.py in python3
mode and use twisted from the python2 venv.