The commit is composed of: (1) distill out top-level dependencies in
common.txt, (2) add -e flag to the vcs-based packages because
pip-compile can't do without, (3) pip-compile/generate the locked files
then remove the -e flags from the lockfile, (4) pin pathlib2 to dev.txt
because it turns out it is a direct requirement of
documentation_crawler, (5) document the structure and add an automation
script (6) remove cryptography==1.9 from requirements/scrapy.txt since
cryptography is automatically added from pyopenssl (7) add sed command
to remove future/futures from the generated lock file in python3 (this
should have been automatically handled by pip-compile, so pending for
the feature from pip-compile)
Tweaked by tabbott to update PROVISION_VERSION and add a missing
`first` dependency.
Apparently the module "cryptography" (used by service_identity) has some
dependencies that weren't initially included in requirements/scrapy.txt.
Python 2 and 3: six, cffi, asn1crypto, idna
Python 2 only: enum34, ipaddress
Out of these modules, the ones that were already included in
requirements/common.txt have been removed, resulting in:
Python 2 and 3: asn1crypto
Python 2 only: ipaddress
Credits as well to @adnrs96, whose help was essential to catch this issue.
- Add scrapy project with rewrited 'crawl' command and spider to check documentation.
Command was rewrited due to return exit status by the exception existing,
it returns exit code 0 if exception happens inside spider in standard behavour.
- Add scrapy requirements to for dev environment. It was added to twisted requirements list.
Fixes#1492
rht
Yago González
Vishnu Ks
Tim Abbott
sinwar
K.Kanakhin