Thumbor and tc-aws have been dragging their feet on Python 3 support
for years, and even the alphas and unofficial forks we’ve been running
don’t seem to be maintained anymore. Depending on these projects is
no longer viable for us.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This guarantees that we don’t accidentally upgrade one without the
other, which could happen for example due to different third-party
version constraints between the two.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
It happens that commonmark, python-jose, and python-twitter don’t
actually use future on Python 3, and moto uses aws-xray-sdk in such a
way that it doesn’t use future, but this was a weird game to be
playing just to remove one dependency, and it caused CI failures after
new releases of future, so let’s just include it.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
The reason that `pip-tools` running on Python 3 didn’t detect the
right requirements for `thumbor` on Python 2 is simply that some of
them are conditional on the Python version.
As for the requirements that had been manually added as a workaround:
`backports-abc` and `singledispatch` are now correctly detected, while
`backports.ssl-match-hostname` was vendored into `urllib3` some time
ago and `certifi` is no longer necessary.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
It’s unclear why pip-tools considers these packages unsafe, and
excluding them from being pinned has resulted in nondeterministic
output that makes our test suite unhappy.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
futures is no longer there to be removed. Be clear about why we’re
removing future (it was never a “pip-tools bug”), and leave evidence
behind to help indicate how long that will be needed.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
Run pip-compile with --quiet so we don’t have to redirect its stderr;
then we can see any exceptions it might throw. Print any resulting
diff in the right order and without extra newlines separating each
line.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
These are not the latest versions, but pip-tools 3.9.0 or 4.0.0 fails
to resolve dependencies from Git URLs:
pip._internal.exceptions.DistributionNotFound: No matching distribution found for zulip==0.6.1_git (from -r requirements/common.in (line 135))
while pip 19.2 breaks pip-tools 3.8.0:
TypeError: __init__() got an unexpected keyword argument 'find_links'
Fixes#10802.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
In tools/update-locked-requirements line 66:
compile_requirements requirements/prod.in $OUTPUT_BASE_DIR/prod.txt
^-- SC2086: Double quote to prevent globbing and word splitting.
In tools/update-locked-requirements line 67:
compile_requirements requirements/dev.in $OUTPUT_BASE_DIR/dev.txt
^-- SC2086: Double quote to prevent globbing and word splitting.
In tools/update-locked-requirements line 68:
compile_requirements requirements/mypy.in $OUTPUT_BASE_DIR/mypy.txt
^-- SC2086: Double quote to prevent globbing and word splitting.
In tools/update-locked-requirements line 69:
compile_requirements requirements/docs.in $OUTPUT_BASE_DIR/docs.txt
^-- SC2086: Double quote to prevent globbing and word splitting.
In tools/update-locked-requirements line 70:
compile_requirements requirements/thumbor.in $OUTPUT_BASE_DIR/thumbor.txt py2
^-- SC2086: Double quote to prevent globbing and word splitting.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
This commit renames various source requirements files like `dev.txt`,
`mypy.txt` etc to `dev.in`, `mypy.in` etc and various locked requirements
files like `dev_lock.txt`, `mypy_lock.txt` etc to `dev.txt`, `mypy.txt`
etc. This will help in emphasizing to the user that *.in are actually
input to `update-locked-requirements` tool which should be run after
updating any of these.
In this commit we add new dependencies needed for running thumbor.
Also we add the script for creating the virtual environment ready
for thumbor.
Note: Thumbor will use python2 and thus have different virtualenv
dedicated to it.
Credits to @TigorC and @joshland as well for there work on this.
In this commit we add a new option which could be used to specify
python version. When 'py2' is specified, future/futures are not
removed from the requirements lock file generated.
My first version of this just replaced the repeated list of two output
files with an array variable, but I decided `"${outputs[@]}"` was too
much to ask people to understand, and the alternative of `$outputs`,
unquoted, encourages bad habits of shell programming. So just handle
one file at a time; the only at all expensive part here is `pip-compile`.
I am tempted to move this to Python, but holding back.
This makes the code a little more concise and also more uniform,
treating `future` the same in prod and in dev. The non-uniformity
looks like an oversight in 2be8a793e, one of the commits that updated
this code for the Python 3-only world.
A couple of remarks still reflected the Python 2 world; and the
"Remove the editable flag" comment was confusingly above a line
that it looks like it could be talking about, but isn't.
First, all the lines of py3_dev.txt except for mypy.txt are mv-ed to
dev.txt. Then dev_lock.txt is generated from dev.txt to be used by
py3_dev.txt. `click` is removed from moto.txt since it is already a
dependency of several libraries and will appear as an autogenerated
dependency.
I think soon we'll put the Python 3 venv at `/srv/zulip-venv` and
make `/srv/zulip-py3-venv` just an alias to that (then remove the
alias too), but for now the old name is helpful for spotting places
that need an update.
The `/srv/zulip-venv` name still appears in codepaths used by Travis
tests.
The commit is composed of: (1) distill out top-level dependencies in
common.txt, (2) add -e flag to the vcs-based packages because
pip-compile can't do without, (3) pip-compile/generate the locked files
then remove the -e flags from the lockfile, (4) pin pathlib2 to dev.txt
because it turns out it is a direct requirement of
documentation_crawler, (5) document the structure and add an automation
script (6) remove cryptography==1.9 from requirements/scrapy.txt since
cryptography is automatically added from pyopenssl (7) add sed command
to remove future/futures from the generated lock file in python3 (this
should have been automatically handled by pip-compile, so pending for
the feature from pip-compile)
Tweaked by tabbott to update PROVISION_VERSION and add a missing
`first` dependency.
Anders Kaseorg
Alex Vandiver
Aman Agrawal
Anders Kaseorg
Anders Kaseorg
Harshit Bansal
Aditya Bansal
Greg Price
rht