The Travis config does this by a `pip install codecov` in setup and
running the resulting `codecov` command at the end. That risks
confusion because it makes the library look like something we might
be using *during* the tests to collect coverage information.
Instead, install and run codecov together at the end.
(The docs also suggest using their bash-based uploader instead;
but following those directions seems to result in coverage.xml
not getting included (maybe not even built), and no coverage data
getting uploaded for Python source files, only JavaScript. Not
sure why that is, but using the pip package is fine.)
Also, because this step is inherently about talking to a
third-party service whose reliability we can't control, and because
it's purely a reporting step that happens after our actual test
suite has succeeded, don't fail if it fails; just print a message.
(I'd prefer a somewhat louder message, like turning that step
yellow or something in the Circle web UI, but Circle doesn't offer
an option like that. If the coverage reports start consistently
failing for some reason, then if we're actually reading them we'll
notice anyway.)
[greg: Edited step name and error message; wrote this commit message.]
This new base image just comes from me doing a fresh `docker build`
and uploading the result. That gives it fresher versions of our
APT dependencies than the image I built a couple of months ago,
speeding up provision by giving it less to install.
From just my experience attempting this change, I've been impressed
with the very fast turnaround in CircleCI for a change -- from `git
push` to the build starting is usually just a couple of seconds.
It also looks like we get much clearer control over the caching
system, and the setup of the environment via a Docker image of our
choice, which can help avoid most of the several minutes Travis takes
to set up our environment before it actually starts running the tests
themselves.
Plus it supports fine-grained parallelism, e.g. by test file:
https://circleci.com/docs/2.0/parallelism-faster-jobs/
So we can probably get much faster builds by making this switch.
Because the environment is much more transparent and even under
our control, I'm also hopeful for more reliable builds.
Vishnu Ks
Greg Price
Greg Price