Mirror
/
zulip
mirror of https://github.com/zulip/zulip.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
zulip/.circleci/config.yml

147 lines
4.7 KiB
YAML
Raw Normal View History

circle: Draft a config for running our builds in CircleCI. From just my experience attempting this change, I've been impressed with the very fast turnaround in CircleCI for a change -- from `git push` to the build starting is usually just a couple of seconds. It also looks like we get much clearer control over the caching system, and the setup of the environment via a Docker image of our choice, which can help avoid most of the several minutes Travis takes to set up our environment before it actually starts running the tests themselves. Plus it supports fine-grained parallelism, e.g. by test file: https://circleci.com/docs/2.0/parallelism-faster-jobs/ So we can probably get much faster builds by making this switch. Because the environment is much more transparent and even under our control, I'm also hopeful for more reliable builds.
2017-08-30 02:09:09 +02:00
# See CircleCI upstream's docs on this config format:
# https://circleci.com/docs/2.0/language-python/
#
version: 2
jobs:
circleci: Run xenial py3.5 backend tests alongside trusty.
2018-01-30 21:20:50 +01:00
"trusty-python-3.4":
circle: Draft a config for running our builds in CircleCI. From just my experience attempting this change, I've been impressed with the very fast turnaround in CircleCI for a change -- from `git push` to the build starting is usually just a couple of seconds. It also looks like we get much clearer control over the caching system, and the setup of the environment via a Docker image of our choice, which can help avoid most of the several minutes Travis takes to set up our environment before it actually starts running the tests themselves. Plus it supports fine-grained parallelism, e.g. by test file: https://circleci.com/docs/2.0/parallelism-faster-jobs/ So we can probably get much faster builds by making this switch. Because the environment is much more transparent and even under our control, I'm also hopeful for more reliable builds.
2017-08-30 02:09:09 +02:00
docker:
circleci: Run xenial py3.5 backend tests alongside trusty.
2018-01-30 21:20:50 +01:00
# This is built from tools/circleci/images/trusty/Dockerfile .
circleci: Bump the Trusty image to one from the new Dockerfile. The new Dockerfile is barely different from the old and shouldn't matter, but in the unlikely case it does, best to find out promptly.
2018-01-31 19:50:36 +01:00
- image: gregprice/circleci:trusty-python-4.test
circle: Draft a config for running our builds in CircleCI. From just my experience attempting this change, I've been impressed with the very fast turnaround in CircleCI for a change -- from `git push` to the build starting is usually just a couple of seconds. It also looks like we get much clearer control over the caching system, and the setup of the environment via a Docker image of our choice, which can help avoid most of the several minutes Travis takes to set up our environment before it actually starts running the tests themselves. Plus it supports fine-grained parallelism, e.g. by test file: https://circleci.com/docs/2.0/parallelism-faster-jobs/ So we can probably get much faster builds by making this switch. Because the environment is much more transparent and even under our control, I'm also hopeful for more reliable builds.
2017-08-30 02:09:09 +02:00
circleci: Set working_directory to zulip. Doesn't do much except that when you ssh into circleci you get a directory called zulip instead of repo.
2018-01-30 21:21:29 +01:00
working_directory: ~/zulip
circle: Draft a config for running our builds in CircleCI. From just my experience attempting this change, I've been impressed with the very fast turnaround in CircleCI for a change -- from `git push` to the build starting is usually just a couple of seconds. It also looks like we get much clearer control over the caching system, and the setup of the environment via a Docker image of our choice, which can help avoid most of the several minutes Travis takes to set up our environment before it actually starts running the tests themselves. Plus it supports fine-grained parallelism, e.g. by test file: https://circleci.com/docs/2.0/parallelism-faster-jobs/ So we can probably get much faster builds by making this switch. Because the environment is much more transparent and even under our control, I'm also hopeful for more reliable builds.
2017-08-30 02:09:09 +02:00
steps:
- checkout
- run:
name: create cache directories
command: |
dirs=(/srv/zulip-{npm,venv}-cache)
sudo mkdir -p "${dirs[@]}"
sudo chown -R circleci "${dirs[@]}"
yarn: Revert back to v0.27.5. Revert yarn version back due to some issue with new version that causes permission issues in ~/.config/yarn directory. Related discussion: https://chat.zulip.org/#narrow/stream/21-provision-help/topic/EACCES.3A.20permission.20denied.2C.20scandir.20'.2Fhome.2Fvagrant.2F.2Econfig.2Fya
2018-04-12 18:46:40 +02:00
circle: Draft a config for running our builds in CircleCI. From just my experience attempting this change, I've been impressed with the very fast turnaround in CircleCI for a change -- from `git push` to the build starting is usually just a couple of seconds. It also looks like we get much clearer control over the caching system, and the setup of the environment via a Docker image of our choice, which can help avoid most of the several minutes Travis takes to set up our environment before it actually starts running the tests themselves. Plus it supports fine-grained parallelism, e.g. by test file: https://circleci.com/docs/2.0/parallelism-faster-jobs/ So we can probably get much faster builds by making this switch. Because the environment is much more transparent and even under our control, I'm also hopeful for more reliable builds.
2017-08-30 02:09:09 +02:00
- restore_cache:
keys:
ci: Make cache keys depend on checksums of dependency manifests Caches in Circle are immutable -- even if a path in a cache changes in builds after the cache was created, the cache is not updated if it already exists. This was making the zulip-venv-cache and zulip-npm-cache directories useless on Circle. This commit changes the cache keys to depend on the checksums of the dependency manifests (requirements/{dev,thumbor}.txt, package.json and yarn.lock). This would ensure that the caches are updated when the environments change. It may result in the occasional build being "uncached" -- when a dependency manifest changes -- but builds without such changes will be much faster, and such builds are a majority.
2018-03-31 22:41:06 +02:00
- v1-npm-base.trusty-{{ checksum "package.json" }}-{{ checksum "yarn.lock" }}
circle: Draft a config for running our builds in CircleCI. From just my experience attempting this change, I've been impressed with the very fast turnaround in CircleCI for a change -- from `git push` to the build starting is usually just a couple of seconds. It also looks like we get much clearer control over the caching system, and the setup of the environment via a Docker image of our choice, which can help avoid most of the several minutes Travis takes to set up our environment before it actually starts running the tests themselves. Plus it supports fine-grained parallelism, e.g. by test file: https://circleci.com/docs/2.0/parallelism-faster-jobs/ So we can probably get much faster builds by making this switch. Because the environment is much more transparent and even under our control, I'm also hopeful for more reliable builds.
2017-08-30 02:09:09 +02:00
- restore_cache:
keys:
ci: Make cache keys depend on checksums of dependency manifests Caches in Circle are immutable -- even if a path in a cache changes in builds after the cache was created, the cache is not updated if it already exists. This was making the zulip-venv-cache and zulip-npm-cache directories useless on Circle. This commit changes the cache keys to depend on the checksums of the dependency manifests (requirements/{dev,thumbor}.txt, package.json and yarn.lock). This would ensure that the caches are updated when the environments change. It may result in the occasional build being "uncached" -- when a dependency manifest changes -- but builds without such changes will be much faster, and such builds are a majority.
2018-03-31 22:41:06 +02:00
- v1-venv-base.trusty-{{ checksum "requirements/thumbor.txt" }}-{{ checksum "requirements/dev.txt" }}
circle: Draft a config for running our builds in CircleCI. From just my experience attempting this change, I've been impressed with the very fast turnaround in CircleCI for a change -- from `git push` to the build starting is usually just a couple of seconds. It also looks like we get much clearer control over the caching system, and the setup of the environment via a Docker image of our choice, which can help avoid most of the several minutes Travis takes to set up our environment before it actually starts running the tests themselves. Plus it supports fine-grained parallelism, e.g. by test file: https://circleci.com/docs/2.0/parallelism-faster-jobs/ So we can probably get much faster builds by making this switch. Because the environment is much more transparent and even under our control, I'm also hopeful for more reliable builds.
2017-08-30 02:09:09 +02:00
- run:
name: install dependencies
command: |
# Install moreutils so we can use `ts` and `mispipe` in the following.
sudo apt-get install -y moreutils
circleci: Remove Circle-created gitconfig file. [Investigation by hackerkid.]
2017-09-20 02:36:30 +02:00
# CircleCI sets the following in Git config at clone time:
# url.ssh://git@github.com.insteadOf https://github.com
# This breaks the Git clones in the NVM `install.sh` we run
# in `install-node`.
# TODO: figure out why that breaks, and whether we want it.
# (Is it an optimization?)
rm -f /home/circleci/.gitconfig
circle: Draft a config for running our builds in CircleCI. From just my experience attempting this change, I've been impressed with the very fast turnaround in CircleCI for a change -- from `git push` to the build starting is usually just a couple of seconds. It also looks like we get much clearer control over the caching system, and the setup of the environment via a Docker image of our choice, which can help avoid most of the several minutes Travis takes to set up our environment before it actually starts running the tests themselves. Plus it supports fine-grained parallelism, e.g. by test file: https://circleci.com/docs/2.0/parallelism-faster-jobs/ So we can probably get much faster builds by making this switch. Because the environment is much more transparent and even under our control, I'm also hopeful for more reliable builds.
2017-08-30 02:09:09 +02:00
# This is the main setup job for the test suite
mispipe "tools/travis/setup-backend" ts
# Cleaning caches is mostly unnecessary in Circle, because
# most builds don't get to write to the cache.
# mispipe "scripts/lib/clean-unused-caches --verbose --threshold 0" ts
- save_cache:
paths:
- /srv/zulip-npm-cache
ci: Make cache keys depend on checksums of dependency manifests Caches in Circle are immutable -- even if a path in a cache changes in builds after the cache was created, the cache is not updated if it already exists. This was making the zulip-venv-cache and zulip-npm-cache directories useless on Circle. This commit changes the cache keys to depend on the checksums of the dependency manifests (requirements/{dev,thumbor}.txt, package.json and yarn.lock). This would ensure that the caches are updated when the environments change. It may result in the occasional build being "uncached" -- when a dependency manifest changes -- but builds without such changes will be much faster, and such builds are a majority.
2018-03-31 22:41:06 +02:00
key: v1-npm-base.trusty-{{ checksum "package.json" }}-{{ checksum "yarn.lock" }}
circle: Draft a config for running our builds in CircleCI. From just my experience attempting this change, I've been impressed with the very fast turnaround in CircleCI for a change -- from `git push` to the build starting is usually just a couple of seconds. It also looks like we get much clearer control over the caching system, and the setup of the environment via a Docker image of our choice, which can help avoid most of the several minutes Travis takes to set up our environment before it actually starts running the tests themselves. Plus it supports fine-grained parallelism, e.g. by test file: https://circleci.com/docs/2.0/parallelism-faster-jobs/ So we can probably get much faster builds by making this switch. Because the environment is much more transparent and even under our control, I'm also hopeful for more reliable builds.
2017-08-30 02:09:09 +02:00
- save_cache:
paths:
- /srv/zulip-venv-cache
ci: Make cache keys depend on checksums of dependency manifests Caches in Circle are immutable -- even if a path in a cache changes in builds after the cache was created, the cache is not updated if it already exists. This was making the zulip-venv-cache and zulip-npm-cache directories useless on Circle. This commit changes the cache keys to depend on the checksums of the dependency manifests (requirements/{dev,thumbor}.txt, package.json and yarn.lock). This would ensure that the caches are updated when the environments change. It may result in the occasional build being "uncached" -- when a dependency manifest changes -- but builds without such changes will be much faster, and such builds are a majority.
2018-03-31 22:41:06 +02:00
key: v1-venv-base.trusty-{{ checksum "requirements/thumbor.txt" }}-{{ checksum "requirements/dev.txt" }}
circle: Draft a config for running our builds in CircleCI. From just my experience attempting this change, I've been impressed with the very fast turnaround in CircleCI for a change -- from `git push` to the build starting is usually just a couple of seconds. It also looks like we get much clearer control over the caching system, and the setup of the environment via a Docker image of our choice, which can help avoid most of the several minutes Travis takes to set up our environment before it actually starts running the tests themselves. Plus it supports fine-grained parallelism, e.g. by test file: https://circleci.com/docs/2.0/parallelism-faster-jobs/ So we can probably get much faster builds by making this switch. Because the environment is much more transparent and even under our control, I'm also hopeful for more reliable builds.
2017-08-30 02:09:09 +02:00
# TODO: in Travis we also cache ~/zulip-emoji-cache, ~/node, ~/misc
circleci: Run frontend tests in circleci.
2018-01-05 12:11:11 +01:00
# The moment of truth! Run the tests.
circle: Draft a config for running our builds in CircleCI. From just my experience attempting this change, I've been impressed with the very fast turnaround in CircleCI for a change -- from `git push` to the build starting is usually just a couple of seconds. It also looks like we get much clearer control over the caching system, and the setup of the environment via a Docker image of our choice, which can help avoid most of the several minutes Travis takes to set up our environment before it actually starts running the tests themselves. Plus it supports fine-grained parallelism, e.g. by test file: https://circleci.com/docs/2.0/parallelism-faster-jobs/ So we can probably get much faster builds by making this switch. Because the environment is much more transparent and even under our control, I'm also hopeful for more reliable builds.
2017-08-30 02:09:09 +02:00
- run:
circleci: Run frontend tests in circleci.
2018-01-05 12:11:11 +01:00
name: run backend tests
circle: Draft a config for running our builds in CircleCI. From just my experience attempting this change, I've been impressed with the very fast turnaround in CircleCI for a change -- from `git push` to the build starting is usually just a couple of seconds. It also looks like we get much clearer control over the caching system, and the setup of the environment via a Docker image of our choice, which can help avoid most of the several minutes Travis takes to set up our environment before it actually starts running the tests themselves. Plus it supports fine-grained parallelism, e.g. by test file: https://circleci.com/docs/2.0/parallelism-faster-jobs/ So we can probably get much faster builds by making this switch. Because the environment is much more transparent and even under our control, I'm also hopeful for more reliable builds.
2017-08-30 02:09:09 +02:00
command: |
. /srv/zulip-py3-venv/bin/activate
mispipe ./tools/travis/backend ts
circleci: Run frontend tests in circleci.
2018-01-05 12:11:11 +01:00
- run:
name: run frontend tests
command: |
. /srv/zulip-py3-venv/bin/activate
mispipe ./tools/travis/frontend ts
circleci: Upload coverage reports to codecov.io. The Travis config does this by a `pip install codecov` in setup and running the resulting `codecov` command at the end. That risks confusion because it makes the library look like something we might be using *during* the tests to collect coverage information. Instead, install and run codecov together at the end. (The docs also suggest using their bash-based uploader instead; but following those directions seems to result in coverage.xml not getting included (maybe not even built), and no coverage data getting uploaded for Python source files, only JavaScript. Not sure why that is, but using the pip package is fine.) Also, because this step is inherently about talking to a third-party service whose reliability we can't control, and because it's purely a reporting step that happens after our actual test suite has succeeded, don't fail if it fails; just print a message. (I'd prefer a somewhat louder message, like turning that step yellow or something in the Circle web UI, but Circle doesn't offer an option like that. If the coverage reports start consistently failing for some reason, then if we're actually reading them we'll notice anyway.) [greg: Edited step name and error message; wrote this commit message.]
2018-01-26 21:08:13 +01:00
- run:
name: upload coverage report
command: |
. /srv/zulip-py3-venv/bin/activate
pip install codecov && codecov \
|| echo "Error in uploading coverage reports to codecov.io."
circle: Draft a config for running our builds in CircleCI. From just my experience attempting this change, I've been impressed with the very fast turnaround in CircleCI for a change -- from `git push` to the build starting is usually just a couple of seconds. It also looks like we get much clearer control over the caching system, and the setup of the environment via a Docker image of our choice, which can help avoid most of the several minutes Travis takes to set up our environment before it actually starts running the tests themselves. Plus it supports fine-grained parallelism, e.g. by test file: https://circleci.com/docs/2.0/parallelism-faster-jobs/ So we can probably get much faster builds by making this switch. Because the environment is much more transparent and even under our control, I'm also hopeful for more reliable builds.
2017-08-30 02:09:09 +02:00
# - store_artifacts: # TODO
# path: var/casper/
# # also /tmp/zulip-test-event-log/
# destination: test-reports
circleci: Run xenial py3.5 backend tests alongside trusty.
2018-01-30 21:20:50 +01:00
"xenial-python-3.5":
docker:
# This is built from tools/circleci/images/xenial/Dockerfile .
- image: gregprice/circleci:xenial-python-3.test
circleci: Set working_directory to zulip. Doesn't do much except that when you ssh into circleci you get a directory called zulip instead of repo.
2018-01-30 21:21:29 +01:00
working_directory: ~/zulip
circleci: Run xenial py3.5 backend tests alongside trusty.
2018-01-30 21:20:50 +01:00
steps:
- checkout
- run:
name: create cache directories
command: |
dirs=(/srv/zulip-{npm,venv}-cache)
sudo mkdir -p "${dirs[@]}"
sudo chown -R circleci "${dirs[@]}"
install-node: Upgrade node, yarn, and nvm. node -> v8.9.4 yarn -> 1.5.1 nvm -> 0.33.8 Also updates a test in timerender.js which depends on time provided by node which is now changed in newer release. Some changes have been made in circeci script, we just create ~/.config directory and chown it to circleci user so installing new version of yarn does not cause any ci failure on circleci during provision.
2018-04-04 20:06:41 +02:00
circleci: Run xenial py3.5 backend tests alongside trusty.
2018-01-30 21:20:50 +01:00
- restore_cache:
keys:
ci: Make cache keys depend on checksums of dependency manifests Caches in Circle are immutable -- even if a path in a cache changes in builds after the cache was created, the cache is not updated if it already exists. This was making the zulip-venv-cache and zulip-npm-cache directories useless on Circle. This commit changes the cache keys to depend on the checksums of the dependency manifests (requirements/{dev,thumbor}.txt, package.json and yarn.lock). This would ensure that the caches are updated when the environments change. It may result in the occasional build being "uncached" -- when a dependency manifest changes -- but builds without such changes will be much faster, and such builds are a majority.
2018-03-31 22:41:06 +02:00
- v1-npm-base.xenial-{{ checksum "package.json" }}-{{ checksum "yarn.lock" }}
circleci: Run xenial py3.5 backend tests alongside trusty.
2018-01-30 21:20:50 +01:00
- restore_cache:
keys:
ci: Make cache keys depend on checksums of dependency manifests Caches in Circle are immutable -- even if a path in a cache changes in builds after the cache was created, the cache is not updated if it already exists. This was making the zulip-venv-cache and zulip-npm-cache directories useless on Circle. This commit changes the cache keys to depend on the checksums of the dependency manifests (requirements/{dev,thumbor}.txt, package.json and yarn.lock). This would ensure that the caches are updated when the environments change. It may result in the occasional build being "uncached" -- when a dependency manifest changes -- but builds without such changes will be much faster, and such builds are a majority.
2018-03-31 22:41:06 +02:00
- v1-venv-base.xenial-{{ checksum "requirements/thumbor.txt" }}-{{ checksum "requirements/dev.txt" }}
circleci: Run xenial py3.5 backend tests alongside trusty.
2018-01-30 21:20:50 +01:00
- run:
name: install dependencies
command: |
CircleCI: Add apt-get update.
2018-05-10 23:27:35 +02:00
sudo apt-get update
circleci: Run xenial py3.5 backend tests alongside trusty.
2018-01-30 21:20:50 +01:00
sudo apt-get install -y moreutils
rm -f /home/circleci/.gitconfig
mispipe "tools/travis/setup-backend" ts
- save_cache:
paths:
- /srv/zulip-npm-cache
ci: Make cache keys depend on checksums of dependency manifests Caches in Circle are immutable -- even if a path in a cache changes in builds after the cache was created, the cache is not updated if it already exists. This was making the zulip-venv-cache and zulip-npm-cache directories useless on Circle. This commit changes the cache keys to depend on the checksums of the dependency manifests (requirements/{dev,thumbor}.txt, package.json and yarn.lock). This would ensure that the caches are updated when the environments change. It may result in the occasional build being "uncached" -- when a dependency manifest changes -- but builds without such changes will be much faster, and such builds are a majority.
2018-03-31 22:41:06 +02:00
key: v1-npm-base.xenial-{{ checksum "package.json" }}-{{ checksum "yarn.lock" }}
circleci: Run xenial py3.5 backend tests alongside trusty.
2018-01-30 21:20:50 +01:00
- save_cache:
paths:
- /srv/zulip-venv-cache
ci: Make cache keys depend on checksums of dependency manifests Caches in Circle are immutable -- even if a path in a cache changes in builds after the cache was created, the cache is not updated if it already exists. This was making the zulip-venv-cache and zulip-npm-cache directories useless on Circle. This commit changes the cache keys to depend on the checksums of the dependency manifests (requirements/{dev,thumbor}.txt, package.json and yarn.lock). This would ensure that the caches are updated when the environments change. It may result in the occasional build being "uncached" -- when a dependency manifest changes -- but builds without such changes will be much faster, and such builds are a majority.
2018-03-31 22:41:06 +02:00
key: v1-venv-base.xenial-{{ checksum "requirements/thumbor.txt" }}-{{ checksum "requirements/dev.txt" }}
circleci: Run xenial py3.5 backend tests alongside trusty.
2018-01-30 21:20:50 +01:00
- run:
name: run backend tests
command: |
. /srv/zulip-py3-venv/bin/activate
mispipe ./tools/travis/backend ts
circleci: Upload to codecov on xenial too. Only takes about 11 seconds; and there may be some codepaths that are exercised on one release and not the other.
2018-01-31 18:41:05 +01:00
- run:
name: upload coverage report
command: |
. /srv/zulip-py3-venv/bin/activate
pip install codecov && codecov \
|| echo "Error in uploading coverage reports to codecov.io."
circleci: Run xenial py3.5 backend tests alongside trusty.
2018-01-30 21:20:50 +01:00
workflows:
version: 2
build:
jobs:
- "trusty-python-3.4"
- "xenial-python-3.5"