Add CircleCI by ruffsl · Pull Request #534 · ros-navigation/navigation2

ruffsl · 2019-01-24T22:10:15Z

This PR add CI support for CircleCI.

Summary

Like Travis, CircleCI is a Continuous Integration service, however CircleCI provides more docker integration such as support for custom images without resorting to machine type executors. This means that CI jobs can start as soon as they are triggered, rather than queuing for an available Virtual Machine to then spin up. The pulling of the Docker Image is perhaps the only main flat overhead in triggering jobs. Further features such as caching and storage of artifacts additionally accelerate job times while providing insight into build and test results.

Overview

The proposed CI would start with pre-build Docker Image, hosted on automated repo on Docker Hub that would trigger tag builds upon new commits into particular GitHub branches or rebuilds of linked Docker Hub repos, i.e. osrf/ros2:nightly base image. Built tags would include all/most the necessary dependencies for the project, include any build dependencies for prerequisite dependencies, i.e. running vcs import, rosdep install, etc. on the master branch. Even when short cutting the install time by building FROM the osrf/ros2:nightly base image, this takes about 90 min on Docker Hub, so it helpful to offload/bake this amount of setup into the image our CI pulls from.

Once the CI pulls the docker image above, the config proceeds to clean the workspaces and checkout the source to inspect. This also entails vcs importing and prerequisite defined in the new checked out source, installing the prerequisite dependencies, building the prerequisites themselves, installing the source packages dependencies, building the source packages, then testing the source packages.

In between steps in the job of the workflow, caching of the build stages for both prerequisite and source packages is attempted. Caches are broken with either the installed dependencies change or the source code has changed. This allows the cache to persist across jobs.

An example of current table of caching in action:

prerequisites	source build	source tests	time
w/o	w/o	w/o	16:19
w	w/o	w/o	05:07
w	w	w/o	02:07

Lastly, the log files for the colcon build and test steps are also archived and made linkable by storing them with the job's public artifacts.

Workflow

DockerHub builds CI Image
- Triggered either from linked base image or branch commits
- Shortcuts CI by installing the majority of decencies
CircleCI
- Triggered by any workflow condition, e.g. commit, PR, web-hook
- Pulls CI image from DockerHub
- Cleans source and prerequisite workspaces
- Checks out respective source from trigger
- Prerequisite
  - Checks out prerequisite from .repo file in source
  - Installs any new/missing prerequisite dependencies
  - Restores prerequisite build cache if possible
  - Builds prerequisite if cache restore falls through
  - Saves prerequisite build cache if needed
- Source
  - Installs any new/missing source dependencies
  - Restores source build cache if possible
  - Builds source if cache restore falls through
  - Saves source build cache if needed
  - Save build log artifacts
- Test
  - Test source build
  - Save test log artifacts

Action Items

DockerHub
- Create the ros-planning DockerHub organization (admin)
- Invite relevant Collaborators to DockerHub org (admin)
- Link GitHub navigation2 automated repo to DockerHub org (admin)
- Configure build rules for automated repo (@ruffs)
  - Add rule for branchs, e.g. ros-planning/navigation2:master
  - Add repo link for Base Image: osrf/ros2:nightly
CircleCI
- Enable GitHub integration (admin)
- Invite relevant Collaborators to GitHub Team (admin)
- Configure Advanced Settings for ci repo (@ruffs)
  - Only build pull requests (On/Off)?
    - Save build usage by only testing open PRed branches
  - Auto-cancel redundant builds (On/Off)?
    - Save build usage by only testing current HEAD of branch
  - Build forked pull requests (On/Off)?
    - Looking at repo's history, most merged PRs seem to come from forks

References:

CircleCI

Configuration Reference
- To get the gist of the config file in PR
Building Open Source Projects
- Special considerations for Open Source Projects
Migrating From Travis CI
- Other notes/suggestions and highlighting differences in CI
Caching
- More on Caching Strategy Tradeoffs when used between workflows and jobs
Debugging with SSH
- This was helpful when writing/debugging this config
Caching apt-get packages
- Using 2.0, this is now done by priming the docker image
Storing Build Artifacts
- Used make colcon build and test log files more accessible

@crdelsey , its partly both. Most of the upfront time is saved by starting from a Docker Image prepped for the CI with common dependence pre-installed, but then some more is also saved by making use of the caching between jobs. We could perhaps save even more by enabling Docker Layer Caching, but this currently seems to be a premium feature from CircleCI.

If we switch to CircleCI, do we still need to have Travis as well, or can we do everything within Circle to keep things simpler?

Well, I'm not sure. You could have both if you wanted, don't think it would be much drain. You might want to have only one or the other CI enabled for PRs to prevent too much integration noise. CircleCI might be better for this, as long as we can keep the quick turnaround times I've tabled above.

I'm not sure Travis has the same monthly limit as CircleCI, so it may still be well suited for longer builds via periodic cron jobs. E.g. just rebuilding the whole Dockerfile daily via master branch and run the stock tests and code coverage.

Also ping @mkhansen-intel , @SteveMacenski , I've updated the top PR comment with more details.

.circleci/config.yml

ghost

I'll admit I don't understand the details of docker cache breaks yet, but I see no harm in adding Circle CI. It would definitely be nice to have faster feedback from the CI system, and it seems to best way to get a feel for Circle vs Travis is to turn this on and get some experience with both side by side.

for CI testing and example nightly jobs

Add distro digest to checksum chain to break caches Compare .repo files from checkout to determine if upstream needs cleaning

navigation2 -> source

thus overwrite checksum upon first write

ghost · 2019-02-01T18:33:25Z

docker hub doesn't allow symbols in an organization name, so I called the org rosplanning instead of ros-planning

ruffsl · 2019-02-01T19:38:29Z

Ok, I've polished up the caching chain so it should skip the upstream re-build if the .repo file is unchanged from what already existed in the CI docker image, as well as breaking the cache if the ros distro files update. Also updated the test the ctest retry to match travis, and updated the image used to point to the rosplanning org.

docker hub doesn't allow symbols in an organization name, so I called the org rosplanning instead of ros-planning

Sounds fine. @crdelsey , it looks like these items were left undone, so my own account can not complete the first due to the absence of the second.

Link GitHub navigation2 automated repo to DockerHub org (admin)

Configure build rules for automated repo (@ruffs)

I.e I'm unable to setup the Build configurations for the rosplanning/navigation2 repo

https://cloud.docker.com/repository/docker/rosplanning/navigation2/builds/edit

Your account should be able to configure the repo to resemble what I have on my fork:

ghost · 2019-02-01T19:53:06Z

I just linked DockerHub to Github, so you should have access now. I also set up the Dockerhub build as you showed above and triggered it.

To ensure test package is built

ruffsl · 2019-02-01T22:32:28Z

Alright, 2f02de1 is passing on my end here:
https://circleci.com/gh/ruffsl/navigation2/80

I don't see the CircleCI from ros-planning though:
https://circleci.com/gh/ros-planning/navigation2

@crdelsey, did you enable Build forked pull requests in the CircleCI repo settings for navigation2?

After we see green light from ros-planning's circleci on this PR, this should be good to merge.
I may have to rebase or something to retrigger a fork PR job after that setting is enabled.

ghost · 2019-02-04T17:42:37Z

@ruffsl I did enable "Build forked pull requests". I made you an admin of this repo so I'm not slowing things down. You should be able to check any settings directly.

ghost · 2019-02-04T17:53:54Z

@ruffsl Circle CI is running on #553, so it could just be that it doesn't run on this PR because it was created before CircleCI was enabled. I'm willing to just merge to master and see what happens. We can then quickly fix any problems that come up or just revert the change if the problem is big, which seems unlikely.

ruffsl · 2019-02-04T18:26:55Z

it could just be that it doesn't run on this PR because it was created before CircleCI was enabled.

That just might be it.

I'm willing to just merge to master and see what happens. We can then quickly fix any problems that come up or just revert the change if the problem is big, which seems unlikely.

Sounds good to me.

ghost · 2019-02-04T19:59:18Z

@orduno @mhpanah @mkhansen-intel Can one of you review this so we can merge?

mkhansenbot

I also am new to CircleCI, but this looks good to me.

ruffsl · 2019-02-05T18:20:59Z

Well, after the merge commit triggered a fresh CI job, it looks like the CTest for nav2_dynamic_params is a little flaky. Remember that the test logs are accessible via the artifacts tab:

https://circleci.com/gh/ros-planning/navigation2/16#artifacts

Although the artifacts tab may seem invisible if your not logging into github, one can force it visible by just adding #artifacts to the job url. Related:

https://circleci.com/ideas/?idea=CCI-I-257

ghost · 2019-02-05T18:27:04Z

I had seen occasional failures with that test on my local machine, but never in CI. I figured it was only flaky after a few runs and various stops and restarts. Looks like it is flaky in CI too now. :-(

I restarted the build and it passed on the second try, so it seems good. Now I just need to figure out what's going on with that test.

ghost · 2019-02-05T18:34:44Z

I opened #556 for the flaky test. By the way, thanks for doing all this work cleaning up the CI stuff for us. I really appreciate it!

* Add description for new BtNavigator param * Fix typo in Iron.rst * Add description for new BtActionServer/Navigator param * Add exemplary param config

Bumps [ros-tooling/setup-ros](https://github.com/ros-tooling/setup-ros) from 0.5.0 to 0.6.0. - [Release notes](https://github.com/ros-tooling/setup-ros/releases) - [Commits](ros-tooling/setup-ros@0.5.0...0.6.0) --- updated-dependencies: - dependency-name: ros-tooling/setup-ros dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

ruffsl and others added 10 commits January 16, 2019 19:56

Refactor Dockerfile

14487f0

Use upstream images Reorganize build stages

Cleaning dockerfile

de873e2

use env to export ROS_DOMAIN_ID remove mapserver to be managed by rosdep don't ignore errors during rosdep install

Source only navigation workspace

a92094b

workspace chaining done durring initializion should suffice

tag docker container so it registers to the repo

9614dc1

Disable unstable system test in Travis.

c753747

Merge commit 'c753747ae347dfb2433cc650adc172d062592492' into nightly

76c1a37

Merge commit '9614dc1dd456ed9c74d1380a52f4e54e1c4be5ff' into nightly

69accc7

Cleaning up Travis CI

5b460ff

Copy repo from local build context

5bfe740

rather than cloning/checking branch URL

Ensure proxy setting is lowercase

6a5a6d5

https://serverfault.com/questions/571887/why-wget-doesnt-read-http-proxy-in-uppercase-format

ruffsl force-pushed the circleci branch 3 times, most recently from c82e784 to a16de31 Compare January 26, 2019 01:50

Add CircleCI

c603159

ruffsl force-pushed the circleci branch from ae85927 to c603159 Compare January 28, 2019 02:20

ruffsl commented Jan 28, 2019

View reviewed changes

.circleci/config.yml Outdated Show resolved Hide resolved

ghost requested review from a user, SteveMacenski, mkhansenbot and orduno January 28, 2019 23:10

ruffsl mentioned this pull request Jan 30, 2019

Update CI Dockerfiles #525

Merged

ghost approved these changes Jan 30, 2019

View reviewed changes

ruffsl added 6 commits January 31, 2019 12:41

Change testing and log handling into references

e21e47a

Add workflow

ad09292

for CI testing and example nightly jobs

Prerequisites -> Upstream

c2f7179

Use command to compose steps

398bde2

Rename executor

7dff98d

Enable distro changes to break build cache

e7e4e81

ruffsl added 9 commits January 31, 2019 19:37

Add distro to checksum and preserve upstream build if .repos unchanged

37d88df

Add distro digest to checksum chain to break caches Compare .repo files from checkout to determine if upstream needs cleaning

Generalize element names

d286ce0

navigation2 -> source

Add info message

8b2f762

Ensure chaining checksum does not append to restored checksum

d086997

thus overwrite checksum upon first write

Store the build checksum for debugging the cache

c37ef38

Try setting mtime to get repeatable digests

9a3fe42

Use directory timestamp instead

9fcb6aa

Use a file timestamp for reference instead of directory

e2b8f75

Merge branch 'master' into circleci

c3c175f

ruffsl added 2 commits February 1, 2019 11:07

Add missing command for ctest retry

ce56067

Switch to using rosplanning DockerHub org

48ef768

Remove COLCON_IGNORE in nav2_system_tests after checkout

2f02de1

To ensure test package is built

mkhansenbot approved these changes Feb 5, 2019

View reviewed changes

ghost merged commit 69d9e04 into ros-navigation:master Feb 5, 2019

ruffsl mentioned this pull request Feb 23, 2019

Add CircleCI Code Coverage to MoveIt2 moveit/moveit2#24

Closed

ruffsl mentioned this pull request Mar 13, 2019

Update Docker build and add CircleCI + CodeCov support moveit/moveit2#44

Closed

15 tasks

ruffsl deleted the circleci branch July 30, 2019 05:42

This pull request was closed.

Conversation

ruffsl commented Jan 24, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Overview

Workflow

Action Items

References:

CircleCI

Related:

Uh oh!

ghost commented Jan 25, 2019

Uh oh!

ruffsl commented Jan 28, 2019

Uh oh!

Uh oh!

ghost left a comment

Choose a reason for hiding this comment

Uh oh!

ghost commented Feb 1, 2019

Uh oh!

ruffsl commented Feb 1, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ghost commented Feb 1, 2019

Uh oh!

ruffsl commented Feb 1, 2019

Uh oh!

ghost commented Feb 4, 2019

Uh oh!

ghost commented Feb 4, 2019

Uh oh!

ruffsl commented Feb 4, 2019

Uh oh!

ghost commented Feb 4, 2019

Uh oh!

mkhansenbot left a comment

Choose a reason for hiding this comment

Uh oh!

ruffsl commented Feb 5, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ghost commented Feb 5, 2019

Uh oh!

ghost commented Feb 5, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ruffsl commented Jan 24, 2019 •

edited

Loading

ruffsl commented Feb 1, 2019 •

edited

Loading

ruffsl commented Feb 5, 2019 •

edited

Loading