Adding code coverage report to build · Pull Request #553 · ros-navigation/navigation2

ghost · 2019-02-02T00:52:00Z

Basic Info

Adding code coverage generation to the travis build with a report out on Codecov.io.

I've split the travis CI build into two:
- a release build that eventually gets pushed to dockerhub, and
- a debug build that includes code coverage generation. The results to that build get pushed to codecov.io
I added a script to to the tools folder that runs the test suite, so that only one line is needed in the travis.yml file instead of 4 or so.
I added a script to the tools folder that generates the code coverage data. If you run it without options it generates an html report you can view locally. If you run it with the option codecovio, it uploads the data to codecov.io which generates a report.
I could not use colcon/colcon-lcov-result on account of Header files in install directories are filtered out of the results colcon/colcon-lcov-result#9, but will keep an eye on it for future refactoring.
The reports generated on codecov.io are a bit wonky because we don't have a baseline report yet. I think I can't clean up any further issues with codecov.io until we integrate this PR to get baseline data.

codecov · 2019-02-03T20:04:51Z

Codecov Report

❗ No coverage uploaded for pull request base (master@0deebc1). Click here to learn what that means.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master     #553   +/-   ##
=========================================
  Coverage          ?   37.88%           
=========================================
  Files             ?      216           
  Lines             ?     9006           
  Branches          ?     3516           
=========================================
  Hits              ?     3412           
  Misses            ?     3384           
  Partials          ?     2210

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0deebc1...6cf7498. Read the comment docs.

ghost · 2019-02-06T17:33:18Z

Fixes #498

ruffsl · 2019-02-07T20:48:44Z

Is this something we'd could for PRs as well? What is the time overhead like when running those code coverage steps?

ghost · 2019-02-08T16:26:25Z

Is this something we'd could for PRs as well? What is the time overhead like when running those code coverage steps?

Yes. This will run on PRs. The coverage steps themselves take on the order of 30 seconds. The biggest overhead is that now we build the stack a second time just to collect coverage data,

mkhansenbot

Looks good to me, thanks for adding this!

ruffsl · 2019-02-08T19:43:56Z

.circleci/config.yml

+        name: Report Code Coverage
+        command: |
+          export ci_env=`bash <(curl -s https://codecov.io/env)`
+          . install/setup.sh


Do we need to source the workspace to run codecove?

I think so, but I'm not 100% sure. It might be that we have to do it because we don't have all the right exec dependencies listed in our package.xml files.

.circleci/config.yml

ruffsl · 2019-02-08T19:48:18Z

Yes. This will run on PRs. The coverage steps themselves take on the order of 30 seconds. The biggest overhead is that now we build the stack a second time just to collect coverage data,

It looks like the CircleCI was missing the run steps to upload the codecove files for PRs, so I added 38bb5fa to try and include that. Though I'm not sure I understand the environment setup codecov needs, so please check that I've invoked it correctly. Travis RP job's codcov upload might overwrite CircleCI upload though, so we may want to pick just one for PRs.

.circleci/config.yml

ghost · 2019-02-09T01:02:58Z

.circleci/config.yml

@@ -186,6 +199,7 @@ executors:
    working_directory: /opt/nav2_ws
    environment:
      CMAKE_BUILD_TYPE: "Release"


For a COVERAGE_ENABLED build, it'll need to be CMAKE_BUILD_TYPE=Debug to produce a useful report. With the release optimizations, the coverage data will probably be unusable.

We'll also need to set a token in the CI environment that the upload script uses. See https://codecov.io/gh/ros-planning/navigation2/settings
I believe you have access.

However, before we set that for circle CI, I think we should only upload from either Travis or Circle. Uploading twice for the same PR might be an issue. I was going to see how much of our 1000 minutes we were using before adding more to the CircleCI build. Do you know how to check the amount of time we've used?

Nevermind. I just noticed you said the exact same thing.

For a COVERAGE_ENABLED build, it'll need to be CMAKE_BUILD_TYPE=Debug to produce a useful report. With the release optimizations, the coverage data will probably be unusable.

Should we just default to Debug builds for PRs, using a nightly cron on master to build/test both. The cron could have a forked workflow:

testing with Debug (and thus codecove for master branch)

testing Release (to make sure at least master is periodically Release tested)

I was going to see how much of our 1000 minutes we were using before adding more to the CircleCI build. Do you know how to check the amount of time we've used?

I can't see this, but org admins should be able to view the stats here:
https://circleci.com/gh/organizations/ros-planning/settings#usage

Although, open source projects (jobs for public repos I presume) shouldn't be affected.

https://discuss.circleci.com/t/has-the-build-minutes-per-month-in-the-free-tier-been-reduced/26399/3

However, before we set that for circle CI, I think we should only upload from either Travis or Circle.

I could comment out the codecov line in Travis to just test out Circleci. Once we merge, circleci config used for PRs would be updated, so we could then profile the added time for codecov steps.

ruffsl · 2019-02-09T02:52:04Z

I changed the workflow to trigger parallel builds for Release and Debug. I'm not sure if we need to test release on PRs as well, but currently only the debug job includes the command to upload to codecov. We can rework the trigger filter to be whatever combination we'd need, like only testing both on master branch, ect.

ghost · 2019-02-13T00:56:17Z

@orduno @mhpanah @ruffsl Looks like this is now passing all the CI it can at the moment. It needs one more approver before it can go in. If one of you can review and approve, I'd appreciate it.

ruffsl

LGTM

…nterface (ros-navigation#553) * [AdmittanceController] Addintional argument in methods of ControllerInterface * Update admittance_controller.cpp

ghost mentioned this pull request Feb 4, 2019

Add CircleCI #534

Merged

12 tasks

ghost changed the title ~~[WIP] Adding code coverage report to build~~ Adding code coverage report to build Feb 4, 2019

ghost requested review from mhpanah and orduno February 5, 2019 23:40

ghost mentioned this pull request Feb 6, 2019

Add BT Navigator System Test #557

Merged

mkhansenbot approved these changes Feb 8, 2019

View reviewed changes

ruffsl reviewed Feb 8, 2019

View reviewed changes

.circleci/config.yml Outdated Show resolved Hide resolved

ghost commented Feb 9, 2019

View reviewed changes

Carl Delsey added 15 commits February 12, 2019 14:18

Script to generate coverage report. Should be run after test.

12e56ac

Add helper script for test suite

857ba30

Modified travis to use runtestsuite.bash script

ac4fe7f

Add coverage to .travis.yml file.

a8e772f

Add coverage badge to main page

f7a28df

Run the code coverage report in the docker container.

d0d8264

Add lcov dependency.

aa76783

Trying to add lcov tool to build.

3d71653

Add coverage args to build script.

0b38026

Fixing passing code coverage args.

4225761

Fixing setting env variable.

9d43623

Fixing docker file coverage_enabled flag.

ad95353

Fixing coverage report script.

a48a7b5

Pass ci env through to docker

9f13e55

Fixing the filter rules for this workspace.

a5f5f44

Carl Delsey and others added 10 commits February 12, 2019 14:18

Removing unneeded dependency install line.

a3404b8

Add comments explaining what lcov lines do.

abea802

Rebasing on top of circle CI and common Cmake functions changes.

68aea56

Fixing test suite script name.

72f2856

Add codecov to CircleCI jobs

0521577

Add conditional to skip codecov if enable is false

11acdb7

Remove unnecessary export for codecov

f124aa3

Update workflow for paralel release and debug jobs

70d2783

Forgot to append coverage step to job

bb856c6

Comment out travis codecove

6cf7498

ruffsl approved these changes Feb 13, 2019

View reviewed changes

ghost merged commit 357e303 into ros-navigation:master Feb 13, 2019

ruffsl mentioned this pull request Mar 19, 2019

Update Docker build and add CircleCI + CodeCov support moveit/moveit2#44

Closed

15 tasks

orduno mentioned this pull request May 16, 2019

enabling closed-loop control in spinning #723

Closed

This pull request was closed.

Conversation

ghost commented Feb 2, 2019 • edited by ghost Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Basic Info

Uh oh!

codecov bot commented Feb 3, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

ghost commented Feb 6, 2019

Uh oh!

ruffsl commented Feb 7, 2019

Uh oh!

ghost commented Feb 8, 2019

Uh oh!

mkhansenbot left a comment

Choose a reason for hiding this comment

Uh oh!

ruffsl Feb 8, 2019

Choose a reason for hiding this comment

Uh oh!

ghost Feb 9, 2019

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ruffsl commented Feb 8, 2019

Uh oh!

Uh oh!

ghost Feb 9, 2019

Choose a reason for hiding this comment

Uh oh!

ghost Feb 9, 2019

Choose a reason for hiding this comment

Uh oh!

ruffsl Feb 9, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ruffsl commented Feb 9, 2019

Uh oh!

ghost commented Feb 13, 2019

Uh oh!

ruffsl left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ghost commented Feb 2, 2019 •

edited by ghost

Loading

codecov bot commented Feb 3, 2019 •

edited

Loading

ruffsl Feb 9, 2019 •

edited

Loading