Java code coverage for sh_tests missing #5331
Description
Description of the problem:
Copying from bazel-discuss thread per request:
We have a mix of JUnit tests and sh_tests that invoke a binary in a test scenario. bazel coverage //... --jobs 1 [0] works great for the unit tests, but does not generate a coverage.dat for the sh_tests.
Repo is open source: https://github.com/batfish/batfish@c6658e88 is base, and https://github.com/batfish/batfish/blob/c6658e88c8c9d48e393547281cc119d99b90670d/tests/parsing-tests/BUILD is the integration sh_test.
I tried figuring out how the Java coverage works under the hood, but I got fairly stymied. Any guidance?
[0] --jobs 1 workaround for crash: #4398
Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.
Check out the linked repository at the linked commit and run bazel coverage --jobs 1 //.... Command log below:
➜ batfish git:(master) git checkout c6658e88c8c9d48e393547281cc119d99b90670d
Note: checking out 'c6658e88c8c9d48e393547281cc119d99b90670d'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:
git checkout -b <new-branch-name>
HEAD is now at c6658e88c Local interface in bgp session status (#1588)
➜ batfish git:(c6658e88c) bazel clean --expunge
INFO: Starting clean.
➜ batfish git:(c6658e88c) bazel coverage --jobs 1 //...
INFO: SHA256 (https://services.gradle.org/distributions/gradle-4.1-bin.zip) = d55dfa9cfb5a3da86a1c9e75bb0b9507f9a8c8c100793ccec7beb6e259f9ed43
INFO: Using default value for --instrumentation_filter: "//projects,//tests/parsing-tests".
INFO: Override the above default with --instrumentation_filter
INFO: Analysed 59 targets (190 packages loaded).
INFO: Found 52 targets and 7 test targets...
INFO: Elapsed time: 289.417s, Critical Path: 47.31s
INFO: 324 processes: 261 darwin-sandbox, 13 local, 50 worker.
INFO: Build completed successfully, 500 total actions
//projects/allinone:allinone_tests PASSED in 6.0s
/private/var/tmp/_bazel_dan/ffd2f268833102995456f3fc3b57cee1/execroot/batfish/bazel-out/darwin-fastbuild/testlogs/projects/allinone/allinone_tests/coverage.dat
//projects/batfish:batfish_tests PASSED in 23.8s
/private/var/tmp/_bazel_dan/ffd2f268833102995456f3fc3b57cee1/execroot/batfish/bazel-out/darwin-fastbuild/testlogs/projects/batfish/batfish_tests/coverage.dat
//projects/batfish-client:client_tests PASSED in 2.1s
/private/var/tmp/_bazel_dan/ffd2f268833102995456f3fc3b57cee1/execroot/batfish/bazel-out/darwin-fastbuild/testlogs/projects/batfish-client/client_tests/coverage.dat
//projects/batfish-common-protocol:common_tests PASSED in 3.1s
/private/var/tmp/_bazel_dan/ffd2f268833102995456f3fc3b57cee1/execroot/batfish/bazel-out/darwin-fastbuild/testlogs/projects/batfish-common-protocol/common_tests/coverage.dat
//projects/coordinator:coordinator_tests PASSED in 5.7s
/private/var/tmp/_bazel_dan/ffd2f268833102995456f3fc3b57cee1/execroot/batfish/bazel-out/darwin-fastbuild/testlogs/projects/coordinator/coordinator_tests/coverage.dat
//projects/question:question_tests PASSED in 1.8s
/private/var/tmp/_bazel_dan/ffd2f268833102995456f3fc3b57cee1/execroot/batfish/bazel-out/darwin-fastbuild/testlogs/projects/question/question_tests/coverage.dat
//tests/parsing-tests:ref_tests PASSED in 25.0s
Executed 7 out of 7 tests: 7 tests pass.
There were tests whose specified size is too big. Use the --test_verbose_timeout_warnings command line option to see wINFO: Build completed successfully, 500 total actions
Note that there were 7 tests run, only 6 coverage.dat files produced. The 6 tests are junit_tests and the 7th is a sh_test.
Note that I would actually be (very pleasantly) surprised if this worked as-is... I'd expect I might have to do something in the generated sh_test to indicate I wanted Java coverage. I just don't know what that something might be...
What operating system are you running Bazel on?
macOS High Sierra 10.13.5 (17F77)
What's the output of bazel info release?
release 0.14.0-homebrew
If bazel info release returns "development version" or "(@Non-Git)", tell us how you built Bazel.
N/A
What's the output of git remote get-url origin ; git rev-parse master ; git rev-parse HEAD ?
N/A I believe.
Have you found anything relevant by searching the web?
-
bazel-discuss thread linked above.
-
I was perhaps wondering if a workaround would be a Skylark rule using
coverage_instrumentedbut have not had a chance to try this yet.