Bazel branch coverage ignores Jacoco filtering #12696
Description
Description of the problem / feature request:
Bazel counts branch coverage also for code that would be filtered out by Jacoco.
Function and line coverage (when looking at reports generated by genhtml) seem to be respecting Jacoco filtering.
Jacoco is constantly improving filtering (https://www.jacoco.org/jacoco/trunk/doc/changes.html), but these changes (e.g. filtering out generated bytecode for Java and Kotlin) aren't then taken by Bazel branch coverage.
Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.
- Check out https://github.com/gergelyfabian/bazel-scala-example/tree/jacoco_coverage_fix.
- In bazel repo: bazel build src/java_tools/junitrunner/java/com/google/testing/coverage:JacocoCoverage_jarjar_deploy.jar
- Copy bazel's bazel-bin/src/java_tools/junitrunner/java/com/google/testing/coverage/JacocoCoverage_jarjar_deploy.jar to bazel-scala-example's tools/
- Run tools/coverage.sh
- You'll see in the coverage report branch coverage for lines that are not covered by line coverage (example-lib/src/main/scala/mypackage/Foo.scala):
23 [ + + ]: : d <- if (foo.a < 10) {
24 : : List(4)
25 : : } else {
26 : : List(14)
27 : : }
- Branch coverage is 5/30 for example-lib/src/main/scala/mypackage/Foo.scala.
Note: These lines are not covered because of a bug in Jacoco 0.8.3 (that Bazel currently uses), it was fixed in Jacoco 0.8.5.
When in contrast I follow the instructions to upgrade Jacoco in #11674 (comment) (to Jacoco 0.8.3 with the backported Scala 2.12 lambda fix) and then run the above steps the line coverage changes (the above lines 23-27 are now covered) and the branch coverage stays the same for example-lib/src/main/scala/mypackage/Foo.scala (5/30).
Compare with plain Jacoco:
- Download http://search.maven.org/remotecontent?filepath=org/jacoco/jacoco/0.8.5/jacoco-0.8.5.zip and http://search.maven.org/remotecontent?filepath=org/jacoco/jacoco/0.8.3/jacoco-0.8.3.zip.
- Unpack them into /tmp/jacoco_0.8.3 and /tmp/jacoco_0.8.5.
- Install Scala 2.12.12 into ~/opt/scala-2.12.12
- Copy bazel-scala-example/example-lib/src/main/scala/mypackage/Foo.scala to /tmp/src/mypackage/Foo.scala
- cd /tmp
- ~/opt/scala-2.12.12/bin/scalac src/mypackage/Foo.scala -d classes
- cd /tmp/jacoco_0.8.3
- java -javaagent:lib/jacocoagent.jar -cp ~/opt/scala-2.12.12/lib/scala-library.jar:../classes mypackage.Foo
- java -jar lib/jacococli.jar report jacoco.exec --classfiles ../classes --sourcefiles ../src --html report
- cd /tmp/jacoco_0.8.5
- java -javaagent:lib/jacocoagent.jar -cp ~/opt/scala-2.12.12/lib/scala-library.jar:../classes mypackage.Foo
- java -jar lib/jacococli.jar report jacoco.exec --classfiles ../classes --sourcefiles ../src --html report
- Compare the reports for Jacoco 0.8.3 and 0.8.5. You'll see that branch coverage is changing and that for 0.8.3 lines that have been filtered aren't counted as branches hit or total.
What operating system are you running Bazel on?
Ubuntu 20.04.
What's the output of bazel info release?
release 3.7.1
If bazel info release returns "development version" or "(@Non-Git)", tell us how you built Bazel.
Replace this line with your answer.
What's the output of git remote get-url origin ; git rev-parse master ; git rev-parse HEAD ?
https://github.com/gergelyfabian/bazel-scala-example
8a9b58a48b86c456a0593427fbe217d6f193b404
aa8039f593df6628ef749f0c7bcb462976d45419
Have you found anything relevant by searching the web?
No.
Any other information, logs, or outputs that you want to share?
genhtml output segment for Foo.scala with Bazel 3.7.1 default Jacoco (0.8.3):
5 : 1 : object Foo {
6 : 1 : val message = "hello world"
7 : 1 : val message1 = 1
8 : :
9 : : def testLambdas(a: Int) = {
10 : 1 : println(message1)
11 : 1 : val foo = Foo(a, message, Set.empty[String])
12 [ + - ]: 1 : foo match {
13 [ + + ]: 1 : case Foo(a, _, _) if a > 10 =>
14 : 1 : println("a is bigger than 10")
15 : : case _ =>
16 : 1 : println("a is not bigger than 10")
17 : : }
18 : 1 : println(foo)
19 : : for {
20 : 1 : a <- List(foo)
21 : : b <- List(2)
22 : : c <- List(3)
23 [ + + ]: : d <- if (foo.a < 10) {
24 : : List(4)
25 : : } else {
26 : : List(14)
27 : : }
28 : : } yield a.a + b + c + d
29 : : }
30 : :
31 : : def main(args: Array[String]) {
32 : 1 : testLambdas(1)
33 : 1 : testLambdas(11)
34 : : }
35 : 1 : }
With Jacoco upgraded to 0.8.3 with Scala 2.12 lambda fix patch:
5 : 1 : object Foo {
6 : 1 : val message = "hello world"
7 : 1 : val message1 = 1
8 : :
9 : : def testLambdas(a: Int) = {
10 : 1 : println(message1)
11 : 1 : val foo = Foo(a, message, Set.empty[String])
12 [ + - ]: 1 : foo match {
13 [ + + ]: 1 : case Foo(a, _, _) if a > 10 =>
14 : 1 : println("a is bigger than 10")
15 : : case _ =>
16 : 1 : println("a is not bigger than 10")
17 : : }
18 : 1 : println(foo)
19 : : for {
20 : 1 : a <- List(foo)
21 : 1 : b <- List(2)
22 : 1 : c <- List(3)
23 [ + + ]: 1 : d <- if (foo.a < 10) {
24 : 1 : List(4)
25 : : } else {
26 : 1 : List(14)
27 : : }
28 : 1 : } yield a.a + b + c + d
29 : : }
30 : :
31 : : def main(args: Array[String]) {
32 : 1 : testLambdas(1)
33 : 1 : testLambdas(11)
34 : : }
35 : 1 : }