Implement the new test-runner's API usage by andre15silva · Pull Request #35 · ASSERT-KTH/flacoco

andre15silva · 2021-06-24T13:22:23Z

This PR implements the usage of the new test-runner's API.

On important aspect is that the CoverageCollectorDetailed transformer doesn't consider constructor coverage (See https://github.com/STAMP-project/test-runner/blob/5b84d1a940564a1c1ee0b22b695503fab4600d97/src/main/java/eu/stamp_project/testrunner/listener/impl/CoverageCollectorDetailed.java#L45-L52), so I have modified the existing tests to follow that.

Also, test-runner doesn't support test coverage by default, so to support that we need to implement it in EntryPoint as an option too.

I'm not entirely sure this is what we want to do tho, as some constructors may have bugs and the coverage there will be important. WDYT?

andre15silva · 2021-06-24T13:24:49Z

Another important issue is that the test suite is failing (at least locally) when the tests are run all together due to:

Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.NoClassDefFoundError: org/junit/platform/launcher/TestExecutionListener
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:756)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
at java.lang.Class.getMethod0(Class.java:3018)
at java.lang.Class.getMethod(Class.java:1784)
at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:650)
at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:632)
Caused by: java.lang.ClassNotFoundException: org.junit.platform.launcher.TestExecutionListener
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 19 more

It's really weird, specially because test-runner itself runs several tests too in its test suite and doesn't have this issue (I've also checked the pre-conditions and there don't seem to be any that affect this).

Have you ever come across an issue like this? (maybe @danglotb?)

martinezmatias · 2021-06-24T13:27:31Z

Hi @andre15silva

On important aspect is that the CoverageCollectorDetailed transformer doesn't consider constructor coverage

Does it mean it should the necessary to remove this if condition on test-runner?

andre15silva · 2021-06-24T13:30:04Z

Does it mean it should the necessary to remove this if condition on test-runner?

In this case, we would add a new option to EntryPoint, but yes, if we want coverage for the constructor instructions we need to do it. WDYT?

martinezmatias · 2021-06-24T13:32:00Z

we would add a new option to EntryPoint

An option would be fine, which by default does not compute coverage of constructor. Otherwise the change will break the test-runner.

andre15silva · 2021-06-24T13:34:36Z

An option would be fine, which by default does not compute coverage of constructor. Otherwise the change will break the test-runner.

Alright. I suggest we do it separately from STAMP-project/test-runner#95, so it doesn't get bigger than it already is.

martinezmatias · 2021-06-24T13:35:27Z

I suggest we do it separately from STAMP-project/test-runner#95, so it doesn't get bigger than it already is.

Fine.

danglotb · 2021-06-24T14:48:32Z

Hello,

I remember to have already the mentioned issue:

Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.NoClassDefFoundError: org/junit/platform/launcher/TestExecutionListener

AFAIR, the classpath was missing a dependency like junit-platform-launcher:

<dependency>
            <groupId>org.junit.platform</groupId>
            <artifactId>junit-platform-launcher</artifactId>
            <version>1.3.2</version>
 </dependency>

This happens when you run the test suite of flacoco locally, isn't it? When I manage to have some time, I'll try to have a look at it.

andre15silva · 2021-06-24T15:23:10Z

Hello,

Hi @danglotb

This happens when you run the test suite of flacoco locally, isn't it? When I manage to have some time, I'll try to have a look at it.

It happens when I run the whole test suite, and sometimes when I run test methods individually.

For example:

# set up test projects
./.github/install_examples.sh

# this works
mvn -Dtest="CoverageRunnerTest#testExampleFL1" clean test

# this results in several failures, including the test that passed above
mvn clean test

In CI the behavior seems to be the same.

Also, I tried adding that dependency but didn't do anything.

danglotb · 2021-06-29T07:48:13Z

Hi @andre15silva,

It seems that here you gather the required jars from the folder ./examples/libs/.

However, one of them, i.e. junit-jupiter-api-5.7.2.jar, is missing (./examples/libs/).
Thus, it seems that test-runner needs (for now) the following jar:

org/junit/platform/junit-platform-launcher/X.Y.Z/junit-platform-launcher-X.Y.Z.jar

On my side, X.Y.Z = 1.3.2. Adding this jar might require adding some others jars to make JUnit5 works.

The actions that could be taken are:

Add the required jars to the ./examples/libs/ folder.
Or refactor test-runner in order to not need the junit-platform-launcher

Hope it can help! :)

andre15silva · 2021-06-29T09:12:34Z

Hi @danglotb ,

Thank you for your time.

You are right, when I added the remaining jar files to the classpath some of the tests started working.

I implemented a way of having default classpaths and configurable ones. I think for now it works, but integrating it in test-runner probably wouldn't be a bad idea either.

In other news, I spend way too much time debugging test-runner and flacoco just to find, in the end, that because EntryPoint options weren't getting reset (I thought they were) and that ended up running the JUnit5 runner for JUnit4 examples.

Anyway, that should now be fixed and this PR is now ready for review (@martinezmatias ?)

Signed-off-by: André Silva <andre15andre@hotmail.com>

Constructors are no longer considered in the computation of coverage, so we remove them from tests. Cover tests option is not yet supported through test-runner's EntryPoint, so we make it unsupported temporarily. Signed-off-by: André Silva <andre15andre@hotmail.com>

Signed-off-by: André Silva <andre15andre@hotmail.com>

I literally spent 5 or 6 hours debugging this and test-runner, just to in the end realize it was simply running the wrong test framework. Signed-off-by: André Silva <andre15andre@hotmail.com>

Signed-off-by: André Silva <andre15andre@hotmail.com>

andre15silva · 2021-06-29T12:20:31Z

Rebased and ready for review.

martinezmatias · 2021-06-29T13:14:35Z

src/main/java/fr/spoonlabs/flacoco/FlacocoMain.java

 public class FlacocoMain implements Callable<Integer> {
 	static boolean log = true;

+	@Option(names = { "-w", "--workspace"}, description = "Path to the workspace directory of flacoco.", defaultValue = "./")


I wonder how it would work when we use Flacoco with a Jar: it will not be a workspace

I'm not sure I understood. I just ran the jar with that option and it works fine.

That option, for now, is only passed to test-runner, which will then save and load serialized files inside of it.

How could running it through the packaged jar result in different behavior?

Ok, perfect. I misunderstood. Sorry :)

Signed-off-by: André Silva <andre15andre@hotmail.com>

This was referenced Jun 24, 2021

CoveredTestResultPerTestMethod API STAMP-project/test-runner#95

Merged

Timeout for test running #7

Closed

andre15silva changed the title ~~WIP: Implement the new test-runner's API usage~~ Implement the new test-runner's API usage Jun 29, 2021

andre15silva mentioned this pull request Jun 29, 2021

Command line interface using picocli #40

Merged

andre15silva added 8 commits June 29, 2021 13:59

Implement the new test-runner's API usage

37e9acc

Signed-off-by: André Silva <andre15andre@hotmail.com>

Update tests.

8147f5e

Constructors are no longer considered in the computation of coverage, so we remove them from tests. Cover tests option is not yet supported through test-runner's EntryPoint, so we make it unsupported temporarily. Signed-off-by: André Silva <andre15andre@hotmail.com>

Add snapshot repository

dcd09c9

Signed-off-by: André Silva <andre15andre@hotmail.com>

Set test-runner to be verbose

a8d862a

Signed-off-by: André Silva <andre15andre@hotmail.com>

Fix classpath issue. Try to run in CI to see if other issue is gone.

2da8a0d

Signed-off-by: André Silva <andre15andre@hotmail.com>

Fix flakyness.

237c979

I literally spent 5 or 6 hours debugging this and test-runner, just to in the end realize it was simply running the wrong test framework. Signed-off-by: André Silva <andre15andre@hotmail.com>

Clean up custom classpaths

da08b2e

Signed-off-by: André Silva <andre15andre@hotmail.com>

Implement new options in the cli

06e6aa6

Signed-off-by: André Silva <andre15andre@hotmail.com>

andre15silva force-pushed the covered-test-result-api branch from 1f82b17 to 06e6aa6 Compare June 29, 2021 12:20

martinezmatias reviewed Jun 29, 2021

View reviewed changes

Fix mainclass in jar

58ae203

Signed-off-by: André Silva <andre15andre@hotmail.com>

martinezmatias merged commit 1bb153b into ASSERT-KTH:master Jun 29, 2021

This was referenced Jun 29, 2021

Fix cover tests #41

Open

Fix constructor coverage #42

Open

Option for covering tests STAMP-project/test-runner#98

Closed

Uh oh!

Conversation

andre15silva commented Jun 24, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

andre15silva commented Jun 24, 2021

Uh oh!

martinezmatias commented Jun 24, 2021

Uh oh!

andre15silva commented Jun 24, 2021

Uh oh!

martinezmatias commented Jun 24, 2021

Uh oh!

andre15silva commented Jun 24, 2021

Uh oh!

martinezmatias commented Jun 24, 2021

Uh oh!

danglotb commented Jun 24, 2021

Uh oh!

andre15silva commented Jun 24, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

danglotb commented Jun 29, 2021

Uh oh!

andre15silva commented Jun 29, 2021

Uh oh!

andre15silva commented Jun 29, 2021

Uh oh!

martinezmatias Jun 29, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andre15silva Jun 29, 2021

Choose a reason for hiding this comment

Uh oh!

martinezmatias Jun 29, 2021

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

andre15silva commented Jun 24, 2021 •

edited

Loading

andre15silva commented Jun 24, 2021 •

edited

Loading

martinezmatias Jun 29, 2021 •

edited

Loading