Draft: Testing Module Testing with Maven for #2760 #2773

aepfli · 2022-09-09T08:49:38Z

Sorry i simply started experimenting out of curiosity regarding #2760 , and I want to share what I tried and what worked and did not work for such modular tests. Please just close this pr, if you do not think it is fitting. Although if you think, this needs just polishing, let me know, i can adapt and help to make it better.

This pr works, but only has one test - so theoretically it is usable :)

BUT... i wanted to experiment a little bit more, and maybe make the original unit tests work within this modular module.

test-jar approach

I tried adding following to the assertj-core to generate a test jar

<plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-jar-plugin</artifactId>
        <executions>
          <execution>
            <goals>
              <goal>test-jar</goal>
            </goals>
          </execution>
        </executions>
      </plugin>

and tried to apply it as a dependency within my modular test. Sadly this ended up with a lot of package errors

errors

[ERROR] the unnamed module reads package org.assertj.core.api from both assertj.core and org.assertj.core
[ERROR] the unnamed module reads package org.assertj.core.api.filter from both assertj.core and org.assertj.core
[ERROR] the unnamed module reads package org.assertj.core.api.iterable from both assertj.core and org.assertj.core
[ERROR] the unnamed module reads package org.assertj.core.api.junit.jupiter from both assertj.core and org.assertj.core
[ERROR] the unnamed module reads package org.assertj.core.api.recursive.comparison from both assertj.core and org.assertj.core
[ERROR] the unnamed module reads package org.assertj.core.condition from both assertj.core and org.assertj.core
[ERROR] the unnamed module reads package org.assertj.core.configuration from both assertj.core and org.assertj.core
[ERROR] the unnamed module reads package org.assertj.core.data from both assertj.core and org.assertj.core
[ERROR] the unnamed module reads package org.assertj.core.description from both assertj.core and org.assertj.core
[ERROR] the unnamed module reads package org.assertj.core.error from both assertj.core and org.assertj.core
[ERROR] the unnamed module reads package org.assertj.core.error.future from both assertj.core and org.assertj.core
[ERROR] the unnamed module reads package org.assertj.core.error.uri from both assertj.core and org.assertj.core
[ERROR] the unnamed module reads package org.assertj.core.extractor from both assertj.core and org.assertj.core
[ERROR] the unnamed module reads package org.assertj.core.groups from both assertj.core and org.assertj.core
[ERROR] the unnamed module reads package org.assertj.core.matcher from both assertj.core and org.assertj.core
[ERROR] the unnamed module reads package org.assertj.core.presentation from both assertj.core and org.assertj.core
[ERROR] the unnamed module reads package org.assertj.core.util from both assertj.core and org.assertj.core
[ERROR] the unnamed module reads package org.assertj.core.util.diff from both assertj.core and org.assertj.core
[ERROR] the unnamed module reads package org.assertj.core.util.diff.myers from both assertj.core and org.assertj.core
[ERROR] the unnamed module reads package org.assertj.core.util.introspection from both assertj.core and org.assertj.core
[ERROR] the unnamed module reads package org.assertj.core.util.xml from both assertj.core and org.assertj.core
[ERROR] module assertj.core reads package org.assertj.core.util from both org.assertj.core and assertj.core
[ERROR] module assertj.core reads package org.assertj.core.util.introspection from both org.assertj.core and assertj.core
[ERROR] module assertj.core reads package org.assertj.core.util.diff from both org.assertj.core and assertj.core
[ERROR] module assertj.core reads package org.assertj.core.util.diff.myers from both org.assertj.core and assertj.core
[ERROR] module assertj.core reads package org.assertj.core.util.xml from both org.assertj.core and assertj.core
[ERROR] module assertj.core reads package org.assertj.core.groups from both org.assertj.core and assertj.core
[ERROR] module assertj.core reads package org.assertj.core.configuration from both org.assertj.core and assertj.core
[ERROR] module assertj.core reads package org.assertj.core.condition from both org.assertj.core and assertj.core
[ERROR] module assertj.core reads package org.assertj.core.description from both org.assertj.core and assertj.core
[ERROR] module assertj.core reads package org.assertj.core.extractor from both org.assertj.core and assertj.core
[ERROR] module assertj.core reads package org.assertj.core.api from both org.assertj.core and assertj.core
[ERROR] module assertj.core reads package org.assertj.core.api.filter from both org.assertj.core and assertj.core
[ERROR] module assertj.core reads package org.assertj.core.api.junit.jupiter from both org.assertj.core and assertj.core
[ERROR] module assertj.core reads package org.assertj.core.api.iterable from both org.assertj.core and assertj.core
[ERROR] module assertj.core reads package org.assertj.core.api.recursive.comparison from both org.assertj.core and assertj.core
[ERROR] module assertj.core reads package org.assertj.core.matcher from both org.assertj.core and assertj.core
[ERROR] module assertj.core reads package org.assertj.core.data from both org.assertj.core and assertj.core
[ERROR] module assertj.core reads package org.assertj.core.error from both org.assertj.core and assertj.core
[ERROR] module assertj.core reads package org.assertj.core.error.future from both org.assertj.core and assertj.core
[ERROR] module assertj.core reads package org.assertj.core.error.uri from both org.assertj.core and assertj.core
[ERROR] module assertj.core reads package org.assertj.core.presentation from both org.assertj.core and assertj.core
[ERROR] module org.hamcrest reads package org.assertj.core.util from both org.assertj.core and assertj.core
[ERROR] module org.hamcrest reads package org.assertj.core.util.introspection from both org.assertj.core and assertj.core
[ERROR] module org.hamcrest reads package org.assertj.core.util.diff from both org.assertj.core and assertj.core
[ERROR] module org.hamcrest reads package org.assertj.core.util.diff.myers from both org.assertj.core and assertj.core
[ERROR] module org.hamcrest reads package org.assertj.core.util.xml from both org.assertj.core and assertj.core
[ERROR] module org.hamcrest reads package org.assertj.core.groups from both org.assertj.core and assertj.core
[ERROR] module org.hamcrest reads package org.assertj.core.configuration from both org.assertj.core and assertj.core
[ERROR] module org.hamcrest reads package org.assertj.core.condition from both org.assertj.core and assertj.core
[ERROR] module org.hamcrest reads package org.assertj.core.description from both org.assertj.core and assertj.core
[ERROR] module org.hamcrest reads package org.assertj.core.extractor from both org.assertj.core and assertj.core
[ERROR] module org.hamcrest reads package org.assertj.core.api from both org.assertj.core and assertj.core
[ERROR] module org.hamcrest reads package org.assertj.core.api.filter from both org.assertj.core and assertj.core
[ERROR] module org.hamcrest reads package org.assertj.core.api.junit.jupiter from both org.assertj.core and assertj.core
[ERROR] module org.hamcrest reads package org.assertj.core.api.iterable from both org.assertj.core and assertj.core
[ERROR] module org.hamcrest reads package org.assertj.core.api.recursive.comparison from both org.assertj.core and assertj.core
[ERROR] module org.hamcrest reads package org.assertj.core.matcher from both org.assertj.core and assertj.core
[ERROR] module org.hamcrest reads package org.assertj.core.data from both org.assertj.core and assertj.core
[ERROR] module org.hamcrest reads package org.assertj.core.error from both org.assertj.core and assertj.core
[ERROR] module org.hamcrest reads package org.assertj.core.error.future from both org.assertj.core and assertj.core
[ERROR] module org.hamcrest reads package org.assertj.core.error.uri from both org.assertj.core and assertj.core
[ERROR] module org.hamcrest reads package org.assertj.core.presentation from both org.assertj.core and assertj.core
[INFO] 63 errors 
[INFO] -------------------------------------------------------------

additional source directory

Then i thought maybe extracting them into target/test-classes could also do the trick - sadly not
Then i thought maybe i could simply add the test source directory to the modularTest module, but this somehow needed more than 5 minutes and did not finish, so i also stepped away from this approach.

Adding an own module for testing in the modular world.

scordio · 2022-09-09T09:15:54Z

Hi Simon, thanks for the PR! This probably overlaps with #2759, which is already stable from the Maven perspective but:

Has an issue with IntelliJ, see IDEA-301032
Lacks a test case for extractingResultOf fails on the module path #2760 and the Pioneer issue - I'm still looking at your branch and thinking how to reproduce the issue, maybe I'll steal the whole PioneerAnnotationUtils as a test target

scordio · 2022-09-09T09:22:41Z

make the original unit tests work within this modular module.

This is a cool point I didn't consider yet: it would be much better to re-run our entire test suite in the modular world instead of adding dedicated tests for it.

aepfli · 2022-09-09T09:33:14Z

Hi Simon, thanks for the PR! This probably overlaps with #2759

yes it does, sorry I did not see that one :)

Lacks a test case for #2760 and the Pioneer issue - I'm still looking at your branch and thinking how to reproduce the issue, maybe I'll steal the whole PioneerAnnotationUtils as a test target

I added one specially for this case within this pr.

This is a cool point I didn't consider yet: it would be much better to re-run our entire test suite in the modular world instead of adding dedicated tests for it.

maybe somebody has suggestions for this error the unnamed module reads package org.assertj.core.api from both assertj.core and org.assertj.core and how we can easily fix it, but i have no clue who i can ask here. This error only applies as soon as i add the test-jar as a dependency. Most likely because it does not have a module-info.java but on the other hand, i also did not find a way to produce it with a module-info.java . As i am hooked i will experiment a little bit more. and see what i might discover.

scordio · 2022-09-09T09:47:35Z

maybe somebody has suggestions for this error the unnamed module reads package org.assertj.core.api from both assertj.core and org.assertj.core and how we can easily fix it, but i have no clue who i can ask here.

Are you getting this error in the IDE or with Maven?

I'll try to play with it as well this evening.

aepfli · 2022-09-09T10:29:17Z

I pushed the changes to use a test-jar. i get this error with maven during testCompile of the modular-testing module

scordio

I'm still checking what could be done in this area. Below are just a few comments but feel free to ignore them until it's clear where we want to go.

About these errors:

the unnamed module reads package org.assertj.core.api from both assertj.core and org.assertj.core

I believe this is caused by these three actors:

The assertj-core multi-release JAR with this module-info
The assertj-core test JAR that has no module descriptor so it becomes an automatic module
The assertj-core-moduletest with the module-info under src/test/java

I assume the unnamed module in the error message is 3, while assertj.core corresponds to 2 and org.assertj.core corresponds to 1.

I believe the assertj-core test JAR should be created with a descriptor, but right now I believe that having a test JAR would be a backup option (see #2347 for the rationale).

I'm trying another strategy with the branch of #2759:

Setting the project.build.testSourceDirectory to ../../../assertj-core/src/test/java
Defining an additional execution of the maven-compiler-plugin where the module-info under src/test/java gets compiled

This should work when running the maven-surefire-plugin under the assertj-core-module-path module (I'm fixing now the necessary declarations in the local module-info).

Also, it requires a bit of duplication for all the test scoped dependencies of assertj-core and the assertj-core/src/test/java folder is detected by IntelliJ as a test source root for two separate modules, leading to weird IDE errors about imports.

I'm still 100% sure it's the direction we want to take, might be OK but I still need to digest the idea 🙂

Lastly, we'll soon start the development of AssertJ 4 which will drop the multi-release configuration, hopefully simplifying the overall test setup, so we might stay with a basic setup in the 3.x branch to be used ad-hoc for specific investigations.

scordio · 2022-09-10T07:14:00Z

assertj-core/pom.xml

+      <plugin>
+        <groupId>org.apache.maven.plugins</groupId>
+        <artifactId>maven-jar-plugin</artifactId>
+        <executions>
+          <execution>
+            <goals>
+              <goal>test-jar</goal>
+            </goals>
+          </execution>
+        </executions>
+      </plugin>


In general, we'd like to avoid the creation of the test jar, see #2347. If we are really forced to have it, we should prevent its deployment to the Nexus.

scordio · 2022-09-10T08:22:22Z

assertj-tests/assertj-module-tests/pom.xml

+        <maven.compiler.release>${java.version}</maven.compiler.release>
+        <maven.compiler.source>${java.version}</maven.compiler.source>
+        <maven.compiler.target>${java.version}</maven.compiler.target>


java.version should be enough, all of them are set in assertj-parent

scordio · 2022-09-10T08:23:06Z

assertj-tests/assertj-module-tests/pom.xml

+            <plugin>
+                <groupId>org.apache.maven.plugins</groupId>
+                <artifactId>maven-compiler-plugin</artifactId>
+            </plugin>
+            <plugin>
+                <groupId>org.apache.maven.plugins</groupId>
+                <artifactId>maven-surefire-plugin</artifactId>
+                <version>3.0.0-M7</version>
+<!--                    <configuration>-->
+<!--                        <dependenciesToScan>-->
+<!--                            <dependency>org.assertj:assertj-core:test-jar</dependency>-->
+<!--                        </dependenciesToScan>-->
+<!--                    </configuration>-->
+
+            </plugin>


Unless there is a special config for the two plugins, declaration is not necessary as we get already the latest versions from assertj-parent

scordio · 2022-09-10T08:25:36Z

assertj-tests/pom.xml

-                <bannedDependencies>
+<!--                <bannedDependencies>
                  <includes>
                    <include>*:*:*:jar:test</include>


If we would need to use a test-jar dependency, we could relax this condition while keeping the enforced scope:

Suggested change

<include>*:*:*:jar:test</include>

<include>*:*:*:*:test</include>

aepfli · 2022-09-10T10:06:06Z

I tried the path you want to use, I did not manage to get it to compile. Not sure why but test compile took ages ;)

I even abused the approach from the mutli release jar and patched the jar with a module-info out of the Modulartest module. But as said I had some kind of compile issues, as it tool ages and I was too impatient to wait more than 5 minutes.

I will also try to experiment some more in the evening. Looks really interesting.

Setting the project.build.testSourceDirectory to ../../../assertj-core/src/test/java

Be aware to also set the test resources ;)

scordio · 2022-09-10T11:34:14Z

I've pushed my WIP changes in e138460, I'll try to finalize them later in the day.

aepfli · 2022-09-10T14:48:06Z

i used part of your code, and I figured out it is just hamcrest - by adding the optional junit dependency and excluding hamcrest, I made it work locally :) with your testdirectory approach. i will check later again, but it looks promising


        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <scope>test</scope>
            <exclusions>
                <!-- This has to be excluded because it's subjecting the code to the incorrect version of certain hamcrest packages. -->
                <exclusion>
                    <groupId>org.hamcrest</groupId>
                    <artifactId>hamcrest-core</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

There are tests failing currently due to missing resources, which I will check later

aepfli · 2022-09-10T19:30:18Z

please do not be overwhelmed by my recent changes

I discovered an issue with all files which have been accessed during the test, as the path was always module specific and not loaded via resources, i adapted the tests to do so (most likely there is a good reason for that, i just thought a fast replace in the IDE could do the trick to see which errors are left)

now i am left with 6 errors, and one of those is my test for those methods. strangely enough there has not been a lot of other detections regarding package violations, except the deepdifference tests.

[ERROR] Errors: 
[ERROR]   RecursiveAssertionDriver_JavaClassLibraryRecursionTest.should_assert_over_and_recurse_into_jcl_classes_when_configured_to_recurse_into_JCL:45 » Introspection
[ERROR]   DeepDifference_Test.testBigDecimals:80->assertHaveNoDifferences:327->assertHaveNoDifferences:331 » Introspection
[ERROR]   DeepDifference_Test.testEquivalentCollections:249->assertHaveNoDifferences:327->assertHaveNoDifferences:331 » Introspection
[ERROR]   DeepDifference_Test.testSameObject:66->assertHaveNoDifferences:327->assertHaveNoDifferences:331 » Introspection
[ERROR]   DeepDifference_Test.testUnorderedCollection:140->assertHaveNoDifferences:327->assertHaveNoDifferences:331 » Introspection
[ERROR]   IterableAssert_extracting_Test.should_allow_assertions_on_property_values_extracted_from_given_iterable:36 » IllegalState
[INFO] 
[ERROR] Tests run: 18673, Failures: 0, Errors: 6, Skipped: 16

I know, those changes are now a lot and blew up the whole PR, and they might not be desired, this was again just exploration and fact finding. So please if you do not see those fitting the need, ignore them :), but we would need to find another solution for the tests and the resources. (most likely also doable by switching the working directory)

.. and I always have to run a clean, before I can run the modular tests

scordio · 2022-09-10T20:27:16Z

Thank you so much for taking the time to do this already!

I'm using your attempts as inspiration and I'm also trying to take a slightly different direction to avoid the weird IDE behavior I was mentioning: instead of setting testSourceDirectory and testResources, I keep the configuration in the corresponding plugins so the IDE doesn't get influenced (066cc49).

I'm not sure how much time I'll have in the next days but I'll continue on this topic as soon as I can.

aepfli · 2022-09-12T08:04:29Z

if I can support you somehow further, or if there is anything I should test/do, etc. please let me know :) I am glad if I can help you.

aepfli · 2022-09-12T09:52:42Z

just something which popped to my head, instead of recompiling the tests why not just reuse the compiled files?

            <plugin>
                <artifactId>maven-resources-plugin</artifactId>
                <groupId>org.apache.maven.plugins</groupId>
                <executions>
                    <execution>
                        <id>copy-resource-one</id>
                        <phase>generate-test-resources</phase>
                        <goals>
                            <goal>copy-resources</goal>
                        </goals>

                        <configuration>
                            <outputDirectory>${project.build.testOutputDirectory}</outputDirectory>
                            <resources>
                                <resource>
                                    <directory>../../assertj-core/target/test-classes</directory>
                                    <includes>
                                        <include>**/*</include>
                                    </includes>
                                </resource>
                            </resources>
                        </configuration>
                    </execution>
                </executions>
            </plugin>

this way the test is executed faster, and it should actually not cause any problems? see 1d1af6f

Downsides:

it might not be clear to the user that he has to also run testCompile for assertj-core if he changes something
it is not 100% clear what is happening and what dependencies we do have on the build
awareness of packages, and that there might be an overwrite of a test if package and class name are an exact match within modulartest and assertj-core

Ups:

pretty fast, as we don't need to recompile 4k files
no IDE issues etc.
moduletests will be executed quiet fast, if you only adapt those, as the recompile will be much much faster

aepfli · 2022-09-22T11:47:59Z

also an interesting twitter thread regarding this topic: https://twitter.com/lukaseder/status/1389236184472301583

scordio · 2022-09-22T14:50:15Z

also an interesting twitter thread regarding this topic: https://twitter.com/lukaseder/status/1389236184472301583

Thanks! I think I've seen it last year and I agree especially with:

Instead, there can be proper test library modules. Modules that are only used by other test modules, and will never be shipped to users.

Perhaps, it's time to let go of the "old" Maven directory layouts?

which is somehow the strategy I'm trying to use with #2424.

It might be we'll go with the approach of moving all the tests to a separate module. I need to digest the idea a bit more 🙂

scordio · 2022-09-23T11:58:35Z

One more thought: I'm not really thrilled to move tests to a separate module-based artifact until IDEA-301032 is fixed, otherwise it will be an unpleasant experience for contributors using IDEA (myself included 😄).

scordio · 2025-02-16T14:18:23Z

Superseded by #3771.

Testing Module Testing with Maven for assertj#2760

8468d98

Adding an own module for testing in the modular world.

aepfli marked this pull request as draft September 9, 2022 08:49

aepfli changed the title ~~Testing Module Testing with Maven for #2760~~ Draft: Testing Module Testing with Maven for #2760 Sep 9, 2022

reverting unneeded change

d5581a3

scordio reviewed Sep 10, 2022

View reviewed changes

making it work

bd71c07

aepfli force-pushed the testing-maven-moduletesting branch from aa9301f to bd71c07 Compare September 10, 2022 14:46

Suggesting adaption of tests to make them run in both modules

dd3a2db

aepfli force-pushed the testing-maven-moduletesting branch from 461a993 to dd3a2db Compare September 10, 2022 19:27

using copying of compiled files instead of compiling

1d1af6f

scordio added theme: module system An issue related to the Java module system type: enhancement labels Sep 22, 2022

scordio mentioned this pull request Oct 18, 2022

Add "it" infrastructure for integration tests #1778

Closed

scordio force-pushed the main branch from d69cd04 to 9902a7a Compare May 20, 2023 15:13

scordio force-pushed the main branch from 136cd31 to 38c7b4b Compare July 20, 2023 15:10

scordio force-pushed the main branch from 38c7b4b to 3cde5bf Compare July 20, 2023 15:20

scordio force-pushed the 3.x branch 2 times, most recently from 301ca01 to c730d18 Compare June 1, 2024 16:04

scordio removed the type: improvement label Oct 2, 2024

scordio closed this Feb 16, 2025

	<include>::*:jar:test</include>
	<include>::::test</include>

Uh oh!

Draft: Testing Module Testing with Maven for #2760 #2773

Draft: Testing Module Testing with Maven for #2760 #2773

Uh oh!

Conversation

aepfli commented Sep 9, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

test-jar approach

additional source directory

Uh oh!

scordio commented Sep 9, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

scordio commented Sep 9, 2022

Uh oh!

aepfli commented Sep 9, 2022

Uh oh!

scordio commented Sep 9, 2022

Uh oh!

aepfli commented Sep 9, 2022

Uh oh!

scordio left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

scordio Sep 10, 2022

Choose a reason for hiding this comment

Uh oh!

scordio Sep 10, 2022

Choose a reason for hiding this comment

Uh oh!

scordio Sep 10, 2022

Choose a reason for hiding this comment

Uh oh!

scordio Sep 10, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aepfli commented Sep 10, 2022

Uh oh!

scordio commented Sep 10, 2022

Uh oh!

aepfli commented Sep 10, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aepfli commented Sep 10, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

scordio commented Sep 10, 2022

Uh oh!

aepfli commented Sep 12, 2022

Uh oh!

aepfli commented Sep 12, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aepfli commented Sep 22, 2022

Uh oh!

scordio commented Sep 22, 2022

Uh oh!

scordio commented Sep 23, 2022

Uh oh!

scordio commented Feb 16, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

aepfli commented Sep 9, 2022 •

edited

Loading

scordio commented Sep 9, 2022 •

edited

Loading

scordio left a comment •

edited

Loading

scordio Sep 10, 2022 •

edited

Loading

aepfli commented Sep 10, 2022 •

edited

Loading

aepfli commented Sep 10, 2022 •

edited

Loading

aepfli commented Sep 12, 2022 •

edited

Loading