Credit

Credit goes to @eyalkoren for the discovery of this issue

Overview

It appears that certain tests are duplicated across sets of smoke tests. For example, take the greetingShouldReturnDefaultMessage test. This test is located in the integrations/test directory, and can be seen here.

If one examines a test run by opening a run and going to the Blue Ocean page, they can see that the same test appears to have run as a part of both the Smoke Tests 01 group and the Smoke Tests 02 group.

Details

Tests are segmented through the use of simple bash scripts. The scripts which provide this segmentation are below:

Smoke Tests 01
Smoke Tests 02(https://github.com/elastic/apm-agent-java/blob/master/scripts/jenkins/smoketests-02.sh)

Each script contains a preparatory step to generate the modules to be tested. Below is the output of each:

smoketest01

find apm-agent-plugins -maxdepth 1 -mindepth 1 -type d|grep -v "target"

apm-agent-plugins/apm-mongoclient-plugin
apm-agent-plugins/apm-redis-plugin
apm-agent-plugins/apm-spring-webmvc-plugin
apm-agent-plugins/apm-quartz-job-plugin
apm-agent-plugins/apm-jaxrs-plugin
apm-agent-plugins/apm-jsf-plugin
apm-agent-plugins/apm-httpclient-core
apm-agent-plugins/apm-kafka-plugin
apm-agent-plugins/apm-java-concurrent-plugin
apm-agent-plugins/apm-jdbc-plugin
apm-agent-plugins/apm-jmx-plugin
apm-agent-plugins/apm-servlet-plugin
apm-agent-plugins/apm-es-restclient-plugin
apm-agent-plugins/apm-opentracing-plugin
apm-agent-plugins/apm-scheduled-annotation-plugin
apm-agent-plugins/apm-mule4-plugin
apm-agent-plugins/apm-jaxws-plugin
apm-agent-plugins/apm-api-plugin
apm-agent-plugins/apm-jms-plugin
apm-agent-plugins/apm-slf4j-plugin
apm-agent-plugins/apm-process-plugin
apm-agent-plugins/apm-urlconnection-plugin
apm-agent-plugins/apm-okhttp-plugin
apm-agent-plugins/apm-apache-httpclient-plugin
apm-agent-plugins/apm-asynchttpclient-plugin
apm-agent-plugins/apm-error-logging-plugin
apm-agent-plugins/apm-hibernate-search-plugin
apm-agent-plugins/apm-spring-resttemplate-plugin

smoketest02

find integration-tests -maxdepth 1 -mindepth 1 -type d|grep -v "target"|

integration-tests/soap-test
integration-tests/cdi-app
integration-tests/spring-boot-2
integration-tests/application-server-integration-tests
integration-tests/spring-boot-1-5
integration-tests/jsf-app
integration-tests/simple-webapp

Reasoning

It appears as though the preparatory step correctly separates the two sets of tests but that Maven may be running the same test(s) in each instance. Maven is executed with the following parameters:

./mvnw -q -Dmaven.javadoc.skip=true -am -amd -pl ${MOD} -P integration-test-only verify (where $MOD is the set of modules from each prepatory step.)

Could, perhaps, the -am or the -amd flag inadvertently be triggering a larger set of tests? Or, is something in one each tests linking to the other set through a dependency of some kind?