With some recent refactoring we have addressed the common use cases for core, modules, and (ES) plugins by creating specific Gradle plugin's for the common use cases (see below). However, the one-off use cases have not been addressed leaving behind a bit of mess of Gradle plugins which can be confusing about which one to use. Generally these one-off use cases are part of qa projects and were not included in the recent refactoring.

This issue is to consider the removal of the three described plugins and find suitable alternative(s). Most of the qa projects will need to be converted to no longer use one of the three below plugins. Some of the qa projects may indeed fit into a well defined use case, but simply have not been converted yet.

Gradle test plugins

Base plugins (which should be kept)

ElasticsearchTestBasePlugin - This plugin creates the test task.
RestTestBasePlugin - this task finds all RestIntegTestTask's and wires them up as necessary to execute against an external cluster.

Applied plugins (which should be kept)

YamlRestTestPlugin - uses RestTestBasePlugin with a custom source set to run the YAML based REST tests
JavaRestTestPlugin - uses RestTestBasePlugin with a custom source set to run the Java based REST tests
YamlCompatTestPlugin - uses RestTestBasePlugin with a custom source set to run the prior major version YAML based REST tests
InternalClusterTestPlugin - uses ElasticsearchTestBasePlugin with a custom source set to run in-process cluster based tests

All of the above have well defined use cases and are generally applicable across many projects. However, there are still exists many project, mostly under qa projects that don't fit well into any of these categories.

For example, :qa:rolling-upgrade spins up prior versions of Elasticsearch and then runs YAML and Java sourced tests at each stage of upgrading a cluster. The task registration and orchestration is via the project's build.gradle, but some of the setup is still done via a plugin(s). In this case the project applies StandaloneTestPlugin which applies StandaloneRestTestPlugin which in turn applies RestTestBasePlugin (along with some other plugins).

Applied plugins which should be considered for removal

StandaloneTestPlugin - registers the test task, and applies StandaloneRestTestPlugin
StandaloneRestTestPlugin - applies some plugins needed to run only the REST tests
RestTestPlugin - creates an integTest task then applies other plugins to run the REST tests.