Automatically create test results elements for each test suite by jvandort · Pull Request #31706 · gradle/gradle

jvandort · 2024-12-13T19:38:29Z

Previously, we created a binary test results elements for each test target. When performing aggregation, this only worked if the target test suite had a single target. Otherwise, we'd run into ambiguity.

To resolve this for multi-target test suites, we only create a single variant for each suite and add all test results variants for all targets to the suite's variant. This means when aggregating against a project, we will aggregate the results of all targets.

Eventually we will want to go back to a test results variant for each target, but until we have the targets able to declare their identity via attributes, we will have a single variant per suite.

We remove the concept of a test suite type, as this was a very simple mapping over the suite name, and only differed for the case of the default test suite, which has a test type of UNIT_TEST instead of its suite name. We will eventually want to not use the suite name as an attribute either, but rather use the suite name as part of a capability.

The reports will eventually want to use attributes to distinguish which targets to aggregate instead of aggregating all targets from each suite.

We also make similar changes to Jacoco.

Reviewing cheatsheet

Before merging the PR, comments starting with

❌ ❓must be fixed
🤔 💅 should be fixed
💭 may be fixed
🎉 celebrate happy things

bot-gradle · 2024-12-17T17:45:02Z

The following builds have failed:

Pull Request Feedback (Trigger) (Check)
- Build Scan

bot-gradle · 2024-12-17T18:14:26Z

The following builds have failed:

Pull Request Feedback (Trigger) (Check)
- Build Scan

Previously, we created a binary test results elements for each test target. When performing aggregation, this only worked if the target test suite had a single target. Otherwise, we'd run into ambiguity. To resolve this for multi-target test suites, we only create a single variant for each sutie and add all test results variants for all targets to the suite's variant. This means when aggregating against a project, we will aggregate the results of all targets. Eventually we will want to go back to a test results variant for each target, but until we have the targets able to declare their identity via attributes, we will have a single variant per suite. We remove the concept of a test suite type, as this was a very simple mapping over the suite name, and only differed for the case of the default test suite, which has a test type of UNIT_TEST instead of its suite name. We will eventually want to not use the suite name as an attribute either, but rather use the suite name as part of a capability. The reports will eventually want to use attributes to distinguish which targets to aggregate instead of aggregating all targets from each suite. We also make similar changes to Jacoco.

jvandort · 2024-12-17T19:57:03Z

@bot-gradle test this

bot-gradle · 2024-12-17T20:55:01Z

The following builds have passed:

Pull Request Feedback (Trigger) (Check)
- Build Scan

big-guy

As a follow up, please add something to the upgrade guide that warns about changes in the JaCoCo and test suite plugins + aggregation.

big-guy · 2024-12-17T22:28:05Z


 package org.gradle.testing.base.plugins;

+import org.apache.commons.lang3.StringUtils;


Can we keep using org.apache.commons.lang.StringUtils? I'm not sure why we have both versions...

Shouldn't we prefer lang3 over lang?

Bah, it looks like we have a mix of both. I thought we were only using lang, not lang3

I guess lang3 is good enough. Ignore me

jvandort · 2024-12-18T05:37:12Z

@big-guy #31809

jvandort force-pushed the jvandort/testing/aggregation/automatically-create-results-elements-for-each-suite branch from 8dfdd58 to 212a951 Compare December 17, 2024 06:33