Is your feature request related to a problem? Please describe.

Tests can be run as part of a package's main derivation (that produces the binaries, etc.), or the tests can be run separately.
Currently this presents a trade-off between build latency and reliability.

• By putting the tests in a separate derivation, dependencies do not need to wait for the tests to run, allowing for increased parallelism, and a shorter critical path (ie shortening the chain of dependencies with the longest cumulative build time).
• However, by putting them in a separate derivation, it's not guaranteed that all variations that package consumers come up with pass the tests. Such variations appear through flake follows, overlays, overrides, etc; so CI can not guarantee all to pass.

Describe the solution you'd like

Break the trade-off by tracking test dependencies in the string context.

• string context: add a new "test item" constructor to the variant
• builtins.derivationStrict: perform the build by ignoring the test items in the string context, but re-add these test items to the context of the returned output strings.
• nix build, etc: do build the whole string context, including the test items
• optionally, prioritize tests when scheduling the build graph, to fail early

This has the effect of making all the test derivations fairly independently schedulable tasks without the risk of skipping any.

Describe alternatives considered

• Track test derivations and parallelize building and testing #7662 (comment)

• Status quo, which is the following

Track it in a package ("passthru") attribute instead. This isn't airtight, because dependencies added through string interpolations won't be captured. Example:

''
  ${hello}/bin/hello >$out
''

In the above string, the test dependencies of hello aren't necessary for the string with context to be realized. Without adding them as regular dependencies (and therefore linearizing the build, which we don't want) it's not possible to extract test dependencies from the string. It's not feasible to do this manually with a helper function involving builtins.getContext and unsafeDiscardStringContext, because this helper would have to be used in far too many places. Furthermore, it has two return values, necessitating a let binding at every call site, making the thing far too cumbersome as well.

Additional context
Add any other context or screenshots about the feature request here.

Priorities

Add 👍 to issues you find important.