We're working with the libp2p team on multi-dimensional testing:
libp2p/test-plans#53

They need a way to describe large test matrixes,
one composition might combine every libp2p version able to communicate over webrtc and use yamux,
another might combine every libp2p version able to use mutex X and transport Y.

The main risk is the composition complexity: Every test run would need a new composition file at the moment.
And the configuration used in libp2p/test-plan is quite complex.
This is the build configuration for go libp2p master: https://github.com/libp2p/test-plans/blob/c71602e88c8a20c6ecc7c18fe1f5b5cf570d3b11/ping/_compositions/go-rust-interop.toml#L33-L56

Contributes to Milestone #1514

Current Solution

Add a [runs] field to compositions

[[groups]]
# the list of instances

[[runs]]
# the list of runs
# a run is of the form:
    id = 'run_id'

    [runs.test_params]
    # custom param for the whole run

    [[runs.groups]]
    id = 'id of a test instance'
    # TODO: maybe use 2 values: `id` and `group_id`,
    #       and default `id` to `group_id`
    instances = {count = 1}

    [[runs.groups.test_params]]
    # custom param for this instance

    [[run.groups]]
    id = 'another instances'
    # and so on.

• demonstration: https://github.com/libp2p/test-plans/tree/feat/interop-dashboard/composer
• discussion: Create a "simple" interop dashboard libp2p/test-plans#55 (comment)

testground run composition --file=./composition-interop.toml --run-index=42 # build and run a single test
testground run composition --file=./composition-interop.toml # build & run all tests
testground run composition --file=./composition-interop.toml ... --result ./results.csv

Tasks

• multiple runs per compositions (client-side) #1526
• document features
  • new composition parameters
  • new cli parameters
• Server-side implementation
  • Drop the concept of build groups request during a run
    • sending a composition with missing artifacts is enough
  • Refactor how we manage tasks and generate outcomes,
    • Make outcomes data type shared between every runner
    • Make the outcome for build tasks consistent with the others
  • implement server-side runs feature
    • refactors tasks to support task groups
    • refactor the supervisor to support task groups
    • refactor the request system to support task groups
    • remove the need for waiting on the client side
• QOL improvements (optional)
  • Add a feature to extract the list of artifacts at the end of a build?
    • https://github.com/libp2p/test-plans/tree/feat/interop-dashboard/composer#follow-ups
  • Fix the composition -w parameter, do not overwrite the file, accept a path instead
  • implement a (partial "./template.toml")
  • implement a --continue-on-build-error that runs "as many tests as possible" even on build failures
  • Update go versions (enable generics & fix issues with golangci-lint)
  • Refactor the integrations tests to make them maintainable,
  • Templating follow-ups
    • Extract templating to another cli command (composable testground)
    • Investigate: https://github.com/Masterminds/sprig
    • Investigate "missingkey=error"
  • improve client-side run/builds
    • make the request retriable
    • generalize the use of strategy, which should reduce code required and make it unit-testable

Something similar was built in lotus:
https://github.com/filecoin-project/lotus/tree/master/testplans/composer

Other discussions

How to deal with build & run failures:

If I try to run N tests for K different versions, and the build for 2 versions is failing:
Should Testground keep building, and run the tests for the builds that succeeded?

Proposal:

Add an option that tells testground: keep going:

testground run composition .... --continue-on-build-error

Are we happy with current templates?

Templating is a very useful scripting engine we integrate into testground. For "simple" use case like parametrization (custom go version, generating large tests, etc) it makes a lot of sense.

Are we happy with this approach? Another approach could be: the user generates toml files with their own scripting engine (js, go, etc).

Proposal: make template reusables with partial

(partial "./groups-go.toml" .) 

This would import another file and use it as a template.