The original roachtest mixed-version framework [1] was born sometime in 2017. While it has gone through many iterations and improvements since then, its limitations have inspired a new, declarative approach [2], which attempts to increase coverage of the mixed-version states while reducing boilerplate, thereby allowing the author to focus on the salient bits, namely workloads and invariants. The new mixed-version framework has already proven its worth by finding 12+ new bugs, having ported only two pre-existing roachtests,

• acceptance/version-upgrade
• backup-restore/mixed-version

Most recently, an initial port of cdc/mixed-versions was merged. This leaves the following roachtests which remain to be ported,

• import/mixed-versions (roachtest: port import/mixed-versions to the new framework #113546)
• decommission/mixed-versions (roachtest: port decommission/mixed-versions #131252)
• change-replicas/mixed-version (roachtest: port change-replicas/mixed-version to mixedversion #113222)
• schemachange/mixed-versions (roachtest: move schemachange/mixed-version to new framework #113890)
• schemachange/secondary-index-multi-version (roachtest: move schemachange/secondary-index-multi-version to new framework #113696)
• systemschema/validate-after-version-upgrade (roachtest: move validate-after-version-upgrade to new framework #113643)
• declarative_schema_changer/job-compatibility-mixed-version-V222-V231(roachtest: move decl_schema_change/job-compat to new framework #114043)
• tpcc/mixed-headroom/n5cpu16 (roachtest: port tpcc/mixed-headroom to the new framework #113706)
• version/mixed/nodes=%d (roachtest: delete version/mixed tests #113323)
• follower-reads/mixed-version/single-region (roachtest: port follower-reads/mixed-version/single-region to new framework #115317)
• rebalance/by-load/leases/mixed-version (roachtest: port rebalance/by-load/*/mixed-version to new framework #115559)
• rebalance/by-load/replicas/mixed-version (roachtest: port rebalance/by-load/*/mixed-version to new framework #115559)
• generate-fixtures (roachtest: update generate-fixtures to use clusterupgrade functions #131467)

In virtue of the new framework, even a mechanical approach to porting the above is likely to yield a higher coverage of the mixed-version states. However, we ask that the authors take a deliberate approach by studying the limitations of the current implementation and correcting them, when feasible, with the help of the new framework.

In addition to the above, TestEng is planning to add a long-running workload utilizing sqlancer in the near term. (Unlike existing workloads, sqlancer is more antagonistic and randomized, albeit synthetic.) Mid-to-longer term, automated failure injection and other features will be added to the framework, thereby increasing overall coverage of error-handling (e.g., verifying that migration steps are indeed idempotent).

[1] #18057
[2] https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/roachtestutil/mixedversion/README.md

Jira issue: CRDB-31478