In #24846 we started migrating the static index backwards compatibility tests to a gradle project. We did this for a few reasons:

1. The static tests played a lot of tricks to get everything to run in the same JVM so they weren't "real". This caused them to miss things from time to time. The qa:full-cluster-restart project has no such problems.
2. It is more difficult to reason about the static backwards compatibility tests because the static indexes are just binary blobs built by a python script. We have to be careful to keep it up to date, etc. The indices built by qa:full-cluster-restart are built at testing time so any changes to the index building code apply immediately.
3. Checking binary blobs into the code is icky.

There are a bunch of pieces left to port from the static tests to qa:full-cluster-restart:

• RestoreBackwardsCompatIT.testRestoreOldSnapshots @nik9000 Port most snapshot/restore static bwc tests to qa:full-cluster-restart #25296
• RepositoryUpgradabilityIT @nik9000 Port most snapshot/restore static bwc tests to qa:full-cluster-restart #25296
• OldIndexBackwardsCompatibilityIT#assertBasicSearchWorks @martijnvg Port OldIndexBackwardsCompatibilityIT#assertBasicSearchWorks over to full cluster restart qa module #24954
• OldIndexBackwardsCompatibilityIT#assertAllSearchWorks @martijnvg Port more OldIndexBackwardsCompatibilityIT tests to full cluster restart qa tests #25173
• OldIndexBackwardsCompatibilityIT#assertBasicAggregationWorks @martijnvg Port more OldIndexBackwardsCompatibilityIT tests to full cluster restart qa tests #25173
• OldIndexBackwardsCompatibilityIT#assertRealtimeGetWorks @martijnvg Port more OldIndexBackwardsCompatibilityIT tests to full cluster restart qa tests #25173
• OldIndexBackwardsCompatibilityIT#assertNewReplicasWork @martijnvg Port more OldIndexBackwardsCompatibilityIT tests to full cluster restart qa tests #25173
• OldIndexBackwardsCompatibilityIT#assertUpgradeWorks @martijnvg Port the remaining old indices search tests to full cluster restart qa module #25275
• OldIndexBackwardsCompatibilityIT#assertPositionIncrementGapDefaults (maybe not a big deal anymore because these are mostly for the 2.x -> 5.0 upgrade)
• OldIndexBackwardsCompatibilityIT#assertAliasWithBadName @martijnvg Port the remaining old indices search tests to full cluster restart qa module #25275
• OldIndexBackwardsCompatibilityIT#assertStoredBinaryFields @martijnvg Port the remaining old indices search tests to full cluster restart qa module #25275

There are a few things we need to clean up before we can remove a bunch of infrastructure for static bwc tests. I don't claim to know how to clean them all up:

• RestoreBackwardsCompatIT.testRestoreUnsupportedSnapshots
• RecoveryWithUnsupportedIndicesIT and StaticIndexBackwardCompatibilityIT
• Remove create_bwc_index.py Remove static indices and repos and the scripts that create them #26109
• Update "how to release" docs to remove "run create_bwc_index.py" (https://github.com/elastic/dev/commit/f12bf0a50bd4534942142b8904481acd8440033b)
• Remove static indices and repositories Remove static indices and repos and the scripts that create them #26109

And there are a bunch of things to add:

• Lenient boolean backwards compatibility tests
• Shrunken indexes (we've had trouble with them in the past) Add a full cluster restart test for index shrinking #25720