We currently have a very limited set of end-to-end tests under test/p2p, mainly focused on testing the P2P networking. We've had multiple issues recently that completely broke functionality in obvious ways (e.g. fast sync and state sync) since we don't do whole-system testing. We need more comprehensive end-to-end tests, which should support e.g.:

• Varying configuration and genesis parameters, for single nodes and entire networks
• Different network topologies, including sentry nodes and adding/removing validators
• Simple chaos testing (e.g. disconnecting the network, killing processes)

The suite needs to be able to run permutations of these in parallel, such that we can e.g. have a nightly run with a few hundred networks.

Comprehensive chaos testing (e.g. Jepsen tests) is out of scope for the initial version, and should possibly be implemented separately.

Related to #2805, #3689, #4641, #2795.

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate contributors tagged
• Contributor assigned/self-assigned