Currently, the allocator simulator assesses the allocator’s behavior under
explicit and user-defined conditions. This could restrict the exposure of
complex scenarios.

Issues: #106311

This issue tracks work that needs to be done to integrate randomness into the
simulator framework, eliminating manual test setup and explicit assertions.

• Extend the gen_cluster command to support randomized parameters for zone
  configurations (including voter and non-voter constraints, localities) and a random number of nodes and stores per node.
• Add randomness to initial node placement across clusters
• Extend gen_ranges, gen_load to support randomized parameters
• Scope out what assertions would be interesting based on the random cluster
  setup
• Ensure consistency on assertion evaluation
• Extend set_liveness to support randomized node status change
• Currently, it is hard to verify stats without using a plot or assertion.
  Simplifying this by generating text-based statistics would make it easier.

Future work:

• Add new types of assertions for zone configuration, range distribution,
  node liveness, load balance, and when multiple samples are run
• Simulate network partition, high latency, load-related latency, and
  randomized inter-region latency based on node proximity
  • Different gossip delay for different regions

Anticipated difficulties:

• Test setup and flakiness: randomized testing could lead to complex and
  unexpected test setups. It could be hard to validate the generated input.

Some ideas to help alleviate these issues:

• Use a seed number for every randomization
• Record test inputs for every failed test
• Write unit tests on things to make them more deterministic
• Add new parameters to allow multiple test runs

Potentially useful libraries:

• https://go.dev/doc/tutorial/fuzz
• https://pkg.go.dev/pgregory.net/rapid

Note that this issue just outlines potential project directions. Some ideas
might be out of scope of this project.

Jira issue: CRDB-29441