Describe the problem

The allocator simulator exhibits non-determinism when overloaded with a high
number of stores and replicas on nodes.

To Reproduce

1. Set the number of stores and replicasPerStore setup in
   TestAllocatorSimulatorDeterministic to 11 and 200.
2. Run ./dev test pkg/kv/kvserver/asim -f TestAllocatorSimulatorDeterministic --ignore-cache --count=500

Additional context

Some observations from playing around:

• This issue seems to be alleviated by gossiping more aggressively. The default
  gossip interval is currently set to 10. Changing this value to 1s avoided the failure.
  But this doesn’t address the root cause since the result should be deterministic
  independent of gossip frequency.
  

  cockroach/pkg/kv/kvserver/asim/config/settings.go

  Line 26 in 2a7edbe

  defaultStateExchangeInterval = 10 * time.Second

• This issue seems to be only reproducible when both replicasPerStore and number
  of stores are overloaded. If only one is overloaded, the test seems to do fine.
  Overloading both parameters might have amplified the exhibited non-deterministic
  behavior.

Jira issue: CRDB-29265