Currently we only move onto a new table once a table has started all of the clients for that table… I think we should investigate doing some kind of randomization to spread the load out across ALL apis because in GCP rate limiting is done on the user as well as project… So this means that we are effectively using up all of our quota then moving on to a different resource rather than using the api quota up consistently across all services

The current scheduling works well with any provider that has their rate limiting aligned to the multiplexer (ie AWS). In those scenarios a round robin scheduling system should not create a regression or reduction in performance.

Another option beyond implementing explicit scheduling algorithms in the SDK is to expose some kind of iterator function that enables each plugin to choose its own ordering.