Overview

Our integ tests need to run in an exclusive environment. Currently we are achieving this roughly by two ways:

• use of sandbox accounts to establish strong boundaries (e.g. canaries vs test pipelines)
• designation of regions per test to separate different tests within the same sandbox

However this model doesn't scale well. We should keep the Sandbox account boundary because we won't have that many
different sandboxes and setting up accounts as a one-off operation is relatively cheap.

Instead of region designations, we should however have a service that reserves and assigns a an account and region pair to a test case. Some test cases (like cross-account deployments #31934) might reserver multiple pairs. The service can also ensure environments are regularly cleaned up from any accidentally left-over resources.

Requirements

A new service an integration test can request exclusive, pre-configured environments to execute tests in.

• MUST support exclusive reservations for pairs of account/region from a pool
• MUST guarantee that test reservations do not starve production reservations (release, canaries...) (for example via the use of different pools)
• MUST support requesting combinations for cross-account deployment use cased
• MUST provide the requesting client with required credentials (i.e. role arn) to execute tests, and other execution-specific metadata like unique prefixes, tags to use, existing resource names, etc...
• MUST be robust against clients dying while holding a reservation (i.e. not hold on to reservations forever)
• SHOULD support metrics about wait time, test run time, failure rates etc.
• SHOULD support bootstrapping of envs for a given reservation, e.g. configuring 2 accounts with cross account trust
• SHOULD support clearing of environments from accidentally left-over resources
• SHOULD support metrics on accidentally left-over resources and the reservations that caused that