Pure code movement. We'll make use of it outside this file in subsequent
commits.

Release note: None

Long-lived disk snapshots can drastically reduce testing time for scale
tests. Tests, whether run by hand or through CI, need only run the
long running fixture generating code (importing some dataset, generating
it organically through workload, etc.) once snapshot fingerprints are
changed, fingerprints that incorporate the major crdb version that
generated them.

Here's an example run that freshly generates disk snapshots:

    === RUN   admission-control/index-backfill
    no existing snapshots found for admission-control/index-backfill (ac-index-backfill), doing pre-work
    created volume snapshot ac-index-backfill-0001-vunknown-1-n2-standard-8 for volume irfansharif-snapshot-0001-1 on irfansharif-snapshot-0001-1/n1
    using 1 newly created snapshot(s) with prefix "ac-index-backfill"
    detached and deleted volume irfansharif-snapshot-0001-1 from irfansharif-snapshot-0001
    created volume irfansharif-snapshot-0001-1
    attached volume irfansharif-snapshot-0001-1 to irfansharif-snapshot-0001
    mounted irfansharif-snapshot-0001-1 to irfansharif-snapshot-0001
    --- PASS: admission-control/index-backfill (79.14s)

Here's a subsequent run that makes use of the aforementioned disk
snapshots:

    === RUN   admission-control/index-backfill
    using 1 pre-existing snapshot(s) with prefix "ac-index-backfill"
    detached and deleted volume irfansharif-snapshot-0001-1 from irfansharif-snapshot-0001
    created volume irfansharif-snapshot-0001-1
    attached volume irfansharif-snapshot-0001-1 to irfansharif-snapshot-0001
    mounted irfansharif-snapshot-0001-1 to irfansharif-snapshot-0001
    --- PASS: admission-control/index-backfill (43.47s)

We add the following APIs to the roachtest.Cluster interface, for tests
to interact with disk snapshots. admission-control/index-backfill is an
example test making use of these APIs.

  type Cluster interface {
      // ...

      // CreateSnapshot creates volume snapshots of the cluster using
      // the given prefix. These snapshots can later be retrieved,
      // deleted or applied to already instantiated clusters.
      CreateSnapshot(ctx context.Context, snapshotPrefix string) error

      // ListSnapshots lists the individual volume snapshots that
      // satisfy the search criteria.
      ListSnapshots(
        ctx context.Context, vslo vm.VolumeSnapshotListOpts,
      ) ([]vm.VolumeSnapshot, error)

      // DeleteSnapshots permanently deletes the given snapshots.
      DeleteSnapshots(
        ctx context.Context, snapshots ...vm.VolumeSnapshot,
      ) error

      // ApplySnapshots applies the given volume snapshots to the
      // underlying cluster. This is a destructive operation as far as
      // existing state is concerned - all already-attached volumes are
      // detached and deleted to make room for new snapshot-derived
      // volumes. The new volumes are created using the same specs
      // (size, disk type, etc.) as the original cluster.
      ApplySnapshots(
        ctx context.Context, snapshots []vm.VolumeSnapshot,
      ) error
  }

These Cluster APIs are in turn is powered by the following additions to
the vm.Provider interface, implemented by each cloud provider. GCE is
the fully spec-ed out one for now.

  type Provider interface {
      // ...

      // CreateVolume creates a new volume using the given options.
      CreateVolume(l *logger.Logger, vco VolumeCreateOpts) (Volume, error)

      // ListVolumes lists all volumes already attached to the given VM.
      ListVolumes(l *logger.Logger, vm *VM) ([]Volume, error)

      // DeleteVolume detaches and deletes the given volume from the
      // given VM.
      DeleteVolume(l *logger.Logger, volume Volume, vm *VM) error

      // AttachVolume attaches the given volume to the given VM.
      AttachVolume(l *logger.Logger, volume Volume, vm *VM) (string, error)

      // CreateVolumeSnapshot creates a snapshot of the given volume,
      // using the given options.
      CreateVolumeSnapshot(
        l *logger.Logger, volume Volume, vsco VolumeSnapshotCreateOpts,
      ) (VolumeSnapshot, error)

      // ListVolumeSnapshots lists the individual volume snapshots that
      // satisfy the search criteria.
      ListVolumeSnapshots(
        l *logger.Logger, vslo VolumeSnapshotListOpts,
      ) ([]VolumeSnapshot, error)

      // DeleteVolumeSnapshot permanently deletes the given snapshot.
      DeleteVolumeSnapshot(l *logger.Logger, snapshot VolumeSnapshot) error
  }

Since these snapshots necessarily outlive the tests, and we don't want
them dangling perpetually, we introduce a prune-dangling roachtest that
acts as a poor man's cron job, sifting through expired snapshots
(>30days) and deleting them. For GCE at least it's not obvious to me how
to create these snapshots in cloud buckets with a TTL built in, hence
this hack. It looks like this (with change to the TTL):

    === RUN   prune-dangling
    pruned old snapshot ac-index-backfill-0001-vunknown-1-n2-standard-8
    --- PASS: prune-dangling (8.59s)

---

We add expose some of these APIs through the roachprod binary directly.

$ roachprod snapshot --help
  snapshot enables creating/listing/deleting/applying cluster snapshots

  Usage:
    roachprod snapshot [command]

  Available Commands:
    create      snapshot a named cluster, using the given snapshot name and description
    list        list all snapshots for the given cloud provider, optionally filtering by the given name
    delete      delete all snapshots for the given cloud provider optionally filtering by the given name
    apply       apply the named snapshots from the given cloud provider to the named cluster

---

About admission-control/index-backfill. It's a fully featured test that
uses the TPC-C 100k dataset and runs a foreground load for 20k
customers. It takes >4hrs to import this data set; with disk snapshots
this step is skipped entirely and takes a few minutes. The actual test
is trivial, we run the foreground load for 1hr and run a large index
backfill concurrently. Before cockroachdb#98308, this results in wild performance
oscillations. It's still a bit wild after flow control, but less so.

We slightly extend the tpc-e harness to make this happen, adding a few
smarts: exposing a 'during' helper to run backfills concurrently with
foreground load, integrate with --skip-init, estimated setup times,
prometheus, and disk snapshots of course.

Release note: None