Skip to content

spanconfig: AUTO SPAN CONFIG job does not handle duplicate after being restored #70173

@adityamaru

Description

@adityamaru

Describe the problem

A full cluster backup will backup all jobs (including automatic jobs) in the cluster. A cluster restore into a fresh cluster will restore the jobs from the backup into the system table of the cluster being restored into.

The AUTO SPAN CONFIG job is automatically created on cluster startup. Performing a full cluster restore, will result in 2 such entries for the AUTO SPAN CONFIG job. One started by the restoring cluster, and one from the backup.

Screen Shot 2021-09-13 at 7 58 58 PM

To Reproduce

  1. Start a single node cluster
  2. BACKUP INTO 'nodelocal://0/foo'
  3. Shut down the cluster
  4. Start a fresh single node cluster that uses the same cockroach-data/extern directory.
  5. RESTORE FROM LATEST IN 'nodelocal://0/foo'
  6. SHOW AUTOMATIC JOBS

Expected behavior
Semantics need to be ironed out for cluster backup and restore performed in a dedicated environment, and also as a secondary tenant.

BACKUP TENANT and RESTORE TENANT performed by the system tenant should be okay and only result in a single entry for the span config job. This is because a tenant restore runs in the host tenants' registry and simply writes all keys from [TenantPrefix, TenantPrefix.End] in a newly created, empty tenant. The creation of a tenant does not lead to a AUTO SPAN CONFIG job being triggered, therefore the restored job entry should be the only one of that kind.

Environment:

  • CockroachDB version: 21.2 onwards

Epic CRDB-8816

Jira issue: CRDB-9970

Metadata

Metadata

Assignees

Labels

A-disaster-recoveryA-zone-configsC-bugCode not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.GA-blockerT-disaster-recoverybranch-release-22.1Used to mark GA and release blockers, technical advisories, and bugs for 22.1

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions