Skip to content

backupccl: TestFullClusterBackup/ensure_that_jobs_are_restored flakes under span configs #75060

@irfansharif

Description

@irfansharif

Describe the problem

Saw a TestFullClusterBackup/ensure_that_jobs_are_restored failure here that looks like fallout from #73876. This issue tracks the investigation for as to why.

------- Stdout: -------
=== RUN   TestFullClusterBackup/ensure_that_jobs_are_restored
    full_cluster_backup_restore_test.go:334: 
        	Error Trace:	full_cluster_backup_restore_test.go:334
        	Error:      	Not equal: 
        	            	expected: []string{"728894745672974337", "running", "2022-01-18 13:10:23.459824 +0000 +0000", "\n\x1freconciling span configurations\x12\x04root\x18\x9b\xc6\xcaﯻ\xf5\x02\xa0\x01\x01\xda\x01\x00", "\x10\xf5\xb3\xc6ﯻ\xf5\x02\xa8\x01\xf2\xf4\xc1\xe9\xb0ւ\xfes\xb2\x01\x02\n\x00", "NULL", "NULL", "1"}
        	            	actual  : []string{"728894745672974337", "running", "2022-01-18 13:10:23.459824 +0000 +0000", "\n\x1freconciling span configurations\x12\x04root\x18\x9b\xc6\xcaﯻ\xf5\x02\xa0\x01\x01\x82\x02\x89!\n\arunning\x10鳥\xfe\xaf\xbb\xf5\x02\x18֝\xe8\xfe\xaf\xbb\xf5\x02 \x01*\xe9 \x12\xe6 \n\xf9\x1a\x12\xf6\x1a\n\xe4\x18\x12\xe1\x18\n\xd3\x17\x12\xd0\x17\n\xb4\x02\n\xb1\x02\n(expected to delete 11 row(s), deleted 10\x12\x84\x02\n8github.com/cockroachdb/errors/errutil/*errutil.leafError\x12:\n8github.com/cockroachdb/errors/errutil/*errutil.leafError\x1a(expected to delete 11 row(s), deleted 10\"b\n4type.googleapis.com/cockroach.errorspb.StringPayload\x12*\n(expected to delete 11 row(s), deleted 10\x1a\x96\x15\n<github.com/cockroachdb/errors/withstack/*withstack.withStack\x12>\n<github.com/cockroachdb/errors/withstack/*withstack.withStack\x1a\x95\x14\ngithub.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigkvaccessor.(*KVAccessor).updateSpanConfigEntriesWithTxn\n\t/go/src/github.com/cockroachd
b/cockroach/pkg/spanconfig/spanconfigkvaccessor/kvaccessor.go:186\ngithub.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigkvaccessor.(*KVAccessor).UpdateSpanConfigEntries.func1\n\t/go/src/github.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigkvaccessor/kvaccessor.go:128\ngithub.com/cockroachdb/cockroach/pkg/kv.runTxn.func1\n\t/go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:849\ngithub.com/cockroachdb/cockroach/pkg/kv.(*Txn).exec\n\t/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn.go:972\ngithub.com/cockroachdb/cockroach/pkg/kv.runTxn\n\t/go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:848\ngithub.com/cockroachdb/cockroach/pkg/kv.(*DB).Txn\n\t/go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:830\ngithub.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigkvaccessor.(*KVAccessor).UpdateSpanConfigEntries\n\t/go/src/github.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigkvaccessor/kvaccessor.go:127\ngithub.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigreconciler.(*fullReconciler).d
eleteExtraneousSpanConfigs\n\t/go/src/github.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigreconciler/reconciler.go:330\ngithub.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigreconciler.(*fullReconciler).reconcile\n\t/go/src/github.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigreconciler/reconciler.go:253\ngithub.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigreconciler.(*Reconciler).Reconcile\n\t/go/src/github.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigreconciler/reconciler.go:150\ngithub.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigjob.(*resumer).Resume\n\t/go/src/github.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigjob/job.go:74\ngithub.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine.func2\n\t/go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1123\ngithub.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine\n\t/go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1124\ngithub.com/cockroachdb/cockroach/pkg/jobs.
(*Registry).runJob\n\t/go/src/github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:401\ngithub.com/cockroachdb/cockroach/pkg/jobs.(*Registry).resumeJob.func1\n\t/go/src/github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:323\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:488\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1581\x1a\x88\x01\nAgithub.com/cockroachdb/errors/assert/*assert.withAssertionFailure\x12C\nAgithub.com/cockroachdb/errors/assert/*assert.withAssertionFailure\x128non-cancelable: expected to delete 11 row(s), deleted 10\x1a\xd2\x01\n9github.com/cockroachdb/errors/errutil/*errutil.withPrefix\x12;\n9github.com/cockroachdb/errors/errutil/*errutil.withPrefix\x1a\x0enon-cancelable\"H\n4type.googleapis.com/cockroach.errorspb.StringPayload\x12\x10\n\x0enon-cancelable\x1a\xe7\x05\n<github.com/cockroachdb/errors/withstack/*withstack.withStack\x12>\n<github.com/cockroachdb/errors/withstack/*withs
tack.withStack\x1a\xe6\x04\ngithub.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine\n\t/go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1148\ngithub.com/cockroachdb/cockroach/pkg/jobs.(*Registry).runJob\n\t/go/src/github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:401\ngithub.com/cockroachdb/cockroach/pkg/jobs.(*Registry).resumeJob.func1\n\t/go/src/github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:323\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:488\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1581\xda\x01\x00", "\x10\x84\xfb\xa0\xfe\xaf\xbb\xf5\x02\xa8\x01\xec\xda\xea\xb3ó\xea\x9dv\xb2\x01\x02\n\x00", "NULL", "NULL", "1"}
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -4,4 +4,4 @@
        	            	  (string) (len=38) "2022-01-18 13:10:23.459824 +0000 +0000",
        	            	- (string) (len=54) "\n\x1freconciling span configurations\x12\x04root\x18\x9b\xc6\xcaﯻ\xf5\x02\xa0\x01\x01\xda\x01\x00",
        	            	- (string) (len=25) "\x10\xf5\xb3\xc6ﯻ\xf5\x02\xa8\x01\xf2\xf4\xc1\xe9\xb0ւ\xfes\xb2\x01\x02\n\x00",
        	            	+ (string) (len=4291) "\n\x1freconciling span configurations\x12\x04root\x18\x9b\xc6\xcaﯻ\xf5\x02\xa0\x01\x01\x82\x02\x89!\n\arunning\x10鳥\xfe\xaf\xbb\xf5\x02\x18֝\xe8\xfe\xaf\xbb\xf5\x02 \x01*\xe9 \x12\xe6 \n\xf9\x1a\x12\xf6\x1a\n\xe4\x18\x12\xe1\x18\n\xd3\x17\x12\xd0\x17\n\xb4\x02\n\xb1\x02\n(expected to delete 11 row(s), deleted 10\x12\x84\x02\n8github.com/cockroachdb/errors/errutil/*errutil.leafError\x12:\n8github.com/cockroachdb/errors/errutil/*errutil.leafError\x1a(expected to delete 11 row(s), deleted 10\"b\n4type.googleapis.com/cockroach.errorspb.StringPayload\x12*\n(expected to delete 11 row(s), deleted 10\x1a\x96\x15\n<github.com/cockroachdb/errors/withstack/*withstack.withStack\x12>\n<github.com/cockroachdb/errors/withstack/*withstack.withStack\x1a\x95\x14\ngithub.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigkvaccessor.(*KVAccessor).updateSpanConfigEntriesWithTxn\n\t/go/src/github.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigkvaccessor/kvaccessor.go:186\ngithu
b.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigkvaccessor.(*KVAccessor).UpdateSpanConfigEntries.func1\n\t/go/src/github.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigkvaccessor/kvaccessor.go:128\ngithub.com/cockroachdb/cockroach/pkg/kv.runTxn.func1\n\t/go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:849\ngithub.com/cockroachdb/cockroach/pkg/kv.(*Txn).exec\n\t/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn.go:972\ngithub.com/cockroachdb/cockroach/pkg/kv.runTxn\n\t/go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:848\ngithub.com/cockroachdb/cockroach/pkg/kv.(*DB).Txn\n\t/go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:830\ngithub.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigkvaccessor.(*KVAccessor).UpdateSpanConfigEntries\n\t/go/src/github.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigkvaccessor/kvaccessor.go:127\ngithub.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigreconciler.(*fullReconciler).deleteExtraneousSpanConfigs\n\t/go/src/github.com/cockroachdb/cockroach/p
kg/spanconfig/spanconfigreconciler/reconciler.go:330\ngithub.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigreconciler.(*fullReconciler).reconcile\n\t/go/src/github.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigreconciler/reconciler.go:253\ngithub.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigreconciler.(*Reconciler).Reconcile\n\t/go/src/github.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigreconciler/reconciler.go:150\ngithub.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigjob.(*resumer).Resume\n\t/go/src/github.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigjob/job.go:74\ngithub.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine.func2\n\t/go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1123\ngithub.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine\n\t/go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1124\ngithub.com/cockroachdb/cockroach/pkg/jobs.(*Registry).runJob\n\t/go/src/github.com/cockroachdb/cockroach/pkg/jobs/
adopt.go:401\ngithub.com/cockroachdb/cockroach/pkg/jobs.(*Registry).resumeJob.func1\n\t/go/src/github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:323\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:488\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1581\x1a\x88\x01\nAgithub.com/cockroachdb/errors/assert/*assert.withAssertionFailure\x12C\nAgithub.com/cockroachdb/errors/assert/*assert.withAssertionFailure\x128non-cancelable: expected to delete 11 row(s), deleted 10\x1a\xd2\x01\n9github.com/cockroachdb/errors/errutil/*errutil.withPrefix\x12;\n9github.com/cockroachdb/errors/errutil/*errutil.withPrefix\x1a\x0enon-cancelable\"H\n4type.googleapis.com/cockroach.errorspb.StringPayload\x12\x10\n\x0enon-cancelable\x1a\xe7\x05\n<github.com/cockroachdb/errors/withstack/*withstack.withStack\x12>\n<github.com/cockroachdb/errors/withstack/*withstack.withStack\x1a\xe6\x04\ngithub.com/cockroachdb/cockroach/pkg/jobs.(*
Registry).stepThroughStateMachine\n\t/go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1148\ngithub.com/cockroachdb/cockroach/pkg/jobs.(*Registry).runJob\n\t/go/src/github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:401\ngithub.com/cockroachdb/cockroach/pkg/jobs.(*Registry).resumeJob.func1\n\t/go/src/github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:323\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:488\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1581\xda\x01\x00",
        	            	+ (string) (len=25) "\x10\x84\xfb\xa0\xfe\xaf\xbb\xf5\x02\xa8\x01\xec\xda\xea\xb3ó\xea\x9dv\xb2\x01\x02\n\x00",
        	            	  (string) (len=4) "NULL",
        	Test:       	TestFullClusterBackup/ensure_that_jobs_are_restored
    --- FAIL: TestFullClusterBackup/ensure_that_jobs_are_restored (0.03s)

To Reproduce

dev test pkg/ccl/backupccl -f=TestFullClusterBackup/ensure_that_jobs_are_restored

Jira issue: CRDB-12446

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions