Skip to content

storage: crash with snapshot widens existing replica, but no replica exists for subsumed key #40257

@andreimatei

Description

@andreimatei

I was stressing the repartitioning tests, as one does.
@tbg or @nvanbenschoten do you have any guesses here? If not, I'm as good as any to look, but in a few days.

make stress PKG=./pkg/ccl/partitionccl TESTS=TestRepartitioning

=== RUN   TestRepartitioning/single_col_range_partitioning/unpartitioned
....
F190827 18:54:51.894556 264384 storage/replica_raft.go:1486  [n2,s2,r192/3:/Table/94/1/{5-6}] snapshot widens existing replica, but no replica exists for subsumed key /Table/94/1/6
goroutine 264384 [running]:
github.com/cockroachdb/cockroach/pkg/util/log.getStacks(0xc000424300, 0xc0004243c0, 0x0, 0x6)
	/home/andrei/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1016 +0xb1
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).outputLogEntry(0x70bc4e0, 0xc000000004, 0x6a3f13e, 0x17, 0x5ce, 0xc00178c800, 0x77)
	/home/andrei/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:872 +0x948
github.com/cockroachdb/cockroach/pkg/util/log.addStructured(0x484fa80, 0xc0076bd710, 0xc000000004, 0x2, 0x40095f8, 0x4b, 0xc003dd4b18, 0x1, 0x1)
	/home/andrei/go/src/github.com/cockroachdb/cockroach/pkg/util/log/structured.go:66 +0x2cc
github.com/cockroachdb/cockroach/pkg/util/log.logDepth(0x484fa80, 0xc0076bd710, 0x1, 0xc000000004, 0x40095f8, 0x4b, 0xc003dd4b18, 0x1, 0x1)
	/home/andrei/go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:69 +0x8c
github.com/cockroachdb/cockroach/pkg/util/log.Fatalf(...)
	/home/andrei/go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:180
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).maybeAcquireSnapshotMergeLock(0xc003c49800, 0x484fa80, 0xc0076bd710, 0x664b41c61ec58c54, 0xe3d23aadb1a1dd8d, 0xc0076f8f40, 0x0, 0x0, 0x0, 0xc005f08c80, ...)
	/home/andrei/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_raft.go:1486 +0x294
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).handleRaftReadyRaftMuLocked(0xc003c49800, 0x484fa80, 0xc0076bd710, 0x664b41c61ec58c54, 0xe3d23aadb1a1dd8d, 0xc0076f8f40, 0x0, 0x0, 0x0, 0xc005f08c80, ...)
	/home/andrei/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_raft.go:521 +0x1893
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRaftSnapshotRequest.func1(0x484fa80, 0xc0076bd710, 0xc003c49800, 0x0)
	/home/andrei/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3490 +0x19d
github.com/cockroachdb/cockroach/pkg/storage.(*Store).withReplicaForRequest(0xc000ce0700, 0x484fa80, 0xc0076bd710, 0xc005f08a48, 0xc007b1f4d8, 0x0)
	/home/andrei/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3370 +0x150
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRaftSnapshotRequest(0xc000ce0700, 0x484fa80, 0xc005a90d80, 0xc005f08a00, 0x664b41c61ec58c54, 0xe3d23aadb1a1dd8d, 0xc0076f8f40, 0x0, 0x0, 0x0, ...)
	/home/andrei/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3432 +0x1a4
github.com/cockroachdb/cockroach/pkg/storage.(*Store).receiveSnapshot(0xc000ce0700, 0x484fa80, 0xc005a90d80, 0xc005f08a00, 0x7f3371250df8, 0xc004a2c870, 0x0, 0x0)
	/home/andrei/go/src/github.com/cockroachdb/cockroach/pkg/storage/store_snapshot.go:871 +0x6d3
github.com/cockroachdb/cockroach/pkg/storage.(*Store).HandleSnapshot(0xc000ce0700, 0xc005f08a00, 0x7f3371250dc8, 0xc004a2c870, 0xc004a2c870, 0x30)
	/home/andrei/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3241 +0x201
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftSnapshot.func1.1(0x48a46e0, 0xc004a2c870, 0xc001294e10, 0x484fa80, 0xc005a90cc0, 0x707e7f, 0xc003d695b0)
	/home/andrei/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:415 +0x138
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftSnapshot.func1(0x484fa80, 0xc005a90cc0)
	/home/andrei/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:416 +0x5d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc001ca2480, 0x484fa80, 0xc005a90cc0, 0xc005120400, 0x32, 0x0, 0x0, 0xc005a90cf0)
	/home/andrei/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
	/home/andrei/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131


419 runs completed, 1 failures, over 22m10s

Metadata

Metadata

Assignees

Labels

A-kv-replicationRelating to Raft, consensus, and coordination.C-bugCode not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.C-test-failureBroken test (automatically or manually discovered).S-1High impact: many users impacted, serious risk of high unavailability or data loss

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions