-
Notifications
You must be signed in to change notification settings - Fork 4.1k
storage: crash with snapshot widens existing replica, but no replica exists for subsumed key #40257
Copy link
Copy link
Closed
Labels
A-kv-replicationRelating to Raft, consensus, and coordination.Relating to Raft, consensus, and coordination.C-bugCode not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.C-test-failureBroken test (automatically or manually discovered).Broken test (automatically or manually discovered).S-1High impact: many users impacted, serious risk of high unavailability or data lossHigh impact: many users impacted, serious risk of high unavailability or data loss
Description
I was stressing the repartitioning tests, as one does.
@tbg or @nvanbenschoten do you have any guesses here? If not, I'm as good as any to look, but in a few days.
make stress PKG=./pkg/ccl/partitionccl TESTS=TestRepartitioning
=== RUN TestRepartitioning/single_col_range_partitioning/unpartitioned
....
F190827 18:54:51.894556 264384 storage/replica_raft.go:1486 [n2,s2,r192/3:/Table/94/1/{5-6}] snapshot widens existing replica, but no replica exists for subsumed key /Table/94/1/6
goroutine 264384 [running]:
github.com/cockroachdb/cockroach/pkg/util/log.getStacks(0xc000424300, 0xc0004243c0, 0x0, 0x6)
/home/andrei/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1016 +0xb1
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).outputLogEntry(0x70bc4e0, 0xc000000004, 0x6a3f13e, 0x17, 0x5ce, 0xc00178c800, 0x77)
/home/andrei/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:872 +0x948
github.com/cockroachdb/cockroach/pkg/util/log.addStructured(0x484fa80, 0xc0076bd710, 0xc000000004, 0x2, 0x40095f8, 0x4b, 0xc003dd4b18, 0x1, 0x1)
/home/andrei/go/src/github.com/cockroachdb/cockroach/pkg/util/log/structured.go:66 +0x2cc
github.com/cockroachdb/cockroach/pkg/util/log.logDepth(0x484fa80, 0xc0076bd710, 0x1, 0xc000000004, 0x40095f8, 0x4b, 0xc003dd4b18, 0x1, 0x1)
/home/andrei/go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:69 +0x8c
github.com/cockroachdb/cockroach/pkg/util/log.Fatalf(...)
/home/andrei/go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:180
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).maybeAcquireSnapshotMergeLock(0xc003c49800, 0x484fa80, 0xc0076bd710, 0x664b41c61ec58c54, 0xe3d23aadb1a1dd8d, 0xc0076f8f40, 0x0, 0x0, 0x0, 0xc005f08c80, ...)
/home/andrei/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_raft.go:1486 +0x294
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).handleRaftReadyRaftMuLocked(0xc003c49800, 0x484fa80, 0xc0076bd710, 0x664b41c61ec58c54, 0xe3d23aadb1a1dd8d, 0xc0076f8f40, 0x0, 0x0, 0x0, 0xc005f08c80, ...)
/home/andrei/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_raft.go:521 +0x1893
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRaftSnapshotRequest.func1(0x484fa80, 0xc0076bd710, 0xc003c49800, 0x0)
/home/andrei/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3490 +0x19d
github.com/cockroachdb/cockroach/pkg/storage.(*Store).withReplicaForRequest(0xc000ce0700, 0x484fa80, 0xc0076bd710, 0xc005f08a48, 0xc007b1f4d8, 0x0)
/home/andrei/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3370 +0x150
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRaftSnapshotRequest(0xc000ce0700, 0x484fa80, 0xc005a90d80, 0xc005f08a00, 0x664b41c61ec58c54, 0xe3d23aadb1a1dd8d, 0xc0076f8f40, 0x0, 0x0, 0x0, ...)
/home/andrei/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3432 +0x1a4
github.com/cockroachdb/cockroach/pkg/storage.(*Store).receiveSnapshot(0xc000ce0700, 0x484fa80, 0xc005a90d80, 0xc005f08a00, 0x7f3371250df8, 0xc004a2c870, 0x0, 0x0)
/home/andrei/go/src/github.com/cockroachdb/cockroach/pkg/storage/store_snapshot.go:871 +0x6d3
github.com/cockroachdb/cockroach/pkg/storage.(*Store).HandleSnapshot(0xc000ce0700, 0xc005f08a00, 0x7f3371250dc8, 0xc004a2c870, 0xc004a2c870, 0x30)
/home/andrei/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3241 +0x201
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftSnapshot.func1.1(0x48a46e0, 0xc004a2c870, 0xc001294e10, 0x484fa80, 0xc005a90cc0, 0x707e7f, 0xc003d695b0)
/home/andrei/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:415 +0x138
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftSnapshot.func1(0x484fa80, 0xc005a90cc0)
/home/andrei/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:416 +0x5d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc001ca2480, 0x484fa80, 0xc005a90cc0, 0xc005120400, 0x32, 0x0, 0x0, 0xc005a90cf0)
/home/andrei/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
/home/andrei/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131
419 runs completed, 1 failures, over 22m10s
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
A-kv-replicationRelating to Raft, consensus, and coordination.Relating to Raft, consensus, and coordination.C-bugCode not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.C-test-failureBroken test (automatically or manually discovered).Broken test (automatically or manually discovered).S-1High impact: many users impacted, serious risk of high unavailability or data lossHigh impact: many users impacted, serious risk of high unavailability or data loss