-
Notifications
You must be signed in to change notification settings - Fork 4.1k
storage: pick up upstream fix for Raft entries handling #28918
Copy link
Copy link
Closed
Labels
A-kv-replicationRelating to Raft, consensus, and coordination.Relating to Raft, consensus, and coordination.C-bugCode not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.
Milestone
Description
Running a 3-node TPCC cluster on roachprod. Under no load, one of the nodes panicked:
E180821 22:00:16.110425 180 storage/replica.go:5367 [n1,s1,r604/1:/Table/54/{2/945/1…-3}] unable to update the state machine: applied index jumped from 26183 to 26185
E180821 22:00:16.110649 180 util/log/crash_reporting.go:477 [n1,s1,r604/1:/Table/54/{2/945/1…-3}] Reported as error bcb7784105ae42b9b06ca477937f8521
F180821 22:00:16.110679 180 storage/replica.go:5372 [n1,s1,r604/1:/Table/54/{2/945/1…-3}] applied index jumped from 26183 to 26185
goroutine 180 [running]:
github.com/cockroachdb/cockroach/pkg/util/log.getStacks(0xc4204a0500, 0xc4204a05a0, 0x3c3bf00, 0x12)
/home/jordan/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:965 +0xcf
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).outputLogEntry(0x453ba00, 0xc400000004, 0x3c3bf48, 0x12, 0x14fc, 0xc420c74370, 0x50)
/home/jordan/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:834 +0x804
github.com/cockroachdb/cockroach/pkg/util/log.addStructured(0x2e91a20, 0xc4208da600, 0xc400000004, 0x2, 0x0, 0x0, 0xc4287c90d8, 0x1, 0x1)
/home/jordan/go/src/github.com/cockroachdb/cockroach/pkg/util/log/structured.go:154 +0x2e5
github.com/cockroachdb/cockroach/pkg/util/log.logDepth(0x2e91a20, 0xc4208da600, 0x1, 0x4, 0x0, 0x0, 0xc4287c90d8, 0x1, 0x1)
/home/jordan/go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:69 +0x8c
github.com/cockroachdb/cockroach/pkg/util/log.Fatal(0x2e91a20, 0xc4208da600, 0xc4287c90d8, 0x1, 0x1)
/home/jordan/go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:181 +0x6c
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).processRaftCommand(0xc420859180, 0x2e91a20, 0xc4208da600, 0xc43075e5f0, 0x8, 0x9, 0x6649, 0x300000003, 0x2, 0x7, ...)
/home/jordan/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:5372 +0xaa8
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).handleRaftReadyRaftMuLocked(0xc420859180, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/home/jordan/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:4232 +0x12fa
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRequestQueue.func1(0x2e91a20, 0xc42d378d50, 0xc420859180, 0x2e91a20)
/home/jordan/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3685 +0x109
github.com/cockroachdb/cockroach/pkg/storage.(*Store).withReplicaForRequest(0xc420035400, 0x2e91a20, 0xc42d378d50, 0xc4296b3040, 0xc4287c9ed0, 0x0)
/home/jordan/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3300 +0x135
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRequestQueue(0xc420035400, 0x2e91a20, 0xc426982210, 0x25c)
/home/jordan/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3673 +0x229
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4201cfd40, 0x2e91a20, 0xc426982210)
/home/jordan/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:226 +0x21b
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2e91a20, 0xc426982210)
/home/jordan/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc424ea8cf0, 0xc420032480, 0xc424ea8ce0)
/home/jordan/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/jordan/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xad
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
A-kv-replicationRelating to Raft, consensus, and coordination.Relating to Raft, consensus, and coordination.C-bugCode not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.