Skip to content

storage: pick up upstream fix for Raft entries handling #28918

@jordanlewis

Description

@jordanlewis

Running a 3-node TPCC cluster on roachprod. Under no load, one of the nodes panicked:

E180821 22:00:16.110425 180 storage/replica.go:5367  [n1,s1,r604/1:/Table/54/{2/945/1…-3}] unable to update the state machine: applied index jumped from 26183 to 26185
E180821 22:00:16.110649 180 util/log/crash_reporting.go:477  [n1,s1,r604/1:/Table/54/{2/945/1…-3}] Reported as error bcb7784105ae42b9b06ca477937f8521
F180821 22:00:16.110679 180 storage/replica.go:5372  [n1,s1,r604/1:/Table/54/{2/945/1…-3}] applied index jumped from 26183 to 26185
goroutine 180 [running]:
github.com/cockroachdb/cockroach/pkg/util/log.getStacks(0xc4204a0500, 0xc4204a05a0, 0x3c3bf00, 0x12)
        /home/jordan/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:965 +0xcf
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).outputLogEntry(0x453ba00, 0xc400000004, 0x3c3bf48, 0x12, 0x14fc, 0xc420c74370, 0x50)
        /home/jordan/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:834 +0x804
github.com/cockroachdb/cockroach/pkg/util/log.addStructured(0x2e91a20, 0xc4208da600, 0xc400000004, 0x2, 0x0, 0x0, 0xc4287c90d8, 0x1, 0x1)
        /home/jordan/go/src/github.com/cockroachdb/cockroach/pkg/util/log/structured.go:154 +0x2e5
github.com/cockroachdb/cockroach/pkg/util/log.logDepth(0x2e91a20, 0xc4208da600, 0x1, 0x4, 0x0, 0x0, 0xc4287c90d8, 0x1, 0x1)
        /home/jordan/go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:69 +0x8c
github.com/cockroachdb/cockroach/pkg/util/log.Fatal(0x2e91a20, 0xc4208da600, 0xc4287c90d8, 0x1, 0x1)
        /home/jordan/go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:181 +0x6c
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).processRaftCommand(0xc420859180, 0x2e91a20, 0xc4208da600, 0xc43075e5f0, 0x8, 0x9, 0x6649, 0x300000003, 0x2, 0x7, ...)
        /home/jordan/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:5372 +0xaa8
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).handleRaftReadyRaftMuLocked(0xc420859180, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /home/jordan/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:4232 +0x12fa
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRequestQueue.func1(0x2e91a20, 0xc42d378d50, 0xc420859180, 0x2e91a20)
        /home/jordan/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3685 +0x109
github.com/cockroachdb/cockroach/pkg/storage.(*Store).withReplicaForRequest(0xc420035400, 0x2e91a20, 0xc42d378d50, 0xc4296b3040, 0xc4287c9ed0, 0x0)
        /home/jordan/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3300 +0x135
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRequestQueue(0xc420035400, 0x2e91a20, 0xc426982210, 0x25c)
        /home/jordan/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3673 +0x229
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4201cfd40, 0x2e91a20, 0xc426982210)
        /home/jordan/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:226 +0x21b
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2e91a20, 0xc426982210)
        /home/jordan/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc424ea8cf0, 0xc420032480, 0xc424ea8ce0)
        /home/jordan/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
        /home/jordan/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xad

Metadata

Metadata

Assignees

Labels

A-kv-replicationRelating to Raft, consensus, and coordination.C-bugCode not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions