kvserver: purging outdated replicas during migrations run into nil pointers #58378
Description
Describe the problem
On this build I see the following crash:
I201230 17:15:54.784810 1 util/stop/stopper.go:566 â‹® quiescing
W201230 17:15:54.784885 377 sql/sqlliveness/slinstance/slinstance.go:183 â‹® [n2] exiting heartbeat loop
F201230 17:15:54.770224 110793 kv/kvserver/replica_proposal.go:783 ⋮ [n2,s2,r85/1:‹/Table/52/1/{49-50}›] not using applied state key in v21.1
goroutine 110793 [running]:
github.com/cockroachdb/cockroach/pkg/util/log.getStacks(0x7fdd001, 0xe9a5d6, 0x203000, 0x452890)
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/get_stacks.go:25 +0xb9
github.com/cockroachdb/cockroach/pkg/util/log.(*loggerT).outputLogEntry(0xc000913000, 0x16558dfbdede14da, 0x4, 0x1b0c9, 0x72e7526, 0x1f, 0x30f, 0x0, 0xc001019040, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:270 +0xc52
github.com/cockroachdb/cockroach/pkg/util/log.logfDepth(0x54d7220, 0xc0038f7000, 0x1, 0x4, 0x48433ac, 0x24, 0x0, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/channels.go:58 +0x1c5
github.com/cockroachdb/cockroach/pkg/util/log.Fatalf(...)
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/log_channels_generated.go:804
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evaluateProposal(0xc002994800, 0x54d7220, 0xc0038f7000, 0xc003412830, 0x8, 0xc003b350e0, 0xc002476840, 0x0, 0xc003412800, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_proposal.go:783 +0x690
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).requestToProposal(0xc002994800, 0x54d7220, 0xc0038f7000, 0xc003412830, 0x8, 0xc003b350e0, 0xc002476840, 0x484, 0x84818)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_proposal.go:816 +0x8e
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).evalAndPropose(0xc002994800, 0x54d7220, 0xc0038f7000, 0xc003b350e0, 0xc003fcc3f0, 0xc001b8d528, 0x0, 0x0, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_raft.go:74 +0xf1
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeWriteBatch(0xc002994800, 0x54d7220, 0xc0038f7000, 0xc003b350e0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_write.go:136 +0x73d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries(0xc002994800, 0x54d7220, 0xc0038f7000, 0xc003b350e0, 0x4e95740, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:352 +0x491
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).sendWithRangeID(0xc002994800, 0x54d7220, 0xc0038f7000, 0x55, 0xc003b350e0, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:97 +0x53d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).Send(...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:36
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*pendingLeaseRequest).requestLeaseAsync.func2(0x54d7220, 0xc0038f7000)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_range_lease.go:402 +0x632
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000ec4240, 0x54d7220, 0xc0038f7000, 0xc0038eaa00, 0x35, 0xc000e8f500, 0xc002281b80)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:347 +0xdd
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:342 +0x11d
This code was introduced in #58088. I suppose it's possible that my PR caused this issue, but I doubt it. Another thing to note is that I don't think this cluster was upgraded or anything like that so I have to assume that there's some initialization invariant that code is not expecting.
Metadata
Metadata
Assignees
Labels
No labels