Skip to content

kvserver: replicastats can produce nil ptr on perlocalitydecayingrate #94696

@kvoli

Description

@kvoli

Describe the problem

When calling PerLocalityDecayingRate() can produce a nil ptr panic de-referencing the locality counter for a record.

To Reproduce

TestCCLLogic_multi_region_zone_configs/force_repaving_all_zone_configs

(better reproduction in progress)

* ERROR: a panic has occurred!
* runtime error: invalid memory address or nil pointer dereference
* (1) attached stack trace
*   -- stack trace:
*   | runtime.gopanic
*   |   GOROOT/src/runtime/panic.go:884
*   | runtime.panicmem
*   |   GOROOT/src/runtime/panic.go:260
*   | runtime.sigpanic
*   |   GOROOT/src/runtime/signal_unix.go:835
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver/replicastats.(*ReplicaStats).PerLocalityDecayingRate
*   |   github.com/cockroachdb/cockroach/pkg/kv/kvserver/replicastats/replica_stats.go:314
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver/replicastats.(*ReplicaStats).SnapshotRatedSummary
*   |   github.com/cockroachdb/cockroach/pkg/kv/kvserver/replicastats/replica_stats.go:385
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver/load.(*ReplicaLoad).RequestLocalityInfo
*   |   github.com/cockroachdb/cockroach/pkg/kv/kvserver/load/replica_load.go:207
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.RangeUsageInfoForRepl
*   |   github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replicate_queue.go:2193
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).Capacity.func1
*   |   github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store.go:2919
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*storeReplicaVisitor).Visit
*   |   github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store.go:439
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).Capacity
*   |   github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store.go:2914
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).Descriptor
*   |   github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store.go:2991
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).updateCapacityGauges
*   |   github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:920
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).computeMetrics
*   |   github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store.go:3248
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).ComputeMetricsPeriodically
*   |   github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store.go:3282
*   | github.com/cockroachdb/cockroach/pkg/server.(*Node).computeMetricsPeriodically.func1
*   |   github.com/cockroachdb/cockroach/pkg/server/node.go:773
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).VisitStores.func1
*   |   github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores.go:150
*   | github.com/cockroachdb/cockroach/pkg/util/syncutil.(*IntMap).Range
*   |   github.com/cockroachdb/cockroach/pkg/util/syncutil/int_map.go:352
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).VisitStores
*   |   github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores.go:149
*   | github.com/cockroachdb/cockroach/pkg/server.(*Node).computeMetricsPeriodically
*   |   github.com/cockroachdb/cockroach/pkg/server/node.go:772
*   | github.com/cockroachdb/cockroach/pkg/server.(*Node).startComputePeriodicMetrics.func1
*   |   github.com/cockroachdb/cockroach/pkg/server/node.go:757
*   | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
*   |   github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:470
*   | runtime.goexit
*   |   GOROOT/src/runtime/asm_amd64.s:1594
* Wraps: (2) runtime error: invalid memory address or nil pointer dereference

Expected behavior
A nil ptr de-reference doesn't occur in PerLocalityDecayingRate()

Environment:

Jira issue: CRDB-23093

Metadata

Metadata

Assignees

Labels

C-bugCode not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.T-kvKV Team

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions