Skip to content

kvcoord: catchup scan quota acquisition #105058

@miretskiy

Description

@miretskiy

observed on customer test cluster:


goroutine 1271105048 [select, 319 minutes]:
github.com/cockroachdb/cockroach/pkg/util/quotapool.(*AbstractPool).Acquire(0xc0aab5f340, {0x7281950, 0xc0c85c7320}, {0x7255b30, 0xc0afa1f690})
	github.com/cockroachdb/cockroach/pkg/util/quotapool/quotapool.go:281 +0x75c
github.com/cockroachdb/cockroach/pkg/util/quotapool.(*IntPool).acquireMaybeWait(0xc0b9f430f8, {0x7281950, 0xc0c85c7320}, 0x1, 0x1)
	github.com/cockroachdb/cockroach/pkg/util/quotapool/intpool.go:178 +0x13f
github.com/cockroachdb/cockroach/pkg/util/quotapool.(*IntPool).Acquire(...)
	github.com/cockroachdb/cockroach/pkg/util/quotapool/intpool.go:147
github.com/cockroachdb/cockroach/pkg/util/limit.(*ConcurrentRequestLimiter).Begin(0xc0b9f430e0, {0x72818a8, 0xc0b4d9b800})
	github.com/cockroachdb/cockroach/pkg/util/limit/limiter.go:58 +0x22a
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.acquireCatchupScanQuota({0x72818a8, 0xc0b4d9b800}, 0xc0008f1900, 0xc0b9f430e0)
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:574 +0x99
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*rangefeedMuxer).startSingleRangeFeed(0xc05b1e18f0, {0x72818a8, 0xc0b4d9b800}, {{0xc008327890, 0x16, 0x18}, {0xc0083278d8, 0x16, 0x18}}, {0x1768f5473b4aa152, ...}, ...)
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_mux_rangefeed.go:213 +0x159
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.divideSpanOnRangeBoundaries({0x72818a8, 0xc0b4d9b800}, 0x18?, {{0xc008327890, 0x16, 0x18}, {0xc0083278d8, 0x16, 0x18}}, {0x1768f5473b4aa152, ...}, ...)
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:394 +0x42f
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.divideAllSpansOnRangeBoundaries.func1.1({0x72818a8, 0xc0b4d9b800})
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:243 +0x157
github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx.func1()
	github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:168 +0x25
golang.org/x/sync/errgroup.(*Group).Go.func1()
	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75 +0x64
created by golang.org/x/sync/errgroup.(*Group).Go
	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:72 +0xa5

Determine if there is an issue with muxrangefeed catchup scan quota management.

Jira issue: CRDB-28843

Metadata

Metadata

Assignees

Labels

A-kv-replicationRelating to Raft, consensus, and coordination.C-bugCode not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.v23.1.5

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions