-
Notifications
You must be signed in to change notification settings - Fork 4.1k
kvcoord: catchup scan quota acquisition #105058
Copy link
Copy link
Closed
Labels
A-kv-replicationRelating to Raft, consensus, and coordination.Relating to Raft, consensus, and coordination.C-bugCode not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.v23.1.5
Description
observed on customer test cluster:
goroutine 1271105048 [select, 319 minutes]:
github.com/cockroachdb/cockroach/pkg/util/quotapool.(*AbstractPool).Acquire(0xc0aab5f340, {0x7281950, 0xc0c85c7320}, {0x7255b30, 0xc0afa1f690})
github.com/cockroachdb/cockroach/pkg/util/quotapool/quotapool.go:281 +0x75c
github.com/cockroachdb/cockroach/pkg/util/quotapool.(*IntPool).acquireMaybeWait(0xc0b9f430f8, {0x7281950, 0xc0c85c7320}, 0x1, 0x1)
github.com/cockroachdb/cockroach/pkg/util/quotapool/intpool.go:178 +0x13f
github.com/cockroachdb/cockroach/pkg/util/quotapool.(*IntPool).Acquire(...)
github.com/cockroachdb/cockroach/pkg/util/quotapool/intpool.go:147
github.com/cockroachdb/cockroach/pkg/util/limit.(*ConcurrentRequestLimiter).Begin(0xc0b9f430e0, {0x72818a8, 0xc0b4d9b800})
github.com/cockroachdb/cockroach/pkg/util/limit/limiter.go:58 +0x22a
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.acquireCatchupScanQuota({0x72818a8, 0xc0b4d9b800}, 0xc0008f1900, 0xc0b9f430e0)
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:574 +0x99
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*rangefeedMuxer).startSingleRangeFeed(0xc05b1e18f0, {0x72818a8, 0xc0b4d9b800}, {{0xc008327890, 0x16, 0x18}, {0xc0083278d8, 0x16, 0x18}}, {0x1768f5473b4aa152, ...}, ...)
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_mux_rangefeed.go:213 +0x159
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.divideSpanOnRangeBoundaries({0x72818a8, 0xc0b4d9b800}, 0x18?, {{0xc008327890, 0x16, 0x18}, {0xc0083278d8, 0x16, 0x18}}, {0x1768f5473b4aa152, ...}, ...)
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:394 +0x42f
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.divideAllSpansOnRangeBoundaries.func1.1({0x72818a8, 0xc0b4d9b800})
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:243 +0x157
github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx.func1()
github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:168 +0x25
golang.org/x/sync/errgroup.(*Group).Go.func1()
golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75 +0x64
created by golang.org/x/sync/errgroup.(*Group).Go
golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:72 +0xa5
Determine if there is an issue with muxrangefeed catchup scan quota management.
Jira issue: CRDB-28843
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
A-kv-replicationRelating to Raft, consensus, and coordination.Relating to Raft, consensus, and coordination.C-bugCode not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.v23.1.5