Skip to content

kvserver: v23.1.2-custom: failed to decode tenant prefix from key for replica: insufficient bytes to decode uvarint value #104606

Closed
Closed
kvserver: v23.1.2-custom: failed to decode tenant prefix from key for replica: insufficient bytes to decode uvarint value#104606
Labels
C-bugCode not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.O-sentryOriginated from an in-the-wild panic report.T-kvKV Teambranch-release-23.1Used to mark GA and release blockers, technical advisories, and bugs for 23.1release-blockerIndicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked.v23.1.4
@cockroach-teamcity

Description

@cockroach-teamcity
cockroach-teamcity
opened on Jun 8, 2023

This issue was autofiled by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry link: https://cockroach-labs.sentry.io/issues/4236862053/?referrer=webhooks_plugin

Panic message:

replica_init.go:371: log.Fatal: failed to decode tenant prefix from key for replica [n1,s1,r&{5708/1:{-} × ×}]: insufficient bytes to decode uvarint value: ×
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).setDescLockedRaftMuLocked
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_init.go:371
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).initRaftMuLockedReplicaMuLocked
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_init.go:213
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.prepareRightReplicaForSplit
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_split.go:257
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.splitPostApply
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_split.go:181
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleSplitResult
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_application_result.go:299
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaStateMachine).handleNonTrivialReplicatedEvalResult
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_application_state_machine.go:320
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaStateMachine).ApplySideEffects
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_application_state_machine.go:187
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.mapCheckedCmdIter
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/cmd.go:210
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).applyOneBatch
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:295
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).ApplyCommittedEntries
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:251
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:1009
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:722
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:646
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:395
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:302
| github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
| github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:470
| runtime.goexit
| GOROOT/src/runtime/asm_amd64.s:1594
Wraps: (2) secondary error attachment
| insufficient bytes to decode uvarint value: ×
| (1) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/cockroach/pkg/util/encoding.DecodeUvarintAscending
| | github.com/cockroachdb/cockroach/pkg/util/encoding/encoding.go:546
| | github.com/cockroachdb/cockroach/pkg/keys.DecodeTenantPrefix
| | github.com/cockroachdb/cockroach/pkg/keys/sql.go:39
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).setDescLockedRaftMuLocked
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_init.go:369
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).initRaftMuLockedReplicaMuLocked
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_init.go:213
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.prepareRightReplicaForSplit
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_split.go:257
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.splitPostApply
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_split.go:181
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleSplitResult
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_application_result.go:299
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaStateMachine).handleNonTrivialReplicatedEvalResult
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_application_state_machine.go:320
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaStateMachine).ApplySideEffects
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_application_state_machine.go:187
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.mapCheckedCmdIter
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/cmd.go:210
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).applyOneBatch
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:295
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).ApplyCommittedEntries
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:251
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:1009
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:722
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:646
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:395
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2
| | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:302
| | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
| | github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:470
| | runtime.goexit
| | GOROOT/src/runtime/asm_amd64.s:1594
| Wraps: (2) insufficient bytes to decode uvarint value: ×
| Error types: (1) *withstack.withStack (2) *errutil.leafError
Wraps: (3) log.Fatal: failed to decode tenant prefix from key for replica [n1,s1,r&{5708/1:{-} × ×}]: insufficient bytes to decode uvarint value: ×
Error types: (1) *withstack.withStack (2) *secondary.withSecondaryError (3) *errutil.leafError
-- report composition:
*errutil.leafError: log.Fatal: failed to decode tenant prefix from key for replica [n1,s1,r&{5708/1:{-} × ×}]: insufficient bytes to decode uvarint value: ×
*secondary.withSecondaryError: details for github.com/cockroachdb/errors/withstack/*withstack.withStack:::
replica_init.go:371: *withstack.withStack (top exception)

Stacktrace (expand for inline code snippets):

https://github.com/cockroachdb/cockroach/blob/429af1379325bf0521dde42dd284b2aaf6d4282c/pkg/kv/kvserver/pkg/kv/kvserver/replica_init.go#L370-L372 in pkg/kv/kvserver.(*Replica).setDescLockedRaftMuLocked
https://github.com/cockroachdb/cockroach/blob/429af1379325bf0521dde42dd284b2aaf6d4282c/pkg/kv/kvserver/pkg/kv/kvserver/replica_init.go#L212-L214 in pkg/kv/kvserver.(*Replica).initRaftMuLockedReplicaMuLocked
https://github.com/cockroachdb/cockroach/blob/429af1379325bf0521dde42dd284b2aaf6d4282c/pkg/kv/kvserver/pkg/kv/kvserver/store_split.go#L256-L258 in pkg/kv/kvserver.prepareRightReplicaForSplit
https://github.com/cockroachdb/cockroach/blob/429af1379325bf0521dde42dd284b2aaf6d4282c/pkg/kv/kvserver/pkg/kv/kvserver/store_split.go#L180-L182 in pkg/kv/kvserver.splitPostApply
https://github.com/cockroachdb/cockroach/blob/429af1379325bf0521dde42dd284b2aaf6d4282c/pkg/kv/kvserver/pkg/kv/kvserver/replica_application_result.go#L298-L300 in pkg/kv/kvserver.(*Replica).handleSplitResult
https://github.com/cockroachdb/cockroach/blob/429af1379325bf0521dde42dd284b2aaf6d4282c/pkg/kv/kvserver/pkg/kv/kvserver/replica_application_state_machine.go#L319-L321 in pkg/kv/kvserver.(*replicaStateMachine).handleNonTrivialReplicatedEvalResult
https://github.com/cockroachdb/cockroach/blob/429af1379325bf0521dde42dd284b2aaf6d4282c/pkg/kv/kvserver/pkg/kv/kvserver/replica_application_state_machine.go#L186-L188 in pkg/kv/kvserver.(*replicaStateMachine).ApplySideEffects

cockroach/pkg/kv/kvserver/apply/cmd.go

Lines 209 to 211 in 429af13

curChecked := iter.CurChecked()
applied, err := fn(curChecked.Ctx(), curChecked)
if err != nil {
in pkg/kv/kvserver/apply.mapCheckedCmdIter

cockroach/pkg/kv/kvserver/apply/task.go

Lines 294 to 296 in 429af13

// Apply the side-effects of each command to the state machine.
appliedIter, err := mapCheckedCmdIter(stagedIter, t.sm.ApplySideEffects)
if err != nil {
in pkg/kv/kvserver/apply.(*Task).applyOneBatch

cockroach/pkg/kv/kvserver/apply/task.go

Lines 250 to 252 in 429af13

for iter.Valid() {
if err := t.applyOneBatch(ctx, iter); err != nil {
// If the batch threw an error, reject all remaining commands in the
in pkg/kv/kvserver/apply.(*Task).ApplyCommittedEntries
https://github.com/cockroachdb/cockroach/blob/429af1379325bf0521dde42dd284b2aaf6d4282c/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L1008-L1010 in pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked
https://github.com/cockroachdb/cockroach/blob/429af1379325bf0521dde42dd284b2aaf6d4282c/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L721-L723 in pkg/kv/kvserver.(*Replica).handleRaftReady
https://github.com/cockroachdb/cockroach/blob/429af1379325bf0521dde42dd284b2aaf6d4282c/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go#L645-L647 in pkg/kv/kvserver.(*Store).processReady
https://github.com/cockroachdb/cockroach/blob/429af1379325bf0521dde42dd284b2aaf6d4282c/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go#L394-L396 in pkg/kv/kvserver.(*raftSchedulerShard).worker
https://github.com/cockroachdb/cockroach/blob/429af1379325bf0521dde42dd284b2aaf6d4282c/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go#L301-L303 in pkg/kv/kvserver.(*raftScheduler).Start.func2

cockroach/pkg/util/stop/stopper.go

Lines 469 to 471 in 429af13

sp.UpdateGoroutineIDToCurrent()
f(ctx)
}()
in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
GOROOT/src/runtime/asm_amd64.s#L1593-L1595 in runtime.goexit

pkg/kv/kvserver/pkg/kv/kvserver/replica_init.go in pkg/kv/kvserver.(*Replica).setDescLockedRaftMuLocked at line 371
pkg/kv/kvserver/pkg/kv/kvserver/replica_init.go in pkg/kv/kvserver.(*Replica).initRaftMuLockedReplicaMuLocked at line 213
pkg/kv/kvserver/pkg/kv/kvserver/store_split.go in pkg/kv/kvserver.prepareRightReplicaForSplit at line 257
pkg/kv/kvserver/pkg/kv/kvserver/store_split.go in pkg/kv/kvserver.splitPostApply at line 181
pkg/kv/kvserver/pkg/kv/kvserver/replica_application_result.go in pkg/kv/kvserver.(*Replica).handleSplitResult at line 299
pkg/kv/kvserver/pkg/kv/kvserver/replica_application_state_machine.go in pkg/kv/kvserver.(*replicaStateMachine).handleNonTrivialReplicatedEvalResult at line 320
pkg/kv/kvserver/pkg/kv/kvserver/replica_application_state_machine.go in pkg/kv/kvserver.(*replicaStateMachine).ApplySideEffects at line 187
pkg/kv/kvserver/apply/cmd.go in pkg/kv/kvserver/apply.mapCheckedCmdIter at line 210
pkg/kv/kvserver/apply/task.go in pkg/kv/kvserver/apply.(*Task).applyOneBatch at line 295
pkg/kv/kvserver/apply/task.go in pkg/kv/kvserver/apply.(*Task).ApplyCommittedEntries at line 251
pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked at line 1009
pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).handleRaftReady at line 722
pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).processReady at line 646
pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftSchedulerShard).worker at line 395
pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftScheduler).Start.func2 at line 302
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2 at line 470
GOROOT/src/runtime/asm_amd64.s in runtime.goexit at line 1594
Tag Value
Cockroach Release v23.1.2-224-g429af137932
Cockroach SHA: 429af13
Platform linux amd64
Distribution CCL
Environment v23.1.3
Command server
Go Version ``
# of CPUs
# of Goroutines

Jira issue: CRDB-28626

Metadata

Metadata

Assignees

No one assigned

    Labels

    C-bugCode not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.O-sentryOriginated from an in-the-wild panic report.T-kvKV Teambranch-release-23.1Used to mark GA and release blockers, technical advisories, and bugs for 23.1release-blockerIndicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked.v23.1.4

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions