Skip to content

storage: proposal quota assertion failure #39188

@ajwerner

Description

@ajwerner

A cluster deployed from 92fef12 hit the assertion added in #39135. The cluster had just been deployed and was running the kv50 workload in the following way on a 5 node cluster:

workload run kv --init --concurrency 64 --tolerate-errors --min-block-bytes=8 --max-block-bytes=128 --read-percent=50

// Assert the sanity of the base index and the queue. Queue entries should
// correspond to applied entries. It should not be possible for the base
// index and the not yet released applied entries to not equal the applied
// index.
releasableIndex := r.mu.proposalQuotaBaseIndex + uint64(len(r.mu.quotaReleaseQueue))
if releasableIndex != status.Applied {
log.Fatalf(ctx, "proposalQuotaBaseIndex (%d) + quotaReleaseQueueLen (%d) = %d"+
" must equal the applied index (%d)",
r.mu.proposalQuotaBaseIndex, len(r.mu.quotaReleaseQueue), releasableIndex,
status.Applied)
}

F190731 10:16:28.126181 158 storage/replica_proposal_quota.go:246  [n2,s2,r21/3:/Table/53{-/1/-4314…}] proposalQuotaBaseIndex (44321) + quotaReleaseQueueLen (52) = 44373 must equal the applied index (44321)

An observation here is that the applied index exactly equal the proposalQuotaBaseIndex. In theory we should only append to the quotaReleaseQueue when the applied index increases. How did entries make it into the quotaReleaseQueue without incrementing the applied index?

The raft log for range 21 on n2 shows entries up to 44379 and a commit index of 44373 but the RangeAppliedState (below) indicates that entries only up to 44321 have been applied. This however might be taken with a grain of salt as the applied state is not durably synced.
The machine did not die so my assumption would be that the write would at least make it to the WAL when the process exits and thus the current reading would reflect the last call to Commit(false) but I'm not all that confident in that.

0.000000000,0 /Local/RangeID/21/r/RangeAppliedState (0x01699d727261736b00): raft_applied_index:44321 lease_applied_index:44243 range_stats:<last_update_nanos:1564568187766302979 gc_bytes_age:27521006 live_bytes:521113 live_count:5282 key_bytes:353414 key_count:5282 val_bytes:1761302 val_count:23729 sys_bytes:588 sys_count:4 > 

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions