Skip to content

Overly frequent storage stats reporting causing high tail latency when dataset is large #18955

@v01dstar

Description

@v01dstar

Bug Report

TiKV reports store size every 1 second, since it requires reading SST properties with RocksDB DB-wide mutex held, serving tail latency got impacted significantly when single-instance dataset is huge.

What version of TiKV are you using?

What operating system and CPU are you using?

Steps to reproduce

What did you expect?

Background activities have limited impact on online traffic.

What did happened?

Tail latency increase, QPS drop

Metadata

Metadata

Assignees

No one assigned

    Labels

    affects-7.5This bug affects the 7.5.x(LTS) versions.affects-8.1This bug affects the 8.1.x(LTS) versions.affects-8.5This bug affects the 8.5.x(LTS) versions.affects-9.0This bug affects the 9.0.x versions.severity/majortype/bugThe issue is confirmed as a bug.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions