What is changed and how it works?

Issue Number: Close #17884

What's Changed:

The current SlowScore mechanism of TiKV can detect and handle the IO latency issues of Raft disks. As an improvement, the write latency of KV disks can also be taken into account to identify abnormal situations of KV disks.

As the above designs, this pr introduce an extra and individual inspector, triggered by inspect-kvdb-interval, to inspect the I/O latency of kvdb if deployed with separate mount path. And the inspector will periodically sends the inspecting latency to SlowScore algorithm, to make it can detect whether there exists disk I/O hung issues on kvdb disk.

Additionally, to mitigate the effects of complex foreground and background I/O operations triggered by RocksDB, the inspector simply writes a string to a designated file and records the time cost for this operation, logging it as the apply_process_duration. And by testing, it's proved to be valid and more accurate than directly recording the duration of applying on RocksDB.

Moreover, if raft-engine and kvdb uses the same mount path when deploying, this newly introduced inspector will not be created to make the inspecting of disk health triggered by inspect-inerval as previous work does.

This pr introduces an extra and individual inspector to detect whether there exists I/O hung issues on kvdb disk, if
the kvdb is deployed with a separate mount path.

Related changes

• PR to update pingcap/docs/pingcap/docs-cn:
• Need to cherry-pick to the release branch

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No code

And the following example shows that this mechanism make senses when injecting I/O delays to kvdb disk, using tpcc-1k workloads:

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Release note

This pr introduces an extra and individual inspector to detect whether there exists I/O hung issues on kvdb disk, if
the kvdb is deployed with a separate mount path.