Search before asking

• I searched in the issues and found nothing similar.

Motivation

Primary key table read/write operations involving RocksDB are currently executed on Netty worker threads while holding the bucket lock. When RocksDB slows down — due to disk I/O bottlenecks, compaction pressure, or compute saturation — the bucket lock is held for extended periods. This causes a cascading failure pattern:

1. RocksDB latency increases.
2. Netty worker threads block on the bucket lock waiting for the slow RocksDB operation to complete.
3. RPC requests of all types (not just KV writes) pile up in the queue because worker threads are exhausted.
4. The cluster enters a severe "stalling oscillation" — request timeouts cascade, healthy buckets become unreachable, and overall cluster health degrades far beyond the scope of the original slowdown.

The core issue: a single slow bucket's RocksDB degradation propagates into a cluster-wide availability problem.

Solution

Introduce a cooperative two-tier backpressure model between the tablet server and the writer client, driven by the column family's RocksDB L0 file count. The goal is to let the client absorb pressure via per-bucket throttling, and to fail fast at the server before RocksDB's own write stall is allowed to block the RPC handler thread.

Tier 1 — Piggyback throttle (soft, proactive)

• On each PutKv request the server reads the column family's L0 file count once via RocksDB#getLongProperty (no per-call string allocation).
• When L0 reaches kv.backpressure.l0-slowdown-trigger (default 12), the server piggybacks a normalized pressure value p ∈ [0, 1) on PbPutKvRespForBucket, where p linearly maps the gap between the Fluss trigger and RocksDB's own level0_slowdown_writes_trigger.
• The client throttles the affected bucket for client.writer.kv-backpressure.max-throttle-ms * p³ (default ceiling 5000ms). The cubic mapping gives a smooth ramp-up: small delays at low pressure, sharply growing delays as L0 approaches the storage trigger.

Tier 2 — Hard rejection (fast-fail, fallback)

• When L0 reaches RocksDB's level0_slowdown_writes_trigger, the server throws StorageBackpressureException (a RetriableException, Errors code 71) instead of submitting the write.
• This short-circuits the request so the RPC handler thread is never blocked by RocksDB's internal sleep.
• The client treats this as the maximum throttle window and lets the standard retry path re-enqueue the batch, making the transition between throttle and rejection continuous.

Why client-cooperative throttling rather than async execution alone

• Pushing the work onto an internal server-side queue without a feedback signal only relocates the queue — it does not reduce inbound pressure. Cooperative throttling slows producers before the queue forms.
• Storage-side hard rejection still bounds the queue depth and protects the Netty worker pool against the worst case.

Anything else?

No response

Willingness to contribute

• I'm willing to submit a PR!