-
Notifications
You must be signed in to change notification settings - Fork 4.1k
storage: measure/optimize intent resolution #7503
Description
When operations on a Range encounter intents which they manage to push (see also #627), they propose ResolveIntent operations to Raft (which in turn either proposes them directly, or relays to the leader).
It's wasteful to propose individual commands like that, especially if there are a lot of them. There are two immediate optimizations that come to mind:
- bundle various intent resolutions into one command and
- even better, let these bundles piggy-back on existing Raft traffic.
This should be possible because there's no client waiting at the sending end of an intent resolution, and so it can be carried out as a "best effort", the result of the Raft command being whatever it would be without the attempt at (separately) resolving the intents.
This is written for a pre-leader-proposed-Raft world, but translates over.
We don't have definitive evidence that intent resolution is a problem, but it's obvious that it clogs Raft for certain transactional workloads since one write = one intent (or even \infinity intents for range deletes) in some scenarios, and timeouts during intent resolution are commonly observed on even moderately busy clusters, and since we put back-pressure in this path, it may lead to all sorts of issues.
Measuring the amount of intent resolution traffic appropriately is a good first step.