-
Notifications
You must be signed in to change notification settings - Fork 38.7k
fees: Introduce Mempool Based Fee Estimation to reduce overestimation #34075
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
fees: Introduce Mempool Based Fee Estimation to reduce overestimation #34075
Conversation
|
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers. Code Coverage & BenchmarksFor details see: https://corecheck.dev/bitcoin/bitcoin/pulls/34075. ReviewsSee the guideline for information on the review process. ConflictsReviewers, this pull request conflicts with the following ones:
If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first. LLM Linter (✨ experimental)Possible places where named args for integral literals may be used (e.g.
Possible places where comparison-specific test macros should replace generic comparisons:
No other added Python lines contain bare assert comparisons that should be replaced with the provided test helpers. 2026-01-12 |
dafb945 to
6ab7665
Compare
|
I think this is a good idea, especially if it there was a 'very conservative' that just didn't use this.
I don't agree here-- rather time based expiration should probably be eliminated over time. If there really is a censored transaction then we want nodes to work on getting it mined, and not just give up if the censors manage to control the blocks for a long time. Also two weeks is already long time to allow a non-minable transaction to distort fee estimates. And it's a short time for low rate confirmation-- there have been times in history when transactions weren't confirmed for months after their initial announcement (though those were also at time so far where size based expiration would have eliminated them). So I suggest instead that future work (not this PR) could improve the estimator by excluding from considerations transactions that should have been mined according to our template but weren't for more than a threshold number of blocks, perhaps 6 or something like that. Such code could also be used to log/warn the user that there is either ongoing censorship or that their node's policy appears to be less restrictive than the network. But in any case it doesn't need to be done immediate to make use of this kind of estimator.
One thing to keep in mind is that if this approach is widely used the success rate will probably change. It might make sense to initially deploy with more conservative settings (/defaults) and slowly increase them to avoid causing an abrupt change. |
6ab7665 to
41076c6
Compare
Indeed. In the context of censorship, we would want to keep such transactions if they are incentive compatible. However, there may also be legitimate reasons miners deliberately exclude them for example, a legacy node that is unaware of a soft fork that invalidates a certain spend (e.g., post-quantum). Hence as you also mentioned, I would like to see the direction of #33510, leaning toward deferring this improvement to future work.
I considered this and even had a branch that added an option to However, the users this targets are those running FWIW, I also did some research over a short block interval to get a sense of what Bitcoin core wallet users uses for feerate estimation, and I find that the majority of Bitcoin Core wallet users (that I was able to fingerprints) are not using the block-policy estimator, we’ve seen self-hosted users 1 2 switch approaches in the past, I’ve also personally spoken to large Bitcoin services that rely on third party but want this feature. |
|
I think enabling it by default probably makes sense, that particular sub comment was more about having a ready way to choose not to use it when transacting. As far as the initial defaults I think there I meant that that it shouldn't target too close to the bottom of the mempool as even if that currently gets high success that may not be true after wide deployment. Though on reflection I'm not sure that thats true as this only lowers feerates, so other people using this should only increase your odds of success while using it. |
41076c6 to
fee2612
Compare
de42f89 to
9e2070f
Compare
|
🚧 At least one of the CI tasks failed. HintsTry to run the tests locally, according to the documentation. However, a CI failure may still
Leave a comment here, if you need help tracking down a confusing failure. |
i - FeeRateEstimator abstract class is the base class of fee rate estimators
Derived classes must provide concrete implementation
of the virtual methods.
ii - FeeRateEstimatorResult struct represent the response returned by
a fee rate estimator.
iii - FeeRateEstimatorType will be used to identify fee estimators.
Each time a new fee estimator is added, a corresponding
enum value should be added.
Co-authored-by: willcl-ark <will@256k1.dev>
- This commit updates CBlockPolicyEstimator class to implement the FeeRateEstimator interface. Co-authored-by: willcl-ark <will@256k1.dev>
- Introduce FeeRateEstimatorManager, a module for managing multiple fee rate estimators. This manager allows registration of multiple FeeRateEstimator instances, enabling having more fee rate estimation strategies to be used in the bitcoin core node and wallet. Co-authored-by: willcl-ark <will@256k1.dev>
- Add GetFeeRateEstimator(), a templated method that returns a specific estimator instance by type.
- Initialize FeeEstimatorManager in node instead of BlockPolicyEstimator. - Forward fee rate estimation requests through FeeEstimatorManager. - Update Chain interface to return FeeRateEstimatorResult, removing the out parameter requirement. - Retain selected BlockPolicyEstimator methods for test-only estimaterawfee RPC compatibility.
- Given a vector of chunk feerates in the order they were added to the block, CalculatePercentiles function will return the 25th, 50th, 75th and 95th percentile feerates. - This commit also add a unit test for the CalculatePercentile function.
- The mempool fee rate estimator uses the unconfirmed transactions in the mempool to generate a fee rate that a package should have for it to confirm as soon as possible. Co-authored-by: willcl-ark <will@256k1.dev>
- Only refresh the cached fee rate estimate if it is older
than 7 seconds.
- This avoids generating block templates too often.
Co-authored-by: willcl-ark <will@256k1.dev>
- This ensures that the users mempool has seen at least 75% of the last 6 blocks transactions.
- This commit also enable estimatesmartfee to return the stats when verbosity level is greater than 1.
9e2070f to
76b36fe
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is very impressive and a nice improvement. This is quite a big PR so I've spent some time trying to understand what the PR is doing conceptually, why the design choices were made, and what trade-offs are being accepted, before diving deeply into the code.
I’m still new, so please treat the points below as learning-oriented feedback rather than authoritative critique
High-level understanding/summary
From my understanding, this PR Introduces a mempool-based fee estimator that supplements (but never replaces) the existing Block Policy Estimator i.e:
Final fee = min(Block Policy Estimator, mempool-based estimate)
This means:
Fees are never increased above the Block Policy Estimator using mempool data.
Mempool data can only lower the recommendation when conditions look favorable.
If the local mempool is deemed unreliable (via the “mempool health” heuristic), the system falls back entirely to the Block Policy Estimator.
Threat model, assumptions, and mitigations
Miner manipulation (Finney-like attacks): A miner could try to seed the mempool with fake high-fee transactions or withhold a block to raise mempool fees before releasing it (a variant of a Finney attack). However, because this PR uses min(Block Policy Estimator, mempool_estimate), an attacker cannot force the recommended fee above the safe Block Policy Estimator value (which is based on confirmed blocks). In effect, miners can only induce under-bidding, not over-bidding. As a result, the worst-case is that transactions are slower to confirm (underestimate), but never require overpayment. Underpayment is considered safer since transactions can be fee bumped later if needed. So the assumption here is “it is relatively easy to fee-bump later if a transaction does not confirm, whereas once a fee is overestimated there is no way to recover from that”
Mempool Representativeness & Filtering Issues: Local mempool divergence (e.g. due to policy differences or connectivity) could otherwise spoil the estimator. The PR addresses this through the “mempool health” (sanity) check: it tracks, for each of the last 6 blocks, how much weight of the block’s transactions was present in our mempool when the block arrived. If the average is below 75%, we assume our mempool isn’t representative (e.g. we missed many tx, or we hold tx miners wouldn’t mine) therefore we will not use the mempool estimator, falling back to Block Policy Estimator.
To be more elaborate IIUC, based on the mempool health, choose estimator:
If mempool is healthy: Bitcoin Core may use the mempool estimator by simulating the next block and computing a percentile fee (e.g., 50th or 75th percentile). Then it takes:
final_fee = min(Block Policy Estimator, mempool fee)
If mempool is not healthy: Bitcoin Core skips the mempool estimator entirely and just uses the Block Policy Estimator
Mempool health check make sure our mempool ≈ to the miners’ mempool. If my understanding is correct.
So I’m guessing this will be dynamic i.e always performing mempool health checks before choosing either to use Block Policy Estimator or mempool Estimator depending on the above conditions.
Question:
Are we treating the 75% mempool-health threshold as a statistically meaningful guarantee of fee-estimate correctness, or merely as a coarse guardrail — and do we have empirical evidence about false positives where the guardrail passes but the estimate is still misleading?
Adversarial flooding of mempool: The assumption here is that malicious peer could flood the local mempool with many very low-fee or very high-fee transactions. If low-fee spam dominates the top of the mempool histogram, the chosen percentile fee rate drops sharply, causing an underestimate. This is tolerated (the user may pay less but risk delay), and in practice the mempool “health” checks will likely disable mempool-based estimates if the mempool is dominated by attacker traffic. Flooding with high-fee spam cannot raise the fee estimate beyond Block Policy Estimator, limiting damage.
Choice of Percentiles and Other Heuristics: The PR chooses fixed fee percentiles for the mempool-based estimator based on empirical testing. In economical mode, it returns the 75th percentile fee from a simulated block template built from the current mempool, while conservative mode returns the 50th percentile. The percentile represents the fee rate at which that fraction of the block’s total weight would be filled when transactions are ordered by fee rate. These choices aim to balance cost and reliability: higher percentiles lower fees but increase delay risk, while lower percentiles raise fees and improve confirmation odds. Mainnet testing (Feb–Mar 2024) showed almost no overpayments and about 26% underpayments, compared to roughly 29% overpayments when using the historical Block Policy Estimator alone.
Beyond percentiles, the PR introduces two additional heuristics: a 7-second cache and a 75% mempool weight threshold. The cache avoids rebuilding block templates too frequently, based on the assumption that mempool contents do not change meaningfully within a few seconds. The weight threshold checks whether the local mempool is representative by requiring that, on average, at least 75% of the transaction weight in recent blocks was already present in the node’s mempool before mining. If this condition is not met, the estimator ignores the mempool and falls back to Block Policy Estimator.
These constants are explicitly acknowledged as tunable and somewhat arbitrary. Overall, the heuristics are intended as a practical starting point that improves fee accuracy without compromising safety, while leaving room for refinement.
Handling Sudden Inflows (“Unfixed Case”): A key concern is when the mempool suddenly fills (fee demand surges) after the estimate was computed but before the transaction is mined. In this case the mempool-based estimate was too low and Block Policy Estimator will catch up only gradually via future blocks. Historical data suggests this happens roughly 26% of the time, accounting for most underestimation cases. In such cases users may need an RBF bump once the fee crisis becomes apparent.
Mempool health check suggestions
The PR already exposes detailed mempool_health_statistics via estimatesmartfee (verbosity > 2), which is a great foundation. However, I think node operators and wallet developers currently lack guidance on how to interpret and act on this data. I suggest the following improvements and documentation clarifications:
Clarify how mempool health is interpreted
Document that each entry shows how much of a mined block’s weight was already present in the local mempool.
Explain that the estimator averages the ratios over recent blocks (typically 6), and considers the mempool healthy if the average ≥ 75%.
Explicitly state that if the threshold is not met (unless my understanding about this is incorrect), Bitcoin Core falls back to the Block Policy Estimator.
Consider exposing:
"mempool_healthy": true|false
I think this avoids forcing node operators and tooling to recompute the average manually and makes monitoring simpler.
Improve logs when switching estimators:
I don’t know whether this already exists. But I think adding explicit log lines when mempool estimator is enabled or disabled due to health and which estimator was chosen for a given estimatesmartfee call will also be beneficial by making debugging easier when a node suddenly stops using mempool estimates.
Examples:
INFO: mempool health avg=0.94 ≥ 0.75 — using mempool estimator
WARN: mempool health avg=0.62 < 0.75 — falling back to Block Policy Estimator
Suggest sanity warnings / metrics
Operators monitoring nodes could benefit from:
Alerts when mempool health drops below threshold for a long duration
A gauge or counter for mempool health ratio over time
I think this helps detect network connectivity issues or local policy divergences before they impact fee estimates.
Document interpretation & edge cases
Finally I think adding a brief section in the PR or documentation covering:
What does it mean when mempool health is low?
Should users be concerned if the mempool estimator is frequently disabled? If yes, then when should they be concerned?
I will be reviewing the code very soon.
This PR is another attempt to fix #27995 using a better approach.
For background and motivation, see #27995 and the discussion in the Delving Bitcoin post Mempool Based Fee Estimation on Bitcoin Core
This PR is currently limited to using the mempool only to lower what is recommended by the Block Policy Estimator.
Accurate and safe fee estimation using the mempool is challenging. There are open questions about how to prevent mempool games that are theoretically possible for miners (a variant of the Finney attack)
This is one of the reasons I opted to use the mempool only to lower the Block Policy Estimator, which itself is not gameable, and therefore this mechanism is not susceptible to this attack.
The underlying assumption here is that, with the current tools and work done to make RBF and CPFP feasible and guarantee (TRUC transaction relay, ephemeral anchors, cluster size 2 package rbf), it is safer to underestimate rather than overestimate.
We now assume it is relatively easy to fee-bump later if a transaction does not confirm, whereas once a fee is overestimated there is no way to recover from that.
Another open question when using the mempool for fee estimation is how to account for incoming transaction inflow.
Bitcoin Augur does this by using past inflow plus a constant expected inflow to predict future inflow. I find this unconvincing for fee estimation and potentially prone to more overestimation, as past conditions are not always representative of the future. See my review of the Augur fee rate estimator and open questions.
This PR uses a much simpler approach based on current user behavior, similar to the widely used method employed by mempool.space: looking at the top block of the mempool and selecting a percentile feerate depending on whether the user is economical or conservative.
Empirical data from both myself and Clara Shikhelman shows that the 75th percentile feerate for economical users and the 50th percentile feerate for conservative users provide positive confirmation guarantees, hence this is what is used in this PR.
Parallel research by René Pickhardt and his student suggests that using the average fee per byte of the block template performs well.
All of these are constants that can be adjusted, there is parallel work exploring these constants and running benchmarks across fee estimators to find a sweet spot.
See also work in LND, the LND Budget Sweeper, which applies this idea successfully. Their approach is to estimate fees initially with bitcoind, then increment gradually as the confirmation deadline approaches, using a fixed fee budget.
Historical data indicates that by doing only the approach in this PR, we are able to reduce overestimation quite significantly (~29%).
This is particularly useful in scenarios where the Block Policy Estimator recommends a high feerate while the mempool is empty.

As seen in the image above, there is only one remaining unfixed case: when there is a sudden inflow of transactions and the feerate rises, the Block Policy Estimator takes time to reflect this. In that case, users will continue to see a low feerate estimate until it slowly updates. From the historical data linked above, this occurs about ~26% of the time).
Overall, we observe a 73% success rate with 0% overestimation, and 26% underestimation with this approach.
This PR also includes refactors that enable this work. Rather than splitting the PR and implementing changes incrementally, I opted for an end-to-end implementation:
1. Refactors
createTransactionInternalto log only the fee and its source (fee estimator, fallback fee, etc.), which is more appropriate.StringForReasonenum fromcommonto the Block Policy Estimator (also separation of concerns)2. Introduce Mempool-Based Fee Estimator and Fee Estimator Manager
Introduce a new abstract fee estimator base class that all fee estimators must implement
Introduce
FeeRateEstimateResultas the response type, containing metadata and avoiding the use of out-parameters. This is easily extensible and makes future changes less invasive, as method signatures do not need to change.Add
FeeEstimatorTypeenum to identify different fee estimatorsMake
CBlockPolicyEstimatorderive from the new fee estimator base classAdd
FeeRateEstimatorManager, responsible for managing all fee estimatorsUpdate the node context to store a
unique_ptrtoFeeRateEstimatorManagerinstead ofCBlockPolicyEstimatorUpdate
CBlockPolicyEstimatorto no longer subscribe directly to the validation interface; instead,FeeRateEstimatorManagersubscribes and forwards relevant notificationsAdd a percentile calculation function that takes the chunk feerates of a block template and returns feerate percentiles; this can also be reused by the
getblockstatsRPCAdd a mempool fee estimator that generates a block template when called, calculates a percentile feerate, and returns the 75th percentile for economical mode or the 50th percentile for conservative mode
Add caching to the mempool estimator so that new estimates are generated only every 7 seconds, assuming enough transactions have propagated to make a meaningful difference.
This heuristic will likely be replaced by requesting block templates via the general-purpose block template cache proposed here: RFC: Bitcoin Core Node
BlockTemplateManager#33389Update
MempoolTransactionRemovedForBlockto return the actual block transactions as wellTrack the weight of block transactions and evicted transactions after each block connection
This data is tracked for the last 6 mined blocks. A mempool feerate estimate is returned only when the average ratio of mempool transaction weight evicted due to
BLOCKreasons to block transaction weight is greater than 75%. This heuristic provides rough confidence that the node’s mempool matches that of the majority of the hashrate. The 75% threshold is arbitrary and can be adjusted.There is a caveat when transactions in the local mempool are consistently not mined by the network, as described in #27995 (e.g., due to filtering).
Accounting for these transactions during fee estimation is not necessary, as they should be evicted from the mempool itself (see #33510). Handling this again within fee estimation would be redundant.
estimatesmartfeewith verbosity > 2see example output