[DO NOT MERGE] Erlay: bandwidth-efficient transaction relay protocol (Full implementation) #30277
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
|
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers. Code Coverage & BenchmarksFor details see: https://corecheck.dev/bitcoin/bitcoin/pulls/30277. ReviewsSee the guideline for information on the review process. ConflictsReviewers, this pull request conflicts with the following ones:
If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first. LLM Linter (✨ experimental)Possible typos and grammar issues:
No other grammatical or typographic errors impacting comprehension were found. drahtbot_id_4_m |
8d72d2b to
3ad2364
Compare
This was referenced Jun 12, 2024
Contributor
|
🚧 At least one of the CI tasks failed. Make sure to run all tests locally, according to the Possibly this is due to a silent merge conflict (the changes in this pull request being Leave a comment here, if you need help tracking down a confusing failure. |
This was referenced Jun 13, 2024
2f61f66 to
45b153f
Compare
This was referenced Jun 13, 2024
218d65a to
b5aab7c
Compare
8 tasks
Member
|
Feel free to grab hebasto@ffed2ab to fix the MSVC build. |
4d348c1 to
ff47dfb
Compare
ff47dfb to
bcf0519
Compare
4 tasks
Co-authored-by: Gleb Naumenko <naumenko.gs@gmail.com>
When the time comes, we should send a sketch of our local reconciliation set to the reconciliation initiator. Co-authored-by: Gleb Naumenko <naumenko.gs@gmail.com>
When the sketches from both sides are combined successfully, the diff is produced. Then this diff can (together with the local txs) be used to identified which transactions are missing locally and remotely. Co-authored-by: Gleb Naumenko <naumenko.gs@gmail.com>
This will help to reuse the code later on in the function to announce transactions. Co-authored-by: Gleb Naumenko <naumenko.gs@gmail.com>
Co-authored-by: Gleb Naumenko <naumenko.gs@gmail.com>
…ondiff TODO: We may be OK defining a smaller m_recently_requested_short_ids, since its contents only really matters for less than a minute
If after decoding a reconciliation sketch it turned out to be insufficient to find set difference, request extension. Co-authored-by: Gleb Naumenko <naumenko.gs@gmail.com>
Store the initial sketches so that we are able to process extension sketch while avoiding transmitting the same data. Co-authored-by: Gleb Naumenko <naumenko.gs@gmail.com>
To be ready to respond to a sketch extension request from our peer, we should store a snapshot of our state and capacity of the initial sketch, so that we compute extension of the same size and over the exact same transactions. Transactions arriving during this reconciliation will be instead stored in the regular set. Co-authored-by: Gleb Naumenko <naumenko.gs@gmail.com>
Co-authored-by: Gleb Naumenko <naumenko.gs@gmail.com>
If peer failed to reconcile based on our initial response sketch, they will ask us for a sketch extension. Store this request to respond later. Co-authored-by: Gleb Naumenko <naumenko.gs@gmail.com>
Sending an extension may allow the peer to reconcile transactions, because now the full sketch has twice as much capacity. Co-authored-by: Gleb Naumenko <naumenko.gs@gmail.com>
If a peer sent us an extension sketch, we should reconstruct a full sketch from it with the snapshot we stored initially, and attempt to decode the difference. Co-authored-by: Gleb Naumenko <naumenko.gs@gmail.com>
This currently unused function is supposed to be used once a reconciliation round is done. It cleans the state corresponding to the passed reconciliation. Co-authored-by: Gleb Naumenko <naumenko.gs@gmail.com>
Once a peer tells us reconciliation is done, we should behave as follows: - if it was successful, just respond them with the transactions they asked by short ID. - if it was a full failure, respond with all local transactions from the reconciliation set snapshot - if it was a partial failure (only low or high part was failed after a bisection), respond with all transactions which were asked for by short id, and announce local txs which belong to the failed chunk. Co-authored-by: Gleb Naumenko <naumenko.gs@gmail.com>
We may still need to add more tests, specially around extensions (if we keep them) Co-authored-by: Gleb Naumenko <naumenko.gs@gmail.com>
575ecfa to
f6d810a
Compare
This was referenced Oct 7, 2025
Draft
This was referenced Oct 15, 2025
Merged
Contributor
|
🐙 This pull request conflicts with the target branch and needs rebase. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Erlay Project Tracking: #30249
This is a full implementation of Erlay. Its purpose is to check the integrity and correctness of the implementation against changes/additions that may originate from the review process and/or rebases on top of newer functionality.
This is not to be merged. Functionality will be spread across multiple smaller PRs to ease the review process.
Approach
The implementation approach builds on the following assumptions:
The general approach works as follows:
Reconciliation is used alongside fanout to relay transactions across the network. For Erlay nodes, the relay method will be decided per-transaction, instead of per connection, meaning that Erlay connections will do both fanout and reconciliation depending on the transaction (legacy connections will do only fanout, obviously).
The parameters selected for fanout are minimized to maximize the bandwidth saving. The current selected defaults are 4 outbound peers and 10% of inbounds. The relay logic depends on the type of connection and how the transaction has been received:
m_tx_inventory_known_filter, so their announcements also count. If the transaction was received via reconciliation, we simply reconcile with the rest of our peers.The reasoning for this is trying to guess how far the transaction has made it into the network with imperfect information. Knowing that fanout is faster than reconciliation, we want to have a higher fanout rate at the very beginning of the propagation, to get as far as we can, being fully efficient. This can be tied to how many of our peers know about the transaction already. Once the transaction is sufficiently spread, we can just reconcile it with the rest of our peers.
This does not apply to inbounds, as they are not trusted, and the metric will be easily abused, plus it may be used to leak transaction origin information. For them, we just keep a low fanout rate.
Testing and simulating
The last two commits of this PR are currently for simulation only. They allow to easily config the inbound/outbound fanout rate without having to recompile the code, and make full reconciliation more efficient.