perf(core): block snapshots with layers#2664
Closed
edg-l wants to merge 126 commits into
Closed
Conversation
Lines of code reportTotal lines added: Detailed view |
Benchmark for c144ed8Click to view benchmark
|
Benchmark Block Execution Results Comparison Against Main
|
Benchmark for 084ba79Click to view benchmark
|
Benchmark for 9cc084fClick to view benchmark
|
Benchmark for 23432e3Click to view benchmark
|
Benchmark for c6fa39cClick to view benchmark
|
Benchmark for 3275d89Click to view benchmark
|
Benchmark for 795388aClick to view benchmark
|
Benchmark for e40a6a8Click to view benchmark
|
Benchmark for 4fe7658Click to view benchmark
|
pablodeymo
reviewed
May 29, 2025
| context.payload.header.requests_hash = context.requests_hash; | ||
| context.payload.header.gas_used = context.payload.header.gas_limit - context.remaining_gas; | ||
| context.account_updates = account_updates; | ||
|
|
Contributor
There was a problem hiding this comment.
Try not to include pretifier changes in such a big PR. This makes review more complicated.
Benchmark for 7349e91Click to view benchmark
|
pablodeymo
reviewed
May 29, 2025
| let other: Vec<_> = account_proof.iter().skip(1).cloned().collect(); | ||
| let trie = Trie::from_nodes(Some(root), &other)?; | ||
| if trie.get(&hash_address(address))?.is_none() { | ||
| if trie.get(hash_address(address))?.is_none() { |
Contributor
There was a problem hiding this comment.
I think it's better to fix clippy warnings in a ad hoc PR and then rebase main to this one.
Contributor
There was a problem hiding this comment.
Thanks for pointing this out 😄.
This wasn't happening in main, I figured out this was due to a change in the signature of Trie::get. Originally, this function received a &PathRLP (type alias to Vec<u8>), and this PR
changed said parameter to impl AsRef<u8>. I don't think this change is worthwile and also affected a lot of tests, incrementing the diff count and diff view unnecessarily. So I went and restored the original parameter.
…o edgl_snapshots_v2
…o edgl_snapshots_v2
…o edgl_snapshots_v2
MegaRedHand
reviewed
Jun 4, 2025
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
12 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Adds block snapshots using disk and diff layers. Improving performance by reducing trie database queries.
Description
Heavily inspired by https://github.com/ethereum/go-ethereum/blob/master/core/state/snapshot/
Uses a base disk layer at block X, adding diff layers as blocks come at X + 1, X + 2, until the cap of max layers is reached, when that happens, flattening of layers is done, moving the bottom disk layer closer to the head block like a moving window.
When one queries for example for the account state at HEAD-4 block. The snapshot tree requests the snapshot at HEAD-4, then using a bloom filter which aggregates all accounts from previous diff layers, we can now if we skip to the disk layer directly or traverse the diff layers. Traversing the diff layers is faster than quering the database.
The disk layer has a flat database table mapping hashes to accounts and storages
Missing features:
Last load tests done:
pr commit: a8ca81f
macOS M3 36gb
pr load-test:
First blocks: 0.58, 0.55, 0.54
Last blocks 0.38, 0.37, 0.32
pr load-test-erc20:
First blocks: 1.04, 0.86, 0.83, 0.99, 0.99
Last blocks: 0.58, 0.58, 0.56
Main:
load-test:
First blocks: 0.38, 0.23, 0.23, 0.30, 0.27
Last blocks: 0.21, 0.20, 0.12
load-test-erc20:
First blocks: 0.60, 0.53, 0.47, 0.48, 0.58, 0.56, 0.54
Last blocks: 0.40, 0.40, 0.36