Perf/parallel block downloader by asdacap · Pull Request #8255 · NethermindEth/nethermind

asdacap · 2025-02-24T10:28:54Z

Include 4444-ish: Refactor/forward header provider #8254
Currently forward sync only download from a single peer.
This is a problem as the upload of a peer is more limited than download therefore the download speed really depends on the peer that the forward sync is being pinned to.
- Depending on the peer, the download speed of the forward sync can be 0 (hang), 15 blk/s or 150 blks.
- Since nethermind can consistently process at least 30blk/s a lot of the time when catching up, the node is waiting until it got lucky with a peer with fast upload.
- This makes it hard to know if something is wrong with it or not.
- Also makes performance benchmarking inconsistent as most of the time, it waiting for block download.
This PR add the ability to dowload blocks and receipts in parallel.
This provide a much more consistent forward sync download speed of around 200 block per sec.
Works by having a map of hash->(block, receipts) that stores download requests.

Types of changes

What types of changes does your code introduce?

Optimization

Testing

Requires testing

Yes
No

If yes, did you write tests?

Yes
No

Notes on testing

benaadams · 2025-03-29T16:59:03Z

Needs rebasing

Copilot

Pull Request Overview

This PR improves forward sync performance by enabling parallel block and receipt downloads. Key changes include:

Adding a padded hash conversion method in Hash256.
Introducing a new blockchain ID ("Hoodi") and corresponding name mapping.
Extending ITxSource implementations with a new SupportsBlobs property and updating related transaction sources.

Reviewed Changes

Copilot reviewed 211 out of 213 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
src/Nethermind/Nethermind.Core/Crypto/Hash256.cs	Added FromBytesWithPadding method to pad hash bytes for non-32-byte inputs
src/Nethermind/Nethermind.Core/BlockchainIds.cs	Added a new blockchain ID (Hoodi) and its name mapping
src/Nethermind/Nethermind.Consensus/Transactions/*	Added SupportsBlobs property to various ITxSource implementations
src/Nethermind/Nethermind.Blockchain/Blocks/BlockhashStore.cs	Updated to use the new padded hash conversion method
src/Nethermind/Nethermind.Blockchain.Test/BlockhashProviderTests.cs	Introduced a new test to validate trimmed hash behavior
scripts/superchain.py	Adjusted log file name formatting

Files not reviewed (2)

scripts/private-networking/clique-validators.sh: Language not supported
scripts/systemd-utils/tail.sh: Language not supported

Copilot · 2025-03-29T16:59:12Z


+        public static Hash256 FromBytesWithPadding(ReadOnlySpan<byte> bytes)
+        {
+            if (bytes.Length != 32)


The FromBytesWithPadding method does not handle the case when bytes.Length > 32, which may result in a negative slice index. Consider changing the condition to 'if (bytes.Length < 32)' and adding appropriate handling or error reporting for inputs longer than 32 bytes.

Suggested change

if (bytes.Length != 32)

if (bytes.Length < 32)

…-downloader

asdacap · 2025-04-04T09:15:02Z

Request size adjustment is a bit flaky. Need to find a solution first or batch size is going to be limited by the slowest node.

…ownloader

Copilot

Pull Request Overview

This PR improves forward synchronization performance by introducing parallel block and receipt downloading while refactoring related tests and dependency registration. Key changes include:

Implementation of a MultiBlockDownloader to download blocks and receipts concurrently.
Refactoring of synchronization feeds and tests to use a new SyncBatchSize constant.
Updates to dependency injection lifetimes in merge-related components.

Reviewed Changes

Copilot reviewed 30 out of 30 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
MultiBlockDownloader.cs	Adds parallel download logic for block bodies and receipts.
IForwardSyncController.cs	Introduces a shared interface for forward sync handling.
FullSyncFeed.cs and FastSyncFeed.cs	Refactors feeds to use the forward sync controller and updates multi-feed status.
BlocksSyncPeerSelectionStrategy.cs	Adjusts peer selection logic using a temporary difficulty variable.
BlockDownloadRequest.cs	Changes BlocksRequest to implement IDisposable and manage pooled resources.
BlockDownloadContext.cs	Removes obsolete download context logic.
SynchronizerTests.cs, OldStyleFullSynchronizerTests.cs, ForwardHeaderProviderTests*.cs, BlockDownloaderTests.Merge.cs	Updates tests to use the new SyncBatchSizeMax constant and revised sync methods.
OwnedBlockBodies.cs	Revises enumerator implementation for owned block bodies.
MergeSynchronizer.cs	Updates dependency lifetimes for BlockDownloader and IForwardHeaderProvider to Singleton.
MergeBlockDownloader.cs, ChainLevelHelper.cs, BlockTreeTests.cs, SyncConfig.cs, ISyncConfig.cs	Minor adjustments and additions to support the new sync behavior.

Comments suppressed due to low confidence (1)

src/Nethermind/Nethermind.Merge.Plugin/Synchronization/MergeSynchronizer.cs:84

Changing the registration of BlockDownloader (and IForwardHeaderProvider) to Singleton may introduce thread-safety or state management issues if these services are not fully stateless and thread-safe. Please verify that the services are intended for a Singleton lifetime.

.AddSingleton<BlockDownloader, MergeBlockDownloader>()

Copilot · 2025-04-18T18:02:43Z

-        {
-            yield return blockBody;
-        }
+        return GetEnumerator();


The implementation of IEnumerable.GetEnumerator() calls GetEnumerator() recursively. Replace it with a call to _rawBodies.GetEnumerator() to correctly iterate over the collection.

…-downloader

LukaszRozmej

Looks good, few things to clarify before merging.

You probably don't want to update submodule!

…er.cs Co-authored-by: Lukasz Rozmej <lukasz.rozmej@gmail.com>

…rSelectionStrategy.cs Co-authored-by: Lukasz Rozmej <lukasz.rozmej@gmail.com>

This reverts commit 8403a08.

Revert "Perf/parallel block downloader (#8255)" This reverts commit 8403a08.

asdacap force-pushed the perf/parallel-block-downloader branch from b3fc46a to 0ff8706 Compare March 19, 2025 23:55

asdacap added 8 commits March 20, 2025 07:59

Parallel forward sync

5b9500a

Singleton for the block downloader

1d4b8c6

Some test

9578d59

Add test for parallel download

974220a

Whitespace

b0542ff

Map pruner

ab144f8

Whitespace

a8b2c89

Fix build

c24a4b9

asdacap force-pushed the perf/parallel-block-downloader branch from 0ff8706 to c24a4b9 Compare March 20, 2025 00:05

benaadams requested a review from Copilot March 29, 2025 16:58

benaadams added the performance is good label Mar 29, 2025

Copilot AI reviewed Mar 29, 2025

View reviewed changes

asdacap changed the base branch from refactor/forward-header-provider to master April 1, 2025 00:49

Merge remote-tracking branch 'origin/master' into perf/parallel-block…

8db4564

…-downloader

asdacap force-pushed the perf/parallel-block-downloader branch from f1c07fa to 8db4564 Compare April 4, 2025 09:04

asdacap added 12 commits April 4, 2025 17:53

Move request sizer to node stats

6b15144

Expose request size

71744a3

For bodies

000cbde

Simplify

b117cee

Fast headers

02eb597

Receipt sync feed

b44fa64

Also check for response count

da67b69

DLS and fix test

7344021

Fix wrong signature

6f3bef8

Merge branch 'perf/accurate-request-limit' into perf/parallel-block-d…

2932654

…ownloader

Use tx count to estimate buffer size

93fa71f

Nicer dsl

1472799

asdacap added 3 commits April 15, 2025 16:02

Slight cleanup

38d0e27

Minor cleanup

b398fda

Whitespace

8dec4fc

asdacap marked this pull request as ready for review April 15, 2025 08:25

asdacap requested a review from rubo as a code owner April 15, 2025 08:25

Fix test

0163cfc

benaadams requested a review from Copilot April 18, 2025 18:02

Copilot AI reviewed Apr 18, 2025

View reviewed changes

asdacap added 4 commits April 22, 2025 19:54

Address copilot

bb5b817

Merge remote-tracking branch 'origin/master' into perf/parallel-block…

afd701b

…-downloader

Merge branch 'master' into perf/parallel-block-downloader

d9394bc

Fix test

588ae03

LukaszRozmej reviewed Apr 28, 2025

View reviewed changes

asdacap and others added 10 commits April 30, 2025 09:58

Update src/Nethermind/Nethermind.Synchronization/Blocks/BlockDownload…

d191d70

…er.cs Co-authored-by: Lukasz Rozmej <lukasz.rozmej@gmail.com>

Update src/Nethermind/Nethermind.Synchronization/Blocks/BlockDownload…

364cbce

…er.cs Co-authored-by: Lukasz Rozmej <lukasz.rozmej@gmail.com>

Update src/Nethermind/Nethermind.Synchronization/Blocks/BlockDownload…

1d3b1b1

…er.cs Co-authored-by: Lukasz Rozmej <lukasz.rozmej@gmail.com>

Update src/Nethermind/Nethermind.Synchronization/Blocks/BlockDownload…

6ff84c6

…er.cs Co-authored-by: Lukasz Rozmej <lukasz.rozmej@gmail.com>

Update src/Nethermind/Nethermind.Synchronization/Blocks/BlockDownload…

cb9bd34

…er.cs Co-authored-by: Lukasz Rozmej <lukasz.rozmej@gmail.com>

Update src/Nethermind/Nethermind.Synchronization/Blocks/BlockDownload…

823f6c4

…er.cs Co-authored-by: Lukasz Rozmej <lukasz.rozmej@gmail.com>

Update src/Nethermind/Nethermind.Synchronization/Blocks/BlockDownload…

24730c0

…er.cs Co-authored-by: Lukasz Rozmej <lukasz.rozmej@gmail.com>

Update src/Nethermind/Nethermind.Synchronization/Blocks/BlocksSyncPee…

78a2116

…rSelectionStrategy.cs Co-authored-by: Lukasz Rozmej <lukasz.rozmej@gmail.com>

Address comment

603b109

Dont change tests

0902310

LukaszRozmej approved these changes Apr 30, 2025

View reviewed changes

Introduce allocate delay to prevent spinning

5688e32

asdacap merged commit 8403a08 into master Apr 30, 2025
80 checks passed

asdacap deleted the perf/parallel-block-downloader branch April 30, 2025 13:43

kamilchodola added a commit that referenced this pull request May 1, 2025

Revert "Perf/parallel block downloader (#8255)"

3bfd6ce

This reverts commit 8403a08.

kamilchodola mentioned this pull request May 1, 2025

Revert "Perf/parallel block downloader" #8574

Merged

kamilchodola added a commit that referenced this pull request May 1, 2025

Revert "Perf/parallel block downloader" (#8574)

b9a2a93

Revert "Perf/parallel block downloader (#8255)" This reverts commit 8403a08.

Conversation

asdacap commented Feb 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Types of changes

What types of changes does your code introduce?

Testing

Requires testing

If yes, did you write tests?

Notes on testing

Uh oh!

benaadams commented Mar 29, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Mar 29, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

asdacap commented Apr 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Apr 18, 2025

Choose a reason for hiding this comment

Uh oh!

LukaszRozmej left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

asdacap commented Feb 24, 2025 •

edited

Loading

asdacap commented Apr 4, 2025 •

edited

Loading