Fix/more flaky tests by LukaszRozmej · Pull Request #9474 · NethermindEth/nethermind

LukaszRozmej · 2025-10-15T09:04:15Z

Types of changes

What types of changes does your code introduce?

Other: tests

Copilot

Pull Request Overview

This PR addresses flaky test issues by improving test reliability and modernizing C# syntax patterns. The changes focus on making asynchronous operations more robust in end-to-end synchronization tests and engine module tests.

Key changes:

Enhanced synchronization in E2E tests by adding proper wait mechanisms for block processing
Improved timing-sensitive payload production tests with retry mechanisms and better timeout handling
Updated method calls to use async versions where appropriate

Reviewed Changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File	Description
E2ESyncTests.cs	Added block processing wait mechanism, modernized variable declarations, and switched to async block suggestion
EngineModuleTests.PayloadProduction.cs	Added retry attribute for flaky test, improved timing calculations, and modernized TaskCompletionSource syntax
ParityStyleTracerTests.cs	Converted synchronous test to async and updated to use async block suggestion method

…_after

… Big_test((storage_hash_and_code_hash_same, System.Action`3[Nethermind.State.StateTree,Nethermind.Trie.Pruning.ITrieStore,Nethermind.Db.IDb]))

LukaszRozmej · 2025-10-15T11:20:28Z

@asdacap I don't have good idea what to do with:

failed Big_test((extension_create_new_extension, System.Action`3[Nethermind.State.StateTree,Nethermind.Trie.Pruning.ITrieStore,Nethermind.Db.IDb])) (103ms)
    END
  STATE TREE
  BRANCH | -> 5a717ee8d7ffe2e336590cfe9486084772c3fa0b503271507d357ad0086986e9
  ++++ACCOUNT  0e0c0107070b00070a0405000901020706080b000909000e030f030904020d0a0e0c0107070b00070a04050009010207000f0e0f06020b0d0c03060e0d020b -> d0087cb36afae313b22983588be5cad530011f3b089d8201ee94ed55d0572cf1
  ++++  NONCE: 2
  ++++  BALANCE: 4
  ++++  IS_CONTRACT: True
  ++++++STORAGE BRANCH | -> 80d9db0b7a1ab84066a618e4c0ba8795790390d6fdcfca44f70661a0d180c894
  ++01 BRANCH | -> 26ae5b6c96f8fc1f5615b4272d8006291fabfca3c12c372a27beee0a809c9a1e
  ++++01 BRANCH | -> 10fd949e620d08b396abd61821c54bf522e70a15685b02e920ff8e1dc0a6bf5e
  ++++++++ACCOUNT  030d0f080206080b09040b0d060209020f0d0d070c090b080a0f03090a07090f030d0f080206080b09040b0d060209020404060b0c0901020003090207 -> 41ce86c1d6a762182b69004e19741e10a83eacd123a75a332816e369b7384d5c
  ++++++++  NONCE: 2
  ++++++++  BALANCE: 3
  ++++++++  IS_CONTRACT: True
  ++++++++++STORAGE BRANCH | -> 285c62def2102bbcde16aa40ec5cb458bbca3ebec845358f26e2fbc45e863a45
  ++++++++ACCOUNT  080f0b0801000e040e0e080f030d0b0e01010401010007020604070203050e05080f0b0801000e040e0e080f030d0b0e0a07070b060908080c0b0a0f0d -> 5f2e0b98bed600b9c100a0bf58c192832e433a82dbe16d76d66814550bb40425
  ++++++++  NONCE: 2
  ++++++++  BALANCE: 10
  ++++++++  IS_CONTRACT: True
  ++++++++++STORAGE BRANCH | -> 285c62def2102bbcde16aa40ec5cb458bbca3ebec845358f26e2fbc45e863a45
  ++++++ACCOUNT  0e0c070c0303030d030d000c030e0e0f080c060109090a0400020805060500090e0c070c0303030d030d000c030e0e0f0c020200080d000307010d0b000e -> 94921a37bfe58a80fa0e1d4642f9606ded43662ed78eb6f88dd4ca9ae3451f31
  ++++++  NONCE: 2
  ++++++  BALANCE: 2
  ++++++  IS_CONTRACT: True
  ++++++++STORAGE BRANCH | -> b86873d99b6757b88746ab3117ff2e7065151d7ef34b04151beb0b6eed09a078
  ++++ACCOUNT  01040c0e080f0b070800070e080a060d030a0a0e02000a000404070e0804080c01040c0e080f0b070800070e080a060d060b0a0d0a0e0d06060a020f080107 -> 4fa60d17033accd44ad86ea4eb85b9c60bc8002b1b83ec043d02ccd22bd07ac9
  ++++  NONCE: 2
  ++++  BALANCE: 5
  ++++  IS_CONTRACT: True
  ++++++STORAGE BRANCH | -> 0652ef331dd9c6bd7d8715f1c065e262b2cbe3df91780e84e8344ba288693708
  ++++ACCOUNT  0302020f03030904060a030c0500030c0901060c080f0c02090706080a0504070302020f03030904060a030c0500030c0f06060c0f070e050a020602000102 -> 28a1e0126715276a67f7e654da7fc615f351f8e321fdde4aff468b7edebbcd36
  ++++  NONCE: 2
  ++++  BALANCE: 0
  ++++  IS_CONTRACT: True
  ++04 BRANCH | -> 5c9f3c5b1d7a2267ff51d51657fdeba7663000aa8da5f19d7f41df2840c2d938
  ++++++ACCOUNT  070a000f0a090f0e010207010b0e0d0e010a0b0e0905080e0f0d040a030e0406070a000f0a090f0e010207010b0e0d0e060a0c0502070e0f0f0e0f0d0b02 -> 3527d270b7d99cd0cd347e9d26f46f31a64dcd98f15bd6146b23538f1e50d30f
  ++++++  NONCE: 2
  ++++++  BALANCE: 6
  ++++++  IS_CONTRACT: True
  ++++++++STORAGE BRANCH | -> 2b75180714f508808fbd5f737c5d2ed6581910b9d8d93cb99aed97c9f294734c
  ++++++ACCOUNT  040105030d0502070e05020805000a0a040106050a04070800060d08070c0708040105030d0502070e05020805000a0a0506000d0d08060103080c0e0c0c -> 36e7b66a658c9dbc51c553dafb93eae6588c81768cbe6fd3248084b440bced90
  ++++++  NONCE: 2
  ++++++  BALANCE: 15
  ++++++  IS_CONTRACT: True
  ++++ACCOUNT  0605060e0f0107050b04020b030600040801050403020f0b060a0b0c0b0708050605060e0f0107050b04020b030600040f03050e020a020b04070b090e0104 -> 7800d200a1ed34bc0aa46b9e3b75d635c1badaa93ef5c0c300a12099bf2332ee
  ++++  NONCE: 2
  ++++  BALANCE: 12
  ++++  IS_CONTRACT: True
  ++++++STORAGE BRANCH | -> 0652ef331dd9c6bd7d8715f1c065e262b2cbe3df91780e84e8344ba288693708
  ++++ACCOUNT  090a000c0108050a0805010e0e00060e0f0002030f0b03080f0e0907030c010a090a000c0108050a0805010e0e00060e07040d0901040901040409080c000f -> d904f810986b83f88a47621be3148d4040d5d2d671f84008b7b47261f583ca26
  ++++  NONCE: 2
  ++++  BALANCE: 8
  ++++  IS_CONTRACT: True
  ++09 BRANCH | -> 5e7ef7cf57be2b75b9805e423af6d24f33291a60aebb57d584676bebc46efaf1
  ++++++ACCOUNT  0a060f0c02090a04040405060b03060203020603080a070004020403010c090c0a060f0c02090a04040405060b03060207090008050f0a0c010506000e09 -> 25aca44d174d873e5760387154c79de20122683c9f5aa3a3c36c427dddcda6b9
  ++++++  NONCE: 2
  ++++++  BALANCE: 1
  ++++++  IS_CONTRACT: True
  ++++++ACCOUNT  070a0d0f0f0f0509060b0c00070506050c0b08030e050f020a0506090b000c0a070a0d0f0f0f0509060b0c00070506050e0a04010e070609050c06070207 -> 592f5444319d00189f3a2ce8f8cf2f49a32da233178c1ab833ab5200f8be7c7f
  ++++++  NONCE: 2
  ++++++  BALANCE: 7
  ++++++  IS_CONTRACT: True
  ++++ACCOUNT  0b0402050d090a0606020a0f0a0f00030a0d00050e090b050e0a07030208000e0b0402050d090a0606020a0f0a0f000301060d0b000409030d07010207070c -> 8cfcebd6e200eeaa86610c0a1917eeece0536103837a281832bf2aa05d5a8f1e
  ++++  NONCE: 2
  ++++  BALANCE: 13
  ++++  IS_CONTRACT: True
  ++++++STORAGE BRANCH | -> 2b75180714f508808fbd5f737c5d2ed6581910b9d8d93cb99aed97c9f294734c
  ++++ACCOUNT  05030d0d09090e0b030d0106000e0f06010e030b08010c09050d0d060d080a0a05030d0d09090e0b030d0106000e0f06060b0506030e02060e0e000f0e0b03 -> bb03c4b45dfe2003967eff6acb8319f0e3fdab5f11ae24b8bc079bc34d509897
  ++++  NONCE: 2
  ++++  BALANCE: 11
  ++++  IS_CONTRACT: True
  ++++++STORAGE BRANCH | -> 80d9db0b7a1ab84066a618e4c0ba8795790390d6fdcfca44f70661a0d180c894
  ++0c BRANCH | -> 474ac0f123ce3977b8dc6b1ca6b3994e004e85408c173f6ab30456c0dcff8079
  ++++++ACCOUNT  050e00010a010b0f060705030f080102070e0b030000090a010e0e090c0a0109050e00010a010b0f060705030f08010202040d0e07010e0b04070f050700 -> e867a581aada9152b61ad18420b7ba70961b20e4d7112b9c480b831bdb0cc1cf
  ++++++  NONCE: 2
  ++++++  BALANCE: 9
  ++++++  IS_CONTRACT: True
  ++++++++STORAGE BRANCH | -> b86873d99b6757b88746ab3117ff2e7065151d7ef34b04151beb0b6eed09a078
  ++++++ACCOUNT  0e0f09070c0b0908000c070f02080109040a08030409020507010500040b04050e0f09070c0b0908000c070f020801090c0806040e02090904010e0b0603 -> 21ab12ae8907dcd25a1d37dca57b657b768cf8e612ceeb12187e64cf2ba308b8
  ++++++  NONCE: 2
  ++++++  BALANCE: 14
  ++++++  IS_CONTRACT: True
  ++0e EXTENSION 0e0e0e0e0e0e0e0e0e0e0e0e0e0e0e0e0e0e0e0e0e0e0e0e0e0e0e0e0e0e0e0e0e0e0e0e0e0e -> 229d639cfe4fcc90e32f9567f8afa33ae1df42f41a1d99b2ed8f5dd37bee6b24
  ++++BRANCH | -> ecd82138eb96b7cd941b323cba998c5a8b0a003f30e91a96618e68bb815f5b67
  ++++++0a EXTENSION 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0b -> 56a05c3cb0385ddc9f2e087d75285fd16eca7a36bc38522bee810bea62802bcb
  ++++++++BRANCH | -> e2c6273d45a4bf852c7a43adbfc57b3e8c37ca42fece8569a8b7f5584eede578
  ++++++++++++ACCOUNT  00000000000000 -> 7756315afaaca699f99f5672ea479ece083f1d19d82ca37baadba88464472c9c
  ++++++++++++  NONCE: 0
  ++++++++++++  BALANCE: 3
  ++++++++++++  IS_CONTRACT: True
  ++++++++++++++STORAGE BRANCH | -> dc32c7ad75fcf89fee15fbf13779382f4238809602674b977a336252b9f3ad9f
  ++++++++++++++++STORAGE 08 EXTENSION 0a -> 56baa6cb1f9056edfa702d2f69996b55dd48011bb4909f753e190132122b4008
  ++++++++++++++++++STORAGE BRANCH | -> 947e9644b83d8999a039871d2f2e20d9b62df25dc2c7d667e5b279425d69fdd2
  ++++++++++++ACCOUNT  01010101010101 -> 5dd5f10b62049417ad51f3b50a9331a39467f0f1cba80897687f1a65a3ee3de7
  ++++++++++++  NONCE: 0
  ++++++++++++  BALANCE: 4
  ++++++++++++  IS_CONTRACT: True
  ++++++++++++++STORAGE BRANCH | -> dc32c7ad75fcf89fee15fbf13779382f4238809602674b977a336252b9f3ad9f
  ++++++++++++++++STORAGE 08 EXTENSION 0a -> 56baa6cb1f9056edfa702d2f69996b55dd48011bb4909f753e190132122b4008
  ++++++++++++++++++STORAGE BRANCH | -> 947e9644b83d8999a039871d2f2e20d9b62df25dc2c7d667e5b279425d69fdd2
  ++++++0e EXTENSION 0e0e0e0e0e0e0e0e0e0e0e0e0e0e0e0b -> 8a89f134eb55d36f11161c6d0aed926bbac6cf8503e0ac005f8a1304088cd4fa
  ++++++++BRANCH | -> dedc5359baaf4f1350d7738b0d5eeaed8aa6f7ff2a8905ac2ee6cae3f4d67c4c
  ++++++++++++ACCOUNT  00000000000000 -> a7262ccd894b95754832b9e5edd80a6e609cc9edaaeb2f2ad7c58f85b03a9ef8
  ++++++++++++  NONCE: 0
  ++++++++++++  BALANCE: 1
  ++++++++++++  IS_CONTRACT: True
  ++++++++++++++STORAGE BRANCH | -> dc32c7ad75fcf89fee15fbf13779382f4238809602674b977a336252b9f3ad9f
  ++++++++++++++++STORAGE 08 EXTENSION 0a -> 56baa6cb1f9056edfa702d2f69996b55dd48011bb4909f753e190132122b4008
  ++++++++++++++++++STORAGE BRANCH | -> 947e9644b83d8999a039871d2f2e20d9b62df25dc2c7d667e5b279425d69fdd2
  ++++++++++++ACCOUNT  01010101010101 -> 9bd56070fa589ea8392a489d23a58f01e83336c68c994e4e57829e4f2fc77008
  ++++++++++++  NONCE: 0
  ++++++++++++  BALANCE: 2
  ++++++++++++  IS_CONTRACT: True
  ++++++++++++++STORAGE BRANCH | -> dc32c7ad75fcf89fee15fbf13779382f4238809602674b977a336252b9f3ad9f
  ++++++++++++++++STORAGE 08 EXTENSION 0a -> 56baa6cb1f9056edfa702d2f69996b55dd48011bb4909f753e190132122b4008
  ++++++++++++++++++STORAGE BRANCH | -> 947e9644b83d8999a039871d2f2e20d9b62df25dc2c7d667e5b279425d69fdd2
  
  MISSING 0x5a717ee8d7ffe2e336590cfe9486084772c3fa0b503271507d357ad0086986e9
  
  Assert.That(local, Is.EqualTo(remote))
    Expected string length 8347 but was 75. Strings differ at index 0.
    Expected: "STATE TREE\nBRANCH | -> 5a717ee8d7ffe2e336590cfe9486084772c3fa..."
    But was:  "MISSING 0x5a717ee8d7ffe2e336590cfe9486084772c3fa0b503271507d3..."
    -----------^
  
    at Nethermind.Synchronization.Test.FastSync.StateSyncFeedTestsBase.DbContext.CompareTrees(String stage, Boolean skipLogs) in /_/src/Nethermind/Nethermind.Synchronization.Test/FastSync/StateSyncFeedTestsBase.cs:259
    at Nethermind.Synchronization.Test.FastSync.StateSyncFeedTests.Big_test(ValueTuple`2 testCase) in /_/src/Nethermind/Nethermind.Synchronization.Test/FastSync/StateSyncFeedTests.cs:108
    at Nethermind.Synchronization.Test.FastSync.StateSyncFeedTests.Big_test(ValueTuple`2 testCase) in /_/src/Nethermind/Nethermind.Synchronization.Test/FastSync/StateSyncFeedTests.cs:109
    at NUnit.Framework.Internal.TaskAwaitAdapter.GenericAdapter`1.BlockUntilCompleted()
    at NUnit.Framework.Internal.MessagePumpStrategy.NoMessagePumpStrategy.WaitForCompletion(AwaitAdapter awaiter)
    at NUnit.Framework.Internal.AsyncToSyncAdapter.Await[TResult](TestExecutionContext context, Func`1 invoke)
    at NUnit.Framework.Internal.AsyncToSyncAdapter.Await(TestExecutionContext context, Func`1 invoke)
    at NUnit.Framework.Internal.Commands.TestMethodCommand.RunTestMethod(TestExecutionContext context)
    at NUnit.Framework.Internal.Commands.TestMethodCommand.Execute(TestExecutionContext context)
    at NUnit.Framework.Internal.Commands.BeforeAndAfterTestCommand.<>c__DisplayClass1_0.<Execute>b__0()
    at NUnit.Framework.Internal.Commands.DelegatingTestCommand.RunTestMethodInThreadAbortSafeZone(TestExecutionContext context, Action action)
    
    1)    at Nethermind.Synchronization.Test.FastSync.StateSyncFeedTestsBase.DbContext.CompareTrees(String stage, Boolean skipLogs) in /_/src/Nethermind/Nethermind.Synchronization.Test/FastSync/StateSyncFeedTestsBase.cs:line 259
    at Nethermind.Synchronization.Test.FastSync.StateSyncFeedTests.Big_test(ValueTuple`2 testCase) in /_/src/Nethermind/Nethermind.Synchronization.Test/FastSync/StateSyncFeedTests.cs:108
    at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)

benaadams · 2025-10-15T12:09:19Z

+                .AddScoped<IBlockchainProcessor>(ctx => ctx.Resolve<BlockchainProcessor>())
+                .AddScoped<IBlockProcessingQueue>(ctx => ctx.Resolve<BlockchainProcessor>())


Are these both same?

Copilot

Pull Request Overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (1)

src/Nethermind/Nethermind.Consensus/Processing/IMainProcessingContext.cs:1

Adding BlockProcessingQueue to a public interface is a breaking change for any external implementers. Consider exposing this via an extension method (e.g., GetBlockProcessingQueue(this IMainProcessingContext) => (IBlockProcessingQueue)context.BlockchainProcessor) or clearly document the breaking change and ensure all implementers are updated.

// SPDX-FileCopyrightText: 2025 Demerzel Solutions Limited

LukaszRozmej added 4 commits October 15, 2025 11:03

fix Should_return_correct_block_reward

6ac8604

fix getPayloadV1_waits_for_block_production

97dfd57

Fix E2ESyncTests by waiting for block processing in setup

b1d2c29

whitespace

fc1bc20

LukaszRozmej requested review from Marchhill, asdacap, benaadams, Copilot and rubo and removed request for asdacap and Copilot October 15, 2025 09:48

Copilot AI reviewed Oct 15, 2025

View reviewed changes

Comment thread src/Nethermind/Nethermind.Synchronization.Test/E2ESyncTests.cs

benaadams approved these changes Oct 15, 2025

View reviewed changes

LukaszRozmej added 5 commits October 15, 2025 12:12

more fixes to E2ESyncTests

201f6d1

Another fix for Should_return_correct_block_reward

c4ee94f

fix Test_task_that_is_scheduled_during_block_processing_will_continue…

69dc196

…_after

more sensible timings

389d7d6

try fix Maintain_correct_pointers_for_beacon_sync_in_archive_sync and…

56ce1b2

… Big_test((storage_hash_and_code_hash_same, System.Action`3[Nethermind.State.StateTree,Nethermind.Trie.Pruning.ITrieStore,Nethermind.Db.IDb]))

LukaszRozmej added 2 commits October 15, 2025 13:26

Try wait for BlockProcessingQueue in Big_test

a90e80b

Revert Big_test changes I give up

4611ca7

LukaszRozmej marked this pull request as ready for review October 15, 2025 11:42

LukaszRozmej requested a review from benaadams October 15, 2025 11:42

benaadams approved these changes Oct 15, 2025

View reviewed changes

benaadams requested a review from Copilot October 15, 2025 12:11

Copilot AI reviewed Oct 15, 2025

View reviewed changes

LukaszRozmej added 2 commits October 15, 2025 15:40

fix

48d3f83

Try fix ImportAsArchiveSync_WillPaceSuggestBlock

755cf38

Try fix ImportAsArchiveSync_WillPaceSuggestBlock further

53dca54

LukaszRozmej merged commit 999133a into master Oct 15, 2025
80 checks passed

LukaszRozmej deleted the fix/more-flaky-tests branch October 15, 2025 14:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix/more flaky tests#9474

Fix/more flaky tests#9474
LukaszRozmej merged 14 commits into
masterfrom
fix/more-flaky-tests

LukaszRozmej commented Oct 15, 2025

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

LukaszRozmej commented Oct 15, 2025 •

edited

Loading

Uh oh!

benaadams Oct 15, 2025

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		.AddScoped<IBlockchainProcessor>(ctx => ctx.Resolve<BlockchainProcessor>())
		.AddScoped<IBlockProcessingQueue>(ctx => ctx.Resolve<BlockchainProcessor>())

Conversation

LukaszRozmej commented Oct 15, 2025

Types of changes

What types of changes does your code introduce?

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

LukaszRozmej commented Oct 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

benaadams Oct 15, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

LukaszRozmej commented Oct 15, 2025 •

edited

Loading