Skip to content

Block lookup tests with SyncTester#5530

Closed
jimmygchen wants to merge 6 commits intosigp:unstablefrom
jimmygchen:bn-p2p-tests
Closed

Block lookup tests with SyncTester#5530
jimmygchen wants to merge 6 commits intosigp:unstablefrom
jimmygchen:bn-p2p-tests

Conversation

@jimmygchen
Copy link
Member

@jimmygchen jimmygchen commented Apr 5, 2024

Issue Addressed

This PR adds a new test utility for Lighthouse sync/lookup and a regression test that uses it.

Currently we're lacking integration tests around sync components and we're finding it difficult to reproduce production issues (especially ones that are related to race conditions). Hopefully this test utility would help increase coverage in this area and make reproducing sync/lookup related issues easier.

Proposed Changes

Additional Info

The SyncTester enables sending messages to SyncManager, and then making assertions on outgoing messages from it.

It covers the following:

  1. Sending SyncMessage to SyncManager to trigger RangeSync, BackFillSync and BlockLookups behaviours.
  2. Making assertions on WorkEvents received from sync
  3. Making assertion on NetworkMessage received from sync (e.g. outgoing RPC requests).

With these tests we can trigger behaviour and verify outcome, i.e. testing of all the arrows in the below diagram:

                     +-----------------+
                     | BeaconProcessor |
                     +---------+-------+
                            ^  |
                            |  |
                  WorkEvent |  | SyncMsg
                            |  | (Result)
                            |  v
+--------+            +-----+-----------+             +----------------+
| Router +----------->|  SyncManager    +------------>| NetworkService |
+--------+  SyncMsg   +-----------------+ NetworkMsg  +----------------+
          (RPC resp)  |  - RangeSync    |  (RPC req)
                      +-----------------+
                      |  - BackFillSync |
                      +-----------------+
                      |  - BlockLookups |
                      +-----------------+

@jimmygchen jimmygchen requested a review from realbigsean April 5, 2024 14:27
@jimmygchen jimmygchen added the work-in-progress PR is a work-in-progress label Apr 5, 2024
@dapplion
Copy link
Collaborator

dapplion commented Apr 6, 2024

This is a great initiative! Writing the tests as a sequence of events allows us to change the internals safely without breaking tests. I would love to change some parts of sync, and this would help a lot.

if you can inject any event you want into sync, why do you need to have traits for RangeSync, BackFillSync and BlockLookups? At least IMO, the whole point of this re-factor is to be able to change the APIs without breaking tests. Now, we should not have to couple testing to the specific API of its internal components.

See this example of a test re-written in event-only form: 624dc8a

    // The peer provides the correct block, should not be penalized. Now the block should be sent
    // for processing.
    rig.send_sync_message(SyncMessage::RpcBlock {
        request_id: SyncRequestId::SingleBlock { id },
        peer_id,
        beacon_block: Some(block.into()),
        seen_timestamp: D,
    });
    rig.expect_empty_network();
    rig.expect_block_process(response_type);

    // The request should still be active.
    assert_eq!(rig.active_lookup_count(), 1);

    // Send the stream termination. Peer should have not been penalized, and the request removed
    // after processing.
    rig.send_sync_message(SyncMessage::RpcBlock {
        request_id: SyncRequestId::SingleBlock { id },
        peer_id,
        beacon_block: None,
        seen_timestamp: D,
    });
    rig.send_sync_message(SyncMessage::BlockComponentProcessed {
        process_type: BlockProcessType::SingleBlock { id: id.id },
        result: BlockProcessingResult::Ok(AvailabilityProcessingStatus::Imported(block_root)),
    });
    rig.expect_empty_network();
    assert_eq!(rig.active_lookup_count(), 0);

@jimmygchen
Copy link
Member Author

Nice, yeah I agree and prefer the simplicity too! I've removed the traits.

…e extra beacon chain harness instance created in tests.
@jimmygchen jimmygchen added ready-for-review The code is ready for review and removed work-in-progress PR is a work-in-progress labels Apr 8, 2024
@jimmygchen jimmygchen marked this pull request as ready for review April 8, 2024 00:31
@jimmygchen
Copy link
Member Author

@dapplion @realbigsean
This PR is ready for review! SyncTester still doesn't have everything we have in Deneb tester, but I think the structure is working and we can iterate the SyncTester over time as we add more tests.

@jimmygchen jimmygchen requested a review from dapplion April 8, 2024 00:33
@jimmygchen jimmygchen added the test improvement Improve tests label Apr 8, 2024
@jmcph4
Copy link
Contributor

jmcph4 commented Apr 8, 2024

@jimmygchen I really love this diagram and I think it should live in the repo somewhere as a Mermaid diagram. Thoughts?

@jimmygchen
Copy link
Member Author

jimmygchen commented Apr 8, 2024

Thanks @jmcph4! ☺️ This is in the repo as comment above SyncTester:

/// +-----------------+
// | BeaconProcessor |
// +---------+-------+
// ^ |
// | |
// WorkEvent | | SyncMsg
// | | (Result)
// | v
// +--------+ +-----+-----------+ +----------------+
// | Router +----------->| SyncManager +------------>| NetworkService |
// +--------+ SyncMsg +-----------------+ NetworkMsg +----------------+
// (RPC resp) | - RangeSync | (RPC req)
// +-----------------+
// | - BackFillSync |
// +-----------------+
// | - BlockLookups |
// +-----------------+
pub struct SyncTester {

I initially wanted to do Mermaid diagram too but struggled a bit as the initial diagram was a bit more complicated, so I ended up doing it in ascii. (I'm also a noob with mermaid 😅)

@jimmygchen
Copy link
Member Author

Superseded by #5534

@jimmygchen jimmygchen closed this Apr 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-review The code is ready for review test improvement Improve tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants