Skip to content

Put parent_header to PreloadUnverified, and keep small block in memory#4461

Closed
eval-exec wants to merge 360 commits intonervosnetwork:ckb-async-downloadfrom
eval-exec:exec/ckb-async-download/reduce-load-block-timecost
Closed

Put parent_header to PreloadUnverified, and keep small block in memory#4461
eval-exec wants to merge 360 commits intonervosnetwork:ckb-async-downloadfrom
eval-exec:exec/ckb-async-download/reduce-load-block-timecost

Conversation

@eval-exec
Copy link
Collaborator

@eval-exec eval-exec commented May 15, 2024 •
edited
Loading

LonelyBlockHash

What problem does this PR solve?

  1. keep small block in LonelyBlockHash
    Change
pub struct LonelyBlockHash {
    /// block
    pub block_number_and_hash: BlockNumberAndHash,

to

pub struct LonelyBlockHash {
    /// block
    pub block_number_and_hash: Either<BlockNumberAndHash, Arc<BlockView>>,
  1. put parent_header to PreloadUnverified

What's Changed:

Related changes

  • Keep small block in PreloadUnverified's channel, then we can reduce many get operation from RocksDB.
  • put parent_header to PreloadUnverified, then we may reduce one get operation from RocksDB.

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code ci-runs-only: [ quick_checks,linters ]

Side effects

  • None

Release note 

None: Exclude this PR from the release note.

eval-exec added 30 commits May 13, 2024 10:38
@eval-exec
Remove useless import orphan_block_pool in ckb-sync
f12644f
@eval-exec
Fix blocking process block usage in benches/benches/benchmarks/always…
1f83b3e 
…_success.rs
@eval-exec
Fix blocking process block usage in benches/benches/benchmarks/overal…
6123d60 
…l.rs
@eval-exec
Fix blocking process block usage in benches/benches/benchmarks/resolv…
9d4ae55 
…e.rs
@eval-exec
Fix blocking process block usage in benches/benches/benchmarks/secp_2…
17a571e 
…in2out.rs
@eval-exec
let ckb-import and ckb-replay construct ChainService with SharedPackage
d3420e3
@eval-exec
Fix Unit test: ckb-light-client-protocol-server::tests/utils/chain.rs
3896d3b
@eval-exec
Fix Unit test: ckb-verification::contextual/src/tests/uncle_verifier.rs
0e4e859
@eval-exec
Fix Unit test: ckb-verification::contextual/src/tests/contextual_bloc…
684654c 
…k_verifier.rs
@eval-exec
Fix Unit test: ckb-sync::src/tests/util.rs
50eae6e
@eval-exec
Fix blocking process block usage in sync/src/relayer/tests/helper.rs
89da541
@eval-exec
Fix Unit test of blocking process block usage in ckb-sync
6b9cb22
@eval-exec
Use blocking process_block method for benches/benches/benchmarks/secp…
e6d45af 
…_2in2out.rs
@eval-exec
Benches: Construct ChainService with SharedPackage provided verify_fa…
bae4525 
…iled_block_rx
@eval-exec
Modify BlockStatus by SyncShared.Shared
7b2297b
@eval-exec
Unit Test: Use SharedPackage to construct ChainService
e32648a
@eval-exec
Unit Test: Use SharedPackage to construct ChainService in ckb-chain::…
8019a93 
…find_fork.rs
@eval-exec
Unit test: Modify ChainService initialization to include `pack.take…
c5fa529 
…_verify_failed_block_tx()` parameter
@eval-exec
Unit test: Modify the new function in ChainService to include ver…
4fb9013 
…ify_failed_block_tx parameter
@eval-exec
Unit test: Initialization of ChainService need pack.take_verify_faile…
10eba3a 
…d_block_tx
@eval-exec
Add more minor ticks to sync progress chart
b2d51c5
@eval-exec
Add draft mermaid sequence diagram for develop branch
bd7bad0
@eval-exec
Add draft mermaid sequence diagram for asynchronous block download
1b4256d
@eval-exec
Add unverified block info to sync_state rpc
0c05d58
@eval-exec
Upgrade Synchronizer disconnect log level from debug to info
cd6d91f
@eval-exec
Add tip_hash and tip_number to sync_state rpc
d1d5fec
@eval-exec
Set receive exit signal log from debug to info
2008b2d
@eval-exec
Let HeaderMap stats feature use info log
5d425a2
@eval-exec
Activate HeaderMap stats profiling feature in make profiling
cac179c 
Signed-off-by: Eval EXEC <execvy@gmail.com>
@eval-exec
Return removed inflight blocks count when disconnect
6a8bb07 
Signed-off-by: Eval EXEC <execvy@gmail.com>
eval-exec and others added 24 commits May 13, 2024 10:39
@eval-exec
Fix sync and relayer unit test by verify_callback
a0c6ff8
@eval-exec
Remove VerifyFailedInfo, remove verify_failed_block channel
86deeb3
@eval-exec
Remove broadcast_compact_block channel
565800b
@eval-exec
Add debug log for block_fetcher->get_ancestor
156b86e
@quake @eval-exec
chore: remove channel send error callback
00fae3d
@quake @eval-exec
chore: remove UnverifiedBlock
e066c68
@eval-exec
Find and verify unverified blocks on ckb startup
75a8007
@eval-exec
Fix cargo clippy warnings
e33733f 
Signed-off-by: Eval EXEC <execvy@gmail.com>
@eval-exec
Use Release to store, Acquire to read for atomic: `is_verifyi…
63f767a 
…ng_unverified_blocks_on_startup`

Signed-off-by: Eval EXEC <execvy@gmail.com>
@eval-exec
Do not accept descendants if parent is invalid
b098b1f 
Signed-off-by: Eval EXEC <execvy@gmail.com>
@eval-exec
Update rpc docs, fix CI test
f1ad952 
Signed-off-by: Eval EXEC <execvy@gmail.com>
@eval-exec
Release ckb-async-download rc0
8f18471
@eval-exec
Do not re-store_unverified_block if it is still verifying by ConsumeU…
fab3055 
…nverified
@eval-exec
Apply cargo clippy suggestions
6e7bbcf
@eval-exec
Rebased develop, regenerate rpc docs, remove orphan pool total_size l…
e86b6c9 
…imitation

Signed-off-by: Eval EXEC <execvy@gmail.com>
@eval-exec
Prestore block to db before orphan pool, use fill unverified thread t…
c9617c4 
…o pre load unverified block

Signed-off-by: Eval EXEC <execvy@gmail.com>
@eval-exec
Shared::get_block_status always read BlockExt from snapshot
23386c1 
Signed-off-by: Eval EXEC <execvy@gmail.com>
@eval-exec
Release ckb-async-download rc1
47e2ff3
@eval-exec
Apply code review suggestions
0f8c8c8 
Signed-off-by: Eval EXEC <execvy@gmail.com>
@eval-exec
Remove block_status_map if block is is_internal_db_error
e6ca07b
@eval-exec
Delete unverified block if verify failed
f7d6b00
@eval-exec
Add SyncInvalid integration test
a0532cb
@eval-exec
Fix cargo clippy for integration test
95c26d9 
Signed-off-by: Eval EXEC <execvy@gmail.com>
@eval-exec
Temporary put parent header to PreloadUnverified, put small BlockView to
ff3cab9 
LonelyBlockHash

Signed-off-by: Eval EXEC <execvy@gmail.com>
@eval-exec eval-exec requested a review from a team as a code owner May 15, 2024 00:47
@eval-exec eval-exec requested review from zhangsoledad and removed request for a team May 15, 2024 00:47
@eval-exec eval-exec marked this pull request as draft May 15, 2024 00:50
@eval-exec eval-exec force-pushed the ckb-async-download branch from 8f38e14 to 8bcdea7 Compare May 22, 2024 01:28
@eval-exec eval-exec closed this May 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@zhangsoledad zhangsoledad Awaiting requested review from zhangsoledad zhangsoledad is a code owner automatically assigned from nervosnetwork/ckb-dev

Copilot code review Copilot Awaiting requested review from Copilot Copilot will automatically review once the pull request is marked ready for review

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@eval-exec @contrun @quake