Skip to content

cmd/geth: add hbss to pbss convert tool#1882

Merged
fynnss merged 1 commit intobnb-chain:developfrom
fynnss:add_hash_trie_node_prune_tool
Oct 5, 2023
Merged

cmd/geth: add hbss to pbss convert tool#1882
fynnss merged 1 commit intobnb-chain:developfrom
fynnss:add_hash_trie_node_prune_tool

Conversation

@fynnss
Copy link
Copy Markdown
Contributor

@fynnss fynnss commented Sep 20, 2023 •
edited
Loading

Description

add hbss to pbss convert tool for generate bsc snapshot more quickly.

Rationale

If starts from the genesis block to make the bsc snaoshot, it will takes 1.5~2 months.

if using the convert tools and convert hbss trie node to pbss trie node, it will takes 2~3 days.

Example

./geth db hash2path --datadir ./node 1000 
./geth db prune-hash-trie  --datadir ./node
./geth db compact --datadir ./node

NOTICE: it will take a long time to iterate the entrie databse.

Changes

Notable changes:

  • add hash2path/prune-hash-trie tools for convert and prune the hbss trie node
  • add trieGet/trieDelete for pbss to debug and handle exceptions

@fynnss fynnss force-pushed the add_hash_trie_node_prune_tool branch 3 times, most recently from 2c99dfe to 82fe65e Compare September 20, 2023 09:28
@fynnss fynnss requested review from RenRick and joey0612 September 20, 2023 09:30
@fynnss fynnss force-pushed the add_hash_trie_node_prune_tool branch from d21a74a to ac1779a Compare September 21, 2023 08:50
@fynnss fynnss changed the title feat: add hash trie node prune tool cmd/geth: add hash trie node prune tool Sep 21, 2023
@fynnss fynnss self-assigned this Sep 21, 2023
@fynnss fynnss added the r4r ready for review label Sep 21, 2023
@fynnss fynnss removed their assignment Sep 21, 2023
@fynnss fynnss force-pushed the add_hash_trie_node_prune_tool branch 3 times, most recently from ac71133 to 85097dd Compare September 22, 2023 10:28
@fynnss fynnss force-pushed the add_hash_trie_node_prune_tool branch from 85097dd to f60e666 Compare September 25, 2023 03:34
@fynnss fynnss changed the title cmd/geth: add hash trie node prune tool cmd/geth: add hbss to pbss convert tool Sep 25, 2023
@fynnss fynnss force-pushed the add_hash_trie_node_prune_tool branch from f60e666 to cd8817c Compare September 25, 2023 03:42
joey0612
joey0612 reviewed Sep 25, 2023
View reviewed changes
cmd/geth/dbcmd.go Outdated
config := trie.HashDefaults
triedb := trie.NewDatabase(db, config)
triedb.Cap(0)
fmt.Println("triedb scheme: ", triedb.Scheme())
Copy link
Copy Markdown
Contributor

@joey0612 joey0612 Sep 25, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

log is better, consistent with other cmds.

Copy link
Copy Markdown
Contributor

@joey0612 joey0612 Sep 25, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and ditto to other fmt.Println

Copy link
Copy Markdown
Contributor Author

@fynnss fynnss Sep 26, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed.

cmd/geth/dbcmd.go
db := utils.MakeChainDatabase(ctx, stack, false, false)
defer db.Close()

scheme := ctx.String(utils.StateSchemeFlag.Name)
Copy link
Copy Markdown
Contributor

@joey0612 joey0612 Sep 25, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

change to ParseStateScheme?

Copy link
Copy Markdown
Contributor Author

@fynnss fynnss Sep 26, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This tool may be used in the scenarios that the database which both the HBSS and PBSS trienode exist, so flags are required

Copy link
Copy Markdown
Contributor

@joey0612 joey0612 Sep 27, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ParseStateScheme is uesed to handle utils.StateSchemeFlag.

@joey0612 joey0612 force-pushed the pbss_active branch 2 times, most recently from 4145f32 to e4b69f7 Compare September 26, 2023 07:47
@fynnss fynnss force-pushed the add_hash_trie_node_prune_tool branch from e928905 to b260cc9 Compare September 26, 2023 09:13
@fynnss fynnss changed the base branch from pbss_active to develop September 26, 2023 09:13
@fynnss fynnss force-pushed the add_hash_trie_node_prune_tool branch 3 times, most recently from 6d75830 to 5c6dd3e Compare September 26, 2023 10:57
NathanBSC
NathanBSC reviewed Sep 26, 2023
View reviewed changes
cmd/geth/dbcmd.go Outdated
}

func ancientInspect(ctx *cli.Context) error {
func inspect(ctx *cli.Context) error {
Copy link
Copy Markdown
Contributor

@NathanBSC NathanBSC Sep 26, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why change the function of inspect?

from code logic, new logic is totally for different usage

Copy link
Copy Markdown
Contributor Author

@fynnss fynnss Sep 26, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed. Made a mistake when rebase code.

@fynnss fynnss force-pushed the add_hash_trie_node_prune_tool branch from 5c6dd3e to e3a7214 Compare September 26, 2023 12:44
@fynnss fynnss force-pushed the add_hash_trie_node_prune_tool branch from e3a7214 to b8bad31 Compare September 26, 2023 12:45
@fynnss fynnss requested a review from joey0612 September 27, 2023 01:54
@fynnss fynnss requested a review from NathanBSC September 27, 2023 01:55
RenRick
RenRick approved these changes Sep 27, 2023
View reviewed changes
Copy link
Copy Markdown

@RenRick RenRick left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@joey0612
Copy link
Copy Markdown
Contributor

joey0612 commented Sep 27, 2023

LGTM

@NathanBSC
Copy link
Copy Markdown
Contributor

NathanBSC commented Sep 27, 2023

LGTM

@fynnss fynnss merged commit c26a303 into bnb-chain:develop Oct 5, 2023
@joey0612 joey0612 mentioned this pull request Oct 18, 2023
Merged
This was referenced Oct 18, 2023
Merged
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@joey0612 joey0612 joey0612 approved these changes

+2 more reviewers

@RenRick RenRick RenRick approved these changes

@NathanBSC NathanBSC NathanBSC approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

r4r ready for review

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@fynnss @joey0612 @NathanBSC @RenRick