Skip to content

feat: more compact exported snapshot#703

Merged
tac0turtle merged 6 commits intocosmos:masterfrom
yihuang:compress-snapshot
Apr 6, 2023
Merged

feat: more compact exported snapshot#703
tac0turtle merged 6 commits intocosmos:masterfrom
yihuang:compress-snapshot

Conversation

@yihuang
Copy link

@yihuang yihuang commented Mar 13, 2023 •
edited
Loading

Closes: #690 #688

  • Purely additive changes for iavl library, add new APIs to support a more compact snapshot format.
  • Still use the existing ExportNode type as the exchange format with sdk, sdk can still use the protobuf encoding as before, which skips empty fields, which works well with the new optimizations.
  • Leave the issue of format negotiation and migration to sdk.

Where to put the code

With the approach taken in this PR, it's totally fine to move the logic to the sdk side as well, if it makes more sense.

Test Result

Dumped from cronos mainnet data, evm module, version 2000000:

Snapshot Type Size
Normal snapshot 469.4M
Compact snapshot 402M
memiavl snapshot 2.3G
  • The new format further reduce the zlib compressed size by 14%, not as big as expected, I guess mainly because zlib compression already done a good job.
  • Both normal/compact snapshots are dumped with sdk's snapshot manager, with and without the compact format introduced in this PR, sum the size of all the chunks.
  • memiavl snapshot is just put for reference, it's a different snapshot format that contains all the node hashes, with zero compression applied.

EDIT:

Just to confirm my suspicion, I recreate the snapshot with zlib compression disabled, it goes like this:

Snapshot Type Size
Normal snapshot 1.82G
Compact snapshot 856M

The compression rate is 54%.

@yihuang yihuang requested a review from a team March 13, 2023 08:21
This was referenced Mar 13, 2023
Closed
Closed
yihuang
yihuang commented Mar 13, 2023
View reviewed changes
@yihuang yihuang changed the title more compact exported snapshot feat: more compact exported snapshot Mar 13, 2023
@yihuang yihuang force-pushed the compress-snapshot branch from be44270 to 7c9c9d0 Compare March 13, 2023 08:47
tac0turtle
tac0turtle reviewed Mar 13, 2023
View reviewed changes
Copy link
Contributor

@tac0turtle tac0turtle left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lets leave this here for now, in store v2 this could be moved higher, into the sdk

cool-develope
cool-develope reviewed Mar 16, 2023
View reviewed changes
compress.go
n.Key = nil

// delta encode the version
maxVersion := maxInt64(e.versionStack[len(e.versionStack)-1], e.versionStack[len(e.versionStack)-2])
Copy link
Contributor

@cool-develope cool-develope Mar 16, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just curious about the efficiency of version delta encode?

Copy link
Author

@yihuang yihuang Mar 17, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#703 (comment)

cool-develope
cool-develope reviewed Mar 16, 2023
View reviewed changes
@yihuang
Copy link
Author

yihuang commented Mar 17, 2023 •
edited
Loading

just curious about the efficiency of version delta encode?

Just did some more detailed comparisons, in the "without zlib compression" table, version delta only contributes 2% more compression rate, but those contributions are carried over to the with-zlib scenario, where it has more significant impact, I guess it's a part that is not handled well by zlib compression:

With Zlib Compression(level 7)

Size (bytes) Compression Rate Time
Original 492170475 - 54s
Skip branch key 455839686 7% 52s
Skip branch key + key delta 444661628 10% 50s
Skip branch key + key delta + version delta 421542633 14% 43s

Without Zlib Compression

Size (bytes) Compression Rate Time
Original 1955535429 - 8.3s
Skip branch key 1308448614 33% 7.8s
Skip branch key + key delta 936722517 52% 8.8s
Skip branch key + key delta + version delta 897597115 54% 8.4s
  • Compression Rate is computed as: (original - new) / original.
  • Time is the elapsed wall time of the whole cmd recorded by time in shell, the cmd reads an existing memiavl snapshot and write the state sync snapshot.
  • It looks strange that in some cases the run time decrease while the compression rate increase, but it indeed reproducible with a few repeated runs here.

Zlib Level 3

I did another round with zlib level 3 compression, it seems provide better trade-off in general, lose some marginal compression rate but being much faster.
If you compare new compact format plus zlib level 3 to the original format with zlib level 7, it reduce 11% size while being much faster.
But of course the bottleneck of snapshot exporting right now is probably on the iavl tree traversal in db, so the compress speed improvements probably matters less in real world.

Size (bytes) Compression Rate Time
Original 508743095 - 23s
Skip branch key 474433915 6.7% 22s
Skip branch key + key delta 461333965 9.3% 22s
Skip branch key + key delta + version delta 436761440 14% 21s

@yihuang yihuang requested a review from cool-develope March 17, 2023 01:57
@cool-develope
Copy link
Contributor

cool-develope commented Mar 17, 2023

txs for detail report

@tac0turtle tac0turtle merged commit b544dc0 into cosmos:master Apr 6, 2023
@yihuang yihuang deleted the compress-snapshot branch April 6, 2023 14:23
@yihuang yihuang mentioned this pull request Apr 12, 2023
Closed
@tac0turtle
Copy link
Contributor

tac0turtle commented Apr 13, 2023

@Mergifyio backport release/v1.x.x

@mergify
Copy link
Contributor

mergify bot commented Apr 13, 2023 •
edited
Loading

backport release/v1.x.x

✅ Backports have been created

Details

mergify bot pushed a commit that referenced this pull request Apr 13, 2023
@mergify
feat: more compact exported snapshot (#703)
b435b63 
Co-authored-by: cool-developer <51834436+cool-develope@users.noreply.github.com>
Co-authored-by: Marko <marbar3778@yahoo.com>
(cherry picked from commit b544dc0)

# Conflicts:
#	CHANGELOG.md
@mergify mergify bot mentioned this pull request Apr 13, 2023
Merged
tac0turtle added a commit that referenced this pull request Apr 13, 2023
@mergify @tac0turtle
feat: more compact exported snapshot (backport #703) (#743)
2e91f88 
Co-authored-by: yihuang <huang@crypto.com>
Co-authored-by: Marko <marbar3778@yahoo.com>
@yihuang yihuang mentioned this pull request Nov 15, 2023
Closed
@coderabbitai coderabbitai bot mentioned this pull request May 13, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

2 more reviewers

@tac0turtle tac0turtle tac0turtle left review comments

@cool-develope cool-develope cool-develope approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

@kocubinski kocubinski

@cool-develope cool-develope

Labels

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

export: apply delta encoding for better compression rate

4 participants

@yihuang @cool-develope @tac0turtle @kocubinski