feat(javascript): Implement Xlang map #1549
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Member
Author
|
@chaokunyang @LiangliangSui Hi，Can you help take a review？ |
chaokunyang
reviewed
Apr 21, 2024
chaokunyang
reviewed
Apr 21, 2024
2 tasks
pandalee99
pushed a commit
that referenced
this pull request
Jan 26, 2025
…p serialization (#2025) ## What does this PR do? This pr provides a new implementation for chunk based map serialization. ## Related issues #1571 #1549 #1722 Closes #925 ## Does this PR introduce any user-facing change? <!-- If any user-facing interface changes, please [open an issue](https://github.com/apache/fury/issues/new/choose) describing the need to do so and update the document if necessary. --> - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark Deserialization are much faster than no-chunk version, serialization are faster if map size are bigger Using the benchmark code in #1722 (comment): This PR has run faster, it gets up to **3x faster** : ```java Benchmark (size) (tracking) Mode Cnt Score Error Units HnBenchmark.testGeneralChunkWriteWithNull 64 true avgt 3 965.521 ± 1830.936 ns/op HnBenchmark.testGeneralChunkWriteWithNull 64 false avgt 3 1060.411 ± 3424.719 ns/op HnBenchmark.testGeneralChunkWriteWithNull 128 true avgt 3 2404.445 ± 8687.122 ns/op HnBenchmark.testGeneralChunkWriteWithNull 128 false avgt 3 1814.507 ± 1722.751 ns/op HnBenchmark.testGeneralChunkWriteWithNull 256 true avgt 3 3944.632 ± 2203.076 ns/op HnBenchmark.testGeneralChunkWriteWithNull 256 false avgt 3 3288.805 ± 867.047 ns/op HnBenchmark.testGeneralWriteWithNull 64 true avgt 3 1962.688 ± 2828.210 ns/op HnBenchmark.testGeneralWriteWithNull 64 false avgt 3 1490.634 ± 962.836 ns/op HnBenchmark.testGeneralWriteWithNull 128 true avgt 3 3659.806 ± 7227.436 ns/op HnBenchmark.testGeneralWriteWithNull 128 false avgt 3 4084.654 ± 7374.774 ns/op HnBenchmark.testGeneralWriteWithNull 256 true avgt 3 9596.658 ± 20767.262 ns/op HnBenchmark.testGeneralWriteWithNull 256 false avgt 3 6679.325 ± 5472.179 ns/op ``` With StringMap and IntMap benchmark: ```java Benchmark (enableChunkEncoding) (mapSize) Mode Cnt Score Error Units MapSerializationSuite.deserializeIntMap false 5 thrpt 3 3804604.842 ± 15328547.705 ops/s MapSerializationSuite.deserializeIntMap false 20 thrpt 3 1254687.969 ± 388949.724 ops/s MapSerializationSuite.deserializeIntMap false 50 thrpt 3 495176.849 ± 335702.097 ops/s MapSerializationSuite.deserializeIntMap false 100 thrpt 3 258875.012 ± 32886.176 ops/s MapSerializationSuite.deserializeIntMap false 200 thrpt 3 134137.015 ± 114908.454 ops/s MapSerializationSuite.deserializeIntMap true 5 thrpt 3 5997383.562 ± 4598913.048 ops/s MapSerializationSuite.deserializeIntMap true 20 thrpt 3 1797855.524 ± 3853406.173 ops/s MapSerializationSuite.deserializeIntMap true 50 thrpt 3 582412.110 ± 1047668.070 ops/s MapSerializationSuite.deserializeIntMap true 100 thrpt 3 389066.866 ± 151297.708 ops/s MapSerializationSuite.deserializeIntMap true 200 thrpt 3 188316.860 ± 35331.909 ops/s MapSerializationSuite.deserializeStringMap false 5 thrpt 3 2898963.533 ± 1930240.310 ops/s MapSerializationSuite.deserializeStringMap false 20 thrpt 3 872196.086 ± 871637.268 ops/s MapSerializationSuite.deserializeStringMap false 50 thrpt 3 308761.737 ± 58099.196 ops/s MapSerializationSuite.deserializeStringMap false 100 thrpt 3 157261.914 ± 397356.241 ops/s MapSerializationSuite.deserializeStringMap false 200 thrpt 3 86576.549 ± 102489.156 ops/s MapSerializationSuite.deserializeStringMap true 5 thrpt 3 3701089.567 ± 1529899.331 ops/s MapSerializationSuite.deserializeStringMap true 20 thrpt 3 1048550.399 ± 130102.760 ops/s MapSerializationSuite.deserializeStringMap true 50 thrpt 3 407559.246 ± 38205.273 ops/s MapSerializationSuite.deserializeStringMap true 100 thrpt 3 172109.437 ± 397927.346 ops/s MapSerializationSuite.deserializeStringMap true 200 thrpt 3 92525.977 ± 379321.772 ops/s MapSerializationSuite.serializeIntMap false 5 thrpt 3 7958692.983 ± 1934287.574 ops/s MapSerializationSuite.serializeIntMap false 20 thrpt 3 2425269.897 ± 3763706.776 ops/s MapSerializationSuite.serializeIntMap false 50 thrpt 3 1079804.122 ± 215967.411 ops/s MapSerializationSuite.serializeIntMap false 100 thrpt 3 369848.671 ± 433172.821 ops/s MapSerializationSuite.serializeIntMap false 200 thrpt 3 192858.945 ± 71543.709 ops/s MapSerializationSuite.serializeIntMap true 5 thrpt 3 7239453.648 ± 3855324.170 ops/s MapSerializationSuite.serializeIntMap true 20 thrpt 3 2137006.685 ± 3823762.656 ops/s MapSerializationSuite.serializeIntMap true 50 thrpt 3 811639.511 ± 2407986.801 ops/s MapSerializationSuite.serializeIntMap true 100 thrpt 3 412728.569 ± 149199.142 ops/s MapSerializationSuite.serializeIntMap true 200 thrpt 3 236602.475 ± 253662.098 ops/s MapSerializationSuite.serializeStringMap false 5 thrpt 3 5821603.026 ± 1397740.496 ops/s MapSerializationSuite.serializeStringMap false 20 thrpt 3 1712819.341 ± 321017.433 ops/s MapSerializationSuite.serializeStringMap false 50 thrpt 3 615260.241 ± 806075.165 ops/s MapSerializationSuite.serializeStringMap false 100 thrpt 3 265117.558 ± 146904.745 ops/s MapSerializationSuite.serializeStringMap false 200 thrpt 3 128618.697 ± 94723.953 ops/s MapSerializationSuite.serializeStringMap true 5 thrpt 3 4503474.325 ± 11254674.336 ops/s MapSerializationSuite.serializeStringMap true 20 thrpt 3 1732501.942 ± 373691.778 ops/s MapSerializationSuite.serializeStringMap true 50 thrpt 3 596678.154 ± 173893.988 ops/s MapSerializationSuite.serializeStringMap true 100 thrpt 3 336814.584 ± 134582.563 ops/s MapSerializationSuite.serializeStringMap true 200 thrpt 3 143124.619 ± 200889.695 ops/s ```    
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
In this PR , the code generator is simple call the AnySerializer which is much more slower then jit code, I will implement the code generator next PR.
Does this PR introduce any user-facing change?
No user-facing change.
Benchmark
Before:
After: