Skip to content

adding msgpack benchmarks#1853

Merged
lemire merged 6 commits intomasterfrom
dlemire/msgpack_benchmarks
Jun 30, 2022
Merged

adding msgpack benchmarks#1853
lemire merged 6 commits intomasterfrom
dlemire/msgpack_benchmarks

Conversation

@lemire
Copy link
Member

@lemire lemire commented Jun 25, 2022

Should be self-explanatory: adding benchmarks where we transcode from json to msgpack.

$ cmake --build build --target bench_ondemand && ./build/benchmark/bench_ondemand --benchmark_filter="json2msgpack"
[  0%] Built target generated-data
Consolidate compiler generated dependencies of target simdjson
[  0%] Building CXX object CMakeFiles/simdjson.dir/src/simdjson.cpp.o
[ 10%] Linking CXX static library libsimdjson.a
[ 10%] Built target simdjson
Consolidate compiler generated dependencies of target yyjson
[ 20%] Built target yyjson
Consolidate compiler generated dependencies of target benchmark
[ 90%] Built target benchmark
Consolidate compiler generated dependencies of target bench_ondemand
[100%] Building CXX object benchmark/CMakeFiles/bench_ondemand.dir/bench_ondemand.cpp.o
[100%] Linking CXX executable bench_ondemand
[100%] Built target bench_ondemand
2022-06-25T18:50:33+00:00
Running ./build/benchmark/bench_ondemand
Run on (2 X 3501.45 MHz CPU s)
CPU Caches:
  L1 Data 48 KiB (x1)
  L1 Instruction 32 KiB (x1)
  L2 Unified 1280 KiB (x1)
  L3 Unified 55296 KiB (x1)
Load Average: 0.44, 0.11, 0.04
simdjson::dom implementation:      icelake
simdjson::ondemand implementation (stage 1): icelake
simdjson::ondemand implementation (stage 2): fallback
-------------------------------------------------------------------------------------------------------
Benchmark                                             Time             CPU   Iterations UserCounters...
-------------------------------------------------------------------------------------------------------
json2msgpack<simdjson_ondemand>/manual_time      238494 ns       353227 ns         2947 best_bytes_per_sec=2.69005G best_docs_per_sec=4.25967k best_items_per_sec=4.25967k bytes=631.515k bytes_per_second=2.46608G/s docs_per_sec=4.19299k/s items=1 items_per_second=4.19299k/s [BEST: throughput=  2.69 GB/s doc_throughput=  4259 docs/s items=         1 avg_time=    238493 ns]
json2msgpack<rapidjson_lossless>/manual_time    1811034 ns      1931976 ns          389 best_bytes_per_sec=361.856M best_docs_per_sec=572.996 best_items_per_sec=572.996 bytes=631.515k bytes_per_second=332.55M/s docs_per_sec=552.171/s items=1 items_per_second=552.171/s [BEST: throughput=  0.36 GB/s doc_throughput=   572 docs/s items=         1 avg_time=   1811033 ns]
json2msgpack<rapidjson>/manual_time             1741477 ns      1862749 ns          404 best_bytes_per_sec=378.209M best_docs_per_sec=598.891 best_items_per_sec=598.891 bytes=631.515k bytes_per_second=345.833M/s docs_per_sec=574.225/s items=1 items_per_second=574.225/s [BEST: throughput=  0.38 GB/s doc_throughput=   598 docs/s items=         1 avg_time=   1741476 ns]
json2msgpack<rapidjson_insitu>/manual_time      1458942 ns      1572830 ns          480 best_bytes_per_sec=443.217M best_docs_per_sec=701.832 best_items_per_sec=701.832 bytes=631.515k bytes_per_second=412.806M/s docs_per_sec=685.428/s items=1 items_per_second=685.428/s [BEST: throughput=  0.44 GB/s doc_throughput=   701 docs/s items=         1 avg_time=   1458941 ns]
json2msgpack<yyjson>/manual_time                 783344 ns       905245 ns          902 best_bytes_per_sec=891.715M best_docs_per_sec=1.41202k best_items_per_sec=1.41202k bytes=631.515k bytes_per_second=768.831M/s docs_per_sec=1.27658k/s items=1 items_per_second=1.27658k/s [BEST: throughput=  0.89 GB/s doc_throughput=  1412 docs/s items=         1 avg_time=    783344 ns]
json2msgpack<yyjson_insitu>/manual_time          581970 ns       697006 ns         1228 best_bytes_per_sec=1.23918G best_docs_per_sec=1.96223k best_items_per_sec=1.96223k bytes=631.515k bytes_per_second=1034.86M/s docs_per_sec=1.7183k/s items=1 items_per_second=1.7183k/s [BEST: throughput=  1.24 GB/s doc_throughput=  1962 docs/s items=         1 avg_time=    581970 ns]
json2msgpack<sajson>/manual_time                 570235 ns       684098 ns         1174 best_bytes_per_sec=1.16855G best_docs_per_sec=1.85039k best_items_per_sec=1.85039k bytes=631.515k bytes_per_second=1056.16M/s docs_per_sec=1.75366k/s items=1 items_per_second=1.75366k/s [BEST: throughput=  1.17 GB/s doc_throughput=  1850 docs/s items=         1 avg_time=    570234 ns]
json2msgpack<nlohmann_json>/manual_time        25432545 ns     25596593 ns           28 best_bytes_per_sec=25.4988M best_docs_per_sec=40.3771 best_items_per_sec=40.3771 bytes=631.515k bytes_per_second=23.6807M/s docs_per_sec=39.3197/s items=1 items_per_second=39.3197/s [BEST: throughput=  0.03 GB/s doc_throughput=    40 docs/s items=         1 avg_time=  25432544 ns]

@lemire
Copy link
Member Author

lemire commented Jun 25, 2022

@jkeiser @NicolasJiaxin Really fast JSON to msgpack transcoding.

@NicolasJiaxin
Copy link
Contributor

Nice! Will you also be adding tests?

@lemire
Copy link
Member Author

lemire commented Jun 27, 2022

@NicolasJiaxin

Nice! Will you also be adding tests?

The benchmarks always include 'some testing' and here we test that all library implementation output the same size. Initially, I wanted all my implementation to have the exact same output (byte-by-byte) but that's tricky because of key ordering (there is no unique way to serialize an object if you are allowed to reorder the keys).

However, the result is not well tested... for that, we would need an independent piece of code that can at least validate that we are producing 'valid msgpack'. If you'd like to contribute that, it would be great. There is probably readily available code to validate a msgpack payload.

Another fun project would be to try to write fast code to 'compact' the msgpack output.

@lemire lemire merged commit 509066f into master Jun 30, 2022
@lemire lemire deleted the dlemire/msgpack_benchmarks branch June 30, 2022 14:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants