Skip to content

perf(spilling): Support serializing rows to avoid extracting it as vector#15290

Open
NEUpanning wants to merge 12 commits intofacebookincubator:mainfrom
NEUpanning:presto-ser-row
Open

perf(spilling): Support serializing rows to avoid extracting it as vector#15290
NEUpanning wants to merge 12 commits intofacebookincubator:mainfrom
NEUpanning:presto-ser-row

Conversation

@NEUpanning
Copy link
Copy Markdown
Contributor

@NEUpanning NEUpanning commented Oct 27, 2025

Currently, Velox has to extract rows in RowContainer as vector for serialization when spilling. This PR supports serializing rows to eliminate the cost of extracting and materializing vectors corresponding to spillExtractVectorTime metrics.

Below are the benchmark results:


============================================================================
[...]ec/tests/SpillerSortBenchmarkTest.cpp     relative  time/iter   iters/s
============================================================================
integer_type                                                 1.00s   996.37m
integer_type_serialize_rows                     100.98%   993.88ms      1.01
bigint_type                                                  1.01s   988.61m
bigint_type_serialize_rows                      100.97%      1.00s   998.16m
string_type_10bytes                                          1.11s   900.35m
string_type_10bytes_serialize_rows              105.15%      1.06s   946.74m
string_type_50bytes                                          1.35s   743.15m
string_type_50bytes_long_serialize_rows         106.23%      1.27s   789.46m
array_of_integer_type                                        1.88s   530.53m
array_of_integer_type_serialize_rows            124.22%      1.52s   659.04m
array_of_10bytes_varbinary_type                              2.60s   384.22m
array_of_10bytes_varbinary_type_serialize_rows  137.72%      1.89s   529.16m
array_of_50bytes_varbinary_type                              3.68s   272.06m
array_of_50bytes_varbinary_type_serialize_rows  143.72%      2.56s   391.01m
array_of_100bytes_varbinary_type                             4.97s   201.02m
array_of_100bytes_varbinary_type_serialize_rows 149.28%      3.33s   300.08m
benchmark spilling stats details
============================================================================
[...]ec/tests/SpillerSortBenchmarkTest.cpp     relative  time/iter   iters/s
============================================================================
spillRuns[1] spilledInputBytes[9.54MB] spilledBytes[6.33MB] spilledRows[1000000] spilledPartitions[1] spilledFiles[1] spillFillTimeNanos[298.08ms] spillSortTimeNanos[577.05ms] spillExtractVectorTime[49.48ms] spillSerializationTimeNanos[30.22ms] spillWrites[7] spillFlushTimeNanos[17.91ms] spillWriteTimeNanos[784.38us] maxSpillExceededLimitCount[0] spillReadBytes[0B] spillReads[0] spillReadTimeNanos[0ns] spillReadDeserializationTimeNanos[0ns]
integer_type                                                 1.00s   996.37m

spillRuns[1] spilledInputBytes[12.40MB] spilledBytes[6.33MB] spilledRows[1000000] spilledPartitions[1] spilledFiles[1] spillFillTimeNanos[297.89ms] spillSortTimeNanos[577.48ms] spillExtractVectorTime[11.61ms] spillSerializationTimeNanos[51.17ms] spillWrites[7] spillFlushTimeNanos[17.38ms] spillWriteTimeNanos[832.79us] maxSpillExceededLimitCount[0] spillReadBytes[0B] spillReads[0] spillReadTimeNanos[0ns] spillReadDeserializationTimeNanos[0ns]
integer_type_serialize_rows                     100.98%   993.88ms      1.01

spillRuns[1] spilledInputBytes[15.26MB] spilledBytes[9.41MB] spilledRows[1000000] spilledPartitions[1] spilledFiles[1] spillFillTimeNanos[297.18ms] spillSortTimeNanos[577.24ms] spillExtractVectorTime[50.69ms] spillSerializationTimeNanos[30.16ms] spillWrites[10] spillFlushTimeNanos[25.71ms] spillWriteTimeNanos[1.28ms] maxSpillExceededLimitCount[0] spillReadBytes[0B] spillReads[0] spillReadTimeNanos[0ns] spillReadDeserializationTimeNanos[0ns]
bigint_type                                                  1.01s   988.61m

spillRuns[1] spilledInputBytes[16.21MB] spilledBytes[9.41MB] spilledRows[1000000] spilledPartitions[1] spilledFiles[1] spillFillTimeNanos[294.63ms] spillSortTimeNanos[575.87ms] spillExtractVectorTime[11.61ms] spillSerializationTimeNanos[54.57ms] spillWrites[10] spillFlushTimeNanos[25.36ms] spillWriteTimeNanos[1.17ms] maxSpillExceededLimitCount[0] spillReadBytes[0B] spillReads[0] spillReadTimeNanos[0ns] spillReadDeserializationTimeNanos[0ns]
bigint_type_serialize_rows                      100.97%      1.00s   998.16m

spillRuns[1] spilledInputBytes[26.70MB] spilledBytes[14.77MB] spilledRows[1000000] spilledPartitions[1] spilledFiles[1] spillFillTimeNanos[302.08ms] spillSortTimeNanos[577.42ms] spillExtractVectorTime[83.14ms] spillSerializationTimeNanos[76.10ms] spillWrites[15] spillFlushTimeNanos[40.56ms] spillWriteTimeNanos[1.51ms] maxSpillExceededLimitCount[0] spillReadBytes[0B] spillReads[0] spillReadTimeNanos[0ns] spillReadDeserializationTimeNanos[0ns]
string_type_10bytes                                          1.11s   900.35m

spillRuns[1] spilledInputBytes[27.66MB] spilledBytes[14.77MB] spilledRows[1000000] spilledPartitions[1] spilledFiles[1] spillFillTimeNanos[298.01ms] spillSortTimeNanos[574.23ms] spillExtractVectorTime[20.10ms] spillSerializationTimeNanos[83.73ms] spillWrites[15] spillFlushTimeNanos[39.57ms] spillWriteTimeNanos[1.50ms] maxSpillExceededLimitCount[0] spillReadBytes[0B] spillReads[0] spillReadTimeNanos[0ns] spillReadDeserializationTimeNanos[0ns]
string_type_10bytes_serialize_rows              105.15%      1.06s   946.74m

spillRuns[1] spilledInputBytes[757.69MB] spilledBytes[44.08MB] spilledRows[1000000] spilledPartitions[1] spilledFiles[1] spillFillTimeNanos[297.01ms] spillSortTimeNanos[570.92ms] spillExtractVectorTime[185.83ms] spillSerializationTimeNanos[129.09ms] spillWrites[47] spillFlushTimeNanos[112.08ms] spillWriteTimeNanos[5.73ms] maxSpillExceededLimitCount[0] spillReadBytes[0B] spillReads[0] spillReadTimeNanos[0ns] spillReadDeserializationTimeNanos[0ns]
string_type_50bytes                                          1.35s   743.15m

spillRuns[1] spilledInputBytes[69.36MB] spilledBytes[44.08MB] spilledRows[1000000] spilledPartitions[1] spilledFiles[1] spillFillTimeNanos[297.38ms] spillSortTimeNanos[574.21ms] spillExtractVectorTime[19.52ms] spillSerializationTimeNanos[186.17ms] spillWrites[47] spillFlushTimeNanos[118.13ms] spillWriteTimeNanos[19.35ms] maxSpillExceededLimitCount[0] spillReadBytes[0B] spillReads[0] spillReadTimeNanos[0ns] spillReadDeserializationTimeNanos[0ns]
string_type_50bytes_long_serialize_rows         106.23%      1.27s   789.46m

spillRuns[1] spilledInputBytes[103.43MB] spilledBytes[23.38MB] spilledRows[1000000] spilledPartitions[1] spilledFiles[1] spillFillTimeNanos[299.08ms] spillSortTimeNanos[577.30ms] spillExtractVectorTime[625.65ms] spillSerializationTimeNanos[237.93ms] spillWrites[32] spillFlushTimeNanos[93.37ms] spillWriteTimeNanos[3.34ms] maxSpillExceededLimitCount[0] spillReadBytes[0B] spillReads[0] spillReadTimeNanos[0ns] spillReadDeserializationTimeNanos[0ns]
array_of_integer_type                                        1.88s   530.53m

spillRuns[1] spilledInputBytes[63.40MB] spilledBytes[23.38MB] spilledRows[1000000] spilledPartitions[1] spilledFiles[1] spillFillTimeNanos[299.58ms] spillSortTimeNanos[577.55ms] spillExtractVectorTime[20.12ms] spillSerializationTimeNanos[455.54ms] spillWrites[32] spillFlushTimeNanos[94.09ms] spillWriteTimeNanos[3.36ms] maxSpillExceededLimitCount[0] spillReadBytes[0B] spillReads[0] spillReadTimeNanos[0ns] spillReadDeserializationTimeNanos[0ns]
array_of_integer_type_serialize_rows            124.22%      1.52s   659.04m

spillRuns[1] spilledInputBytes[370.92MB] spilledBytes[73.02MB] spilledRows[1000000] spilledPartitions[1] spilledFiles[1] spillFillTimeNanos[298.38ms] spillSortTimeNanos[576.86ms] spillExtractVectorTime[793.47ms] spillSerializationTimeNanos[590.73ms] spillWrites[97] spillFlushTimeNanos[265.04ms] spillWriteTimeNanos[13.06ms] maxSpillExceededLimitCount[0] spillReadBytes[0B] spillReads[0] spillReadTimeNanos[0ns] spillReadDeserializationTimeNanos[0ns]
array_of_10bytes_varbinary_type                              2.60s   384.22m

spillRuns[1] spilledInputBytes[121.80MB] spilledBytes[72.81MB] spilledRows[1000000] spilledPartitions[1] spilledFiles[1] spillFillTimeNanos[297.92ms] spillSortTimeNanos[571.34ms] spillExtractVectorTime[21.37ms] spillSerializationTimeNanos[666.29ms] spillWrites[97] spillFlushTimeNanos[256.59ms] spillWriteTimeNanos[9.22ms] maxSpillExceededLimitCount[0] spillReadBytes[0B] spillReads[0] spillReadTimeNanos[0ns] spillReadDeserializationTimeNanos[0ns]
array_of_10bytes_varbinary_type_serialize_rows  137.72%      1.89s   529.16m

spillRuns[1] spilledInputBytes[1.06GB] spilledBytes[253.80MB] spilledRows[1000000] spilledPartitions[1] spilledFiles[1] spillFillTimeNanos[297.86ms] spillSortTimeNanos[566.41ms] spillExtractVectorTime[1.00s] spillSerializationTimeNanos[946.97ms] spillWrites[328] spillFlushTimeNanos[725.17ms] spillWriteTimeNanos[35.09ms] maxSpillExceededLimitCount[0] spillReadBytes[0B] spillReads[0] spillReadTimeNanos[0ns] spillReadDeserializationTimeNanos[0ns]
array_of_50bytes_varbinary_type                              3.68s   272.06m

spillRuns[1] spilledInputBytes[353.89MB] spilledBytes[254.86MB] spilledRows[1000000] spilledPartitions[1] spilledFiles[1] spillFillTimeNanos[296.06ms] spillSortTimeNanos[574.13ms] spillExtractVectorTime[20.52ms] spillSerializationTimeNanos[785.47ms] spillWrites[328] spillFlushTimeNanos[728.75ms] spillWriteTimeNanos[45.17ms] maxSpillExceededLimitCount[0] spillReadBytes[0B] spillReads[0] spillReadTimeNanos[0ns] spillReadDeserializationTimeNanos[0ns]
array_of_50bytes_varbinary_type_serialize_rows  143.72%      2.56s   391.01m

spillRuns[1] spilledInputBytes[1.13GB] spilledBytes[476.34MB] spilledRows[1000000] spilledPartitions[1] spilledFiles[1] spillFillTimeNanos[298.18ms] spillSortTimeNanos[571.76ms] spillExtractVectorTime[1.14s] spillSerializationTimeNanos[1.37s] spillWrites[612] spillFlushTimeNanos[1.36s] spillWriteTimeNanos[75.16ms] maxSpillExceededLimitCount[0] spillReadBytes[0B] spillReads[0] spillReadTimeNanos[0ns] spillReadDeserializationTimeNanos[0ns]
array_of_100bytes_varbinary_type                             4.97s   201.02m

spillRuns[1] spilledInputBytes[643.58MB] spilledBytes[475.49MB] spilledRows[1000000] spilledPartitions[1] spilledFiles[1] spillFillTimeNanos[294.76ms] spillSortTimeNanos[565.68ms] spillExtractVectorTime[19.21ms] spillSerializationTimeNanos[905.35ms] spillWrites[612] spillFlushTimeNanos[1.34s] spillWriteTimeNanos[50.25ms] maxSpillExceededLimitCount[0] spillReadBytes[0B] spillReads[0] spillReadTimeNanos[0ns] spillReadDeserializationTimeNanos[0ns]
array_of_100bytes_varbinary_type_serialize_rows 149.28%      3.33s   300.08m

@netlify
Copy link
Copy Markdown

netlify Bot commented Oct 27, 2025

Deploy Preview for meta-velox canceled.

Name Link
🔨 Latest commit 734bdaf
🔍 Latest deploy log https://app.netlify.com/projects/meta-velox/deploys/6908525202dded0008c36fd8

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Oct 27, 2025
@NEUpanning
Copy link
Copy Markdown
Contributor Author

@jinchengchenghh Would you like to take a look? Thank you.

@jinchengchenghh
Copy link
Copy Markdown
Contributor

Could you summarize the benchmark result in the PR description? And add the benchmark result for primitive data type.

Comment thread velox/exec/Driver.cpp
queryConfig.spillFileCreateConfig(),
queryConfig.windowSpillMinReadBatchRows());
queryConfig.windowSpillMinReadBatchRows(),
true);
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add the config

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change is an optimization with no adverse impact. This parameter is only used for benchmarking. So I'm uncertain whether it is necessary to add a configuration for it.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make senses.

Comment thread velox/exec/Spiller.cpp Outdated
Comment thread velox/exec/Spill.cpp Outdated
Comment thread velox/exec/Spill.cpp
initPartitionWriter(id);

uint64_t bytes = 0;
for (const auto& row : rows) {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here you have got the exactly row size, so you don't need an estimated initial size 1'000

batch_->createStreamTree(
          std::static_pointer_cast<const RowType>(type_),
          1'000,
          serdeOptions_.get());

Comment thread velox/exec/ContainerRowSerde.cpp
batch_->createStreamTree(
std::static_pointer_cast<const RowType>(rows->type()),
1'000,
rows->size(),
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here is to write part of the RowVector by indices

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks.

@jinchengchenghh
Copy link
Copy Markdown
Contributor

Would you like to take a look? Thanks! @xiaoxmeng

@NEUpanning
Copy link
Copy Markdown
Contributor Author

Could you summarize the benchmark result in the PR description? And add the benchmark result for primitive data type.

@jinchengchenghh Thank you for reviewing. I've updated the PR description.

@jinchengchenghh
Copy link
Copy Markdown
Contributor

Please make the benchmark spilling stats details readable

@NEUpanning
Copy link
Copy Markdown
Contributor Author

Please make the benchmark spilling stats details readable

@jinchengchenghh I tried to delete unnecessary stats and reformat it. Now the spilling stats text line is on the line above the corresponding benchmark result.

@jinchengchenghh
Copy link
Copy Markdown
Contributor

Overall looks good to me, I ask @xiaoxmeng to help review this PR offline, he can determine this.

@NEUpanning
Copy link
Copy Markdown
Contributor Author

Hi @xiaoxmeng , just a gentle reminder to take a look at this PR when you have a moment. Thanks!

@stale
Copy link
Copy Markdown

stale Bot commented Apr 14, 2026

This pull request has been automatically marked as stale because it has not had recent activity. If you'd still like this PR merged, please comment on the PR, make sure you've addressed reviewer comments, and rebase on the latest main. Thank you for your contributions!

@stale stale Bot added the stale label Apr 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. stale

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants