feat: Add NonMaterializedSortBuffer #15157
Conversation
✅ Deploy Preview for meta-velox canceled.
|
27dc9b0 to
ef59320
Compare
16f988c to
606779e
Compare
|
|
||
| int32_t outputRow = 0; | ||
| int32_t outputSize = 0; | ||
| bool isEndOfBatch = false; |
There was a problem hiding this comment.
s/isEndOfBatch/endOfBatch/
| int32_t outputRow = 0; | ||
| int32_t outputSize = 0; | ||
| bool isEndOfBatch = false; | ||
| while (outputRow + outputSize < output_->size()) { |
There was a problem hiding this comment.
s/outputSize/bufferedOutputRows/
|
|
||
| void HybridSortBuffer::prepareOutputWithSpill() { | ||
| VELOX_CHECK(hasSpilled()); | ||
| if (inputSpiller_ != nullptr) { |
There was a problem hiding this comment.
if () {
...
return;
}
...
| outputSpillPartitionSet_.begin()->second->createUnorderedReader( | ||
| spillConfig_->readBufferSize, pool(), spillStats_); | ||
| } | ||
| outputSpillPartitionSet_.clear(); |
There was a problem hiding this comment.
this belong to the else condition?
468ca20 to
027a584
Compare
a2704bc to
0b7b165
Compare
|
@duanmeng : Thanks for this PR. Please can you explain the motivation for this change. |
@aditi-pandit Hi Aditi, the |
Thanks for your explanation. I got the infrastructure pieces. Which operators are we planning to enhance with this ? Do you have some early results to share ? |
Add a NonMaterializedSortBuffer, which materializes only the sort key columns
and additional index columns into a RowContainer for sorting. After the sort, it
uses these indices to gather and copy the corresponding rows from the original
input vectors into the output vector. This reduces the overhead of materializing
payload columns into the RowContainer, which is especially beneficial for wide
table scenarios. Support for spilling will be added in a follow-up PR.