✅ Deploy Preview for meta-velox canceled.

Hi @wForget!

Thank you for your pull request and welcome to our community.

Action Required

In order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you.

Process

In order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA.

Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with CLA signed. The tagging process may take up to 1 hour after signing. Please give it that time before contacting us about it.

If you have received this in error or have any questions, please contact us at cla@meta.com. Thanks!

Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Meta Open Source project. Thanks!

Thanks!
Please update the configs.rst

Thanks! Please update the configs.rst

Thank you, updated

@xiaoxmeng @majetideepak Could you please take a look?

Gentle ping @xiaoxmeng @majetideepak, sorry to bother you again, could you please review it if you have time?

@wForget can you share the motivation behind this feature?

My motivation is to make the size of the RowVector after deserializing SpillFile closer to this batch size. As observed in #13861, excessively large row sizes may lead to performance regression.

How would users configure both the buffer size and batch size in general?

The spill buffer will be flushed when either the total serialized size reaches spill_write_buffer_size or the number of buffered rows reaches spill_write_batch_size. The default value for spill_write_buffer_size is 4MB, and the default value for spill_write_batch_size is 4096 (Or should we let it be closer to preferred_output_batch_rows ?).

This pull request has been automatically marked as stale because it has not had recent activity. If you'd still like this PR merged, please comment on the PR, make sure you've addressed reviewer comments, and rebase on the latest main. Thank you for your contributions!

This pull request has been automatically marked as stale because it has not had recent activity. If you'd still like this PR merged, please comment on the PR, make sure you've addressed reviewer comments, and rebase on the latest main. Thank you for your contributions!