Skip to content

object_store: Flush buffered multipart only during poll_shutdown#3397

Merged
tustvold merged 1 commit into
apache:masterfrom
askoa:multipart-write
Dec 28, 2022
Merged

object_store: Flush buffered multipart only during poll_shutdown#3397
tustvold merged 1 commit into
apache:masterfrom
askoa:multipart-write

Conversation

@askoa

@askoa askoa commented Dec 26, 2022

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes apache/arrow-rs-object-store#205

Are there any user-facing changes?

Yes, the poll_flush method will not upload current_buffer.

@askoa askoa marked this pull request as ready for review December 26, 2022 23:18
@askoa

askoa commented Dec 26, 2022

Copy link
Copy Markdown
Contributor Author

I don't see any tests for the multipart functionality. Also, not sure how to specifically test flush is working as expected. If there needs to be tests then need some hints on how to add them.

tustvold
tustvold approved these changes Dec 27, 2022
View reviewed changes

@tustvold tustvold left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is tested by stream_get for AWS, Azure and GCP

@tustvold tustvold merged commit 8f99692 into apache:master Dec 28, 2022
@ursabot

ursabot commented Dec 28, 2022

Copy link
Copy Markdown

Benchmark runs are scheduled for baseline = b903384 and contender = 8f99692. 8f99692 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Skipped ⚠️ Benchmarking of arrow-rs-commits is not supported on ec2-t3-xlarge-us-east-2] ec2-t3-xlarge-us-east-2
[Skipped ⚠️ Benchmarking of arrow-rs-commits is not supported on test-mac-arm] test-mac-arm
[Skipped ⚠️ Benchmarking of arrow-rs-commits is not supported on ursa-i9-9960x] ursa-i9-9960x
[Skipped ⚠️ Benchmarking of arrow-rs-commits is not supported on ursa-thinkcentre-m75q] ursa-thinkcentre-m75q
Buildkite builds:
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

@askoa askoa deleted the multipart-write branch January 25, 2023 11:45
alamb pushed a commit to alamb/arrow-rs that referenced this pull request Mar 20, 2025
@askoa
object_store: Flush buffered multipart only during poll_shutdown (apa…
fbf0561 
…che#3397)

Co-authored-by: askoa <askoa@local>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@tustvold tustvold tustvold approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Only flush buffered multi-part data on poll_shutdown not on poll_flush

3 participants

@askoa @ursabot @tustvold