Skip to content

feat(storage): add rewrite_until_done helper as extension trait#3328

Merged
suzmue merged 3 commits intogoogleapis:mainfrom
suzmue:helper
Sep 12, 2025
Merged

feat(storage): add rewrite_until_done helper as extension trait#3328
suzmue merged 3 commits intogoogleapis:mainfrom
suzmue:helper

Conversation

@suzmue
Copy link
Copy Markdown
Collaborator

@suzmue suzmue commented Sep 11, 2025

Several examples for object rewrite operations required a manual loop to poll the operation until it was complete. This boilerplate is cumbersome for users and prone to error.

This change introduces a RewriteObjectExt extension trait, which provides rewrite_until_done() helper method on the RewriteObject builder. This function encapsulates the polling logic, automatically handling the rewrite_token and looping until the operation is finished.

This extension trait lives in the new builder_ext module. This is modeled after the model_ext module which contains other convenience functions and types that are built on top of generated types.

For #3204

@product-auto-label product-auto-label bot added the api: storage Issues related to the Cloud Storage API. label Sep 11, 2025
@codecov
Copy link
Copy Markdown

codecov bot commented Sep 11, 2025

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 96.32%. Comparing base (cf91973) to head (4b24b77).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3328      +/-   ##
==========================================
+ Coverage   96.29%   96.32%   +0.02%     
==========================================
  Files         113      114       +1     
  Lines        4510     4517       +7     
==========================================
+ Hits         4343     4351       +8     
+ Misses        167      166       -1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@suzmue
Copy link
Copy Markdown
Collaborator Author

suzmue commented Sep 11, 2025

This PR still needs tests, but I welcome thoughts on whether this is 1) the right location and 2) the right way to expose this helper.

@dbolduc
Copy link
Copy Markdown
Member

dbolduc commented Sep 11, 2025

thoughts

I really like the ergonomics of the trait extension. I think it is nicer than a fn of a builder.

I am cool with it in model_ext.

The only thing that isn't obvious to me is whether the loop should have some sort of exhaustion policy. Like if we are making progress too slowly.

I don't think we need a policy. I think even if we want such a policy, we would add it to the storage::request_options::RequestOptions. <-- oh wait this doesn't work. I mixed up my clients

@coryan
Copy link
Copy Markdown
Collaborator

coryan commented Sep 12, 2025

thoughts

I really like the ergonomics of the trait extension. I think it is nicer than a fn of a builder.

💯

I am cool with it in model_ext.

We are extending builders, so builder_ext seems slightly better to me?

The only thing that isn't obvious to me is whether the loop should have some sort of exhaustion policy. Like if we are making progress too slowly.

I don't think we need a policy. I think even if we want such a policy, we would add it to the storage::request_options::RequestOptions. <-- oh wait this doesn't work. I mixed up my clients

I would keep this simple. If we want a loop with exhaustion we can add another extension function (in the future) that transforms the rewrite into a stream. Then they have all the power of streams to do things like:

    use futures::stream::StreamExt;
    let builder = client
        .rewrite_object()
        .set_source_bucket(format!("projects/_/buckets/{source_bucket_id}"))
        .set_source_object(SOURCE_NAME)
        .set_destination_bucket(format!("projects/_/buckets/{dest_bucket_id}"))
        .set_destination_name(DEST_NAME);

  let moved = builder
      .to_stream() // A new thing
      .take(100) // at most 100 attempts.
      .map(|v| { /* return some error if the stream is too slow */ })
      .try_fold(None, |object, response| if response.done { response.resource } else { object })

A simple function like the one provided in this PR will handle 90% of the cases. We can add the streams for the next 9%, and customers can fallback to an explicit loop for the remaining 1%.

@suzmue
Copy link
Copy Markdown
Collaborator Author

suzmue commented Sep 12, 2025

I am cool with it in model_ext.

We are extending builders, so builder_ext seems slightly better to me?

Created builder_ext and moved this there.

This PR still needs tests

Tests are added!

@suzmue suzmue marked this pull request as ready for review September 12, 2025 19:28
@suzmue suzmue requested a review from a team September 12, 2025 19:28
Copy link
Copy Markdown
Member

@dbolduc dbolduc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I disagree with the return type, but other than that, I just have nits

dbolduc
dbolduc previously approved these changes Sep 12, 2025
@suzmue suzmue merged commit e9f10a3 into googleapis:main Sep 12, 2025
25 checks passed
@suzmue suzmue deleted the helper branch September 12, 2025 22:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api: storage Issues related to the Cloud Storage API.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants