Skip to content

feat(storage/transfermanager): add DownloadDirectory #10430

Merged
BrennaEpp merged 9 commits into
googleapis:mainfrom
BrennaEpp:tm-directory
Jul 3, 2024
Merged

feat(storage/transfermanager): add DownloadDirectory #10430
BrennaEpp merged 9 commits into
googleapis:mainfrom
BrennaEpp:tm-directory

Conversation

@BrennaEpp

@BrennaEpp BrennaEpp commented Jun 25, 2024

Copy link
Copy Markdown
Contributor

mvp

Follow-up PRs:

  • Clean up file on failure
  • Fail if the file already exists instead of overwriting
  • Skipifexists
  • strip prefix option
  • empty object handling (if necessary)

@product-auto-label product-auto-label Bot added the api: storage Issues related to the Cloud Storage API. label Jun 25, 2024
Prototype.

Follow-up PRs:
- Cleanup on failure
- Skipifexists + option to fail if exists?
- strip prefix ? (strip prefix on local files)
- if empty object, create a directory instead ?
- possibly more integration tests
@BrennaEpp BrennaEpp marked this pull request as ready for review June 25, 2024 20:55
@BrennaEpp BrennaEpp requested review from a team and tritone June 25, 2024 20:55

@tritone tritone left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few minor things; generally looks good

Comment thread storage/transfermanager/downloader.go Outdated
Comment thread storage/transfermanager/downloader.go Outdated
Comment thread storage/transfermanager/downloader.go
Comment thread storage/transfermanager/downloader.go
Comment thread storage/transfermanager/downloader.go Outdated
Comment thread storage/transfermanager/downloader.go
Comment thread storage/transfermanager/downloader.go Outdated
@BrennaEpp BrennaEpp requested a review from tritone June 28, 2024 23:38

@tritone tritone left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One more minor suggestion, otherwise looks good.

Comment thread storage/transfermanager/downloader.go Outdated
Callback func([]DownloadOutput)

// OnObjectDownload will run after every finished object download.
// It can only be set if the Downloader has the [WithCallbacks] option set.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we could probably loosen this requirement? A user could want to run in sync mode and still use this to track/log progress.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

@BrennaEpp BrennaEpp requested a review from tritone July 3, 2024 17:20
@BrennaEpp BrennaEpp enabled auto-merge (squash) July 3, 2024 17:54
@BrennaEpp BrennaEpp merged commit 0d0e5dd into googleapis:main Jul 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api: storage Issues related to the Cloud Storage API.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants