-
Notifications
You must be signed in to change notification settings - Fork 568
Description
Is your feature request related to a problem? Please describe.
gsutil supports parallel composite uploads, but the Google Cloud Storage SDKs do not.
For large files, transfers to Google Cloud Storage could be significantly faster if chunks of data are uploaded simultaneously. The Amazon SDKs natively support multi-threaded, multipart uploads with Golang, Ruby, and Python libraries.
Describe the solution you'd like
gsutil implements this in Python.
Ideally there would be a simple boolean parameter, such as parallel_composite_upload:
# Copy the file to a backup bucket
backup = storage.bucket "task-attachment-backups"
file.copy backup, file.name, parallel_composite_upload: trueDescribe alternatives you've considered
This could be implemented in the application layer, but it really belongs in the SDK.
Additional context
I created an equivalent issue in the Go SDK: googleapis/google-cloud-go#3219