Skip to content

storage: Support parallel composite uploads for faster transfers #8147

@stanhu

Description

@stanhu

Is your feature request related to a problem? Please describe.

gsutil supports parallel composite uploads, but the Google Cloud Storage SDKs do not.

For large files, transfers to Google Cloud Storage could be significantly faster if chunks of data are uploaded simultaneously. The Amazon SDKs natively support multi-threaded, multipart uploads with Golang, Ruby, and Python libraries.

Describe the solution you'd like

gsutil implements this in Python.

Ideally there would be a simple boolean parameter, such as parallel_composite_upload:

# Copy the file to a backup bucket
backup = storage.bucket "task-attachment-backups"
file.copy backup, file.name, parallel_composite_upload: true

Describe alternatives you've considered

This could be implemented in the application layer, but it really belongs in the SDK.

Additional context

I created an equivalent issue in the Go SDK: googleapis/google-cloud-go#3219

Metadata

Metadata

Assignees

Labels

api: storageIssues related to the Cloud Storage API.type: feature request‘Nice-to-have’ improvement, new feature or different behavior or design.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions