Skip to content

storage: Support parallel composite uploads for faster transfers #3219

@stanhu

Description

@stanhu

Is your feature request related to a problem? Please describe.

gsutil supports parallel composite uploads, but the Google Cloud Storage SDKs do not.

For large files, transfers to Google Cloud Storage could be significantly faster if chunks of data are uploaded simultaneously. The Amazon SDKs natively support multi-threaded, multipart uploads with Golang, Ruby, and Python libraries.

Describe the solution you'd like

gsutil implements this in Python.

Ideally there would be a simple boolean parameter, such as ParallelCompositeUpload, that would activate this mode in the bucket Write.

Describe alternatives you've considered

This could be implemented in the application layer, but it really belongs in the SDK.

Additional context
Add any other context or screenshots about the feature request here.

Metadata

Metadata

Assignees

Labels

api: storageIssues related to the Cloud Storage API.type: feature request‘Nice-to-have’ improvement, new feature or different behavior or design.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions