blob/s3blob: make it possible to configure the default integrity protection#3634
Merged
vangent merged 2 commits intoNov 5, 2025
Merged
Conversation
1a17166 to
c69a9c8
Compare
Contributor
|
Can you do this without updating all of the |
Contributor
Author
I need |
c69a9c8 to
a266faf
Compare
Contributor
|
Tests are still failing. It is hard to get them all to pass for contributors when updating dependencies. You can either keep trying, or keep this on hold until I update them. |
a266faf to
1d25529
Compare
Contributor
Author
|
I just needed to run |
…ection As discussed in aws/aws-sdk-go-v2#3003 and aws/aws-sdk-go-v2#2960, `github.com/aws/aws-sdk-go-v2/service/s3` v1.73.0 changed the AWS SDK default object integrity behavior. Third-party S3 providers, such as Linode, may fail with `XAmzContentSHA256Mismatch` error as a result. A workaround is to set the `AWS_REQUEST_CHECKSUM_CALCULATION` and `AWS_RESPONSE_CHECKSUM_VALIDATION` to `when_required`. However, these environment variables do not affect multipart uploads that used the SDK's `manager.Uploader` implementation (aws/aws-sdk-go-v2#3007). Multipart uploads fail with a 400 Bad Request due to the inclusion of `X-Amz-Sdk-Checksum-Algorithm: CRC32` HTTP headers. With aws/aws-sdk-go-v2#3151, the default integrity protection can be explicitly configured for `manager.Uploader`. To ensure backwards compatibility with third-party S3 providers, this commit adds support for two query parameters: * `request_checksum_calculation` - `when_supported`, `when_required` * `response_checksum_calculation` - `when_supported`, `when_required` For example, on Linode, the defaults don't work: ``` % cat main.go | ./gocdk-blob upload "s3://smybucket?endpoint=https://us-sea-1.linodeobjects.com®ion=us-sea-1" main.go gocdk-blob: closing the writer: blob (key "main.go") (code=Unknown): operation error S3: PutObject, https response error StatusCode: 400, RequestID: <redacted>, HostID: <redacted>, api error XAmzContentSHA256Mismatch: UnknownError ``` Using `request_checksum_calculation=when_required` works: ``` % cat main.go | ./gocdk-blob upload "s3://smybucket?endpoint=https://us-sea-1.linodeobjects.com®ion=us-sea-1&request_checksum_calculation=when_required" main.go % ``` This test was repeated with a larger to validate multipart uploads work.
1d25529 to
1beea01
Compare
flyisland
pushed a commit
to flyisland/gitaly
that referenced
this pull request
May 30, 2026
Previously we had implemented manual checksum algorithm setting for s3 sinks, but we hadn't realised that our gocloud version was not updated for some time and actually aws sdk already implemented default integrity checks: aws/aws-sdk-go-v2#2960 This change removes the need for manual checksum configuration and instead updates the gocloud version to latest to leave the checksum handling to underlying sdk. To ensure backwards compatibility with third-party S3 providers, two query parameters can be used: request_checksum_calculation - when_supported, when_required response_checksum_calculation - when_supported, when_required Source: google/go-cloud#3634 Changelog: changed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
As discussed in aws/aws-sdk-go-v2#3003 and aws/aws-sdk-go-v2#2960,
github.com/aws/aws-sdk-go-v2/service/s3v1.73.0 changed the AWS SDK default object integrity behavior. Third-party S3 providers, such as Linode, may fail withXAmzContentSHA256Mismatcherror as a result.A workaround is to set the
AWS_REQUEST_CHECKSUM_CALCULATIONandAWS_RESPONSE_CHECKSUM_VALIDATIONtowhen_required.However, these environment variables do not affect multipart uploads that used the SDK's
manager.Uploaderimplementation. Multipart uploads fail with a 400 Bad Request due to the inclusion ofX-Amz-Sdk-Checksum-Algorithm: CRC32HTTP headers. With aws/aws-sdk-go-v2#3151, the default integrity protection can be configured explicitly formanager.Uploader.To ensure backwards compatibility with third-party S3 providers, this commit adds support for two query parameters:
request_checksum_calculation-when_supported,when_requiredresponse_checksum_calculation-when_supported,when_requiredFor example, on Linode, the defaults don't work:
Using
request_checksum_calculation=when_requiredworks:This test was repeated with a larger to validate multipart uploads work.