feat: flexible checksum updates by wty-Bryant · Pull Request #2808 · aws/aws-sdk-go-v2

wty-Bryant · 2024-09-26T03:33:32Z

Add client cfg to opt-in/out checksum calculation/validation, change default algorithm to calculate checksum

service/internal/checksum/middleware_compute_input_checksum.go

aws/checksum.go

.changelog/9ebe24c4791541e0840da49eab6f9d97.json

aws/checksum.go

merge main into current branch

aws/checksum.go

config/provider.go

service/internal/checksum/middleware_add.go

service/internal/checksum/middleware_compute_input_checksum.go

service/internal/checksum/middleware_compute_input_checksum_test.go

service/internal/checksum/middleware_setup_context.go

service/internal/checksum/middleware_validate_output.go

.changelog/9ebe24c4791541e0840da49eab6f9d97.json

aws/checksum.go

service/internal/checksum/middleware_add.go

service/internal/checksum/middleware_setup_context.go

service/internal/checksum/middleware_compute_input_checksum.go

service/internal/checksum/middleware_add.go

merge main into branch

config/load_options.go

config/shared_config.go

service/internal/checksum/middleware_add.go

service/internal/checksum/middleware_compute_input_checksum_test.go

service/internal/checksum/middleware_compute_input_checksum.go

service/internal/checksum/middleware_compute_input_checksum_test.go

.changelog/9ebe24c4791541e0840da49eab6f9d97.json

aws/config.go

config/load_options.go

service/internal/checksum/middleware_compute_input_checksum.go

.changelog/9ebe24c4791541e0840da49eab6f9d97.json

aws/config.go

merge main into branch

aws/checksum.go

aws/middleware/user_agent.go

service/internal/checksum/middleware_checksum_metrics_tracking.go

Madrigal · 2024-12-18T16:05:05Z

service/internal/checksum/middleware_checksum_metrics_tracking.go

+
+// ID provides the middleware identifier
+func (m *RequestChecksumMetricsTracking) ID() string {
+	return "AWSChecksum:RequestMetricsTracking"


[observation] we don't follow this pattern of x:yon other middlewares, but I like how you used it here

service/internal/checksum/middleware_checksum_metrics_tracking_test.go

feature/s3/manager/integ_upload_test.go

merge main to branch

merge main into branch

Madrigal · 2025-01-14T18:35:05Z

.changelog/9ebe24c4791541e0840da49eab6f9d97.json

+{
+    "id": "9ebe24c4-7915-41e0-840d-a49eab6f9d97",
+    "type": "feature",
+    "description": "S3 client behavior is updated to always calculate a checksum by default for operations that support it (such as PutObject or UploadPart), or require it (such as DeleteObjects). The checksum algorithm used by default now becomes CRC32. Checksum behavior can be configured using `when_supported` and `when_required` options - in code using RequestChecksumCalculation, in shared config using request_checksum_calculation, or as env variable using AWS_REQUEST_CHECKSUM_CALCULATION. The S3 client attempts to validate response checksums for all S3 API operations that support checksums. However, if the SDK has not implemented the specified checksum algorithm then this validation is skipped. Checksum validation behavior can be configured using `when_supported` and `when_required` options - in code using ResponseChecksumValidation, in shared config using response_checksum_validation, or as env variable using AWS_RESPONSE_CHECKSUM_VALIDATION.",


Suggested change

"description": "S3 client behavior is updated to always calculate a checksum by default for operations that support it (such as PutObject or UploadPart), or require it (such as DeleteObjects). The checksum algorithm used by default now becomes CRC32. Checksum behavior can be configured using `when_supported` and `when_required` options - in code using RequestChecksumCalculation, in shared config using request_checksum_calculation, or as env variable using AWS_REQUEST_CHECKSUM_CALCULATION. The S3 client attempts to validate response checksums for all S3 API operations that support checksums. However, if the SDK has not implemented the specified checksum algorithm then this validation is skipped. Checksum validation behavior can be configured using `when_supported` and `when_required` options - in code using ResponseChecksumValidation, in shared config using response_checksum_validation, or as env variable using AWS_RESPONSE_CHECKSUM_VALIDATION.",

"description": "S3 client behavior is updated to always calculate a checksum by default for operations that support it (such as PutObject or UploadPart), or require it (such as DeleteObjects). The checksum algorithm used by default now becomes CRC32. Checksum behavior can be configured using `when_supported` and `when_required` options in 3 different places - in code using RequestChecksumCalculation, in shared config using request_checksum_calculation, or as env variable using AWS_REQUEST_CHECKSUM_CALCULATION. The S3 client attempts to validate response checksums for all S3 API operations that support checksums. However, if a different checksum algorithm is specified and the SDK has not yet implemented it, this validation is skipped. Checksum validation behavior can be configured using `when_supported` and `when_required` options in 3 places as well - in code using ResponseChecksumValidation, in shared config using response_checksum_validation, or as env variable using AWS_RESPONSE_CHECKSUM_VALIDATION.",

As announced in aws#2960, AWS SDK for Go v2 service/s3 v1.73.0 shipped a change (aws#2808) that adopted the new default integrity protections. While it is possible to revert to the previous behavior by setting `AWS_REQUEST_CHECKSUM_CALCULATION=when_required`, this config setting did not apply to the S3 Manager multipart uploader, which always enabled CRC32 checksums by default. This commit checks the S3 options and only attaches the CRC32 checksums if `when_required` is not set. Relates to aws#3007

As announced in aws#2960, AWS SDK for Go v2 service/s3 v1.73.0 shipped a change (aws#2808) that adopted the new default integrity protections. While it is possible to revert to the previous behavior by setting `AWS_REQUEST_CHECKSUM_CALCULATION=when_required`, this config setting did not apply to the S3 Manager multipart uploader, which always enabled CRC32 checksums by default. This commit checks the S3 options and only attaches the CRC32 checksums if `when_required` is not set. Relates to aws#3007 Signed-off-by: Stan Hu <stanhu@gmail.com>

…3148) As announced in #2960, AWS SDK for Go v2 service/s3 v1.73.0 shipped a change (#2808) that adopted the new default integrity protections. While it is possible to revert to the previous behavior by setting `AWS_REQUEST_CHECKSUM_CALCULATION=when_required`, this config setting did not apply to the S3 Manager multipart uploader, which always enabled CRC32 checksums by default. This commit checks the S3 options and only attaches the CRC32 checksums if `when_required` is not set. Relates to #3007 Signed-off-by: Stan Hu <stanhu@gmail.com>

… default checksums (aws#3148) As announced in aws#2960, AWS SDK for Go v2 service/s3 v1.73.0 shipped a change (aws#2808) that adopted new default integrity protections, automatically calculating CRC32 checksums for operations like PutObject and UploadPart. While it is possible to revert to the previous behavior by setting `AWS_REQUEST_CHECKSUM_CALCULATION=when_required`, this config setting did not apply to the S3 Manager multipart uploader, which always enabled CRC32 checksums by default regardless of the global setting. This commit adds a new `RequestChecksumCalculation` field to the Uploader struct that allows users to control checksum behavior: - `RequestChecksumCalculationWhenSupported` (default): Always calculates CRC32 checksums for multipart uploads - `RequestChecksumCalculationWhenRequired`: Only calculates checksums when explicitly set by the user, preserving backwards compatibility For example: ```go uploader := manager.NewUploader(client, func(u *manager.Uploader) { u.RequestChecksumCalculation = aws.RequestChecksumCalculationWhenRequired }) ``` S3 Express One Zone buckets always require CRC32 checksums regardless of this setting, as mandated by the S3 Express service requirements. The uploader automatically detects S3 Express buckets (names ending with `--x-s3`) and applies CRC32 checksums unconditionally. Fixes aws#3007

As announced in aws#2960, AWS SDK for Go v2 service/s3 v1.73.0 shipped a change (aws#2808) that adopted new default integrity protections, automatically calculating CRC32 checksums for operations like PutObject and UploadPart. While it is possible to revert to the previous behavior by setting `AWS_REQUEST_CHECKSUM_CALCULATION=when_required`, this config setting did not apply to the S3 Manager multipart uploader, which always enabled CRC32 checksums by default regardless of the global setting. This commit adds a new `RequestChecksumCalculation` field to the Uploader struct that allows users to control checksum behavior: - `RequestChecksumCalculationWhenSupported` (default): Always calculates CRC32 checksums for multipart uploads - `RequestChecksumCalculationWhenRequired`: Only calculates checksums when explicitly set by the user, preserving backwards compatibility For example: ```go uploader := manager.NewUploader(client, func(u *manager.Uploader) { u.RequestChecksumCalculation = aws.RequestChecksumCalculationWhenRequired }) ``` S3 Express One Zone buckets always require CRC32 checksums regardless of this setting, as mandated by the S3 Express service requirements. The uploader automatically detects S3 Express buckets (names ending with `--x-s3`) and applies CRC32 checksums unconditionally. Fixes aws#3007

As announced in aws#2960, AWS SDK for Go v2 service/s3 v1.73.0 shipped a change (aws#2808) that adopted new default integrity protections, automatically calculating CRC32 checksums for operations like PutObject and UploadPart. While it is possible to revert to the previous behavior by setting `AWS_REQUEST_CHECKSUM_CALCULATION=when_required`, this config setting did not apply to the S3 Manager multipart uploader, which always enabled CRC32 checksums by default regardless of the global setting. This commit adds a new `RequestChecksumCalculation` field to the Uploader struct that allows users to control checksum behavior: - `RequestChecksumCalculationWhenSupported` (default): Always calculates CRC32 checksums for multipart uploads - `RequestChecksumCalculationWhenRequired`: Only calculates checksums when explicitly set by the user, preserving backwards compatibility For example: ```go uploader := manager.NewUploader(client, func(u *manager.Uploader) { u.RequestChecksumCalculation = aws.RequestChecksumCalculationWhenRequired }) ``` S3 Express One Zone buckets always require CRC32 checksums regardless of this setting, as mandated by the S3 Express service requirements. The uploader automatically detects S3 Express buckets (names ending with `--x-s3`) and applies CRC32 checksums unconditionally. Fixes aws#3007 Signed-off-by: Stan Hu <stanhu@gmail.com>

Tianyi Wang added 8 commits September 25, 2024 17:59

add config and change default behavior of checksum middleware

e89e0c9

merge main into branch

ee0fb5b

regenerate client

0efe635

regenerate clients

bc3f437

merge protocol test code from main

58bf509

merge from main

ab72530

add more test cases

b0a53a4

add changelog

f7b5a2d

wty-Bryant requested a review from a team as a code owner September 26, 2024 03:33

modify s3 internal test

b93f439

lucix-aws reviewed Sep 26, 2024

View reviewed changes

service/internal/checksum/middleware_compute_input_checksum.go Outdated Show resolved Hide resolved

aws/checksum.go Show resolved Hide resolved

aws/checksum.go Outdated Show resolved Hide resolved

lucix-aws reviewed Sep 26, 2024

View reviewed changes

.changelog/9ebe24c4791541e0840da49eab6f9d97.json Outdated Show resolved Hide resolved

lucix-aws reviewed Sep 26, 2024

View reviewed changes

aws/checksum.go Outdated Show resolved Hide resolved

Tianyi Wang added 4 commits September 29, 2024 16:53

separate checksum config check and workflow

72c3ba6

Merge branch 'main' into feat-flex-checksum

f70fb1d

merge main into current branch

restore s3 test

7bc5326

remove unused md5 header

28e8ac4

Madrigal reviewed Sep 30, 2024

View reviewed changes

lucix-aws reviewed Sep 30, 2024

View reviewed changes

Tianyi Wang added 3 commits October 1, 2024 15:25

separate checksum config and workflow

23a7d0e

Merge branch 'main' into feat-flex-checksum

e12cb30

merge main into branch

change default checksum to const

df759d3

Madrigal reviewed Oct 2, 2024

View reviewed changes

add checksum unset enum and modify comment of cfg

93aee16

Madrigal reviewed Oct 7, 2024

View reviewed changes

.changelog/9ebe24c4791541e0840da49eab6f9d97.json Show resolved Hide resolved

aws/config.go Outdated Show resolved Hide resolved

config/load_options.go Show resolved Hide resolved

service/internal/checksum/middleware_compute_input_checksum.go Outdated Show resolved Hide resolved

lucix-aws reviewed Oct 7, 2024

View reviewed changes

.changelog/9ebe24c4791541e0840da49eab6f9d97.json Outdated Show resolved Hide resolved

lucix-aws reviewed Oct 7, 2024

View reviewed changes

aws/config.go Outdated Show resolved Hide resolved

Tianyi Wang added 2 commits October 7, 2024 17:42

change comment

b147f0f

Merge branch 'main' into feat-flex-checksum

db21450

merge main into branch

lucix-aws reviewed Oct 8, 2024

View reviewed changes

aws/checksum.go Outdated Show resolved Hide resolved

Madrigal reviewed Dec 18, 2024

View reviewed changes

Tianyi Wang and others added 4 commits December 18, 2024 16:06

reorder feature id and simplify metric tracking test

25d943c

Merge branch 'main' into feat-flex-checksum

bf57f67

merge main to branch

update changelog

cc9af06

Merge branch 'main' into feat-flex-checksum

8d71128

merge main into branch

lucix-aws changed the title ~~[DO NOT MERGE] Feat flex checksum~~ feat: flexible checksum updates Jan 14, 2025

Madrigal reviewed Jan 14, 2025

View reviewed changes

Madrigal approved these changes Jan 14, 2025

View reviewed changes

wty-Bryant merged commit 6636822 into main Jan 14, 2025

wty-Bryant deleted the feat-flex-checksum branch January 14, 2025 22:15

janisz mentioned this pull request Jan 17, 2025

chore(deps): bump github.com/aws/aws-sdk-go-v2/config from 1.28.11 to 1.29.0 stackrox/stackrox#13835

Closed

op mentioned this pull request Mar 7, 2025

s3: Fails to upload because of checksum issue rclone/rclone#8432

Closed

yottta mentioned this pull request Mar 13, 2025

New AWS region "mx-central-1" for the S3 backend opentofu/opentofu#2570

Closed

stanhu mentioned this pull request Jul 21, 2025

[reverted] Make it possible to disable integrity checks for multipart uploads #3148

Merged

stanhu mentioned this pull request Jul 25, 2025

feature(s3/manager): add option to control default checksums #3151

Merged

Conversation

wty-Bryant commented Sep 26, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Madrigal Dec 18, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Madrigal Jan 14, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants