Conversation
Integrity check enforcement was enabled in aws-sdk-go-v2 by default (see aws/aws-sdk-go-v2#2960). This leads to inability to use vmbackup, vmrestore and vmbackupmanager with 3rd party providers which do not support this option. Currently, it is possible to use `AWS_REQUEST_CHECKSUM_CALCULATION=when_required` and `AWS_RESPONSE_CHECKSUM_VALIDATION=when_required` environment variables to restore previous behaviour but it requires a manual intervention from users' side. This change disables this check by default to restore previous behaviour of aws-sdk-go-v2. Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
|
@zekker6 @f41gh7 In my opinion the proposed changes will not resolve the issue. The upload logic decides if there is a need to use a "multipart" mode |
|
@wwojdanowski Did you check the variables which are used in the latest comment here? (note that these have different value compared to earlier reply) Also, this config should be propagated to uploader from S3 client as uploader is created with client base here. Here is a vmbackup image build based on this PR, you could try that to verify if this changes works for you: victoriametrics/vmbackup:heads-gh-8622-0-g52aed046a9 |
|
@zekker6 ok I'll try later with this image, thanks |
|
Hi @zekker6 unfortunately it fails just like before, on a multipart upload
|
|
@wwojdanowski Thank you for testing, I'll try to replicate this behaviour by using some larger chunks of data against Linode S3. |
|
Closing in favor of #8642 as disabling integrity checks does not resolve issues with some of S3-compatible storage providers. Even with integrity checks disabled AWS SDK will still try to verify SHA256 checksums which will fail with some S3-compatible storage providers. |
Pin AWS libraries to version before 2025-01-15 (see https://github.com/aws/aws-sdk-go-v2/releases/tag/release-2025-01-15). This version enabled request and response checksum verification by default which breaks compatibility with non-AWS S3-compatible storage providers. See: #8622 Supersedes #8630 --------- Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Pin AWS libraries to version before 2025-01-15 (see https://github.com/aws/aws-sdk-go-v2/releases/tag/release-2025-01-15). This version enabled request and response checksum verification by default which breaks compatibility with non-AWS S3-compatible storage providers. See: #8622 Supersedes #8630 --------- Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Pin AWS libraries to version before 2025-01-15 (see https://github.com/aws/aws-sdk-go-v2/releases/tag/release-2025-01-15). This version enabled request and response checksum verification by default which breaks compatibility with non-AWS S3-compatible storage providers. See: VictoriaMetrics#8622 Supersedes VictoriaMetrics#8630 --------- Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Pin AWS libraries to version before 2025-01-15 (see https://github.com/aws/aws-sdk-go-v2/releases/tag/release-2025-01-15). This version enabled request and response checksum verification by default which breaks compatibility with non-AWS S3-compatible storage providers. See: VictoriaMetrics#8622 Supersedes VictoriaMetrics#8630 --------- Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
…9844) Updates: - unpin AWS dependencies and run `make vendor-update` - add config options to enable checksums only if required by storage in order to preserve backwards compatibility Related issues: - #9748 - #8622 Tested with: AWS S3, self-hosted MinIO, Linode object storage as it was failing previously with multi-part uploads (reported here - #8630 (comment)). An updated library allows (PR with the fix - aws/aws-sdk-go-v2#3151) overriding multi-part upload configurations so that compatibility can be preserved. Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
…9844) Updates: - unpin AWS dependencies and run `make vendor-update` - add config options to enable checksums only if required by storage in order to preserve backwards compatibility Related issues: - #9748 - #8622 Tested with: AWS S3, self-hosted MinIO, Linode object storage as it was failing previously with multi-part uploads (reported here - #8630 (comment)). An updated library allows (PR with the fix - aws/aws-sdk-go-v2#3151) overriding multi-part upload configurations so that compatibility can be preserved. Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Integrity check enforcement was enabled in aws-sdk-go-v2 by default (see aws/aws-sdk-go-v2#2960). This leads to inability to use vmbackup, vmrestore and vmbackupmanager with 3rd party providers which do not support this option.
Currently, it is possible to use
AWS_REQUEST_CHECKSUM_CALCULATION=when_requiredandAWS_RESPONSE_CHECKSUM_VALIDATION=when_requiredenvironment variables to restore previous behaviour but it requires a manual intervention from users' side.This change disables this check by default to restore previous behaviour of aws-sdk-go-v2.
See: #8622