Skip to content

unpin botocore, disable failing sigv2/checksum tests#619

Merged
cbodley merged 9 commits intoceph:masterfrom
cbodley:wip-unpin-botocore
Feb 19, 2025
Merged

unpin botocore, disable failing sigv2/checksum tests#619
cbodley merged 9 commits intoceph:masterfrom
cbodley:wip-unpin-botocore

Conversation

@cbodley
Copy link
Copy Markdown
Contributor

@cbodley cbodley commented Feb 12, 2025

unpin botocore version and work through failures

disable failing sigv2 tests:

FAILED s3tests_boto3/functional/test_s3.py::test_cors_presigned_get_object_v2 - botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the CreateBucket operation: None                                              
FAILED s3tests_boto3/functional/test_s3.py::test_cors_presigned_get_object_tenant_v2 - botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the CreateBucket operation: None
FAILED s3tests_boto3/functional/test_s3.py::test_cors_presigned_put_object_v2 - botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the CreateBucket operation: None                                              
FAILED s3tests_boto3/functional/test_s3.py::test_cors_presigned_put_object_tenant_v2 - botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the CreateBucket operation: None

resolve tenanted bucket name failures:

FAILED s3tests_boto3/functional/test_s3.py::test_bucket_policy_different_tenant - KeyError: 'bucket'
FAILED s3tests_boto3/functional/test_s3.py::test_bucket_policy_tenanted_bucket - KeyError: 'bucket'

still seeing several checksum-related failures:

FAILED s3tests_boto3/functional/test_s3.py::test_multipart_copy_small - botocore.exceptions.FlexibleChecksumError: Expected checksum 4AAr8A== did not match calculated checksum: WbxXZw==
FAILED s3tests_boto3/functional/test_s3.py::test_multipart_copy_special_names - botocore.exceptions.FlexibleChecksumError: Expected checksum g2eXRw== did not match calculated checksum: qgUmLw==
FAILED s3tests_boto3/functional/test_s3.py::test_multipart_copy_multiple_sizes - botocore.exceptions.FlexibleChecksumError: Expected checksum R9Y9Og== did not match calculated checksum: WRq9Ww==
FAILED s3tests_boto3/functional/test_s3.py::test_ranged_request_response_code - botocore.exceptions.FlexibleChecksumError: Expected checksum kC/QBA== did not match calculated checksum: 50r44A==
FAILED s3tests_boto3/functional/test_s3.py::test_ranged_big_request_response_code - botocore.exceptions.FlexibleChecksumError: Expected checksum oDTwIA== did not match calculated checksum: JW3gew==
FAILED s3tests_boto3/functional/test_s3.py::test_ranged_request_skip_leading_bytes_response_code - botocore.exceptions.FlexibleChecksumError: Expected checksum kC/QBA== did not match calculated checksum: /sUwqQ==
FAILED s3tests_boto3/functional/test_s3.py::test_ranged_request_return_trailing_bytes_response_code - botocore.exceptions.FlexibleChecksumError: Expected checksum kC/QBA== did not match calculated checksum: /sUwqQ==
FAILED s3tests_boto3/functional/test_s3.py::test_multipart_checksum_upload_fallback - botocore.errorfactory.InvalidRequest: An error occurred (InvalidRequest) when calling the CompleteMultipartUpload operation: None

Signed-off-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>
the botocore.handlers.validate_bucket_name validation is enabled by
default, but we can disable it with unregister(). this avoids having to
mess with the signature or url

Signed-off-by: Casey Bodley <cbodley@redhat.com>
…upload_fallback

the test sends create_multipart_upload() with a requested checksum
algorithm, and upload_part() requests without. newer boto automatically
adds `x-amz-sdk-checksum-algorithm: CRC32` when nothing is specified,
so we have to explicitly disable that behavior via botocore config

Signed-off-by: Casey Bodley <cbodley@redhat.com>
boto update exposed some bugs in our checksum feature around multipart
copy and ranged requests. disabling them with fails_on_rgw until
https://tracker.ceph.com/issues/69936 is resolved

Signed-off-by: Casey Bodley <cbodley@redhat.com>
> botocore.exceptions.ParamValidationError: Parameter validation failed:
> Unknown parameter in input: "Bucket", must be one of: MaxBuckets, ContinuationToken, Prefix, BucketRegion

Signed-off-by: Casey Bodley <cbodley@redhat.com>
…ngth_negative

if tls is enabled, boto will switch to STREAMING-UNSIGNED-PAYLOAD-TRAILER
and omit the provided content-length header. this leads to test failure:

> ________________ test_object_create_bad_contentlength_negative _________________
>       e = assert_raises(ClientError, client.put_object, Bucket=bucket_name, Key=key_name, ContentLength=-1)
> AssertionError: ClientError not raised

Signed-off-by: Casey Bodley <cbodley@redhat.com>
work around sns failures due to default region_name='':

> ERROR s3tests_boto3/functional/test_sns.py::test_account_topic_publish - botocore.exceptions.EndpointResolutionError: Invalid region: region was not a valid DNS name.
> ERROR s3tests_boto3/functional/test_sns.py::test_cross_account_topic_publish - botocore.exceptions.EndpointResolutionError: Invalid region: region was not a valid DNS name.

Signed-off-by: Casey Bodley <cbodley@redhat.com>
@cbodley cbodley marked this pull request as ready for review February 17, 2025 16:38
@cbodley
Copy link
Copy Markdown
Contributor Author

cbodley commented Feb 17, 2025

i got a clean run of this branch against ceph main at https://pulpito.ceph.com/cbodley-2025-02-14_20:34:11-rgw-main-distro-default-smithi/

@cbodley cbodley changed the title unpin botocore, disable failing sigv2 tests unpin botocore, disable failing sigv2/checksum tests Feb 17, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants