rgw: implement RestrictPublicBuckets from PublicAccessBlock#57206
Merged
rgw: implement RestrictPublicBuckets from PublicAccessBlock#57206
Conversation
clwluvw
added a commit
to clwluvw/s3-tests
that referenced
this pull request
May 1, 2024
Ref. ceph/ceph#57206 Signed-off-by: Seena Fallah <seenafallah@gmail.com>
d47e8b4 to
007466d
Compare
|
This pull request has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs for another 30 days. |
Member
Author
|
Hi @cbodley, Can you please take a look at this? |
|
This pull request has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs for another 30 days. |
Member
Author
|
unstale me, please. |
adamemerson
approved these changes
Oct 17, 2024
clwluvw
added a commit
to clwluvw/s3-tests
that referenced
this pull request
Oct 17, 2024
Ref. ceph/ceph#57206 Signed-off-by: Seena Fallah <seenafallah@gmail.com>
Member
Author
|
Tests available at ceph/s3-tests#563 |
Member
Author
|
jenkins test api |
Contributor
|
thank you @clwluvw ! |
|
This pull request has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs for another 30 days. |
Member
Author
|
jenkins test api |
Member
Author
|
jenkins test make check arm64 |
Member
Author
|
jenkins test api |
cbodley
reviewed
Dec 17, 2024
Comment on lines
+1132
to
+1134
| // If RestrictPublicBuckets is enabled and the policy allows public access, deny the request if the requester is not in the bucket owner account | ||
| if (restrict_public_buckets && rgw::IAM::is_public(*policy) && !id->is_owner_of(bucket_owner)) { | ||
| return Effect::Deny; |
Contributor
There was a problem hiding this comment.
adding this to the common function eval_or_pass() means it applies to every kind of policy, including identity and session policies
looking at PutPublicAccessBlock, BlockPublicPolicy and RestrictPublicBuckets both specify bucket policy:
Specifies whether Amazon S3 should restrict public bucket policies for this bucket. Setting this element to TRUE restricts access to this bucket to only AWS service principals and authorized users within this account if the bucket has a public policy.
bucket policy is a type of "resource policy", distinct from identity and session policy. do you see any other docs that talk about identity/session policies?
to make this specific to bucket policy, i think you can just add checks for bucket_policy in verify_bucket_permission() and verify_object_permission()
Member
Author
There was a problem hiding this comment.
right, I guess you should not be able to create any identity/session policies with Principal * at all. so rgw::IAM::is_public(*policy) should never be true?
Member
Author
There was a problem hiding this comment.
I have moved this out to evaluate_iam_policies() in respect to only being considered by resource_policy anyway as you said, that doesn't look like the right place to be called.
I'm unsure about verify_topic_permission() and whether we should consider it or not.
What do you think, should still get it exported further to verify_bucket_permission() and verify_object_permission() explicitly?
Contributor
There was a problem hiding this comment.
i'm not sure whether we're strict about Principal in those policies, but yeah
Contributor
|
could you please revert the rest of the changes? and i still think these checks belong in verify_bucket_permission() and verify_object_permission() rather than the common evaluate_iam_policies() those methods should be able to perform this check before their calls to evaluate_iam_policies() |
53e1bdb to
14184fc
Compare
cbodley
approved these changes
Dec 17, 2024
Contributor
cbodley
left a comment
There was a problem hiding this comment.
thanks, looks great. sorry for the late review
Comment on lines
+1353
to
+1354
| ldpp_dout(dpp, 10) << __func__ << ": public policies are blocked by the BlockPublicPolicy block public access setting" << dendl; | ||
| s->err.message = "public policies are blocked by the BlockPublicPolicy block public access setting."; |
Contributor
There was a problem hiding this comment.
is BlockPublicPolicy correct here, or should this say RestrictPublicBuckets instead?
Member
Author
There was a problem hiding this comment.
I'm afraid we cannot say anything due to having struct perm_state_base * const s,. or is there any other way of passing the message back?
Contributor
There was a problem hiding this comment.
oh right, RGWUserPermHandler in rgw_data_sync.cc needs to be able to call this without a req_state. i guess you could pass an optional std::string* message for the callers that have one
Member
Author
There was a problem hiding this comment.
I was thinking of a more generic solution to that due to (https://aws.amazon.com/about-aws/whats-new/2024/08/amazon-s3-http-403-access-denied-error-messages/). Maybe worth doing it in another PR for all type of errors caused by verify_*_permission() and perhaps integrated to perm_state_base?
Contributor
There was a problem hiding this comment.
i know @adamemerson has ideas to improve the client-facing diagnostics for iam policy. i certainly think it could make sense to revisit as part of that work
Contributor
There was a problem hiding this comment.
I was thinking of a more generic solution to that due to (https://aws.amazon.com/about-aws/whats-new/2024/08/amazon-s3-http-403-access-denied-error-messages/). Maybe worth doing it in another PR for all type of errors caused by
verify_*_permission()and perhaps integrated toperm_state_base?
I'd be in favor of it. What I'm focusing on at the moment is just extending what I did with Principals and the policy checker to resources, and see what else we could be more strict about. I'll be starting tomorrow and we could improve 403 as part of that?
200c44a to
fbcfc42
Compare
Member
Author
|
rebased due to the conflict in the PendingReleaseNotes file. |
Member
Author
|
jenkins test api |
|
This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved |
fbcfc42 to
5118516
Compare
|
This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved |
5118516 to
3bec7ae
Compare
|
This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved |
According to the AWS docs (https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-control-block-public-access.html) If RestrictPublicBuckets is set to true and there are public policies in place, Only access from the bucket owner's account is possible. Fixes: https://tracker.ceph.com/issues/65741 Signed-off-by: Seena Fallah <seenafallah@gmail.com>
3bec7ae to
07ad231
Compare
Member
Author
|
jenkins test make check arm64 |
cbodley
pushed a commit
to cbodley/s3-tests
that referenced
this pull request
Apr 8, 2025
Ref. ceph/ceph#57206 Signed-off-by: Seena Fallah <seenafallah@gmail.com>
Contributor
|
https://jenkins.ceph.com/job/ceph-pull-requests-arm64/71766/
commented in https://tracker.ceph.com/issues/70700 |
Contributor
|
jenkins test make check arm64 |
Contributor
|
https://jenkins.ceph.com/job/ceph-pull-requests-arm64/71867/
|
Contributor
|
jenkins test make check arm64 |
Contributor
|
passed qa in https://pulpito.ceph.com/cbodley-2025-04-08_22:25:36-rgw-wip-65741-distro-default-smithi/
|
cbodley
pushed a commit
to ceph/s3-tests
that referenced
this pull request
Apr 9, 2025
Ref. ceph/ceph#57206 Signed-off-by: Seena Fallah <seenafallah@gmail.com> (cherry picked from commit e4285b0)
cwickramanayake-dotcom
added a commit
to splunk/s3-tests
that referenced
this pull request
Jan 28, 2026
* add tests for ObjectSizeGreater(Less)Than Add tests for the new ObjectSizeGreaterThan and ObjectSizeLessThan lifecycle operators. Signed-off-by: Matt Benjamin <mbenjamin@redhat.com> * test etag on mpu complete replies this is to cover the fix of: https://tracker.ceph.com/issues/58879 Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com> * a change is the RGW error-response require changes in s3-tests Signed-off-by: Gal Salomon <gal.salomon@gmail.com> * add handling for EventStreamError exception Signed-off-by: Gal Salomon <gal.salomon@gmail.com> * Fix wrong assertion of the test: `test_buckets_list_ctime` TestName: s3tests_boto3.functional.test_s3:test_buckets_list_ctime Problem: The test creates 5 buckets for a user but in an assertion check, it asserts false if any bucket of the user has CreationTime less than a day prior to current time. Due to this reason the test fails if the user has pre-existing buckets older than a day. Solution: Assert only on the CreationTime of buckets that were created with test execution. Signed-off-by: Sumedh A. Kulkarni <sumedh.a.kulkarni@seagate.com> Co-developed-by: Bob Ham <bham12@bloomberg.net> Signed-off-by: Bob Ham <bham12@bloomberg.net> * requirements: unpin pytz version Fixes: https://tracker.ceph.com/issues/66655 Signed-off-by: Casey Bodley <cbodley@redhat.com> * test Get/HeadObject with partNumber for single-multipart upload test_multipart_get_part() tests 'normal' multipart uploads. add a new test case for a multipart upload with a single part to tests the fix for https://tracker.ceph.com/issues/66705 Signed-off-by: Casey Bodley <cbodley@redhat.com> * add "checksum" marker, since new checksum tests reference it this removes a Pytest warning during execution Signed-off-by: Matt Benjamin <mbenjamin@redhat.com> * test_multipart_upload_sha256: work around failures re-trying complete-multipart As described in https://tracker.ceph.com/issues/65746, retrying complete-multipart after having attempted to complete the same upload with a bad checksum argument fails with an internal error. The status code is 500, but I'm unsure if it can be retried again, or whether the upload can be aborted later. Signed-off-by: Matt Benjamin <mbenjamin@redhat.com> * add test_multipart_checksum_3parts tests a full multipart upload cycle with 3 unique parts, which verifies composite checksum computation and the logic to propagate parts_count to ComleteMultipart Signed-off-by: Matt Benjamin <mbenjamin@redhat.com> * add test_post_object_upload_checksum this tests a two-megabyte binary upload with validated (awscli-computed) SHA256 checksum, and also verifies failure when a bad checksum is provided Signed-off-by: Matt Benjamin <mbenjamin@redhat.com> * remove duplicate size assigment [rkhudov review] Signed-off-by: Matt Benjamin <mbenjamin@redhat.com> * mark two tests that fail on dbstore also add @pytest.mark.checksum for new checksum tests Signed-off-by: Matt Benjamin <mbenjamin@redhat.com> * s3select: align error codes with the new AWS format ref. ceph/ceph#56864 Signed-off-by: Seena Fallah <seenafallah@gmail.com> * BucketPolicy: donot allow NotPrincipal with Allow Effect Ref. ceph/ceph#58686 Signed-off-by: Seena Fallah <seenafallah@gmail.com> * PolicyStatus: add test for policy with Principal Ref. ceph/ceph#58686 Signed-off-by: Seena Fallah <seenafallah@gmail.com> * BlockPublicPolicy: add test when policy has principal Ref. https://tracker.ceph.com/issues/67048 Signed-off-by: Seena Fallah <seenafallah@gmail.com> * BucketPolicy: add test for sse-c in conditions Ref. ceph/ceph#58689 Signed-off-by: Seena Fallah <seenafallah@gmail.com> * BucketPolicy: decouple encryption tests from invalid algo and unencrypted Signed-off-by: Seena Fallah <seenafallah@gmail.com> * s3: reenable tenanted bucket policy test the before-call hook url-encodes the ':' part of tenanted bucket names to resolve SignatureDoesNotMatch errors removed the list-v2 version of the test since it isn't relevant to bucket policy test coverage add a new test case that creates the bucket under the tenanted user, then uses the main client to access it Signed-off-by: Casey Bodley <cbodley@redhat.com> * UploadPartCopy: add test for source bucket with policy Ref: ceph/ceph#59253 Signed-off-by: Seena Fallah <seenafallah@gmail.com> * rgw: adding tests for add_client_id_to_oidc_provider and update_thumbprint_for oidc_provider. Signed-off-by: Pritha Srivastava <prsrivas@redhat.com> * fix the assert per empty results Signed-off-by: Gal Salomon <gal.salomon@gmail.com> * fix comments Signed-off-by: galsalomon66 <gal.salomon@gmail.com> * Add v2 signature presigned get_object tests This adds tests for get_object presigned URLs using signature v2. Also code formatting. Signed-off-by: Tobias Urdin <tobias.urdin@binero.com> * s3: add v2 signature presigned put_object tests Signed-off-by: Casey Bodley <cbodley@redhat.com> * s3: test GetObject with PartNumber and SSE-C encryption Signed-off-by: Casey Bodley <cbodley@redhat.com> * Updating lifecycle_noncur* tests to include objects with instance empty Signed-off-by: Soumya Koduri <skoduri@redhat.com> * BucketPublicAccessBlock: add tests for RestrictPublicBuckets Ref. ceph/ceph#57206 Signed-off-by: Seena Fallah <seenafallah@gmail.com> * rgw/s3-test: adding test to set, get, delete and then get a bucket policy to ensure that delete correctly removes the bucket policy Signed-off-by: Pritha Srivastava <prsrivas@redhat.com> * rgw/s3-tests: adding a test to check put, get, delete and then get to ensure that public block access is deleted successfully. Signed-off-by: Pritha Srivastava <prsrivas@redhat.com> * rgw/s3tests: making the tests generic to check only for 403 error code (and not specific errors) so that the tests pass for any order of auth engines. Signed-off-by: Pritha Srivastava <prsrivas@redhat.com> * rgw:modify testcase rgw header when bucket quota or user is disabled By default quota is disable at user/bucket level. if quota is disable then these value will not return so need to modify test_head_bucket_usages 'X-RGW-Quota-User-Size' 'X-RGW-Quota-User-Objects' 'X-RGW-Quota-Bucket-Size' 'X-RGW-Quota-Bucket-Objects' to Fix: https://tracker.ceph.com/issues/68211 Signed-off-by: Raja Sharma raja@ibm.com * sts: remove test_get_session_token_permanent_creds_denied Fixes: https://tracker.ceph.com/issues/69001 Signed-off-by: Casey Bodley <cbodley@redhat.com> * rgw/logging: basic set of tests for bucket logging Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com> * rgw/logging: tests for flush API as well as test for object key filters Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com> * HeadBucket: add read-stats query string for usage test Refs: ceph/ceph#56756 Signed-off-by: Seena Fallah <seenafallah@gmail.com> * test get_object_attributes Signed-off-by: Matt Benjamin <mbenjamin@redhat.com> * multipart fallback to create-multipart checksum algorithm there seem to be workloads which assume checksum algorithm can be omitted from upload-part Signed-off-by: Matt Benjamin <mbenjamin@redhat.com> * more tests for GetObjectAttributes * multipart upload without checksums * multipart upload with a single part * pagination of multipart parts * non-multipart upload with/without checksum * versioned object, current and non-current * sse-c encrypted object Signed-off-by: Casey Bodley <cbodley@redhat.com> Signed-off-by: Matt Benjamin <mbenjamin@redhat.com> * mark attribute tests as failing on dbstore (for now) Signed-off-by: Matt Benjamin <mbenjamin@redhat.com> * rgw/logging: test tenant support when source and log buckets are on a specific tenant, or on different tenants Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com> * rgw/logging: test source and log have different names Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com> * rgw/logging: test log bucket encryption error Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com> * rgw/logging: add cleanup tests * use flushing api instead of sleep when possible * test flushing concurrency Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com> * rgw/logging: fix typo in readme Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com> * rgw test-case to support x-expected-bucket-owner If the bucket is owned by a different account, the request fails with the HTTP status code 403 Forbidden (access denied). PR: ceph/ceph#61215 Fixes: https://tracker.ceph.com/issues/64526 Signed-off-by: Raja Sharma <raja@ibm.com> * test-case to fix obj by multipart upload cant get tag Signed-off-by: Raja Sharma <raja@ibm.com> * add test case for stacked delete-markers Signed-off-by: Jane Zhu <jzhu116@bloomberg.net> * rgw|BN: fix test to expect authorization failure Signed-off-by: Ali Masarwa <ali.saed.masarwa@gmail.com> * requirements: unpin botocore version Signed-off-by: Casey Bodley <cbodley@redhat.com> * s3: mark sigv2 tests with fails_on_rgw Signed-off-by: Casey Bodley <cbodley@redhat.com> * s3: sse-kms policy tests use v4 client Signed-off-by: Casey Bodley <cbodley@redhat.com> * s3: clean up use of tenanted bucket names the botocore.handlers.validate_bucket_name validation is enabled by default, but we can disable it with unregister(). this avoids having to mess with the signature or url Signed-off-by: Casey Bodley <cbodley@redhat.com> * s3: disable request_checksum_calculation for test_multipart_checksum_upload_fallback the test sends create_multipart_upload() with a requested checksum algorithm, and upload_part() requests without. newer boto automatically adds `x-amz-sdk-checksum-algorithm: CRC32` when nothing is specified, so we have to explicitly disable that behavior via botocore config Signed-off-by: Casey Bodley <cbodley@redhat.com> * s3: disable multipart/ranged tests with checksum failures boto update exposed some bugs in our checksum feature around multipart copy and ranged requests. disabling them with fails_on_rgw until https://tracker.ceph.com/issues/69936 is resolved Signed-off-by: Casey Bodley <cbodley@redhat.com> * iam: remove invalid Bucket param from list_buckets() > botocore.exceptions.ParamValidationError: Parameter validation failed: > Unknown parameter in input: "Bucket", must be one of: MaxBuckets, ContinuationToken, Prefix, BucketRegion Signed-off-by: Casey Bodley <cbodley@redhat.com> * s3: disable checksum calculation for test_object_create_bad_contentlength_negative if tls is enabled, boto will switch to STREAMING-UNSIGNED-PAYLOAD-TRAILER and omit the provided content-length header. this leads to test failure: > ________________ test_object_create_bad_contentlength_negative _________________ > e = assert_raises(ClientError, client.put_object, Bucket=bucket_name, Key=key_name, ContentLength=-1) > AssertionError: ClientError not raised Signed-off-by: Casey Bodley <cbodley@redhat.com> * sns: clear region_name for s3 clients work around sns failures due to default region_name='': > ERROR s3tests_boto3/functional/test_sns.py::test_account_topic_publish - botocore.exceptions.EndpointResolutionError: Invalid region: region was not a valid DNS name. > ERROR s3tests_boto3/functional/test_sns.py::test_cross_account_topic_publish - botocore.exceptions.EndpointResolutionError: Invalid region: region was not a valid DNS name. Signed-off-by: Casey Bodley <cbodley@redhat.com> * rgw/iam: add RemoveClientIDFromOpenIDConnectProvider Signed-off-by: Raja Sharma <raja@ibm.com> * s3: reenable tests with range requests and checksums Signed-off-by: Casey Bodley <cbodley@redhat.com> * rgw/restore: s3tests to test restore object functionality. This tests are added to tests temporary restore, permanent restore and read through resotre object functionality, this includes zonegroup parameters and checks. Signed-off-by: shreyanshjain7174 <ssanchet@redhat.com> Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com> * s3: test that object lock can be enabled on an existing bucket Signed-off-by: Casey Bodley <cbodley@redhat.com> * running cloud restore tests seperately Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com> * rgw/s3tests: removing fails_on_rgw for test_versioned_concurrent_object_create_concurrent_remove as the associated tracker is marked as resolved and the test also passes against main. Signed-off-by: Pritha Srivastava <prsrivas@redhat.com> * s3: test bucket policy evaluation for CreateMultipartUpload test case for https://tracker.ceph.com/issues/70191 Signed-off-by: Casey Bodley <cbodley@redhat.com> * mark test_versioning_stack_delete_merkers fails_on_dbstore Signed-off-by: Jane Zhu <jzhu116@bloomberg.net> * cloud-restore: testcase for non-current versioned object Signed-off-by: Soumya Koduri <skoduri@redhat.com> * rgw/s3tests: handle allow read through values in cloud transitions cases Check the allow_read_through values for cases test_lifecycle_cloud_transition and test_read_through Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com> * BucketPolicy: add tests for ConfirmRemoveSelfBucketAccess header Refrence: ceph/ceph#57629 Signed-off-by: Seena Fallah <seenafallah@gmail.com> * rgw/logging: add missing marker to remove warnings Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com> * test put_bucket_acl with group uri Signed-off-by: Casey Bodley <cbodley@redhat.com> * rgw/logging: add basic policy to target bucket this commit needed to be able to run bucket logging regression against: ceph/ceph#62284 since target bucket requires policy for bucket logging to work this only covers the positive cases from bucket logging policy perspective Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com> * rgw/logging: add more bucket logging error cases this is to cover these checks: * source bucket ownership * "requester pays" on log bucket that were added in: ceph/ceph#62284 Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com> * rgw/logging: add bucket logging permission tests this is to cover new functionality added in: ceph/ceph#62284 Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com> * add minimal put-object for CRC64NVME Signed-off-by: Matt Benjamin <mbenjamin@redhat.com> * enhance additional checksum tests includes tests for CRC64NVME, tests for selecting COMPOSITE and FULL_OBJECT checksums a decomposed matrix of tests for all checksum types also removes the mixed checksum upload case that no longer works in recent boto3 cleanups, add sha1 checksum validation failure (mismatch) returns BadDigest multipart checksum matrix helper now validates checksum and checksum type for all operations which can return them (complete-multipart, head-object, and get-object-attributes) Signed-off-by: Matt Benjamin <mbenjamin@redhat.com> * rgw/logging: fix tenant test tests were failing in teuthology since the tenanted user name already have the tenant in its name Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com> * rgw/logging: test bucket logging with accounts Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com> * rgw/cloud-transition: increase time out for test cases Fixes: https://tracker.ceph.com/issues/69411 Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com> * rgw/logging: mark test that require rollover time set Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com> * add test coverage for DeleteBucketLifecycle Signed-off-by: Casey Bodley <cbodley@redhat.com> * rgw/logging: add tests for flushed object name when using API call to flush the pending logging object of the bucket, the flshed object name is returned. tests were added to verify the right name is returned. Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com> * rgw/logging: add tests to verify wildcrad policy Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com> * Tag copy tests * rgw/logging: test object metadata changes in journal mode Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com> * rgw/logging: test mpu complete in standard mode Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com> * rgw/logging: test log object name order verify that the names of the log objects in the log bucket are lexicographically ordered Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com> * rgw/logging: test part upload in mpu in standard mode Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com> * s3: test retry after error instead of 100-continue Signed-off-by: Casey Bodley <cbodley@redhat.com> * s3: avoid 100-continue bug for test_encryption_sse_c_deny_algo_with_bucket_policy was failing for a reason unrelated to sse-c or bucket policy Fixes: https://tracker.ceph.com/issues/70607 Signed-off-by: Casey Bodley <cbodley@redhat.com> * s3: Add tests for test_upload_part_copy_percent_encoded_key * s3: fix: Changed test so that it accepts the character that has % sign in it as a key * add missing 'assert' to test_bucket_logging_roll_time saw a failure after this: ``` response = client.list_objects_v2(Bucket=log_bucket_name) keys = _get_keys(response) len(keys) == 1 > key = keys[0] E IndexError: list index out of range s3tests_boto3/functional/test_s3.py:16637: IndexError ``` Signed-off-by: Casey Bodley <cbodley@redhat.com> * website: use pytest parametrize for test_routing_generator() resolves error with recent pytest: ==================================== ERRORS ==================================== ____________ ERROR collecting s3tests/functional/test_s3_website.py ____________ 'yield' keyword is allowed in fixtures, but not in tests (test_routing_generator) use https://docs.pytest.org/en/7.1.x/example/parametrize.html instead to generate a test_routing_generator() for each entry in ROUTING_RULES_TESTS Signed-off-by: Casey Bodley <cbodley@redhat.com> * website: routing_setup() as pytest fixture c80e9d2 had removed the use of routing_setup() without adding a fixture to replace it. this had broken routing_check(), but we hadn't noticed it because pytest hadn't supported the 'yield' pattern in test_routing_generator() to call it test_routing_generator() now uses this routing_setup fixture to provide the kwargs that had previously been injected into routing_check() via @common.with_setup_kwargs(setup=routing_setup, teardown=routing_teardown) Signed-off-by: Casey Bodley <cbodley@redhat.com> * increase time out for restore op in test_restore_object_permanent and in test_lifecycle_expiration_size_gt() for lc. Fixes: https://tracker.ceph.com/issues/71348 Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com> * fix: Fix typo Signed-off-by: Emin <emin.sunacoglu@clyso.com> * rgw/logging: test conf last modified time Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com> * rgw/logging: add tests for logging object key format Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com> * rgw/sts: GetAccountSummary test Signed-off-by: Raja Sharma <raja@ibm.com> * readme: fix typo Fixed typos in the example commands. Signed-off-by: N Balachandran <nithya.balachandran@ibm.com> * s3: test_block_public_object_canned_acls() tests that 'private' works Fixes: https://tracker.ceph.com/issues/49135 Signed-off-by: Casey Bodley <cbodley@redhat.com> * bucket-logging: tests for the ACLRequired field Tests that the AclRequired field is set correctly in the bucket logging record. Signed-off-by: N Balachandran <nithya.balachandran@ibm.com> * bucket-logging: fix typo Fixes typo in function name. Signed-off-by: N Balachandran <nithya.balachandran@ibm.com> * bucket-logging: test the AuthType log field Adds a test to check the value of the AuthType field. Signed-off-by: N Balachandran <nithya.balachandran@ibm.com> * rgw/sts: GetCallerIndentity test Signed-off-by: Raja Sharma <raja@ibm.com> * delete, rgwlc: delete marker tests 0. non-creation in non-versioned buckets 1. creation in versioned buckets 2. creation in versioning-suspended buckets 3. delete-marker expiration (lifecycle) Signed-off-by: Matt Benjamin <mbenjamin@redhat.com> * bucket_logging: remove duplicate function Removes the duplicate _verify_record_field function definition. Signed-off-by: N Balachandran <nithya.balachandran@ibm.com> * s3: test put_object() with IfMatch and IfNoneMatch Signed-off-by: Casey Bodley <cbodley@redhat.com> * s3: test delete_object() with IfMatch Signed-off-by: Casey Bodley <cbodley@redhat.com> * s3: test delete_object() with IfMatchLastModifiedTime Signed-off-by: Casey Bodley <cbodley@redhat.com> * s3: test delete_object() with IfMatchSize Signed-off-by: Casey Bodley <cbodley@redhat.com> * s3: test delete_objects() with ETag Signed-off-by: Casey Bodley <cbodley@redhat.com> * s3: test delete_objects() with LastModifiedTime Signed-off-by: Casey Bodley <cbodley@redhat.com> * s3: test delete_objects() with Size Signed-off-by: Casey Bodley <cbodley@redhat.com> * s3: fixed tests we expected an error if we delete non-existent objects but that is not true Signed-off-by: Ali Masarwa <ali.saed.masarwa@gmail.com> * s3: more testing for conditional put Signed-off-by: Ali Masarwa <ali.saed.masarwa@gmail.com> * a fix for empty results.for some of the queries an empty result that may rarely happen, could cause an assertion. the fix handle that cases. Signed-off-by: gal salomon <gal.salomon@gmail.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * rgw/cloud-restore: verify if the expiry-date is updated This is the test-case for ceph/ceph#64804. If the `restoe-object` request is re-issued on a cloud-restored object, it should update restoration period accordingly from the time the request is re-issued. Signed-off-by: Soumya Koduri <skoduri@redhat.com> * s3: test list_buckets with MaxBuckets/ContinuationToken Signed-off-by: Casey Bodley <cbodley@redhat.com> * s3: test for aws-chunked in Content-Encoding responses aws-chunked should not be persisted on upload for return on download Signed-off-by: Casey Bodley <cbodley@redhat.com> * test_s3 versioning multi object delete - add more versions for each object Signed-off-by: Oguzhan Ozmen <oozmen@bloomberg.net> * test_s3: verify list_object_versions after multi-object delete with versioning Extend test_versioning_concurrent_multi_object_delete to also verify that list_object_versions returns no Versions or DeleteMarkers after all objects and their versions have been deleted. This ensures both list_objects and list_object_versions reflect an empty bucket. * extend test_versioning_obj_suspended_copy to test copy to another non-versioned bucket Signed-off-by: Jane Zhu <jzhu116@bloomberg.net> * rgw/logging: modify tests to allow for committing empty objects new tests were added and existing tests that assumed the previous behavior were modified. testing: ceph/ceph#65387 Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com> * s3: Mark tests known to fail on dbstore Fixes: https://tracker.ceph.com/issues/72094 Signed-off-by: Adam C. Emerson <aemerson@redhat.com> * rgw/logging: fix cleanup tests for rollovers This is to test this PR: ceph/ceph#65456 Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com> * rgw/logging: add conf change specific tests and also test updates while writing logs fix: cover all src buckets in cleanup tests Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com> * s3: test conditional multipart_put_object() Signed-off-by: Ali Masarwa <ali.saed.masarwa@gmail.com> Signed-off-by: Ali Masarwa <ALI.SAED.MASARWA@gmail.com> * rgw/logging: verify error message when AccessDenied on target testing: ceph/ceph#65494 Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com> * s3: test object ownership Signed-off-by: Casey Bodley <cbodley@redhat.com> * qa/rgw: Include rgw_restore_processor_period in restore tests This adjusts restore period as per ceph/ceph#64933 Also deleting lifecycle post transition so that temp restored files remain for a while without being re-transitioned immeditately. Signed-off-by: Soumya Koduri <skoduri@redhat.com> * remove boto2 tests and dependency boto2 import fails on recent python with: > E ModuleNotFoundError: No module named 'boto.vendored.six.moves' remove all tests under s3tests/ that weren't converted to boto3 Fixes: https://tracker.ceph.com/issues/72876 Signed-off-by: Casey Bodley <cbodley@redhat.com> * rename s3tests_boto3/ back to s3tests/ Signed-off-by: Casey Bodley <cbodley@redhat.com> * rgw/restore: Fixing readthrough Signed-off-by: Soumya Koduri <skoduri@redhat.com> * s3: test checksum for get_object with part number * rgw/logging: change test to match the new source cleanup process * in "journal" mode, when multiple source buckets share the same log prefix, the first one being deleted will remove the object holding the temporary object name, and only the first would commit the temp object. * in "standard" mode, the other buckets will recreate the object holding the temporary object name, becase the bucket deletion operation will be logged as well. * added tests where only one of the buckets sharing a common prefix is deleted and the rest are not Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com> * CopyObject: add test for encrypted objects Add test for copy on sse-s3 and sse-c encrypted objects. Signed-off-by: Seena Fallah <seenafallah@gmail.com> * CopyObject: add storage class copy tests Signed-off-by: Seena Fallah <seenafallah@gmail.com> * lc-transition: add test with encrypted objects Signed-off-by: Seena Fallah <seenafallah@gmail.com> * UploadPart: add copy part with encryption Signed-off-by: Seena Fallah <seenafallah@gmail.com> * CompleteMultipartUpload: test to verify sse-c headers Signed-off-by: Seena Fallah <seenafallah@gmail.com> * UploadPart: test to verify sse-c headers Signed-off-by: Seena Fallah <seenafallah@gmail.com> * refactor: merge test_copy_enc_ with test_copy_enc_storage_class Signed-off-by: Seena Fallah <seenafallah@gmail.com> * test_multipart_sse_c_get_part: pass enc attrs on mp complete Based on the new fixes around copy enc, those headers are required when completing mp upload. Signed-off-by: Seena Fallah <seenafallah@gmail.com> * functional: fix invalid escape sequence deprication warning This change address multiple messages in the log: DeprecationWarning: invalid escape sequence '\$' Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com> * rgw/bucket-logging: handle async log commits Modified to work with the new asynchronous buckt logging commit mechanism. Fixes: https://tracker.ceph.com/issues/71365 Signed-off-by: Nithya Balachandran <nithya.balachandran@ibm.com> * s3: update etag for get_part tests * Revert "SPL-170872 mark few false negative tests for splunk skip. (#6)" This reverts commit 3dc9362. * Revert "Mark false negative tests in s3-test" This reverts commit ec2aef0. * Revert "update requirements.txt && fix README typo" This reverts commit 4b10f5a. * Revert "splunk S3-compatibility checking tool v1" This reverts commit 69b2678. * Update to latest ceph/s3-tests, fix all tests, and introduce core S3 compliance tests - Rebased branch to latest upstream ceph/s3-tests - Fixed all test failures for compatibility - Added Splunk-specific compliance test lists and skip logic - Updated README with Splunk S3 compliance instructions - Introduced core test set for S3 compliance validation - Improved configuration and test selection for Splunk environments This update ensures the branch is up-to-date, core compliance tests are clearly defined for S3 validation." * s3 core tests shell script after deselecting some tests * fixed the test sets --------- Signed-off-by: Matt Benjamin <mbenjamin@redhat.com> Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com> Signed-off-by: Gal Salomon <gal.salomon@gmail.com> Signed-off-by: Sumedh A. Kulkarni <sumedh.a.kulkarni@seagate.com> Signed-off-by: Bob Ham <bham12@bloomberg.net> Signed-off-by: Casey Bodley <cbodley@redhat.com> Signed-off-by: Seena Fallah <seenafallah@gmail.com> Signed-off-by: Pritha Srivastava <prsrivas@redhat.com> Signed-off-by: galsalomon66 <gal.salomon@gmail.com> Signed-off-by: Tobias Urdin <tobias.urdin@binero.com> Signed-off-by: Soumya Koduri <skoduri@redhat.com> Signed-off-by: Raja Sharma raja@ibm.com Signed-off-by: Raja Sharma <raja@ibm.com> Signed-off-by: Jane Zhu <jzhu116@bloomberg.net> Signed-off-by: Ali Masarwa <ali.saed.masarwa@gmail.com> Signed-off-by: shreyanshjain7174 <ssanchet@redhat.com> Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com> Signed-off-by: Emin <emin.sunacoglu@clyso.com> Signed-off-by: N Balachandran <nithya.balachandran@ibm.com> Signed-off-by: gal salomon <gal.salomon@gmail.com> Signed-off-by: Oguzhan Ozmen <oozmen@bloomberg.net> Signed-off-by: Adam C. Emerson <aemerson@redhat.com> Signed-off-by: Ali Masarwa <ALI.SAED.MASARWA@gmail.com> Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com> Signed-off-by: Nithya Balachandran <nithya.balachandran@ibm.com> Co-authored-by: Matt Benjamin <mbenjamin@redhat.com> Co-authored-by: Yuval Lifshitz <ylifshit@ibm.com> Co-authored-by: Casey Bodley <cbodley@users.noreply.github.com> Co-authored-by: Gal Salomon <gal.salomon@gmail.com> Co-authored-by: Sumedh A. Kulkarni <sumedh.a.kulkarni@seagate.com> Co-authored-by: Casey Bodley <cbodley@redhat.com> Co-authored-by: Seena Fallah <seenafallah@gmail.com> Co-authored-by: Pritha Srivastava <prsrivas@redhat.com> Co-authored-by: Tobias Urdin <tobias.urdin@binero.com> Co-authored-by: Soumya Koduri <skoduri@redhat.com> Co-authored-by: Raja Sharma <raja@ibm.com> Co-authored-by: Adam Emerson <aemerson@redhat.com> Co-authored-by: Yuval Lifshitz <yuvalif@yahoo.com> Co-authored-by: Jane Zhu <jzhu116@bloomberg.net> Co-authored-by: Raja <rajaramsharma422@gmail.com> Co-authored-by: Ali Masarwa <ali.saed.masarwa@gmail.com> Co-authored-by: shreyanshjain7174 <ssanchet@redhat.com> Co-authored-by: Jiffin Tony Thottan <thottanjiffin@gmail.com> Co-authored-by: Andrew Gaul <andrew@gaul.org> Co-authored-by: Emin <emin.sunacoglu@clyso.com> Co-authored-by: N Balachandran <nithya.balachandran@ibm.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Oguzhan Ozmen <oozmen@bloomberg.net> Co-authored-by: sungjoon_koh <sungjoon_koh@linecorp.com> Co-authored-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com> Co-authored-by: chamara-splunk <cwickram@cisco.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
According to the AWS docs (https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-control-block-public-access.html), If RestrictPublicBuckets is set to true and there are public policies in place, Only access from the bucket owner's account is possible.
Fixes: https://tracker.ceph.com/issues/65741