rgw: allow disabling bucket stats on head bucket#56756
Merged
Conversation
49a8c59 to
a53a44b
Compare
cbodley
reviewed
Apr 25, 2024
Contributor
cbodley
left a comment
There was a problem hiding this comment.
i definitely support making HeadBucket requests cheaper by default. instead of a config option, what do you think about requiring some request header like x-rgw-get-bucket-stats to opt in to this extension? that way it's off by default everywhere
Member
Author
While that could indeed be an option, I hold a firm belief that it should be considered as an additional configuration. Customers might deploy monitoring tools with short intervals to retrieve such information. However, if the bucket contains a high number of shards, this could lead to significant resource consumption on the cluster, particularly in terms of IOPS. Combining these two options allows administrators to disable the feature entirely if necessary. Users could then request access to the feature specifically if it's permitted by the administrator. What are your thoughts on this approach? |
|
This pull request has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs for another 30 days. |
Member
Author
|
unstale me please. |
|
This pull request has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs for another 30 days. |
a6a16ba to
7f9cf7a
Compare
7058e3b to
cb88003
Compare
cbodley
reviewed
Nov 13, 2024
cb88003 to
d971951
Compare
|
This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved |
d971951 to
21f6dc0
Compare
|
This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved |
cbodley
approved these changes
Dec 10, 2024
Contributor
cbodley
left a comment
There was a problem hiding this comment.
looks good, just needs rebase
21f6dc0 to
8ceec53
Compare
clwluvw
added a commit
to clwluvw/s3-tests
that referenced
this pull request
Dec 10, 2024
Refs: ceph/ceph#56756 Signed-off-by: Seena Fallah <seenafallah@gmail.com>
Member
Author
|
jenkins test make check |
Member
Author
|
tests available at ceph/s3-tests#605 |
Contributor
|
failed first round of qa in https://pulpito.ceph.com/cbodley-2024-12-12_14:24:46-rgw-wip-65369-distro-default-smithi/ s3tests failure: ex https://qa-proxy.ceph.com/teuthology/cbodley-2024-12-12_14:24:46-rgw-wip-65369-distro-default-smithi/8032154/teuthology.log ceph_test_librgw_file segfault: ex https://qa-proxy.ceph.com/teuthology/cbodley-2024-12-12_14:24:46-rgw-wip-65369-distro-default-smithi/8032166/teuthology.log tempest (swift) test failures: https://qa-proxy.ceph.com/teuthology/cbodley-2024-12-12_14:24:46-rgw-wip-65369-distro-default-smithi/8032181/teuthology.log |
Contributor
see qa/workunits/rgw/test_librgw_file.sh for instructions to run locally against vstart |
8ceec53 to
4a4cd79
Compare
clwluvw
added a commit
to clwluvw/s3-tests
that referenced
this pull request
Dec 12, 2024
Refs: ceph/ceph#56756 Signed-off-by: Seena Fallah <seenafallah@gmail.com>
Member
Author
thanks @cbodley - I have found the issue with the first two failures and pushed a fix here and into s3-tests. but for the swift one, I'm afraid we need to change https://github.com/openstack/tempest/blob/422131d9520ec26e79a698cefdf98dc9a2c244ee/tempest/lib/services/object_storage/container_client.py#L89 to send the arg. should we exclude swift from this feature or open a change there for the integration? |
cbodley
reviewed
Dec 13, 2024
The HeadBucket API now reports the `X-RGW-Bytes-Used` and `X-RGW-Object-Count` headers only when the `read-stats` querystring is explicitly included in the API request. Fixes: https://tracker.ceph.com/issues/65369 Signed-off-by: Seena Fallah <seenafallah@gmail.com>
4a4cd79 to
854bb63
Compare
Contributor
|
passed qa in https://pulpito.ceph.com/cbodley-2024-12-13_17:43:49-rgw-wip-65369-distro-default-smithi/
|
cbodley
pushed a commit
to ceph/s3-tests
that referenced
this pull request
Dec 17, 2024
Refs: ceph/ceph#56756 Signed-off-by: Seena Fallah <seenafallah@gmail.com> (cherry picked from commit 3145b1b)
Moisi
pushed a commit
to Moisi/ceph
that referenced
this pull request
Jun 30, 2025
1. This is a backport based on ceph#56756
Moisi
pushed a commit
to Moisi/ceph
that referenced
this pull request
Jun 30, 2025
…ucket by default Merge in STORAGEENG/ceph from CEPH-343-backport-reef-rgw-allow-disabling-bucket-stats-on-head-bucket to aka_version_18.2.1 * commit '22acc26b974ed3f3c1a438d73fad75ef64db5310': CEPH-343 Disabling bucket stats when S3 head bucket by default 1. This is a backport based on ceph#56756
cwickramanayake-dotcom
added a commit
to splunk/s3-tests
that referenced
this pull request
Jan 28, 2026
* add tests for ObjectSizeGreater(Less)Than Add tests for the new ObjectSizeGreaterThan and ObjectSizeLessThan lifecycle operators. Signed-off-by: Matt Benjamin <mbenjamin@redhat.com> * test etag on mpu complete replies this is to cover the fix of: https://tracker.ceph.com/issues/58879 Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com> * a change is the RGW error-response require changes in s3-tests Signed-off-by: Gal Salomon <gal.salomon@gmail.com> * add handling for EventStreamError exception Signed-off-by: Gal Salomon <gal.salomon@gmail.com> * Fix wrong assertion of the test: `test_buckets_list_ctime` TestName: s3tests_boto3.functional.test_s3:test_buckets_list_ctime Problem: The test creates 5 buckets for a user but in an assertion check, it asserts false if any bucket of the user has CreationTime less than a day prior to current time. Due to this reason the test fails if the user has pre-existing buckets older than a day. Solution: Assert only on the CreationTime of buckets that were created with test execution. Signed-off-by: Sumedh A. Kulkarni <sumedh.a.kulkarni@seagate.com> Co-developed-by: Bob Ham <bham12@bloomberg.net> Signed-off-by: Bob Ham <bham12@bloomberg.net> * requirements: unpin pytz version Fixes: https://tracker.ceph.com/issues/66655 Signed-off-by: Casey Bodley <cbodley@redhat.com> * test Get/HeadObject with partNumber for single-multipart upload test_multipart_get_part() tests 'normal' multipart uploads. add a new test case for a multipart upload with a single part to tests the fix for https://tracker.ceph.com/issues/66705 Signed-off-by: Casey Bodley <cbodley@redhat.com> * add "checksum" marker, since new checksum tests reference it this removes a Pytest warning during execution Signed-off-by: Matt Benjamin <mbenjamin@redhat.com> * test_multipart_upload_sha256: work around failures re-trying complete-multipart As described in https://tracker.ceph.com/issues/65746, retrying complete-multipart after having attempted to complete the same upload with a bad checksum argument fails with an internal error. The status code is 500, but I'm unsure if it can be retried again, or whether the upload can be aborted later. Signed-off-by: Matt Benjamin <mbenjamin@redhat.com> * add test_multipart_checksum_3parts tests a full multipart upload cycle with 3 unique parts, which verifies composite checksum computation and the logic to propagate parts_count to ComleteMultipart Signed-off-by: Matt Benjamin <mbenjamin@redhat.com> * add test_post_object_upload_checksum this tests a two-megabyte binary upload with validated (awscli-computed) SHA256 checksum, and also verifies failure when a bad checksum is provided Signed-off-by: Matt Benjamin <mbenjamin@redhat.com> * remove duplicate size assigment [rkhudov review] Signed-off-by: Matt Benjamin <mbenjamin@redhat.com> * mark two tests that fail on dbstore also add @pytest.mark.checksum for new checksum tests Signed-off-by: Matt Benjamin <mbenjamin@redhat.com> * s3select: align error codes with the new AWS format ref. ceph/ceph#56864 Signed-off-by: Seena Fallah <seenafallah@gmail.com> * BucketPolicy: donot allow NotPrincipal with Allow Effect Ref. ceph/ceph#58686 Signed-off-by: Seena Fallah <seenafallah@gmail.com> * PolicyStatus: add test for policy with Principal Ref. ceph/ceph#58686 Signed-off-by: Seena Fallah <seenafallah@gmail.com> * BlockPublicPolicy: add test when policy has principal Ref. https://tracker.ceph.com/issues/67048 Signed-off-by: Seena Fallah <seenafallah@gmail.com> * BucketPolicy: add test for sse-c in conditions Ref. ceph/ceph#58689 Signed-off-by: Seena Fallah <seenafallah@gmail.com> * BucketPolicy: decouple encryption tests from invalid algo and unencrypted Signed-off-by: Seena Fallah <seenafallah@gmail.com> * s3: reenable tenanted bucket policy test the before-call hook url-encodes the ':' part of tenanted bucket names to resolve SignatureDoesNotMatch errors removed the list-v2 version of the test since it isn't relevant to bucket policy test coverage add a new test case that creates the bucket under the tenanted user, then uses the main client to access it Signed-off-by: Casey Bodley <cbodley@redhat.com> * UploadPartCopy: add test for source bucket with policy Ref: ceph/ceph#59253 Signed-off-by: Seena Fallah <seenafallah@gmail.com> * rgw: adding tests for add_client_id_to_oidc_provider and update_thumbprint_for oidc_provider. Signed-off-by: Pritha Srivastava <prsrivas@redhat.com> * fix the assert per empty results Signed-off-by: Gal Salomon <gal.salomon@gmail.com> * fix comments Signed-off-by: galsalomon66 <gal.salomon@gmail.com> * Add v2 signature presigned get_object tests This adds tests for get_object presigned URLs using signature v2. Also code formatting. Signed-off-by: Tobias Urdin <tobias.urdin@binero.com> * s3: add v2 signature presigned put_object tests Signed-off-by: Casey Bodley <cbodley@redhat.com> * s3: test GetObject with PartNumber and SSE-C encryption Signed-off-by: Casey Bodley <cbodley@redhat.com> * Updating lifecycle_noncur* tests to include objects with instance empty Signed-off-by: Soumya Koduri <skoduri@redhat.com> * BucketPublicAccessBlock: add tests for RestrictPublicBuckets Ref. ceph/ceph#57206 Signed-off-by: Seena Fallah <seenafallah@gmail.com> * rgw/s3-test: adding test to set, get, delete and then get a bucket policy to ensure that delete correctly removes the bucket policy Signed-off-by: Pritha Srivastava <prsrivas@redhat.com> * rgw/s3-tests: adding a test to check put, get, delete and then get to ensure that public block access is deleted successfully. Signed-off-by: Pritha Srivastava <prsrivas@redhat.com> * rgw/s3tests: making the tests generic to check only for 403 error code (and not specific errors) so that the tests pass for any order of auth engines. Signed-off-by: Pritha Srivastava <prsrivas@redhat.com> * rgw:modify testcase rgw header when bucket quota or user is disabled By default quota is disable at user/bucket level. if quota is disable then these value will not return so need to modify test_head_bucket_usages 'X-RGW-Quota-User-Size' 'X-RGW-Quota-User-Objects' 'X-RGW-Quota-Bucket-Size' 'X-RGW-Quota-Bucket-Objects' to Fix: https://tracker.ceph.com/issues/68211 Signed-off-by: Raja Sharma raja@ibm.com * sts: remove test_get_session_token_permanent_creds_denied Fixes: https://tracker.ceph.com/issues/69001 Signed-off-by: Casey Bodley <cbodley@redhat.com> * rgw/logging: basic set of tests for bucket logging Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com> * rgw/logging: tests for flush API as well as test for object key filters Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com> * HeadBucket: add read-stats query string for usage test Refs: ceph/ceph#56756 Signed-off-by: Seena Fallah <seenafallah@gmail.com> * test get_object_attributes Signed-off-by: Matt Benjamin <mbenjamin@redhat.com> * multipart fallback to create-multipart checksum algorithm there seem to be workloads which assume checksum algorithm can be omitted from upload-part Signed-off-by: Matt Benjamin <mbenjamin@redhat.com> * more tests for GetObjectAttributes * multipart upload without checksums * multipart upload with a single part * pagination of multipart parts * non-multipart upload with/without checksum * versioned object, current and non-current * sse-c encrypted object Signed-off-by: Casey Bodley <cbodley@redhat.com> Signed-off-by: Matt Benjamin <mbenjamin@redhat.com> * mark attribute tests as failing on dbstore (for now) Signed-off-by: Matt Benjamin <mbenjamin@redhat.com> * rgw/logging: test tenant support when source and log buckets are on a specific tenant, or on different tenants Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com> * rgw/logging: test source and log have different names Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com> * rgw/logging: test log bucket encryption error Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com> * rgw/logging: add cleanup tests * use flushing api instead of sleep when possible * test flushing concurrency Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com> * rgw/logging: fix typo in readme Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com> * rgw test-case to support x-expected-bucket-owner If the bucket is owned by a different account, the request fails with the HTTP status code 403 Forbidden (access denied). PR: ceph/ceph#61215 Fixes: https://tracker.ceph.com/issues/64526 Signed-off-by: Raja Sharma <raja@ibm.com> * test-case to fix obj by multipart upload cant get tag Signed-off-by: Raja Sharma <raja@ibm.com> * add test case for stacked delete-markers Signed-off-by: Jane Zhu <jzhu116@bloomberg.net> * rgw|BN: fix test to expect authorization failure Signed-off-by: Ali Masarwa <ali.saed.masarwa@gmail.com> * requirements: unpin botocore version Signed-off-by: Casey Bodley <cbodley@redhat.com> * s3: mark sigv2 tests with fails_on_rgw Signed-off-by: Casey Bodley <cbodley@redhat.com> * s3: sse-kms policy tests use v4 client Signed-off-by: Casey Bodley <cbodley@redhat.com> * s3: clean up use of tenanted bucket names the botocore.handlers.validate_bucket_name validation is enabled by default, but we can disable it with unregister(). this avoids having to mess with the signature or url Signed-off-by: Casey Bodley <cbodley@redhat.com> * s3: disable request_checksum_calculation for test_multipart_checksum_upload_fallback the test sends create_multipart_upload() with a requested checksum algorithm, and upload_part() requests without. newer boto automatically adds `x-amz-sdk-checksum-algorithm: CRC32` when nothing is specified, so we have to explicitly disable that behavior via botocore config Signed-off-by: Casey Bodley <cbodley@redhat.com> * s3: disable multipart/ranged tests with checksum failures boto update exposed some bugs in our checksum feature around multipart copy and ranged requests. disabling them with fails_on_rgw until https://tracker.ceph.com/issues/69936 is resolved Signed-off-by: Casey Bodley <cbodley@redhat.com> * iam: remove invalid Bucket param from list_buckets() > botocore.exceptions.ParamValidationError: Parameter validation failed: > Unknown parameter in input: "Bucket", must be one of: MaxBuckets, ContinuationToken, Prefix, BucketRegion Signed-off-by: Casey Bodley <cbodley@redhat.com> * s3: disable checksum calculation for test_object_create_bad_contentlength_negative if tls is enabled, boto will switch to STREAMING-UNSIGNED-PAYLOAD-TRAILER and omit the provided content-length header. this leads to test failure: > ________________ test_object_create_bad_contentlength_negative _________________ > e = assert_raises(ClientError, client.put_object, Bucket=bucket_name, Key=key_name, ContentLength=-1) > AssertionError: ClientError not raised Signed-off-by: Casey Bodley <cbodley@redhat.com> * sns: clear region_name for s3 clients work around sns failures due to default region_name='': > ERROR s3tests_boto3/functional/test_sns.py::test_account_topic_publish - botocore.exceptions.EndpointResolutionError: Invalid region: region was not a valid DNS name. > ERROR s3tests_boto3/functional/test_sns.py::test_cross_account_topic_publish - botocore.exceptions.EndpointResolutionError: Invalid region: region was not a valid DNS name. Signed-off-by: Casey Bodley <cbodley@redhat.com> * rgw/iam: add RemoveClientIDFromOpenIDConnectProvider Signed-off-by: Raja Sharma <raja@ibm.com> * s3: reenable tests with range requests and checksums Signed-off-by: Casey Bodley <cbodley@redhat.com> * rgw/restore: s3tests to test restore object functionality. This tests are added to tests temporary restore, permanent restore and read through resotre object functionality, this includes zonegroup parameters and checks. Signed-off-by: shreyanshjain7174 <ssanchet@redhat.com> Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com> * s3: test that object lock can be enabled on an existing bucket Signed-off-by: Casey Bodley <cbodley@redhat.com> * running cloud restore tests seperately Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com> * rgw/s3tests: removing fails_on_rgw for test_versioned_concurrent_object_create_concurrent_remove as the associated tracker is marked as resolved and the test also passes against main. Signed-off-by: Pritha Srivastava <prsrivas@redhat.com> * s3: test bucket policy evaluation for CreateMultipartUpload test case for https://tracker.ceph.com/issues/70191 Signed-off-by: Casey Bodley <cbodley@redhat.com> * mark test_versioning_stack_delete_merkers fails_on_dbstore Signed-off-by: Jane Zhu <jzhu116@bloomberg.net> * cloud-restore: testcase for non-current versioned object Signed-off-by: Soumya Koduri <skoduri@redhat.com> * rgw/s3tests: handle allow read through values in cloud transitions cases Check the allow_read_through values for cases test_lifecycle_cloud_transition and test_read_through Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com> * BucketPolicy: add tests for ConfirmRemoveSelfBucketAccess header Refrence: ceph/ceph#57629 Signed-off-by: Seena Fallah <seenafallah@gmail.com> * rgw/logging: add missing marker to remove warnings Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com> * test put_bucket_acl with group uri Signed-off-by: Casey Bodley <cbodley@redhat.com> * rgw/logging: add basic policy to target bucket this commit needed to be able to run bucket logging regression against: ceph/ceph#62284 since target bucket requires policy for bucket logging to work this only covers the positive cases from bucket logging policy perspective Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com> * rgw/logging: add more bucket logging error cases this is to cover these checks: * source bucket ownership * "requester pays" on log bucket that were added in: ceph/ceph#62284 Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com> * rgw/logging: add bucket logging permission tests this is to cover new functionality added in: ceph/ceph#62284 Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com> * add minimal put-object for CRC64NVME Signed-off-by: Matt Benjamin <mbenjamin@redhat.com> * enhance additional checksum tests includes tests for CRC64NVME, tests for selecting COMPOSITE and FULL_OBJECT checksums a decomposed matrix of tests for all checksum types also removes the mixed checksum upload case that no longer works in recent boto3 cleanups, add sha1 checksum validation failure (mismatch) returns BadDigest multipart checksum matrix helper now validates checksum and checksum type for all operations which can return them (complete-multipart, head-object, and get-object-attributes) Signed-off-by: Matt Benjamin <mbenjamin@redhat.com> * rgw/logging: fix tenant test tests were failing in teuthology since the tenanted user name already have the tenant in its name Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com> * rgw/logging: test bucket logging with accounts Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com> * rgw/cloud-transition: increase time out for test cases Fixes: https://tracker.ceph.com/issues/69411 Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com> * rgw/logging: mark test that require rollover time set Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com> * add test coverage for DeleteBucketLifecycle Signed-off-by: Casey Bodley <cbodley@redhat.com> * rgw/logging: add tests for flushed object name when using API call to flush the pending logging object of the bucket, the flshed object name is returned. tests were added to verify the right name is returned. Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com> * rgw/logging: add tests to verify wildcrad policy Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com> * Tag copy tests * rgw/logging: test object metadata changes in journal mode Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com> * rgw/logging: test mpu complete in standard mode Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com> * rgw/logging: test log object name order verify that the names of the log objects in the log bucket are lexicographically ordered Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com> * rgw/logging: test part upload in mpu in standard mode Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com> * s3: test retry after error instead of 100-continue Signed-off-by: Casey Bodley <cbodley@redhat.com> * s3: avoid 100-continue bug for test_encryption_sse_c_deny_algo_with_bucket_policy was failing for a reason unrelated to sse-c or bucket policy Fixes: https://tracker.ceph.com/issues/70607 Signed-off-by: Casey Bodley <cbodley@redhat.com> * s3: Add tests for test_upload_part_copy_percent_encoded_key * s3: fix: Changed test so that it accepts the character that has % sign in it as a key * add missing 'assert' to test_bucket_logging_roll_time saw a failure after this: ``` response = client.list_objects_v2(Bucket=log_bucket_name) keys = _get_keys(response) len(keys) == 1 > key = keys[0] E IndexError: list index out of range s3tests_boto3/functional/test_s3.py:16637: IndexError ``` Signed-off-by: Casey Bodley <cbodley@redhat.com> * website: use pytest parametrize for test_routing_generator() resolves error with recent pytest: ==================================== ERRORS ==================================== ____________ ERROR collecting s3tests/functional/test_s3_website.py ____________ 'yield' keyword is allowed in fixtures, but not in tests (test_routing_generator) use https://docs.pytest.org/en/7.1.x/example/parametrize.html instead to generate a test_routing_generator() for each entry in ROUTING_RULES_TESTS Signed-off-by: Casey Bodley <cbodley@redhat.com> * website: routing_setup() as pytest fixture c80e9d2 had removed the use of routing_setup() without adding a fixture to replace it. this had broken routing_check(), but we hadn't noticed it because pytest hadn't supported the 'yield' pattern in test_routing_generator() to call it test_routing_generator() now uses this routing_setup fixture to provide the kwargs that had previously been injected into routing_check() via @common.with_setup_kwargs(setup=routing_setup, teardown=routing_teardown) Signed-off-by: Casey Bodley <cbodley@redhat.com> * increase time out for restore op in test_restore_object_permanent and in test_lifecycle_expiration_size_gt() for lc. Fixes: https://tracker.ceph.com/issues/71348 Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com> * fix: Fix typo Signed-off-by: Emin <emin.sunacoglu@clyso.com> * rgw/logging: test conf last modified time Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com> * rgw/logging: add tests for logging object key format Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com> * rgw/sts: GetAccountSummary test Signed-off-by: Raja Sharma <raja@ibm.com> * readme: fix typo Fixed typos in the example commands. Signed-off-by: N Balachandran <nithya.balachandran@ibm.com> * s3: test_block_public_object_canned_acls() tests that 'private' works Fixes: https://tracker.ceph.com/issues/49135 Signed-off-by: Casey Bodley <cbodley@redhat.com> * bucket-logging: tests for the ACLRequired field Tests that the AclRequired field is set correctly in the bucket logging record. Signed-off-by: N Balachandran <nithya.balachandran@ibm.com> * bucket-logging: fix typo Fixes typo in function name. Signed-off-by: N Balachandran <nithya.balachandran@ibm.com> * bucket-logging: test the AuthType log field Adds a test to check the value of the AuthType field. Signed-off-by: N Balachandran <nithya.balachandran@ibm.com> * rgw/sts: GetCallerIndentity test Signed-off-by: Raja Sharma <raja@ibm.com> * delete, rgwlc: delete marker tests 0. non-creation in non-versioned buckets 1. creation in versioned buckets 2. creation in versioning-suspended buckets 3. delete-marker expiration (lifecycle) Signed-off-by: Matt Benjamin <mbenjamin@redhat.com> * bucket_logging: remove duplicate function Removes the duplicate _verify_record_field function definition. Signed-off-by: N Balachandran <nithya.balachandran@ibm.com> * s3: test put_object() with IfMatch and IfNoneMatch Signed-off-by: Casey Bodley <cbodley@redhat.com> * s3: test delete_object() with IfMatch Signed-off-by: Casey Bodley <cbodley@redhat.com> * s3: test delete_object() with IfMatchLastModifiedTime Signed-off-by: Casey Bodley <cbodley@redhat.com> * s3: test delete_object() with IfMatchSize Signed-off-by: Casey Bodley <cbodley@redhat.com> * s3: test delete_objects() with ETag Signed-off-by: Casey Bodley <cbodley@redhat.com> * s3: test delete_objects() with LastModifiedTime Signed-off-by: Casey Bodley <cbodley@redhat.com> * s3: test delete_objects() with Size Signed-off-by: Casey Bodley <cbodley@redhat.com> * s3: fixed tests we expected an error if we delete non-existent objects but that is not true Signed-off-by: Ali Masarwa <ali.saed.masarwa@gmail.com> * s3: more testing for conditional put Signed-off-by: Ali Masarwa <ali.saed.masarwa@gmail.com> * a fix for empty results.for some of the queries an empty result that may rarely happen, could cause an assertion. the fix handle that cases. Signed-off-by: gal salomon <gal.salomon@gmail.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * rgw/cloud-restore: verify if the expiry-date is updated This is the test-case for ceph/ceph#64804. If the `restoe-object` request is re-issued on a cloud-restored object, it should update restoration period accordingly from the time the request is re-issued. Signed-off-by: Soumya Koduri <skoduri@redhat.com> * s3: test list_buckets with MaxBuckets/ContinuationToken Signed-off-by: Casey Bodley <cbodley@redhat.com> * s3: test for aws-chunked in Content-Encoding responses aws-chunked should not be persisted on upload for return on download Signed-off-by: Casey Bodley <cbodley@redhat.com> * test_s3 versioning multi object delete - add more versions for each object Signed-off-by: Oguzhan Ozmen <oozmen@bloomberg.net> * test_s3: verify list_object_versions after multi-object delete with versioning Extend test_versioning_concurrent_multi_object_delete to also verify that list_object_versions returns no Versions or DeleteMarkers after all objects and their versions have been deleted. This ensures both list_objects and list_object_versions reflect an empty bucket. * extend test_versioning_obj_suspended_copy to test copy to another non-versioned bucket Signed-off-by: Jane Zhu <jzhu116@bloomberg.net> * rgw/logging: modify tests to allow for committing empty objects new tests were added and existing tests that assumed the previous behavior were modified. testing: ceph/ceph#65387 Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com> * s3: Mark tests known to fail on dbstore Fixes: https://tracker.ceph.com/issues/72094 Signed-off-by: Adam C. Emerson <aemerson@redhat.com> * rgw/logging: fix cleanup tests for rollovers This is to test this PR: ceph/ceph#65456 Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com> * rgw/logging: add conf change specific tests and also test updates while writing logs fix: cover all src buckets in cleanup tests Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com> * s3: test conditional multipart_put_object() Signed-off-by: Ali Masarwa <ali.saed.masarwa@gmail.com> Signed-off-by: Ali Masarwa <ALI.SAED.MASARWA@gmail.com> * rgw/logging: verify error message when AccessDenied on target testing: ceph/ceph#65494 Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com> * s3: test object ownership Signed-off-by: Casey Bodley <cbodley@redhat.com> * qa/rgw: Include rgw_restore_processor_period in restore tests This adjusts restore period as per ceph/ceph#64933 Also deleting lifecycle post transition so that temp restored files remain for a while without being re-transitioned immeditately. Signed-off-by: Soumya Koduri <skoduri@redhat.com> * remove boto2 tests and dependency boto2 import fails on recent python with: > E ModuleNotFoundError: No module named 'boto.vendored.six.moves' remove all tests under s3tests/ that weren't converted to boto3 Fixes: https://tracker.ceph.com/issues/72876 Signed-off-by: Casey Bodley <cbodley@redhat.com> * rename s3tests_boto3/ back to s3tests/ Signed-off-by: Casey Bodley <cbodley@redhat.com> * rgw/restore: Fixing readthrough Signed-off-by: Soumya Koduri <skoduri@redhat.com> * s3: test checksum for get_object with part number * rgw/logging: change test to match the new source cleanup process * in "journal" mode, when multiple source buckets share the same log prefix, the first one being deleted will remove the object holding the temporary object name, and only the first would commit the temp object. * in "standard" mode, the other buckets will recreate the object holding the temporary object name, becase the bucket deletion operation will be logged as well. * added tests where only one of the buckets sharing a common prefix is deleted and the rest are not Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com> * CopyObject: add test for encrypted objects Add test for copy on sse-s3 and sse-c encrypted objects. Signed-off-by: Seena Fallah <seenafallah@gmail.com> * CopyObject: add storage class copy tests Signed-off-by: Seena Fallah <seenafallah@gmail.com> * lc-transition: add test with encrypted objects Signed-off-by: Seena Fallah <seenafallah@gmail.com> * UploadPart: add copy part with encryption Signed-off-by: Seena Fallah <seenafallah@gmail.com> * CompleteMultipartUpload: test to verify sse-c headers Signed-off-by: Seena Fallah <seenafallah@gmail.com> * UploadPart: test to verify sse-c headers Signed-off-by: Seena Fallah <seenafallah@gmail.com> * refactor: merge test_copy_enc_ with test_copy_enc_storage_class Signed-off-by: Seena Fallah <seenafallah@gmail.com> * test_multipart_sse_c_get_part: pass enc attrs on mp complete Based on the new fixes around copy enc, those headers are required when completing mp upload. Signed-off-by: Seena Fallah <seenafallah@gmail.com> * functional: fix invalid escape sequence deprication warning This change address multiple messages in the log: DeprecationWarning: invalid escape sequence '\$' Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com> * rgw/bucket-logging: handle async log commits Modified to work with the new asynchronous buckt logging commit mechanism. Fixes: https://tracker.ceph.com/issues/71365 Signed-off-by: Nithya Balachandran <nithya.balachandran@ibm.com> * s3: update etag for get_part tests * Revert "SPL-170872 mark few false negative tests for splunk skip. (#6)" This reverts commit 3dc9362. * Revert "Mark false negative tests in s3-test" This reverts commit ec2aef0. * Revert "update requirements.txt && fix README typo" This reverts commit 4b10f5a. * Revert "splunk S3-compatibility checking tool v1" This reverts commit 69b2678. * Update to latest ceph/s3-tests, fix all tests, and introduce core S3 compliance tests - Rebased branch to latest upstream ceph/s3-tests - Fixed all test failures for compatibility - Added Splunk-specific compliance test lists and skip logic - Updated README with Splunk S3 compliance instructions - Introduced core test set for S3 compliance validation - Improved configuration and test selection for Splunk environments This update ensures the branch is up-to-date, core compliance tests are clearly defined for S3 validation." * s3 core tests shell script after deselecting some tests * fixed the test sets --------- Signed-off-by: Matt Benjamin <mbenjamin@redhat.com> Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com> Signed-off-by: Gal Salomon <gal.salomon@gmail.com> Signed-off-by: Sumedh A. Kulkarni <sumedh.a.kulkarni@seagate.com> Signed-off-by: Bob Ham <bham12@bloomberg.net> Signed-off-by: Casey Bodley <cbodley@redhat.com> Signed-off-by: Seena Fallah <seenafallah@gmail.com> Signed-off-by: Pritha Srivastava <prsrivas@redhat.com> Signed-off-by: galsalomon66 <gal.salomon@gmail.com> Signed-off-by: Tobias Urdin <tobias.urdin@binero.com> Signed-off-by: Soumya Koduri <skoduri@redhat.com> Signed-off-by: Raja Sharma raja@ibm.com Signed-off-by: Raja Sharma <raja@ibm.com> Signed-off-by: Jane Zhu <jzhu116@bloomberg.net> Signed-off-by: Ali Masarwa <ali.saed.masarwa@gmail.com> Signed-off-by: shreyanshjain7174 <ssanchet@redhat.com> Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com> Signed-off-by: Emin <emin.sunacoglu@clyso.com> Signed-off-by: N Balachandran <nithya.balachandran@ibm.com> Signed-off-by: gal salomon <gal.salomon@gmail.com> Signed-off-by: Oguzhan Ozmen <oozmen@bloomberg.net> Signed-off-by: Adam C. Emerson <aemerson@redhat.com> Signed-off-by: Ali Masarwa <ALI.SAED.MASARWA@gmail.com> Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com> Signed-off-by: Nithya Balachandran <nithya.balachandran@ibm.com> Co-authored-by: Matt Benjamin <mbenjamin@redhat.com> Co-authored-by: Yuval Lifshitz <ylifshit@ibm.com> Co-authored-by: Casey Bodley <cbodley@users.noreply.github.com> Co-authored-by: Gal Salomon <gal.salomon@gmail.com> Co-authored-by: Sumedh A. Kulkarni <sumedh.a.kulkarni@seagate.com> Co-authored-by: Casey Bodley <cbodley@redhat.com> Co-authored-by: Seena Fallah <seenafallah@gmail.com> Co-authored-by: Pritha Srivastava <prsrivas@redhat.com> Co-authored-by: Tobias Urdin <tobias.urdin@binero.com> Co-authored-by: Soumya Koduri <skoduri@redhat.com> Co-authored-by: Raja Sharma <raja@ibm.com> Co-authored-by: Adam Emerson <aemerson@redhat.com> Co-authored-by: Yuval Lifshitz <yuvalif@yahoo.com> Co-authored-by: Jane Zhu <jzhu116@bloomberg.net> Co-authored-by: Raja <rajaramsharma422@gmail.com> Co-authored-by: Ali Masarwa <ali.saed.masarwa@gmail.com> Co-authored-by: shreyanshjain7174 <ssanchet@redhat.com> Co-authored-by: Jiffin Tony Thottan <thottanjiffin@gmail.com> Co-authored-by: Andrew Gaul <andrew@gaul.org> Co-authored-by: Emin <emin.sunacoglu@clyso.com> Co-authored-by: N Balachandran <nithya.balachandran@ibm.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Oguzhan Ozmen <oozmen@bloomberg.net> Co-authored-by: sungjoon_koh <sungjoon_koh@linecorp.com> Co-authored-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com> Co-authored-by: chamara-splunk <cwickram@cisco.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This will result in calculating bucket stats on each request which can be manipulated by the user and could be a heavy operation on buckets with a high number of shards.
Fixes: https://tracker.ceph.com/issues/65369