If the bucket is owned by a different account, the request fails with the HTTP status code 403 Forbidden (access denied).
PR: ceph/ceph#61215
Fixes: https://tracker.ceph.com/issues/64526

If the bucket is owned by a different account, the request fails with the HTTP status code 403 Forbidden (access denied).
PR: ceph/ceph#61215
Fixes: https://tracker.ceph.com/issues/64526

Signed-off-by: Raja Sharma <raja@ibm.com>

If the bucket is owned by a different account, the request fails with the HTTP status code 403 Forbidden (access denied).
PR: ceph/ceph#61215
Fixes: https://tracker.ceph.com/issues/64526

Signed-off-by: Raja Sharma <raja@ibm.com>

* add tests for ObjectSizeGreater(Less)Than

Add tests for the new ObjectSizeGreaterThan and
ObjectSizeLessThan lifecycle operators.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

* test etag on mpu complete replies

this is to cover the fix of: https://tracker.ceph.com/issues/58879

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

* a change is the RGW error-response require changes in s3-tests

Signed-off-by: Gal Salomon <gal.salomon@gmail.com>

* add handling for EventStreamError exception

Signed-off-by: Gal Salomon <gal.salomon@gmail.com>

* Fix wrong assertion of the test: `test_buckets_list_ctime`

TestName:
s3tests_boto3.functional.test_s3:test_buckets_list_ctime

Problem:
The test creates 5 buckets for a user but in an assertion check,
it asserts false if any bucket of the user has CreationTime less
than a day prior to current time.
Due to this reason the test fails if the user has pre-existing
buckets older than a day.

Solution:
Assert only on the CreationTime of buckets that were created with
test execution.

Signed-off-by: Sumedh A. Kulkarni <sumedh.a.kulkarni@seagate.com>
Co-developed-by: Bob Ham <bham12@bloomberg.net>
Signed-off-by: Bob Ham <bham12@bloomberg.net>

* requirements: unpin pytz version

Fixes: https://tracker.ceph.com/issues/66655

Signed-off-by: Casey Bodley <cbodley@redhat.com>

* test Get/HeadObject with partNumber for single-multipart upload

test_multipart_get_part() tests 'normal' multipart uploads. add a new
test case for a multipart upload with a single part to tests the fix
for https://tracker.ceph.com/issues/66705

Signed-off-by: Casey Bodley <cbodley@redhat.com>

* add "checksum" marker, since new checksum tests reference it

this removes a Pytest warning during execution

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

* test_multipart_upload_sha256: work around failures re-trying complete-multipart

As described in https://tracker.ceph.com/issues/65746, retrying complete-multipart
after having attempted to complete the same upload with a bad checksum argument
fails with an internal error.

The status code is 500, but I'm unsure if it can be retried again, or whether
the upload can be aborted later.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

* add test_multipart_checksum_3parts

tests a full multipart upload cycle with 3 unique parts, which
verifies composite checksum computation and the logic to propagate
parts_count to ComleteMultipart

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

* add test_post_object_upload_checksum

this tests a two-megabyte binary upload with validated
(awscli-computed) SHA256 checksum, and also verifies failure when
a bad checksum is provided

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

* remove duplicate size assigment [rkhudov review]

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

* mark two tests that fail on dbstore

also add @pytest.mark.checksum for new checksum
tests

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

* s3select: align error codes with the new AWS format

ref. ceph/ceph#56864

Signed-off-by: Seena Fallah <seenafallah@gmail.com>

* BucketPolicy: donot allow NotPrincipal with Allow Effect

Ref. ceph/ceph#58686

Signed-off-by: Seena Fallah <seenafallah@gmail.com>

* PolicyStatus: add test for policy with Principal

Ref. ceph/ceph#58686

Signed-off-by: Seena Fallah <seenafallah@gmail.com>

* BlockPublicPolicy: add test when policy has principal

Ref. https://tracker.ceph.com/issues/67048

Signed-off-by: Seena Fallah <seenafallah@gmail.com>

* BucketPolicy: add test for sse-c in conditions

Ref. ceph/ceph#58689

Signed-off-by: Seena Fallah <seenafallah@gmail.com>

* BucketPolicy: decouple encryption tests from invalid algo and unencrypted

Signed-off-by: Seena Fallah <seenafallah@gmail.com>

* s3: reenable tenanted bucket policy test

the before-call hook url-encodes the ':' part of tenanted bucket names
to resolve SignatureDoesNotMatch errors

removed the list-v2 version of the test since it isn't relevant to
bucket policy test coverage

add a new test case that creates the bucket under the tenanted user,
then uses the main client to access it

Signed-off-by: Casey Bodley <cbodley@redhat.com>

* UploadPartCopy: add test for source bucket with policy

Ref: ceph/ceph#59253

Signed-off-by: Seena Fallah <seenafallah@gmail.com>

* rgw: adding tests for add_client_id_to_oidc_provider
and update_thumbprint_for oidc_provider.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

* fix the assert per empty results

Signed-off-by: Gal Salomon <gal.salomon@gmail.com>

* fix comments

Signed-off-by: galsalomon66 <gal.salomon@gmail.com>

* Add v2 signature presigned get_object tests

This adds tests for get_object presigned URLs
using signature v2.

Also code formatting.

Signed-off-by: Tobias Urdin <tobias.urdin@binero.com>

* s3: add v2 signature presigned put_object tests

Signed-off-by: Casey Bodley <cbodley@redhat.com>

* s3: test GetObject with PartNumber and SSE-C encryption

Signed-off-by: Casey Bodley <cbodley@redhat.com>

* Updating lifecycle_noncur* tests to include objects with instance empty

Signed-off-by: Soumya Koduri <skoduri@redhat.com>

* BucketPublicAccessBlock: add tests for RestrictPublicBuckets

Ref. ceph/ceph#57206
Signed-off-by: Seena Fallah <seenafallah@gmail.com>

* rgw/s3-test: adding test to set, get, delete and then get
a bucket policy to ensure that delete correctly removes
the bucket policy

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

* rgw/s3-tests: adding a test to check put, get, delete and
then get to ensure that public block access is deleted
successfully.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

* rgw/s3tests: making the tests generic to check only
for 403 error code (and not specific errors) so that
the tests pass for any order of auth engines.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

* rgw:modify testcase rgw header when bucket quota or user is disabled
By default quota is disable at user/bucket level.
if quota is disable then these value will not return so need to modify test_head_bucket_usages

'X-RGW-Quota-User-Size'
'X-RGW-Quota-User-Objects'
'X-RGW-Quota-Bucket-Size'
'X-RGW-Quota-Bucket-Objects'
to Fix: https://tracker.ceph.com/issues/68211

Signed-off-by: Raja Sharma raja@ibm.com

* sts: remove test_get_session_token_permanent_creds_denied

Fixes: https://tracker.ceph.com/issues/69001

Signed-off-by: Casey Bodley <cbodley@redhat.com>

* rgw/logging: basic set of tests for bucket logging

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

* rgw/logging: tests for flush API

as well as test for object key filters

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

* HeadBucket: add read-stats query string for usage test

Refs: ceph/ceph#56756

Signed-off-by: Seena Fallah <seenafallah@gmail.com>

* test get_object_attributes

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

* multipart fallback to create-multipart checksum algorithm

there seem to be workloads which assume checksum algorithm can be
omitted from upload-part

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

* more tests for GetObjectAttributes

* multipart upload without checksums
* multipart upload with a single part
* pagination of multipart parts
* non-multipart upload with/without checksum
* versioned object, current and non-current
* sse-c encrypted object

Signed-off-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

* mark attribute tests as failing on dbstore (for now)

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

* rgw/logging: test tenant support

when source and log buckets are on a specific tenant, or
on different tenants

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

* rgw/logging: test source and log have different names

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

* rgw/logging: test log bucket encryption error

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

* rgw/logging: add cleanup tests

* use flushing api instead of sleep when possible
* test flushing concurrency

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

* rgw/logging: fix typo in readme

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

* rgw test-case to support x-expected-bucket-owner
If the bucket is owned by a different account, the request fails with the HTTP status code 403 Forbidden (access denied).
PR: ceph/ceph#61215
Fixes: https://tracker.ceph.com/issues/64526

Signed-off-by: Raja Sharma <raja@ibm.com>

* test-case to fix obj by multipart upload cant get tag

Signed-off-by: Raja Sharma <raja@ibm.com>

* add test case for stacked delete-markers

Signed-off-by: Jane Zhu <jzhu116@bloomberg.net>

* rgw|BN: fix test to expect authorization failure

Signed-off-by: Ali Masarwa <ali.saed.masarwa@gmail.com>

* requirements: unpin botocore version

Signed-off-by: Casey Bodley <cbodley@redhat.com>

* s3: mark sigv2 tests with fails_on_rgw

Signed-off-by: Casey Bodley <cbodley@redhat.com>

* s3: sse-kms policy tests use v4 client

Signed-off-by: Casey Bodley <cbodley@redhat.com>

* s3: clean up use of tenanted bucket names

the botocore.handlers.validate_bucket_name validation is enabled by
default, but we can disable it with unregister(). this avoids having to
mess with the signature or url

Signed-off-by: Casey Bodley <cbodley@redhat.com>

* s3: disable request_checksum_calculation for test_multipart_checksum_upload_fallback

the test sends create_multipart_upload() with a requested checksum
algorithm, and upload_part() requests without. newer boto automatically
adds `x-amz-sdk-checksum-algorithm: CRC32` when nothing is specified,
so we have to explicitly disable that behavior via botocore config

Signed-off-by: Casey Bodley <cbodley@redhat.com>

* s3: disable multipart/ranged tests with checksum failures

boto update exposed some bugs in our checksum feature around multipart
copy and ranged requests. disabling them with fails_on_rgw until
https://tracker.ceph.com/issues/69936 is resolved

Signed-off-by: Casey Bodley <cbodley@redhat.com>

* iam: remove invalid Bucket param from list_buckets()

> botocore.exceptions.ParamValidationError: Parameter validation failed:
> Unknown parameter in input: "Bucket", must be one of: MaxBuckets, ContinuationToken, Prefix, BucketRegion

Signed-off-by: Casey Bodley <cbodley@redhat.com>

* s3: disable checksum calculation for test_object_create_bad_contentlength_negative

if tls is enabled, boto will switch to STREAMING-UNSIGNED-PAYLOAD-TRAILER
and omit the provided content-length header. this leads to test failure:

> ________________ test_object_create_bad_contentlength_negative _________________
>       e = assert_raises(ClientError, client.put_object, Bucket=bucket_name, Key=key_name, ContentLength=-1)
> AssertionError: ClientError not raised

Signed-off-by: Casey Bodley <cbodley@redhat.com>

* sns: clear region_name for s3 clients

work around sns failures due to default region_name='':

> ERROR s3tests_boto3/functional/test_sns.py::test_account_topic_publish - botocore.exceptions.EndpointResolutionError: Invalid region: region was not a valid DNS name.
> ERROR s3tests_boto3/functional/test_sns.py::test_cross_account_topic_publish - botocore.exceptions.EndpointResolutionError: Invalid region: region was not a valid DNS name.

Signed-off-by: Casey Bodley <cbodley@redhat.com>

* rgw/iam: add RemoveClientIDFromOpenIDConnectProvider

Signed-off-by: Raja Sharma <raja@ibm.com>

* s3: reenable tests with range requests and checksums

Signed-off-by: Casey Bodley <cbodley@redhat.com>

* rgw/restore: s3tests to test restore object functionality.

This tests are added to tests temporary restore, permanent restore and
read through resotre object functionality, this includes zonegroup
parameters and checks.

Signed-off-by: shreyanshjain7174 <ssanchet@redhat.com>
Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>

* s3: test that object lock can be enabled on an existing bucket

Signed-off-by: Casey Bodley <cbodley@redhat.com>

* running cloud restore tests seperately

Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>

* rgw/s3tests: removing fails_on_rgw for
test_versioned_concurrent_object_create_concurrent_remove
as the associated tracker is marked as resolved and
the test also passes against main.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

* s3: test bucket policy evaluation for CreateMultipartUpload

test case for https://tracker.ceph.com/issues/70191

Signed-off-by: Casey Bodley <cbodley@redhat.com>

* mark test_versioning_stack_delete_merkers fails_on_dbstore

Signed-off-by: Jane Zhu <jzhu116@bloomberg.net>

* cloud-restore: testcase for non-current versioned object

Signed-off-by: Soumya Koduri <skoduri@redhat.com>

* rgw/s3tests: handle allow read through values in cloud transitions cases

Check the allow_read_through values for cases test_lifecycle_cloud_transition
and test_read_through

Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>

* BucketPolicy: add tests for ConfirmRemoveSelfBucketAccess header

Refrence: ceph/ceph#57629
Signed-off-by: Seena Fallah <seenafallah@gmail.com>

* rgw/logging: add missing marker to remove warnings

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

* test put_bucket_acl with group uri

Signed-off-by: Casey Bodley <cbodley@redhat.com>

* rgw/logging: add basic policy to target bucket

this commit needed to be able to run bucket logging regression
against: ceph/ceph#62284
since target bucket requires policy for bucket logging to work
this only covers the positive cases from bucket logging policy
perspective

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

* rgw/logging: add more bucket logging error cases

this is to cover these checks:
* source bucket ownership
* "requester pays" on log bucket

that were added in: ceph/ceph#62284

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

* rgw/logging: add bucket logging permission tests

this is to cover new functionality added in:
ceph/ceph#62284

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

* add minimal put-object for CRC64NVME

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

* enhance additional checksum tests

includes tests for CRC64NVME, tests for selecting
COMPOSITE and FULL_OBJECT checksums

a decomposed matrix of tests for all checksum types

also removes the mixed checksum upload case that no longer
works in recent boto3

cleanups, add sha1

checksum validation failure (mismatch) returns BadDigest

multipart checksum matrix helper now validates checksum and checksum type
for all operations which can return them (complete-multipart, head-object,
and get-object-attributes)

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

* rgw/logging: fix tenant test

tests were failing in teuthology since the tenanted user name
already have the tenant in its name

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

* rgw/logging: test bucket logging with accounts

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

* rgw/cloud-transition: increase time out for test cases

Fixes: https://tracker.ceph.com/issues/69411
Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>

* rgw/logging: mark test that require rollover time set

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

* add test coverage for DeleteBucketLifecycle

Signed-off-by: Casey Bodley <cbodley@redhat.com>

* rgw/logging: add tests for flushed object name

when using API call to flush the pending logging object of
the bucket, the flshed object name is returned.
tests were added to verify the right name is returned.

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

* rgw/logging: add tests to verify wildcrad policy

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

* Tag copy tests

* rgw/logging: test object metadata changes in journal mode

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

* rgw/logging: test mpu complete in standard mode

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

* rgw/logging: test log object name order

verify that the names of the log objects in the log bucket
are lexicographically ordered

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

* rgw/logging: test part upload in mpu in standard mode

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

* s3: test retry after error instead of 100-continue

Signed-off-by: Casey Bodley <cbodley@redhat.com>

* s3: avoid 100-continue bug for test_encryption_sse_c_deny_algo_with_bucket_policy

was failing for a reason unrelated to sse-c or bucket policy

Fixes: https://tracker.ceph.com/issues/70607

Signed-off-by: Casey Bodley <cbodley@redhat.com>

* s3: Add tests for test_upload_part_copy_percent_encoded_key

* s3: fix: Changed test so that it accepts the character that has % sign in it as a key

* add missing 'assert' to test_bucket_logging_roll_time

saw a failure after this:
```
        response = client.list_objects_v2(Bucket=log_bucket_name)
        keys = _get_keys(response)
        len(keys) == 1

>       key = keys[0]
E       IndexError: list index out of range

s3tests_boto3/functional/test_s3.py:16637: IndexError
```

Signed-off-by: Casey Bodley <cbodley@redhat.com>

* website: use pytest parametrize for test_routing_generator()

resolves error with recent pytest:

==================================== ERRORS ====================================
____________ ERROR collecting s3tests/functional/test_s3_website.py ____________
'yield' keyword is allowed in fixtures, but not in tests (test_routing_generator)

use https://docs.pytest.org/en/7.1.x/example/parametrize.html instead to
generate a test_routing_generator() for each entry in ROUTING_RULES_TESTS

Signed-off-by: Casey Bodley <cbodley@redhat.com>

* website: routing_setup() as pytest fixture

c80e9d2 had removed the use of
routing_setup() without adding a fixture to replace it. this had broken
routing_check(), but we hadn't noticed it because pytest hadn't
supported the 'yield' pattern in test_routing_generator() to call it

test_routing_generator() now uses this routing_setup fixture to provide
the kwargs that had previously been injected into routing_check() via
@common.with_setup_kwargs(setup=routing_setup, teardown=routing_teardown)

Signed-off-by: Casey Bodley <cbodley@redhat.com>

* increase time out for restore op in test_restore_object_permanent
and in test_lifecycle_expiration_size_gt() for lc.

Fixes: https://tracker.ceph.com/issues/71348
Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>

* fix: Fix typo

Signed-off-by: Emin <emin.sunacoglu@clyso.com>

* rgw/logging: test conf last modified time

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

* rgw/logging: add tests for logging object key format

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

* rgw/sts: GetAccountSummary test

Signed-off-by: Raja Sharma <raja@ibm.com>

* readme: fix typo

Fixed typos in the example commands.

Signed-off-by: N Balachandran <nithya.balachandran@ibm.com>

* s3: test_block_public_object_canned_acls() tests that 'private' works

Fixes: https://tracker.ceph.com/issues/49135

Signed-off-by: Casey Bodley <cbodley@redhat.com>

* bucket-logging: tests for the ACLRequired field

Tests that the AclRequired field is set correctly in the bucket logging record.

Signed-off-by: N Balachandran <nithya.balachandran@ibm.com>

* bucket-logging: fix typo

Fixes typo in function name.

Signed-off-by: N Balachandran <nithya.balachandran@ibm.com>

* bucket-logging: test the AuthType log field

Adds a test to check the value of the AuthType field.

Signed-off-by: N Balachandran <nithya.balachandran@ibm.com>

* rgw/sts: GetCallerIndentity test

Signed-off-by: Raja Sharma <raja@ibm.com>

* delete, rgwlc: delete marker tests

0. non-creation in non-versioned buckets
1. creation in versioned buckets
2. creation in versioning-suspended buckets
3. delete-marker expiration (lifecycle)

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

* bucket_logging: remove duplicate function

Removes the duplicate _verify_record_field function definition.

Signed-off-by: N Balachandran <nithya.balachandran@ibm.com>

* s3: test put_object() with IfMatch and IfNoneMatch

Signed-off-by: Casey Bodley <cbodley@redhat.com>

* s3: test delete_object() with IfMatch

Signed-off-by: Casey Bodley <cbodley@redhat.com>

* s3: test delete_object() with IfMatchLastModifiedTime

Signed-off-by: Casey Bodley <cbodley@redhat.com>

* s3: test delete_object() with IfMatchSize

Signed-off-by: Casey Bodley <cbodley@redhat.com>

* s3: test delete_objects() with ETag

Signed-off-by: Casey Bodley <cbodley@redhat.com>

* s3: test delete_objects() with LastModifiedTime

Signed-off-by: Casey Bodley <cbodley@redhat.com>

* s3: test delete_objects() with Size

Signed-off-by: Casey Bodley <cbodley@redhat.com>

* s3: fixed tests

we expected an error if we delete non-existent objects but that is not
true

Signed-off-by: Ali Masarwa <ali.saed.masarwa@gmail.com>

* s3: more testing for conditional put

Signed-off-by: Ali Masarwa <ali.saed.masarwa@gmail.com>

* a fix for empty results.for some of the queries
an empty result that may rarely happen, could cause an assertion.
the fix handle that cases.

Signed-off-by: gal salomon <gal.salomon@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* rgw/cloud-restore: verify if the expiry-date is updated

This is the test-case for ceph/ceph#64804.
If the `restoe-object` request is re-issued on a cloud-restored object,
it should update restoration period accordingly from the time the request
is re-issued.

Signed-off-by: Soumya Koduri <skoduri@redhat.com>

* s3: test list_buckets with MaxBuckets/ContinuationToken

Signed-off-by: Casey Bodley <cbodley@redhat.com>

* s3: test for aws-chunked in Content-Encoding responses

aws-chunked should not be persisted on upload for return on download

Signed-off-by: Casey Bodley <cbodley@redhat.com>

* test_s3 versioning multi object delete - add more versions for each object

Signed-off-by: Oguzhan Ozmen <oozmen@bloomberg.net>

* test_s3: verify list_object_versions after multi-object delete with versioning

Extend test_versioning_concurrent_multi_object_delete to also verify that
list_object_versions returns no Versions or DeleteMarkers after all objects
and their versions have been deleted. This ensures both list_objects and
list_object_versions reflect an empty bucket.

* extend test_versioning_obj_suspended_copy to test copy to another non-versioned bucket

Signed-off-by: Jane Zhu <jzhu116@bloomberg.net>

* rgw/logging: modify tests to allow for committing empty objects

new tests were added and existing tests that assumed the previous
behavior were modified.
testing: ceph/ceph#65387

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

* s3: Mark tests known to fail on dbstore

Fixes: https://tracker.ceph.com/issues/72094

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

* rgw/logging: fix cleanup tests for rollovers

This is to test this PR: ceph/ceph#65456

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

* rgw/logging: add conf change specific tests

and also test updates while writing logs
fix: cover all src buckets in cleanup tests

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

* s3: test conditional multipart_put_object()

Signed-off-by: Ali Masarwa <ali.saed.masarwa@gmail.com>
Signed-off-by: Ali Masarwa <ALI.SAED.MASARWA@gmail.com>

* rgw/logging: verify error message when AccessDenied on target

testing: ceph/ceph#65494

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

* s3: test object ownership

Signed-off-by: Casey Bodley <cbodley@redhat.com>

* qa/rgw: Include rgw_restore_processor_period in restore tests

This adjusts restore period as per ceph/ceph#64933

Also deleting lifecycle post transition so that temp restored files
remain for a while without being re-transitioned immeditately.

Signed-off-by: Soumya Koduri <skoduri@redhat.com>

* remove boto2 tests and dependency

boto2 import fails on recent python with:
> E   ModuleNotFoundError: No module named 'boto.vendored.six.moves'

remove all tests under s3tests/ that weren't converted to boto3

Fixes: https://tracker.ceph.com/issues/72876

Signed-off-by: Casey Bodley <cbodley@redhat.com>

* rename s3tests_boto3/ back to s3tests/

Signed-off-by: Casey Bodley <cbodley@redhat.com>

* rgw/restore: Fixing readthrough

Signed-off-by: Soumya Koduri <skoduri@redhat.com>

* s3: test checksum for get_object with part number

* rgw/logging: change test to match the new source cleanup process

* in "journal" mode, when multiple source buckets share the same log prefix,
the first one being deleted will remove the object holding the temporary
object name, and only the first would commit the temp object.
* in "standard" mode, the other buckets will recreate the object holding
the temporary object name, becase the bucket deletion operation will be
logged as well.
* added tests where only one of the buckets sharing a common prefix is
  deleted and the rest are not

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

* CopyObject: add test for encrypted objects

Add test for copy on sse-s3 and sse-c encrypted objects.

Signed-off-by: Seena Fallah <seenafallah@gmail.com>

* CopyObject: add storage class copy tests

Signed-off-by: Seena Fallah <seenafallah@gmail.com>

* lc-transition: add test with encrypted objects

Signed-off-by: Seena Fallah <seenafallah@gmail.com>

* UploadPart: add copy part with encryption

Signed-off-by: Seena Fallah <seenafallah@gmail.com>

* CompleteMultipartUpload: test to verify sse-c headers

Signed-off-by: Seena Fallah <seenafallah@gmail.com>

* UploadPart: test to verify sse-c headers

Signed-off-by: Seena Fallah <seenafallah@gmail.com>

* refactor: merge test_copy_enc_ with test_copy_enc_storage_class

Signed-off-by: Seena Fallah <seenafallah@gmail.com>

* test_multipart_sse_c_get_part: pass enc attrs on mp complete

Based on the new fixes around copy enc, those headers are required
when completing mp upload.

Signed-off-by: Seena Fallah <seenafallah@gmail.com>

* functional: fix invalid escape sequence deprication warning

This change address multiple messages in the log:

    DeprecationWarning: invalid escape sequence '\$'

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

* rgw/bucket-logging: handle async log commits

Modified to work with the new asynchronous buckt logging commit
mechanism.

Fixes: https://tracker.ceph.com/issues/71365

Signed-off-by: Nithya Balachandran <nithya.balachandran@ibm.com>

* s3: update etag for get_part tests

* Revert "SPL-170872 mark few false negative tests for splunk skip. (#6)"

This reverts commit 3dc9362.

* Revert "Mark false negative tests in s3-test"

This reverts commit ec2aef0.

* Revert "update requirements.txt && fix README typo"

This reverts commit 4b10f5a.

* Revert "splunk S3-compatibility checking tool v1"

This reverts commit 69b2678.

* Update to latest ceph/s3-tests, fix all tests, and introduce core S3 compliance tests

- Rebased branch to latest upstream ceph/s3-tests
- Fixed all test failures for compatibility
- Added Splunk-specific compliance test lists and skip logic
- Updated README with Splunk S3 compliance instructions
- Introduced core test set for S3 compliance validation
- Improved configuration and test selection for Splunk environments

This update ensures the branch is up-to-date, core compliance tests are clearly defined for S3 validation."

* s3 core tests shell script after deselecting some tests

* fixed the test sets

---------

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>
Signed-off-by: Gal Salomon <gal.salomon@gmail.com>
Signed-off-by: Sumedh A. Kulkarni <sumedh.a.kulkarni@seagate.com>
Signed-off-by: Bob Ham <bham12@bloomberg.net>
Signed-off-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Seena Fallah <seenafallah@gmail.com>
Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>
Signed-off-by: galsalomon66 <gal.salomon@gmail.com>
Signed-off-by: Tobias Urdin <tobias.urdin@binero.com>
Signed-off-by: Soumya Koduri <skoduri@redhat.com>
Signed-off-by: Raja Sharma raja@ibm.com
Signed-off-by: Raja Sharma <raja@ibm.com>
Signed-off-by: Jane Zhu <jzhu116@bloomberg.net>
Signed-off-by: Ali Masarwa <ali.saed.masarwa@gmail.com>
Signed-off-by: shreyanshjain7174 <ssanchet@redhat.com>
Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>
Signed-off-by: Emin <emin.sunacoglu@clyso.com>
Signed-off-by: N Balachandran <nithya.balachandran@ibm.com>
Signed-off-by: gal salomon <gal.salomon@gmail.com>
Signed-off-by: Oguzhan Ozmen <oozmen@bloomberg.net>
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Signed-off-by: Ali Masarwa <ALI.SAED.MASARWA@gmail.com>
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>
Signed-off-by: Nithya Balachandran <nithya.balachandran@ibm.com>
Co-authored-by: Matt Benjamin <mbenjamin@redhat.com>
Co-authored-by: Yuval Lifshitz <ylifshit@ibm.com>
Co-authored-by: Casey Bodley <cbodley@users.noreply.github.com>
Co-authored-by: Gal Salomon <gal.salomon@gmail.com>
Co-authored-by: Sumedh A. Kulkarni <sumedh.a.kulkarni@seagate.com>
Co-authored-by: Casey Bodley <cbodley@redhat.com>
Co-authored-by: Seena Fallah <seenafallah@gmail.com>
Co-authored-by: Pritha Srivastava <prsrivas@redhat.com>
Co-authored-by: Tobias Urdin <tobias.urdin@binero.com>
Co-authored-by: Soumya Koduri <skoduri@redhat.com>
Co-authored-by: Raja Sharma <raja@ibm.com>
Co-authored-by: Adam Emerson <aemerson@redhat.com>
Co-authored-by: Yuval Lifshitz <yuvalif@yahoo.com>
Co-authored-by: Jane Zhu <jzhu116@bloomberg.net>
Co-authored-by: Raja <rajaramsharma422@gmail.com>
Co-authored-by: Ali Masarwa <ali.saed.masarwa@gmail.com>
Co-authored-by: shreyanshjain7174 <ssanchet@redhat.com>
Co-authored-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>
Co-authored-by: Andrew Gaul <andrew@gaul.org>
Co-authored-by: Emin <emin.sunacoglu@clyso.com>
Co-authored-by: N Balachandran <nithya.balachandran@ibm.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Oguzhan Ozmen <oozmen@bloomberg.net>
Co-authored-by: sungjoon_koh <sungjoon_koh@linecorp.com>
Co-authored-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>
Co-authored-by: chamara-splunk <cwickram@cisco.com>