Skip to content

RGW | fix conditional Delete, MultiDelete and Put#63348

Merged
AliMasarweh merged 2 commits intoceph:mainfrom
AliMasarweh:wip-alimasa-68183
Jul 28, 2025
Merged

RGW | fix conditional Delete, MultiDelete and Put#63348
AliMasarweh merged 2 commits intoceph:mainfrom
AliMasarweh:wip-alimasa-68183

Conversation

@AliMasarweh
Copy link
Member

https://tracker.ceph.com/issues/68183
https://bugzilla.redhat.com/show_bug.cgi?id=2350732

Contribution Guidelines

  • To sign and title your commits, please refer to Submitting Patches to Ceph.

  • If you are submitting a fix for a stable branch (e.g. "quincy"), please refer to Submitting Patches to Ceph - Backports for the proper workflow.

  • When filling out the below checklist, you may click boxes directly in the GitHub web UI. When entering or editing the entire PR message in the GitHub web UI editor, you may also select a checklist item by adding an x between the brackets: [x]. Spaces and capitalization matter when checking off items this way.

Checklist

  • Tracker (select at least one)
    • References tracker ticket
    • Very recent bug; references commit where it was introduced
    • New feature (ticket optional)
    • Doc update (no ticket needed)
    • Code cleanup (no ticket needed)
  • Component impact
    • Affects Dashboard, opened tracker ticket
    • Affects Orchestrator, opened tracker ticket
    • No impact that needs to be tracked
  • Documentation (select at least one)
    • Updates relevant documentation
    • No doc update is appropriate
  • Tests (select at least one)
Show available Jenkins commands

@AliMasarweh AliMasarweh requested a review from a team as a code owner May 19, 2025 12:15
@github-actions github-actions bot added the rgw label May 19, 2025
@cbodley
Copy link
Contributor

cbodley commented May 19, 2025

looking for IfMatch/IfNoneMatch in s3-tests, i only see test cases for get_object() which wouldn't call through prepare_atomic_modification()

could you please add test cases for put_object() and complete_multipart_upload() that covers both etag values and the * wildcard?

@AliMasarweh
Copy link
Member Author

looking for IfMatch/IfNoneMatch in s3-tests, i only see test cases for get_object() which wouldn't call through prepare_atomic_modification()

could you please add test cases for put_object() and complete_multipart_upload() that covers both etag values and the * wildcard?

sure, @mattbenjamin said that there are more cases that need fix such as:
multidelete and multiput looking for a LastModified
I will do that then write the tests

@AliMasarweh AliMasarweh changed the title RGW: conditional write fix for if-match={etag} RGW: conditional write fix for if-match and if-match-last-modified-time May 22, 2025
@AliMasarweh AliMasarweh changed the title RGW: conditional write fix for if-match and if-match-last-modified-time RGW | fix conditional write for If-Match and If-Unmodified-Since Jun 1, 2025
@AliMasarweh AliMasarweh force-pushed the wip-alimasa-68183 branch 2 times, most recently from e24ca5a to 9d3a326 Compare June 4, 2025 13:53
@github-actions github-actions bot added the tests label Jun 4, 2025
@AliMasarweh AliMasarweh changed the title RGW | fix conditional write for If-Match and If-Unmodified-Since RGW | fix conditional write for If-Match and If-Match-Last-Modified-Time Jun 4, 2025
@AliMasarweh AliMasarweh force-pushed the wip-alimasa-68183 branch 3 times, most recently from 6234330 to 0627bf2 Compare June 10, 2025 09:36
@AliMasarweh AliMasarweh force-pushed the wip-alimasa-68183 branch 3 times, most recently from 02498a1 to 186325b Compare June 11, 2025 11:10
@AliMasarweh
Copy link
Member Author

I think the issue in conditional put with versioned bucket, is this block of code in RGWRados::get_obj_state_impl:

  if (s->has_attrs) {
    if (s->is_olh && need_follow_olh) {
      return get_olh_target_state(dpp, *octx, bucket_info, obj, s, psm, y);
    }
    return 0;
  }

we return from this function and the object doesn't have an etag in attrs

why are we expecting an attrs here?

@dang
Copy link
Contributor

dang commented Jun 18, 2025

I think the issue is that there are two objects in question here. The unversioned object, and the versioned object. That sequence you posted does load the state (and so the attributes) of the versioned object, but it doesn't do anything to the unversioned object. Where is this being called from?

@mattbenjamin
Copy link
Contributor

the subject says "fix conditional write" but is this PR going to address DeleteObject only?

@cbodley
Copy link
Contributor

cbodley commented Jun 18, 2025

I think the issue in conditional put with versioned bucket, is this block of code in RGWRados::get_obj_state_impl:

#60957 reported a similar issue here, but the fix wasn't passing tests :(

edit: i think that was specific to multipart uploads though

@cbodley
Copy link
Contributor

cbodley commented Jun 18, 2025

I think the issue in conditional put with versioned bucket, is this block of code in RGWRados::get_obj_state_impl:

@AliMasarweh what are you expecting conditional put to do in a versioned bucket? with versioning enabled, writes create a new object version instead of overwriting an existing one. so If-Match would not find an existing etag for comparison

edit: maybe i'm mistaken, https://docs.aws.amazon.com/AmazonS3/latest/userguide/conditional-writes.html#conditional-error-response talks about interactions with versioning enabled/suspended

@AliMasarweh AliMasarweh changed the title RGW | fix conditional write for If-Match and If-Match-Last-Modified-Time RGW | fix conditional Delete, MultiDelete and Put Jun 19, 2025
@AliMasarweh
Copy link
Member Author

https://pulpito.ceph.com/alimasa-2025-07-17_02:35:01-rgw-wip-alimasa-68183-distro-default-smithi/
is this run good enough?

@cbodley
Copy link
Contributor

cbodley commented Jul 17, 2025

https://pulpito.ceph.com/alimasa-2025-07-17_02:35:01-rgw-wip-alimasa-68183-distro-default-smithi/ is this run good enough?

yikes, lots of red there. looks mostly unrelated except for one rgw/verify job: https://qa-proxy.ceph.com/teuthology/alimasa-2025-07-17_02:35:01-rgw-wip-alimasa-68183-distro-default-smithi/8392360/teuthology.log

test_s3.py::test_delete_objects_version_if_match_last_modified_time FAILED [ 94%]

___________ test_delete_objects_version_if_match_last_modified_time ____________

    @pytest.mark.fails_on_aws # only supported for directory buckets
    @pytest.mark.conditional_write
    def test_delete_objects_version_if_match_last_modified_time():
        client = get_client()
        bucket = get_new_bucket(client)
        check_configure_versioning_retry(bucket, "Enabled", "Enabled")
        key = 'obj'

        badmtime = datetime.datetime(2015, 1, 1)

        version = client.put_object(Bucket=bucket, Key=key)['VersionId']
        client.put_object(Bucket=bucket, Key=key)
        mtime = client.head_object(Bucket=bucket, Key=key)['LastModified']

        response = client.delete_objects(Bucket=bucket, Delete={'Objects': [{'Key': key, 'VersionId': version, 'LastModifiedTime': badmtime}]})
        assert 'PreconditionFailed' == response['Errors'][0]['Code']

        response = client.delete_objects(Bucket=bucket, Delete={'Objects': [{'Key': key, 'VersionId': version, 'LastModifiedTime': mtime}]})
>       assert key == response['Deleted'][0]['Key'] # success
E       KeyError: 'Deleted'

s3tests_boto3/functional/test_s3.py:18385: KeyError

Copy link
Contributor

@mattbenjamin mattbenjamin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I made a few comments related to readability; in addition, there are now too many commits to cleanly backport; can you please squash this down to no more than a few logically related commits?

@AliMasarweh AliMasarweh requested a review from a team as a code owner July 21, 2025 14:28
@github-actions github-actions bot added the rbd label Jul 21, 2025
size_match supports size 0
checks_preconditions checks for last_modified and size as well
supports versioned object

Signed-off-by: Ali Masarwa <amasarwa@redhat.com>
@idryomov idryomov removed the rbd label Jul 21, 2025
@idryomov idryomov removed the request for review from a team July 21, 2025 15:23
@AliMasarweh
Copy link
Member Author

do we have a stable commit on main that I should rebase on my code, teuthology has too many valgrind error:
https://pulpito.ceph.com/alimasa-2025-07-21_14:44:30-rgw-wip-alimasa-68183-distro-default-smithi/

@mattbenjamin mattbenjamin self-requested a review July 23, 2025 17:48
Copy link
Contributor

@mattbenjamin mattbenjamin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

appreciate the cleanup, new change backported with only the async dispatch conflict, seems to have passed downstream QA; thank you!

@kayrus
Copy link

kayrus commented Jul 23, 2025

qa/rgw/s3tests-branch.yaml modification should be cleaned up as well
#63348 (comment)

Signed-off-by: Ali Masarwa <amasarwa@redhat.com>
@AliMasarweh
Copy link
Member Author

jenkins test make check arm64

@AliMasarweh AliMasarweh merged commit 7850229 into ceph:main Jul 28, 2025
13 checks passed
@ivancich
Copy link
Member

ivancich commented Aug 11, 2025

This PR appears to have caused some significant breakage:

A new error regarding orphan list showed up during the teuthology run that indicated a regression:

https://pulpito.ceph.com/alimasa-2025-07-28_14:08:21-rgw-wip-alimasa-68183-distro-default-smithi/8412051

I bisected the orphan list failure to commit 55f5b76 , which is one of the two commits in this PR.

@AliMasarweh AliMasarweh mentioned this pull request Sep 17, 2025
14 tasks
@cupnes
Copy link

cupnes commented Sep 17, 2025

Do you have any plans to backport this to squid?

@ivancich
Copy link
Member

Do you have any plans to backport this to squid?

The tracker does not list any backports, and we try to record backports there. But @AliMasarweh may know differently....

@cbodley
Copy link
Contributor

cbodley commented Sep 17, 2025

Do you have any plans to backport this to squid?

thanks @cupnes, i updated https://tracker.ceph.com/issues/68183 to request squid and tentacle backports

@cupnes
Copy link

cupnes commented Sep 18, 2025

@ivancich @cbodley
I understood. Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

9 participants