Skip to content

rgw: fix GetObject ETag with part number parameter #65194

Merged
cbodley merged 1 commit intoceph:mainfrom
sungjoon-koh:get-object-part-etag
Dec 4, 2025
Merged

rgw: fix GetObject ETag with part number parameter #65194
cbodley merged 1 commit intoceph:mainfrom
sungjoon-koh:get-object-part-etag

Conversation

@sungjoon-koh
Copy link
Copy Markdown
Contributor

@sungjoon-koh sungjoon-koh commented Aug 23, 2025 •
edited by jmundack
Loading

  • Changed from returning part MD5 to object ETag

Modified ETag to return the object's ETag instead of the individual part's MD5 hash when part number is specified in GetObject requests. This change aligns with AWS S3 specification.

Corresponding S3 test update - ceph/s3-tests#699

Contribution Guidelines

  • To sign and title your commits, please refer to Submitting Patches to Ceph.

  • If you are submitting a fix for a stable branch (e.g. "quincy"), please refer to Submitting Patches to Ceph - Backports for the proper workflow.

  • When filling out the below checklist, you may click boxes directly in the GitHub web UI. When entering or editing the entire PR message in the GitHub web UI editor, you may also select a checklist item by adding an x between the brackets: [x]. Spaces and capitalization matter when checking off items this way.

Checklist

  • Tracker (select at least one)
    • References tracker ticket
    • Very recent bug; references commit where it was introduced
    • New feature (ticket optional)
    • Doc update (no ticket needed)
    • Code cleanup (no ticket needed)
  • Component impact
    • Affects Dashboard, opened tracker ticket
    • Affects Orchestrator, opened tracker ticket
    • No impact that needs to be tracked
  • Documentation (select at least one)
    • Updates relevant documentation
    • No doc update is appropriate
  • Tests (select at least one)
Show available Jenkins commands

@sungjoon-koh sungjoon-koh requested a review from a team as a code owner August 23, 2025 10:13
@github-actions github-actions Bot added the rgw label Aug 23, 2025
@sungjoon-koh
Copy link
Copy Markdown
Contributor Author

sungjoon-koh commented Aug 23, 2025 •
edited
Loading

Test scenario

$ fallocate part --len 5M
$ aws s3api create-bucket --bucket test
$ aws s3api create-multipart-upload --bucket test --key testobj

# set UI=<upload id>
$ aws s3api upload-part --bucket test --key testobj --part-number 1 --body part --upload-id $UI
$ aws s3api upload-part --bucket test --key testobj --part-number 2 --body part --upload-id $UI
$ aws s3api complete-multipart-upload --bucket test --key testobj --upload-id $UI --multipart-upload file://parts.json

Result before fix

$ aws s3api get-object --bucket test --key testobj --part-number=1 mp-object-res
{
    "AcceptRanges": "bytes",
    "LastModified": "2025-08-23T09:40:34+00:00",
    "ContentLength": 5242880,
    "ETag": "\"5f363e0e58a95f06cbe9bbc662c5dfb6\"",
    "ContentType": "binary/octet-stream",
    "Metadata": {},
    "PartsCount": 2
}

$ aws s3api get-object --bucket test --key testobj --part-number=2 mp-object-res
{
    "AcceptRanges": "bytes",
    "LastModified": "2025-08-23T09:40:35+00:00",
    "ContentLength": 5242880,
    "ETag": "\"5f363e0e58a95f06cbe9bbc662c5dfb6\"",
    "ContentType": "binary/octet-stream",
    "Metadata": {},
    "PartsCount": 2
}

$ aws s3api get-object --bucket test --key testobj mp-object-res
{
    "AcceptRanges": "bytes",
    "LastModified": "2025-08-23T09:40:36+00:00",
    "ContentLength": 10485760,
    "ETag": "\"a7d414b9133d6483d9a1c4e04e856e3b-2\"",
    "ContentType": "binary/octet-stream",
    "Metadata": {},
    "PartsCount": 2
}

Result after fix

$ aws s3api get-object --bucket test --key testobj --part-number=1 mp-object-res
{
    "AcceptRanges": "bytes",
    "LastModified": "2025-08-23T10:08:28+00:00",
    "ContentLength": 5242880,
    "ETag": "\"a7d414b9133d6483d9a1c4e04e856e3b-2\"",
    "ContentType": "binary/octet-stream",
    "Metadata": {},
    "PartsCount": 2
}

$ aws s3api get-object --bucket test --key testobj --part-number=2 mp-object-res
{
    "AcceptRanges": "bytes",
    "LastModified": "2025-08-23T10:08:29+00:00",
    "ContentLength": 5242880,
    "ETag": "\"a7d414b9133d6483d9a1c4e04e856e3b-2\"",
    "ContentType": "binary/octet-stream",
    "Metadata": {},
    "PartsCount": 2
}

$ aws s3api get-object --bucket test --key testobj mp-object-res
{
    "AcceptRanges": "bytes",
    "LastModified": "2025-08-23T10:08:32+00:00",
    "ContentLength": 10485760,
    "ETag": "\"a7d414b9133d6483d9a1c4e04e856e3b-2\"",
    "ContentType": "binary/octet-stream",
    "Metadata": {},
    "PartsCount": 2
}

@sungjoon-koh
rgw: fix GetObject ETag with part number parameter
9100fd5 
- Changed from returning part MD5 to object ETag

Signed-off-by: sungjoon_koh <sungjoon_koh@linecorp.com>
@sungjoon-koh
Copy link
Copy Markdown
Contributor Author

sungjoon-koh commented Aug 23, 2025

@cbodley Could you please check this PR? Thank you.

@sungjoon-koh
Copy link
Copy Markdown
Contributor Author

sungjoon-koh commented Aug 23, 2025 •
edited
Loading

aws S3 test result

$ aws s3api get-object --bucket sjtest1aa --key testobj --part-number=1 mp-object-res
{
    "AcceptRanges": "bytes",
    "LastModified": "2025-08-23T10:27:51+00:00",
    "ContentLength": 5242880,
    "ETag": "\"a7d414b9133d6483d9a1c4e04e856e3b-2\"",
    "ContentRange": "bytes 0-5242879/10485760",
    "ContentType": "binary/octet-stream",
    "ServerSideEncryption": "AES256",
    "Metadata": {},
    "PartsCount": 2
}

$ aws s3api get-object --bucket sjtest1aa --key testobj --part-number=2 mp-object-res
{
    "AcceptRanges": "bytes",
    "LastModified": "2025-08-23T10:27:51+00:00",
    "ContentLength": 5242880,
    "ETag": "\"a7d414b9133d6483d9a1c4e04e856e3b-2\"",
    "ContentRange": "bytes 5242880-10485759/10485760",
    "ContentType": "binary/octet-stream",
    "ServerSideEncryption": "AES256",
    "Metadata": {},
    "PartsCount": 2
}

$ aws s3api get-object --bucket sjtest1aa --key testobj  mp-object-res
{
    "AcceptRanges": "bytes",
    "LastModified": "2025-08-23T10:27:51+00:00",
    "ContentLength": 10485760,
    "ETag": "\"a7d414b9133d6483d9a1c4e04e856e3b-2\"",
    "ChecksumCRC64NVME": "r3XRsudNx4U=",
    "ChecksumType": "FULL_OBJECT",
    "ContentType": "binary/octet-stream",
    "ServerSideEncryption": "AES256",
    "Metadata": {}
}

@adamemerson
Copy link
Copy Markdown
Contributor

adamemerson commented Sep 12, 2025

@clwluvw Can you take a look and see if this looks reasonable to you?

@adamemerson adamemerson requested a review from clwluvw September 12, 2025 20:41
@clwluvw
Copy link
Copy Markdown
Member

clwluvw commented Sep 12, 2025

Perhaps it would be worth a test case in https://github.com/ceph/s3-tests as well.

@adamemerson adamemerson self-assigned this Sep 12, 2025
@sungjoon-koh sungjoon-koh mentioned this pull request Sep 19, 2025
Merged
@sungjoon-koh
Copy link
Copy Markdown
Contributor Author

sungjoon-koh commented Sep 19, 2025

@clwluvw I’ve updated the get_part tests and opened an s3-tests PR: ceph/s3-tests#699.
I’d greatly appreciate it if you could take a look at your convenience.

@adamemerson
Copy link
Copy Markdown
Contributor

adamemerson commented Sep 19, 2025

As a note: This has new s3tests and needs to be QA'd separately. It should not receive the needs-qa tag.

@clwluvw If the s3tests meet with your approval could you make a staging branch and run RGW QA against the new s3-tests?

@github-actions
Copy link
Copy Markdown

github-actions Bot commented Nov 18, 2025

This pull request has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs for another 30 days.
If you are a maintainer or core committer, please follow-up on this pull request to identify what steps should be taken by the author to move this proposed change forward.
If you are the author of this pull request, thank you for your proposed contribution. If you believe this change is still appropriate, please ensure that any feedback has been addressed and ask for a code review.

@github-actions github-actions Bot added the stale label Nov 18, 2025
@jmundack
Copy link
Copy Markdown
Contributor

jmundack commented Nov 18, 2025

can we get a tracker for this? and potential backports for the active releases?

@github-actions github-actions Bot removed the stale label Nov 18, 2025
@sungjoon-koh
Copy link
Copy Markdown
Contributor Author

sungjoon-koh commented Dec 2, 2025

@clwluvw Could you please review this PR at your convenience?

@cbodley cbodley mentioned this pull request Dec 2, 2025
Merged
14 tasks
@cbodley
Copy link
Copy Markdown
Contributor

cbodley commented Dec 2, 2025

i prepared a test branch https://github.com/ceph/ceph-ci/commits/wip-cbodley-testing that includes #66042 and #65194, and points to an s3-tests branch https://github.com/cbodley/s3-tests/commits/wip-cbodley-testing that includes ceph/s3-tests#704 and ceph/s3-tests#699

@cbodley
Copy link
Copy Markdown
Contributor

cbodley commented Dec 2, 2025

can we get a tracker for this? and potential backports for the active releases?

created https://tracker.ceph.com/issues/74060 for squid/tentacle backports

@cbodley
Copy link
Copy Markdown
Contributor

cbodley commented Dec 2, 2025

qa pending in https://pulpito.ceph.com/cbodley-2025-12-02_17:25:51-rgw-wip-cbodley-testing-distro-default-smithi/

@cbodley
Copy link
Copy Markdown
Contributor

cbodley commented Dec 4, 2025

qa pending in https://pulpito.ceph.com/cbodley-2025-12-02_17:25:51-rgw-wip-cbodley-testing-distro-default-smithi/

lots of failures, but i think they're all unrelated. the rgw/sts jobs showed new test failures, but i ran them against main in https://pulpito.ceph.com/cbodley-2025-12-04_17:21:30-rgw:sts-main-distro-default-smithi/ and saw the same - now tracking in https://tracker.ceph.com/issues/74099

confirmed that the updated test cases are passing (see example teuthology.log):

s3tests/functional/test_s3.py::test_multipart_get_part PASSED            [ 36%]
s3tests/functional/test_s3.py::test_multipart_sse_c_get_part PASSED      [ 37%]
s3tests/functional/test_s3.py::test_multipart_single_get_part PASSED     [ 37%]

@cbodley cbodley changed the title rgw: fix GetObject ETag with part number parameter rgw: fix GetObject ETag with part number parameter Dec 4, 2025
@cbodley cbodley merged commit 1c1bd1d into ceph:main Dec 4, 2025
15 of 19 checks passed
This was referenced Feb 4, 2026
Open
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@cbodley cbodley cbodley approved these changes

@clwluvw clwluvw Awaiting requested review from clwluvw

Assignees

@adamemerson adamemerson

Labels

rgw wip-cbodley-testing

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@sungjoon-koh @adamemerson @clwluvw @jmundack @cbodley