rgw: implement CopyObject for encrypted objects by clwluvw · Pull Request #63794 · ceph/ceph

clwluvw · 2025-06-07T20:27:33Z

Implement decrypt/decompress while reading and encrypt/compress while writing the data on CopyObject API call if needed.

S3 Tests: ceph/s3-tests#595
Fixes: https://tracker.ceph.com/issues/23264

Checklist

Tracker (select at least one)
- References tracker ticket
- Very recent bug; references commit where it was introduced
- New feature (ticket optional)
- Doc update (no ticket needed)
- Code cleanup (no ticket needed)
Component impact
- Affects Dashboard, opened tracker ticket
- Affects Orchestrator, opened tracker ticket
- No impact that needs to be tracked
Documentation (select at least one)
- Updates relevant documentation
- No doc update is appropriate
Tests (select at least one)
- Includes unit test(s)
- Includes integration test(s)
- Includes bug reproducer
- No tests

Show available Jenkins commands

jenkins test classic perf Jenkins Job | Jenkins Job Definition
jenkins test crimson perf Jenkins Job | Jenkins Job Definition
jenkins test signed Jenkins Job | Jenkins Job Definition
jenkins test make check Jenkins Job | Jenkins Job Definition
jenkins test make check arm64 Jenkins Job | Jenkins Job Definition
jenkins test submodules Jenkins Job | Jenkins Job Definition
jenkins test dashboard Jenkins Job | Jenkins Job Definition
jenkins test dashboard cephadm Jenkins Job | Jenkins Job Definition
jenkins test api Jenkins Job | Jenkins Job Definition
jenkins test docs ReadTheDocs | Github Workflow Definition
jenkins test ceph-volume all Jenkins Jobs | Jenkins Jobs Definition
jenkins test windows Jenkins Job | Jenkins Job Definition
jenkins test rook e2e Jenkins Job | Jenkins Job Definition

clwluvw · 2025-06-08T14:16:10Z

jenkins retest this please

clwluvw · 2025-06-09T02:00:07Z

jenkins test make check arm64

mattbenjamin · 2025-06-09T18:00:56Z

@clwluvw how does this differ from #54543?

clwluvw · 2025-06-09T19:32:12Z

@clwluvw how does this differ from #54543?

I have added the topic to the agenda for Wednesday to discuss.

clwluvw · 2025-06-16T10:45:24Z

@cbodley @mdw-at-linuxbox - fyi, I have added some fixes for UploadPart API when copying all forms of encryption and compression and are passing the tests from ceph/s3-tests@9d95bbf

github-actions · 2025-08-15T12:02:49Z

This pull request has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs for another 30 days.
If you are a maintainer or core committer, please follow-up on this pull request to identify what steps should be taken by the author to move this proposed change forward.
If you are the author of this pull request, thank you for your proposed contribution. If you believe this change is still appropriate, please ensure that any feedback has been addressed and ask for a code review.

taxilian · 2025-09-10T20:20:48Z

I'd sure appreciate having this finished; I use sseCustomerKey etc a lot, but having it break any time you use it with CopyObject makes it unusable for a lot of different projects.

github-actions · 2025-11-09T23:01:56Z

This pull request has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs for another 30 days.
If you are a maintainer or core committer, please follow-up on this pull request to identify what steps should be taken by the author to move this proposed change forward.
If you are the author of this pull request, thank you for your proposed contribution. If you believe this change is still appropriate, please ensure that any feedback has been addressed and ask for a code review.

taxilian · 2025-11-10T17:41:55Z

just as an example, this feature is required in order to use it with percona backup for mongodb

Co-authored-by: Marcus Watts <mwatts@redhat.com> Signed-off-by: Seena Fallah <seenafallah@gmail.com>

clwluvw · 2025-11-20T10:48:40Z

pending qa against ceph/s3-tests#595 in https://pulpito.ceph.com/cbodley-2025-11-19_21:47:16-rgw-wip-23264-distro-default-gibba/

Had a minor err when populating crypt header responses for complete multipart upload. Now tests on my local are passing as well:

$ S3TEST_CONF=s3tests.conf.SAMPLE tox -- -v -m 'not fails_on_rgw and not lifecycle_expiration and not lifecycle_transition and not cloud_transition and not test_of_sts and not webidentity_test and not fails_with_subdomain and not bucket_logging'
...
========================================================================================= 820 passed, 4 skipped, 217 deselected in 545.89s (0:09:05) =========================================================================================
__________________________________________________________________________________________________________________ summary ___________________________________________________________________________________________________________________
  py: commands succeeded
  congratulations :)

pushed new build for shaman on the same branch: https://shaman.ceph.com/builds/ceph/wip-23264/2157ea33c8bddc8cda3037df288f3bfa37d7b594/

cbodley · 2025-11-20T13:29:19Z

pushed new build for shaman on the same branch: https://shaman.ceph.com/builds/ceph/wip-23264/2157ea33c8bddc8cda3037df288f3bfa37d7b594/

rescheduled qa in https://pulpito.ceph.com/cbodley-2025-11-20_13:28:10-rgw-wip-23264-distro-default-gibba

clwluvw · 2025-11-20T15:46:02Z

pushed new build for shaman on the same branch: https://shaman.ceph.com/builds/ceph/wip-23264/2157ea33c8bddc8cda3037df288f3bfa37d7b594/

rescheduled qa in https://pulpito.ceph.com/cbodley-2025-11-20_13:28:10-rgw-wip-23264-distro-default-gibba

Missed sanitizing the copy source attrs while logging, pushed new build to shaman: https://shaman.ceph.com/builds/ceph/wip-23264/65a75ff9be13539fb1d4bbf4a0fd22f8b6638e55/

clwluvw · 2025-11-20T18:57:04Z

pushed new build to shaman: https://shaman.ceph.com/builds/ceph/wip-23264/65a75ff9be13539fb1d4bbf4a0fd22f8b6638e55/

rescheduled qa in https://pulpito.ceph.com/sfallah-2025-11-20_18:54:10-rgw-wip-23264-distro-default-gibba/

Co-authored-by: Marcus Watts <mwatts@redhat.com> Signed-off-by: Seena Fallah <seenafallah@gmail.com>

clwluvw · 2025-11-20T20:24:24Z

pushed new build to shaman: https://shaman.ceph.com/builds/ceph/wip-23264/65a75ff9be13539fb1d4bbf4a0fd22f8b6638e55/

rescheduled qa in https://pulpito.ceph.com/sfallah-2025-11-20_18:54:10-rgw-wip-23264-distro-default-gibba/

Found out another logic in sanitization here:

ceph/src/rgw/rgw_op.h

Lines 2378 to 2382 in d8a62ea

    
           if (blocklisted_headers.count(name) == 1) { 
        
             ldpp_subdout(dpp, rgw, 10) << "skipping x>> " << name << dendl; 
        
             continue; 
        
           } else if (allow_empty_attrs || !xattr.empty()) { 
        
             ldpp_subdout(dpp, rgw, 10) << "x>> " << name << ":" << xattr << dendl;

building the new one: https://shaman.ceph.com/builds/ceph/wip-23264/196191fb6218b2f61c92cbc2b3d1e45f2ec9e8ba/
new run: https://pulpito.ceph.com/sfallah-2025-11-21_00:26:32-rgw-wip-23264-distro-default-gibba/
rerun: https://pulpito.ceph.com/sfallah-2025-11-21_13:51:14-rgw-wip-23264-distro-default-gibba/

KervyN · 2025-11-21T09:50:45Z

@clwluvw does this only implement copy/move objects for SSE-C or also for SSE-S3 and SSE-KMS?

clwluvw · 2025-11-21T10:06:34Z

@clwluvw does this only implement copy/move objects for SSE-C or also for SSE-S3 and SSE-KMS?

It does implement copy objects for all variant types of encryption that rgw supports, which includes (SSE-C, SSE-S3, and SSE-KMS).

clwluvw · 2025-11-21T21:33:06Z

new run: https://pulpito.ceph.com/sfallah-2025-11-21_00:26:32-rgw-wip-23264-distro-default-gibba/
rerun: https://pulpito.ceph.com/sfallah-2025-11-21_13:51:14-rgw-wip-23264-distro-default-gibba/

@cbodley - could you please review the qa?

cbodley · 2025-11-25T16:37:22Z

mostly looks good, but the rgw/tempest job failed in both runs with an error i don't recognize. from logs:
http://qa-proxy.ceph.com/teuthology/sfallah-2025-11-21_00:26:32-rgw-wip-23264-distro-default-gibba/8616852/teuthology.log
https://qa-proxy.ceph.com/teuthology/sfallah-2025-11-21_13:51:14-rgw-wip-23264-distro-default-gibba/8618415/teuthology.log

2025-11-21T16:51:08.510 INFO:teuthology.orchestra.run.gibba022.stdout:==============================
2025-11-21T16:51:08.510 INFO:teuthology.orchestra.run.gibba022.stdout:Failed 1 tests - output below:
2025-11-21T16:51:08.510 INFO:teuthology.orchestra.run.gibba022.stdout:==============================
2025-11-21T16:51:08.510 INFO:teuthology.orchestra.run.gibba022.stdout:
2025-11-21T16:51:08.510 INFO:teuthology.orchestra.run.gibba022.stdout:tempest.api.object_storage.test_object_version.ContainerTest.test_versioned_container[id-a151e158-dcbf-4a1f-a1e7-46cd65895a6f]
2025-11-21T16:51:08.510 INFO:teuthology.orchestra.run.gibba022.stdout:------------------------------------------------------------------------------------------------------------------------------
2025-11-21T16:51:08.511 INFO:teuthology.orchestra.run.gibba022.stdout:
2025-11-21T16:51:08.511 INFO:teuthology.orchestra.run.gibba022.stdout:Captured traceback:
2025-11-21T16:51:08.511 INFO:teuthology.orchestra.run.gibba022.stdout:~~~~~~~~~~~~~~~~~~~
2025-11-21T16:51:08.511 INFO:teuthology.orchestra.run.gibba022.stdout:    Traceback (most recent call last):
2025-11-21T16:51:08.511 INFO:teuthology.orchestra.run.gibba022.stdout:
2025-11-21T16:51:08.511 INFO:teuthology.orchestra.run.gibba022.stdout:      File "/home/ubuntu/cephtest/tempest/tempest/api/object_storage/test_object_version.py", line 90, in test_versioned_container
2025-11-21T16:51:08.511 INFO:teuthology.orchestra.run.gibba022.stdout:    self.assertContainer(base_container_name, '1', '1024',
2025-11-21T16:51:08.511 INFO:teuthology.orchestra.run.gibba022.stdout:
2025-11-21T16:51:08.511 INFO:teuthology.orchestra.run.gibba022.stdout:      File "/home/ubuntu/cephtest/tempest/tempest/api/object_storage/test_object_version.py", line 35, in assertContainer
2025-11-21T16:51:08.511 INFO:teuthology.orchestra.run.gibba022.stdout:    self.assertEqual(header_value, byte)
2025-11-21T16:51:08.511 INFO:teuthology.orchestra.run.gibba022.stdout:
2025-11-21T16:51:08.512 INFO:teuthology.orchestra.run.gibba022.stdout:      File "/home/ubuntu/cephtest/tempest/.tox/venv/lib/python3.10/site-packages/testtools/testcase.py", line 393, in assertEqual
2025-11-21T16:51:08.512 INFO:teuthology.orchestra.run.gibba022.stdout:    self.assertThat(observed, matcher, message)
2025-11-21T16:51:08.512 INFO:teuthology.orchestra.run.gibba022.stdout:
2025-11-21T16:51:08.512 INFO:teuthology.orchestra.run.gibba022.stdout:      File "/home/ubuntu/cephtest/tempest/.tox/venv/lib/python3.10/site-packages/testtools/testcase.py", line 480, in assertThat
2025-11-21T16:51:08.512 INFO:teuthology.orchestra.run.gibba022.stdout:    raise mismatch_error
2025-11-21T16:51:08.512 INFO:teuthology.orchestra.run.gibba022.stdout:
2025-11-21T16:51:08.512 INFO:teuthology.orchestra.run.gibba022.stdout:    testtools.matchers._impl.MismatchError: '1023' != '1024'

this is asserting on the response header value "x-container-bytes-used" in https://github.com/openstack/tempest/blob/34.1.0/tempest/api/object_storage/test_object_version.py#L34-L35

for reference, they passed on the most recent baseline https://pulpito.ceph.com/teuthology-2025-11-21_20:40:24-rgw-main-distro-default-smithi/

Set ofs to total bytes read by adding 1 to end offset. Since 'end' represents the last byte offset (zero-indexed), we need to add 1 to get the actual number of bytes copied. Signed-off-by: Seena Fallah <seenafallah@gmail.com>

clwluvw · 2025-11-25T18:25:42Z

this is asserting on the response header value "x-container-bytes-used" in https://github.com/openstack/tempest/blob/34.1.0/tempest/api/object_storage/test_object_version.py#L34-L35

found the issue, had a wrong ofs to set for accounted_size as described in the last commit.
new build: https://shaman.ceph.com/builds/ceph/wip-23264/8d55ba26f8e1f35a2498f4c6eacc14cfb2c1ac7c/
qa run: https://pulpito.ceph.com/sfallah-2025-11-26_15:44:01-rgw-wip-23264-distro-default-gibba/
qa rerun: https://pulpito.ceph.com/sfallah-2025-11-27_08:50:27-rgw-wip-23264-distro-default-gibba/

clwluvw · 2025-11-26T15:50:48Z

jenkins test make check

clwluvw · 2025-11-27T08:50:52Z

jenkins test make check

clwluvw · 2025-11-27T09:55:34Z

jenkins test make check

clwluvw · 2025-11-27T15:06:27Z

qa run: https://pulpito.ceph.com/sfallah-2025-11-26_15:44:01-rgw-wip-23264-distro-default-gibba/
qa rerun: https://pulpito.ceph.com/sfallah-2025-11-27_08:50:27-rgw-wip-23264-distro-default-gibba/

@cbodley - could you please take another look at the new results?

clwluvw · 2025-11-27T15:06:48Z

jenkins test make check

clwluvw · 2025-11-27T16:06:35Z

jenkins test make check

clwluvw · 2025-12-01T14:23:06Z

jenkins test make check

cbodley · 2025-12-01T15:21:54Z

qa run: https://pulpito.ceph.com/sfallah-2025-11-26_15:44:01-rgw-wip-23264-distro-default-gibba/
qa rerun: https://pulpito.ceph.com/sfallah-2025-11-27_08:50:27-rgw-wip-23264-distro-default-gibba/

@cbodley - could you please take another look at the new results?

results approved. 👍 the tempest jobs passed test_versioned_container, but are failing for unrelated reasons tracked in https://tracker.ceph.com/issues/72968

github-actions · 2025-12-01T15:34:39Z

This is an automated message by src/script/redmine-upkeep.py.

I found one or more Fixes: tags in the commit messages in

git log 7e15bef24039ae1f6d916958ad517a3589d2303e^..7e15bef24039ae1f6d916958ad517a3589d2303e

The referenced tickets are:

https://tracker.ceph.com/issues/23264

Those tickets do not reference this merged Pull Request. If this Pull Request merge resolves any of those tickets, please update the "Pull Request ID" field on each ticket. A future run of this script will appropriately update them.

Update Log: https://github.com/ceph/ceph/actions/runs/19828113367

cbodley · 2025-12-01T15:38:24Z

tyvm @clwluvw @mdw-at-linuxbox!

AdrianSilaghi · 2026-03-31T16:48:09Z

Hi, thanks for this excellent fix!

We're running Ceph v19.2.3 (Squid) in production and our users are affected by the inability to CopyObject on encrypted objects. We currently have server-side encryption disabled as a workaround.

Are there any plans to backport this to the Squid (v19.x) release line? This would be very valuable for production deployments that aren't yet in a position to upgrade to v21.x.

Thanks!

github-actions Bot added the rgw label Jun 7, 2025

clwluvw force-pushed the enc-copy branch from 60d2d20 to 834a3df Compare June 7, 2025 20:47

github-actions Bot added the tests label Jun 7, 2025

clwluvw force-pushed the enc-copy branch 4 times, most recently from 45a0219 to 1359a66 Compare June 8, 2025 13:49

clwluvw marked this pull request as ready for review June 8, 2025 13:49

clwluvw requested a review from a team as a code owner June 8, 2025 13:49

clwluvw requested a review from cbodley June 8, 2025 16:20

clwluvw force-pushed the enc-copy branch from 1359a66 to 56fc992 Compare June 8, 2025 18:30

github-actions Bot added the stale label Aug 15, 2025

github-actions Bot removed the stale label Sep 10, 2025

github-actions Bot added the stale label Nov 9, 2025

clwluvw removed the stale label Nov 10, 2025

clwluvw force-pushed the enc-copy branch 3 times, most recently from 6f5a1f9 to 3393797 Compare November 19, 2025 17:10

clwluvw and others added 3 commits November 19, 2025 18:36

rgw: involve next on RGWGetObj_BlockDecrypt::fixup_range

b415ece

Co-authored-by: Marcus Watts <mwatts@redhat.com> Signed-off-by: Seena Fallah <seenafallah@gmail.com>

rgw: involve next on RGWGetObj_BlockDecrypt::flush

bfa58a6

Co-authored-by: Marcus Watts <mwatts@redhat.com> Signed-off-by: Seena Fallah <seenafallah@gmail.com>

rgw: teach rgw_s3_prepare_decrypt about source decrypt headers

1c9e356

Co-authored-by: Marcus Watts <mwatts@redhat.com> Signed-off-by: Seena Fallah <seenafallah@gmail.com>

clwluvw force-pushed the enc-copy branch from 05e4759 to 5a72832 Compare November 20, 2025 15:42

cbodley approved these changes Nov 20, 2025

View reviewed changes

rgw: sanitize copy source attrs

e4d6835

Co-authored-by: Marcus Watts <mwatts@redhat.com> Signed-off-by: Seena Fallah <seenafallah@gmail.com>

clwluvw force-pushed the enc-copy branch from 5a72832 to e4d6835 Compare November 20, 2025 20:21

rgw: fix offset calculation in copy_obj_data

16e477d

Set ofs to total bytes read by adding 1 to end offset. Since 'end' represents the last byte offset (zero-indexed), we need to add 1 to get the actual number of bytes copied. Signed-off-by: Seena Fallah <seenafallah@gmail.com>

cbodley merged commit 7e15bef into ceph:main Dec 1, 2025
13 checks passed

cbodley mentioned this pull request Dec 2, 2025

rgw: copy object encryption fixes #54543

Closed

This was referenced Mar 24, 2026

rgw: fix fixup_range cascade for compressed+encrypted objects #67966

Merged

rgw: always decompress in CopyObject when source is compressed #68011

Merged

Conversation

clwluvw commented Jun 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Uh oh!

clwluvw commented Jun 8, 2025

Uh oh!

clwluvw commented Jun 9, 2025

Uh oh!

mattbenjamin commented Jun 9, 2025

Uh oh!

clwluvw commented Jun 9, 2025

Uh oh!

clwluvw commented Jun 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Aug 15, 2025

Uh oh!

taxilian commented Sep 10, 2025

Uh oh!

github-actions Bot commented Nov 9, 2025

Uh oh!

taxilian commented Nov 10, 2025

Uh oh!

clwluvw commented Nov 20, 2025

Uh oh!

cbodley commented Nov 20, 2025

Uh oh!

clwluvw commented Nov 20, 2025

Uh oh!

clwluvw commented Nov 20, 2025

Uh oh!

clwluvw commented Nov 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

KervyN commented Nov 21, 2025

Uh oh!

clwluvw commented Nov 21, 2025

Uh oh!

clwluvw commented Nov 21, 2025

Uh oh!

cbodley commented Nov 25, 2025

Uh oh!

clwluvw commented Nov 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

clwluvw commented Nov 26, 2025

Uh oh!

clwluvw commented Nov 27, 2025

Uh oh!

clwluvw commented Nov 27, 2025

Uh oh!

clwluvw commented Nov 27, 2025

Uh oh!

clwluvw commented Nov 27, 2025

Uh oh!

clwluvw commented Nov 27, 2025

Uh oh!

clwluvw commented Dec 1, 2025

Uh oh!

cbodley commented Dec 1, 2025

Uh oh!

Uh oh!

github-actions Bot commented Dec 1, 2025

Uh oh!

cbodley commented Dec 1, 2025

Uh oh!

AdrianSilaghi commented Mar 31, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

clwluvw commented Jun 7, 2025 •

edited

Loading

clwluvw commented Jun 16, 2025 •

edited

Loading

clwluvw commented Nov 20, 2025 •

edited

Loading

clwluvw commented Nov 25, 2025 •

edited

Loading