rgw/restore: Add notifications for restore events#65023
Conversation
|
@soumyakoduri follwoing commenst are only regarding: 6693ad1
|
Thank Yuval. Could you please elaborate on Init/Complete events?
|
apologize. i see it now |
|
I suspect that the title should read “notifications” |
| This will restore the object ``doc1.rtf`` at an optional version, | ||
| for the duration of 10 days. | ||
|
|
||
| .. note:: Once restored, the restoration period of these temporary copies can be updated by reissuing the request with a new period. |
There was a problem hiding this comment.
I would remove "Once restored," and capitalize The
There was a problem hiding this comment.
This particular commit changes are actually part of #64804 .. I shall address al your comments in that PR..
| overrides: | ||
| s3tests: | ||
| force-branch: ceph-master | ||
| force-repo: https://github.com/soumyakoduri/s3-tests.git |
There was a problem hiding this comment.
Is it appropriate to have merged code refer to your personal fork?
There was a problem hiding this comment.
Ack. Won't pretend to understand these dynamics, but it seemed unusual so I had to ask.
| } | ||
| delete_at = expiration_date; | ||
|
|
||
| ldpp_dout(dpp, 5) << "Setting Restore expiry time to: " << expiration_date << " , restore_time: " << restore_time << ", restore_interval: " << cct->_conf->rgw_restore_debug_interval << dendl; |
|
This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved |
6693ad1 to
12b0610
Compare
12b0610 to
01fc9c8
Compare
|
With the changes events received are as below - `Body: Body: Body: Body: Body: Body: ` |
| @@ -663,6 +729,25 @@ int Restore::restore_obj_from_cloud(rgw::sal::Bucket* pbucket, | |||
| } | |||
|
|
|||
| ldpp_dout(this, 10) << __PRETTY_FUNCTION__ << ": Restore of object " << pobj->get_key() << " is in progress." << dendl; | |||
|
|
|||
| auto& attrs = pobj->get_attrs(); | |||
There was a problem hiding this comment.
yop can put this section inside the if (notify) block, so we don't do that unnecessarily
There was a problem hiding this comment.
do we expect to have size and etag during init?
There was a problem hiding this comment.
size will be non-zero only when restore is complete. Otherwise it remains '0' during init and after expiry. But etag should always be present. I fixed it now.
| @@ -625,6 +675,22 @@ int Restore::restore_obj_from_cloud(rgw::sal::Bucket* pbucket, | |||
| return -EINVAL; | |||
| } | |||
|
|
|||
| auto notify = driver->get_notification( | |||
| dpp, pobj, nullptr, | |||
| {rgw::notify::ObjectRestoreInitiated, rgw::notify::ObjectRestore}, | |||
There was a problem hiding this comment.
| {rgw::notify::ObjectRestoreInitiated, rgw::notify::ObjectRestore}, | |
| {rgw::notify::ObjectRestoreInitiated}, |
specific type is known
|
in "complete" event: we expect |
|
Thanks Yuval. I fixed the {"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2025-10-16T19:28:42.346032Z","eventName":"ObjectRestore:Post","userIdentity":{"principalId":"rgw restore"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"0","x-amz-id-2":"64110-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"notify-http","bucket":{"name":"bucket1","ownerIdentity":{"principalId":"testid"},"arn":"arn:aws:s3:default::bucket1","id":"f46ae69f-ea94-4dda-8fad-ea81f2dd2a98.4179.1"},"object":{"key":"h2","size":0,"eTag":""6f5902ac237024bdd0c176cb93063dc4"","versionId":"","sequencer":"6A47F168F79BA014","metadata":[{"key":"x-amz-meta-s3cmd-attrs","val":"atime:1760366604/ctime:1709487143/gid:0/gname:root/md5:6f5902ac237024bdd0c176cb93063dc4/mode:33188/mtime:1709487143/uid:0/uname:root"}],"tags":[]}},"eventId":"1760642922.346070."6f5902ac237024bdd0c176cb93063dc4"","opaqueData":""}]} {"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2025-10-16T19:28:45.813023Z","eventName":"ObjectRestore:Completed","userIdentity":{"principalId":"rgw restore"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"0","x-amz-id-2":"64110-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"notify-http","bucket":{"name":"bucket1","ownerIdentity":{"principalId":"testid"},"arn":"arn:aws:s3:default::bucket1","id":"f46ae69f-ea94-4dda-8fad-ea81f2dd2a98.4179.1"},"object":{"key":"h2","size":12,"eTag":""6f5902ac237024bdd0c176cb93063dc4"","versionId":"","sequencer":"6D47F1682FE77530","metadata":[{"key":"x-amz-meta-s3cmd-attrs","val":"atime:1760366604/ctime:1709487143/gid:0/gname:root/md5:6f5902ac237024bdd0c176cb93063dc4/mode:33188/mtime:1709487143/uid:0/uname:root"}],"tags":[]}},"eventId":"1760642925.813033."6f5902ac237024bdd0c176cb93063dc4"","opaqueData":""}]} {"Records":[{"eventVersion":"2.2","eventSource":"ceph:s3","awsRegion":"default","eventTime":"2025-10-16T19:30:26.532052Z","eventName":"ObjectRestore:Delete","userIdentity":{"principalId":"rgw restore"},"requestParameters":{"sourceIPAddress":""},"responseElements":{"x-amz-request-id":"0","x-amz-id-2":"64116-default-default"},"s3":{"s3SchemaVersion":"1.0","configurationId":"notify-http","bucket":{"name":"bucket1","ownerIdentity":{"principalId":"testid"},"arn":"arn:aws:s3:default::bucket1","id":"f46ae69f-ea94-4dda-8fad-ea81f2dd2a98.4179.1"},"object":{"key":"h2","size":0,"eTag":""6f5902ac237024bdd0c176cb93063dc4"","versionId":"","sequencer":"D247F168F05BB71F","metadata":[{"key":"x-amz-meta-s3cmd-attrs","val":"atime:1760366604/ctime:1709487143/gid:0/gname:root/md5:6f5902ac237024bdd0c176cb93063dc4/mode:33188/mtime:1709487143/uid:0/uname:root"}],"tags":[]}},"eventId":"1760643026.532110."6f5902ac237024bdd0c176cb93063dc4"","opaqueData":""}]} 127.0.0.1 - - [17/Oct/2025 01:00:26] "POST / HTTP/1.1" 200 - |
| @@ -5693,7 +5693,8 @@ int RGWRados::restore_obj_from_cloud(RGWLCCloudTierCtx& tier_ctx, | |||
| return ret; | |||
| } | |||
|
|
|||
| return ret; | |||
| // this returned size can be used to send bucket notification | |||
| return accounted_size; | |||
There was a problem hiding this comment.
where is this being used?
note that this changes the semantics of the return value, and may break callers that interpret ret == 0 or !ret as "success"
There was a problem hiding this comment.
The only caller of this routine is process_restore_entry - https://github.com/ceph/ceph/pull/65023/files#diff-9e991de9c26277f30eda2574f4a27ccfd4b1b92dc00f6d9bc84f78db4222f195R519 ..
The size returned here is used to send notification ObjectRestoreCompleted - https://github.com/ceph/ceph/pull/65023/files#diff-9e991de9c26277f30eda2574f4a27ccfd4b1b92dc00f6d9bc84f78db4222f195R545
There was a problem hiding this comment.
in process_restore_entry() this value is returned:
https://github.com/soumyakoduri/ceph/blob/394d2afff91e4ccc8c6ee9f5d60e7ee5f5e46ec1/src/rgw/rgw_restore.cc#L555
and later on this return value is checked with !ret which would fail for positive integers:
https://github.com/soumyakoduri/ceph/blob/394d2afff91e4ccc8c6ee9f5d60e7ee5f5e46ec1/src/rgw/rgw_restore.cc#L378-L380
i think it would be much safer to return zero on sucess, and return the as a by ref parameter.
There was a problem hiding this comment.
Ack. Will update.
There was a problem hiding this comment.
Addressed this. Kindly review.
394d2af to
d8520f0
Compare
|
@soumyakoduri (could be done in another PR) but we probably need to add tests for this usecase |
@yuvalif ...sure.. I couldn't find any cloud-transition related tests to extend them for restore. Yes! that may take some time and need to be addressed separately. |
|
jenkins test api |
what is the basic test suite for the "restore" feature? |
currently, we have cloud-transition and cloud-restore tests running as part of cloud-transition test-suite |
d8520f0 to
ffe1003
Compare
|
jenkins test docs |
The following notification events are added for restore operations as stated in https://docs.aws.amazon.com/AmazonS3/latest/userguide/notification-how-to-event-types-and-destinations.html s3:ObjectRestore:* s3:ObjectRestore:Post s3:ObjectRestore:Completed s3:ObjectRestore:Delete By using the ObjectRestore event types, you can receive notifications for event initiation and completion when restoring objects from the S3 Glacier Flexible Retrieval storage class, S3 Glacier Deep Archive storage class, S3 Intelligent-Tiering Archive Access tier, and S3 Intelligent-Tiering Deep Archive Access tier. You can also receive notifications for when the restored copy of an object expires. The s3:ObjectRestore:Post event type notifies you of object restoration initiation. The s3:ObjectRestore:Completed event type notifies you of restoration completion. The s3:ObjectRestore:Delete event type notifies you when the temporary copy of a restored object expires. Signed-off-by: Soumya Koduri <skoduri@redhat.com>
Include `size` param in the restore API to read the actual object size once restore is completed. This is needed to send bucket notification to the client. Signed-off-by: Soumya Koduri <skoduri@redhat.com>
ffe1003 to
5e2d2b5
Compare
if this is currently part of s3tests and does not have any RGW specific test code, we would probably need to either add a restore specific suite, or add that to the bucket_notification suite (and borrow the code and setup from the "cloud_restore" code in s3tests. |
|
jenkins test windows |
|
Pr testing completed , Approved by @ivancich details are in :https://tracker.ceph.com/issues/73724, |
created tracker for this: https://tracker.ceph.com/issues/73780 |
The following notification events are added for restore operations as stated
in https://docs.aws.amazon.com/AmazonS3/latest/userguide/notification-how-to-event-types-and->
s3:ObjectRestore:*
s3:ObjectRestore:Post
s3:ObjectRestore:Completed
s3:ObjectRestore:Delete
By using the ObjectRestore event types, you can receive notifications for event initiation and
completion when restoring objects from the S3 Glacier Flexible Retrieval storage class,
S3 Glacier Deep Archive storage class, S3 Intelligent-Tiering Archive Access tier, and
S3 Intelligent-Tiering Deep Archive Access tier. You can also receive notifications for when >
The s3:ObjectRestore:Post event type notifies you of object restoration initiation. The s3:Ob>
event type notifies you of restoration completion. The s3:ObjectRestore:Delete event type not>
the temporary copy of a restored object expires.
Fixes: https://tracker.ceph.com/issues/69203
Signed-off-by: Soumya Koduri skoduri@redhat.com
Checklist
Show available Jenkins commands
jenkins test classic perfJenkins Job | Jenkins Job Definitionjenkins test crimson perfJenkins Job | Jenkins Job Definitionjenkins test signedJenkins Job | Jenkins Job Definitionjenkins test make checkJenkins Job | Jenkins Job Definitionjenkins test make check arm64Jenkins Job | Jenkins Job Definitionjenkins test submodulesJenkins Job | Jenkins Job Definitionjenkins test dashboardJenkins Job | Jenkins Job Definitionjenkins test dashboard cephadmJenkins Job | Jenkins Job Definitionjenkins test apiJenkins Job | Jenkins Job Definitionjenkins test docsReadTheDocs | Github Workflow Definitionjenkins test ceph-volume allJenkins Jobs | Jenkins Jobs Definitionjenkins test windowsJenkins Job | Jenkins Job Definitionjenkins test rook e2eJenkins Job | Jenkins Job Definition