rgw/cloudtier: initial MVP of the feature RestoreObject from cloud by soumyakoduri · Pull Request #59311 · ceph/ceph

soumyakoduri · 2024-08-19T15:04:23Z

This PR covers initial PoC for the feature to restore objects from the cloud-tier on demand. Details below -

Feature doc: https://docs.google.com/document/d/1rzLJAzHK6cLuzJswgoplgugNOFCKi8NrjPOGfgVrIFg/edit
Updated to - src/doc/rgw/cloud-restore.md in this PR
Fixes: https://tracker.ceph.com/issues/67679

Restore CLI parsing
Added restore cli parameters and xml parsing for restore-object command
Current commit has basic cli checks with their corresponding response, cli parameters like Days.

Response elements:
For first and repetitive request 202 Accepted will be corresponding response code.
For CloudRestored status 200 OK will be corresponding response code.
For conflicting requests 409 Conflict corresponding response code.

TODO:

Improve response messages and error codes.
Fix empty x-amz-restore header in the response
For temporary restored objects, if the request is retried, update expiry_date

GET/RESTORE : calling cloud restore APIs to fetch the object from the remote endpoint.
What are all supported :
It allows read-through for cloud-tiered objects
New tier config options user need to set allow_read_through to true and read_through_restore_days more than 1 for this feature to work, also objects with retain_head_object will be available for this feature.
First get request will fail with restoring in progress error, objects are downloaded asynchronously.
The objects restored are temporary.

TODO :

implement call backs
HEAD should return restore-status in the x-amz-restore header

CloudTier module:

Given <bucket,object,cloudtier-storage-class> & optional<days>, fetch the object from the cloud endpoint and copy it to STANDARD storage-class
if <days> provided, and
a) > 0, the restore is marked temporary with expiry date, but
b) = 0, restore is not done and an error is returned
Without <days>, it is considered as permanent restore.
For temporary restore, return original cloudtier storage-class for x-amz-storage-class .
New attrs are added to store RestoreStatus and RestoreType
Use Object Expirer to delete the temporarily restored objects.
For temporarily restored objects, set delete_at attr to the expiration time.
This will add those objects to ObjectExpirer list. Use LC worker thread to scan that list and delete expired objects. By delete here, it means to delete restored object data and reset HEAD object as Cloud-transitioned object as it was before restore.

In addition below changes are done -

If temporary, object is still marked RGWObj::CloudTiered and mtime is set same as
transition time.
If permanent, object is marked RGWObj::Main and mtime is set to restore time (now()).
rgw_restore_debug_interval option added to set configure restore Days (similar to rgw_lc_debug_interval)

TODO:

Recheck on s3cmd attrs and any other attr related changes

Not addressed in this PR:

Signed-off-by: Shreyansh Sancheti ssancheti@ibm.com
Signed-off-by: Jiffin Tony Thottan thottanjiffin@gmail.com
Signed-off-by: Soumya Koduri skoduri@redhat.com

Checklist

Tracker (select at least one)
- References tracker ticket
- Very recent bug; references commit where it was introduced
- New feature (ticket optional)
- Doc update (no ticket needed)
- Code cleanup (no ticket needed)
Component impact
- Affects Dashboard, opened tracker ticket
- Affects Orchestrator, opened tracker ticket
- No impact that needs to be tracked
Documentation (select at least one)
- Updates relevant documentation
- No doc update is appropriate
Tests (select at least one)
- Includes unit test(s)
- Includes integration test(s)
- Includes bug reproducer
- No tests

Show available Jenkins commands

jenkins retest this please
jenkins test classic perf
jenkins test crimson perf
jenkins test signed
jenkins test make check
jenkins test make check arm64
jenkins test submodules
jenkins test dashboard
jenkins test dashboard cephadm
jenkins test api
jenkins test docs
jenkins render docs
jenkins test ceph-volume all
jenkins test ceph-volume tox
jenkins test windows
jenkins test rook e2e

soumyakoduri · 2024-08-19T19:22:57Z

teuthology results - http://pulpito.front.sepia.ceph.com/soumyakoduri-2024-08-15_06:06:37-rgw-wip-skoduri-cloud-restore-distro-default-smithi/

dang

Looks great! A few tiny nits.

soumyakoduri · 2024-08-22T09:46:29Z

Looks great! A few tiny nits.

Thanks Dan. Addressed your comments and also fixed a bug in ObjectExpirer code.

soumyakoduri · 2024-08-23T18:10:21Z

teuthology results - http://pulpito.front.sepia.ceph.com/soumyakoduri-2024-08-23_11:57:23-rgw-wip-skoduri-cloud-restore-distro-default-smithi/

soumyakoduri · 2024-08-23T18:10:52Z

jenkins test make check

soumyakoduri · 2024-08-23T18:11:00Z

jenkins test make check arm64

thotz · 2024-08-23T07:22:33Z

+    // Read ACLOwner maybe ? 
+  }
+
+  ldpp_dout(tier_ctx.dpp, 20) << __func__ << "(): Sucessfully fetched object from cloud bucket:" << dest_bucket << ", object: " << target_obj_name << dendl;


Suggested change

ldpp_dout(tier_ctx.dpp, 20) << __func__ << "(): Sucessfully fetched object from cloud bucket:" << dest_bucket << ", object: " << target_obj_name << dendl;

ldpp_dout(tier_ctx.dpp, 20) << __func__ << "(): Successfully fetched object from cloud bucket:" << dest_bucket << ", object: " << target_obj_name << dendl;

thotz · 2024-08-26T14:38:23Z

+  ret = processor.complete(accounted_size, etag, &mtime, set_mtime,
+                           attrs, rgw::cksum::no_cksum, delete_at , nullptr, nullptr, nullptr,
+                           (rgw_zone_set *)&zone_set, &canceled, rctx, log_op ? rgw::sal::FLAG_LOG_OP : 0);
+    if (ret < 0) {


indentation

thotz · 2024-08-27T05:25:30Z

+        ldpp_dout(dpp, 5) << "restore is not completed yet, please retry again" << dendl;
+        s->err.message = "restore is not completed yet";
+      }
+      return op_ret;


Suggested change

return op_ret;

thotz · 2024-08-27T05:27:09Z

+    } else { // CloudRestored..return success
+      return 0;
+    }


Suggested change

} else { // CloudRestored..return success

return 0;

}

return op_ret;

thotz · 2024-08-27T05:32:08Z

+  } else {
+    ldpp_dout(this, 20) << " manifest not found" << dendl;
+  }
+  ldpp_dout(this, 20) << "completed restore" << dendl;


most of the errors are alredy handled in the handle_cloud_object_tier, it will reductant IMO

Yeah, I guess only handle_cloudtier is needed there then.

thotz · 2024-08-27T05:32:59Z

+  if (ret < 0) {
+    ldpp_dout(this, 5) << "RGWLC::process_expire_objects: failed, "
+	          << " worker ix: " << worker->ix << dendl;
+  }


return ret;

We should not be failing LC process for any failure with expiring temp objects here.

thotz · 2024-08-27T05:44:06Z

+
+  if (attr_iter == attrs.end() || restore_status != rgw::sal::RGWRestoreStatus::None) {
+    s->err.http_ret = 202; //Accepted
+    dump_header(s, "x-amz-restore", rgw_bl_str(restore_status));


here the header can be empty right, is it right to provide empty values for header?

Not sure about that.

https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadObject.html#API_HeadObject_ResponseElements

x-amz-restore
If the object is an archived object (an object whose storage class is GLACIER), the response includes this header if either the archive restoration is in progress (see RestoreObject or an archive copy is already restored.

If an archive copy is already restored, the header value indicates when Amazon S3 is scheduled to delete the object copy. For example:

x-amz-restore: ongoing-request="false", expiry-date="Fri, 21 Dec 2012 00:00:00 GMT"

If the object restoration is in progress, the header returns the value ongoing-request="true".

expiry-date should be set for only temporary copies.

Okay got it, so in case of read_through at the moment it will be temporary objects only.

thotz · 2024-09-06T10:23:47Z

 }

+int RadosObject::restore_obj_from_cloud(Bucket* bucket,
+                                  rgw::sal::PlacementTier* tier,


yesterday while fixing tier config issue, I was wondering whether we need both tierconf and Placementtier is required for this function

right..I was wondering the same.. now that tier_config is updated with latest Zonegroup tier info, we can avoid this second argument.

github-actions · 2024-09-09T14:02:00Z

This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved

soumyakoduri · 2024-09-22T18:08:25Z

jenkins test windows

soumyakoduri · 2024-09-22T18:08:39Z

jenkins test make check arm64

soumyakoduri · 2024-09-22T18:08:56Z

teuthology test results: http://pulpito.front.sepia.ceph.com/soumyakoduri-2024-09-22_06:37:27-rgw-wip-skoduri-cloud-restore-distro-default-smithi

soumyakoduri · 2024-09-22T18:10:03Z

jenkins test windows

soumyakoduri · 2024-09-22T18:12:44Z

@mattbenjamin @dang @cbodley .. all the review comments so far have been addressed and code is rebased to latest main.
teuthology results have passed - http://pulpito.front.sepia.ceph.com/soumyakoduri-2024-09-22_06:37:27-rgw-wip-skoduri-cloud-restore-distro-default-smithi

Please review and let us know if this PR can be merged. Thanks!

addressed in latest code.

mattbenjamin · 2024-09-23T16:08:40Z

could we maybe squash some of these, keeping summarizing the squashed commits in the commit mesg?

soumyakoduri · 2024-09-25T07:29:58Z

jenkins test api

soumyakoduri · 2024-09-25T07:30:17Z

could we maybe squash some of these, keeping summarizing the squashed commits in the commit mesg?

Done. Please review!

soumyakoduri · 2024-09-30T05:54:58Z

jenkins test api

soumyakoduri · 2024-09-30T05:55:10Z

jekins test make check arm64

soumyakoduri · 2024-09-30T05:56:27Z

teuthology results: http://pulpito.front.sepia.ceph.com/soumyakoduri-2024-09-27_07:57:10-rgw-wip-skoduri-cloud-restore-distro-default-smithi

One test (rgw:multisite) failed which seems unrelated to this PR. I see that test fail without these changes too.

soumyakoduri · 2024-09-30T06:00:23Z

jenkins retest this please

soumyakoduri · 2024-09-30T09:13:38Z

jenkins test api

soumyakoduri · 2024-09-30T09:33:45Z

jenkins test make check arm64

soumyakoduri · 2024-09-30T15:47:54Z

jenkins test make check arm64

soumyakoduri · 2024-10-02T10:07:01Z

@mattbenjamin , do you have any further comments on this PR? If not can we merge this PR? There are couple of other WIP issues to be reviewed & fixed which are blocked and waiting for this PR to be merged. If there are n't any major comments, we can open another PR to address them if needed.

mattbenjamin

is this PR ready for testing by QE? found cosmetic spacing stuff in esp. first commit in the series, but this is not critical

soumyakoduri · 2024-10-03T02:34:49Z

is this PR ready for testing by QE? found cosmetic spacing stuff in esp. first commit in the series, but this is not critical

yes and its already tested by QE in the downstream.

1)Add functionality to restore cloud-transitioned objects on demand. Current commit has below - * Given <bucket,object>, fetch the object from the cloud endpoint. * if days provided and > 0, the restore is marked temporary with expiry date. * Without <days>, it is marked as permanent restore. 2)Use ObjectExpirer/delete_at attr to delete temp objects For temporarily restored objects, set delete_at attr to the expiration time. This will add those objects to ObjectExpirer list. Use LC worker thread to scan that list and delete expired objects. By delete here, it means to delete restored object data and reset HEAD object as Cloud-transitioned object as it was before restore. In addition below changes are done - * If temporary, object is still marked RGWObj::CloudTiered and mtime is set same as transition time. * If permanent, object is marked RGWObj::Main and mtime is set to restore time (now()). * rgw_restore_debug_interval option added to set configure restore Days (similar to rgw_lc_debug_interval) There is an issue with ObjectExpirer code where in if an object is added to ObjectExpirer list and is re-written, it is not deleted from the expirer list and hence the new object may get deleted. Fixed the same and also addressed minor review comments. 3)Design doc added 4) ObjCategory should be set to CloudTiered only for cloud-transitioned objects and temporarily restored objects. Permanent copies are to be treated as regular objects. Signed-off-by: Soumya Koduri <skoduri@redhat.com>

What are all supported : * It allows read-through for cloud-tiered objects via restore_obj_from_cloud * New tier config options user need to set allow_read_through to true and read_through_restore_days more than 1 for this feature to work, also objects with retain_head_object will be available for this feature. * First get request will fail with restoring in progress error, objects are downloaded asynchronously. * The objects restore are temporary. * Tested `aws s3api get-object`, `aws s3api head-object` and `aws s3 cp` In addition send timeout errors for first readthrough request Also addressed lint warning and other cleanup(review comments) Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>

* For first and repititive request 202 Accepted will be corresponding response code. * For CloudRestored status 200 OK will be corresponding response code. * For conflicting requests 409 Conflict corresponding response code. Also Fixed storage class update while listing objects. Earlier while restoring object temporarily list-objects (s3api) and radosgw-admin bucket list didn't have updated storage class. With this fixed it now has the cloudtier storage class. Signed-off-by: shreyanshjain7174 <ssanchet@redhat.com>

mattbenjamin

much appreciated cleanup!

soumyakoduri · 2024-10-03T13:23:10Z

jenkins test make check

soumyakoduri · 2024-10-03T13:23:18Z

jenkins test make check arm64

soumyakoduri requested a review from a team as a code owner August 19, 2024 15:04

github-actions Bot added common rgw documentation labels Aug 19, 2024

soumyakoduri force-pushed the wip-skoduri-cloud-restore branch 2 times, most recently from 8644849 to e49ecc5 Compare August 19, 2024 18:57

soumyakoduri assigned thotz, soumyakoduri and shreyanshjain7174 Aug 19, 2024

soumyakoduri requested review from cbodley, dang and mattbenjamin August 19, 2024 19:09

dang approved these changes Aug 21, 2024

View reviewed changes

Comment thread src/rgw/driver/rados/rgw_rados.cc Outdated

Comment thread src/rgw/driver/rados/rgw_sal_rados.cc Outdated

soumyakoduri force-pushed the wip-skoduri-cloud-restore branch from 939b59e to 78c529a Compare August 26, 2024 06:10

thotz previously requested changes Aug 27, 2024

View reviewed changes

thotz reviewed Sep 6, 2024

View reviewed changes

github-actions Bot added the needs-rebase label Sep 9, 2024

soumyakoduri force-pushed the wip-skoduri-cloud-restore branch 3 times, most recently from d162331 to 8852379 Compare September 22, 2024 07:02

github-actions Bot removed the needs-rebase label Sep 22, 2024

soumyakoduri requested a review from thotz September 22, 2024 18:14

soumyakoduri force-pushed the wip-skoduri-cloud-restore branch from 8852379 to 54873d9 Compare September 25, 2024 04:39

soumyakoduri force-pushed the wip-skoduri-cloud-restore branch 2 times, most recently from a9707f6 to acdf239 Compare September 27, 2024 15:47

soumyakoduri force-pushed the wip-skoduri-cloud-restore branch from acdf239 to a1a0ead Compare September 30, 2024 05:58

mattbenjamin reviewed Oct 2, 2024

View reviewed changes

Comment thread src/rgw/driver/daos/rgw_sal_daos.h Outdated

Comment thread src/rgw/driver/posix/rgw_sal_posix.h Outdated

Comment thread src/rgw/driver/rados/rgw_sal_rados.cc Outdated

soumyakoduri force-pushed the wip-skoduri-cloud-restore branch from a1a0ead to ca665a4 Compare October 3, 2024 02:34

soumyakoduri and others added 3 commits October 3, 2024 13:58

soumyakoduri force-pushed the wip-skoduri-cloud-restore branch from ca665a4 to 1b66bc5 Compare October 3, 2024 08:29

mattbenjamin approved these changes Oct 3, 2024

View reviewed changes

soumyakoduri merged commit d53ffb1 into ceph:main Oct 3, 2024

	ldpp_dout(tier_ctx.dpp, 20) << __func__ << "(): Sucessfully fetched object from cloud bucket:" << dest_bucket << ", object: " << target_obj_name << dendl;
	ldpp_dout(tier_ctx.dpp, 20) << __func__ << "(): Successfully fetched object from cloud bucket:" << dest_bucket << ", object: " << target_obj_name << dendl;

Conversation

soumyakoduri commented Aug 19, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Uh oh!

soumyakoduri commented Aug 19, 2024

Uh oh!

dang left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

soumyakoduri commented Aug 22, 2024

Uh oh!

soumyakoduri commented Aug 23, 2024

Uh oh!

soumyakoduri commented Aug 23, 2024

Uh oh!

soumyakoduri commented Aug 23, 2024

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Sep 9, 2024

Uh oh!

soumyakoduri commented Sep 22, 2024

Uh oh!

soumyakoduri commented Sep 22, 2024

Uh oh!

soumyakoduri commented Sep 22, 2024

Uh oh!

soumyakoduri commented Sep 22, 2024

Uh oh!

soumyakoduri commented Sep 22, 2024

Uh oh!

mattbenjamin commented Sep 23, 2024

Uh oh!

soumyakoduri commented Sep 25, 2024

Uh oh!

soumyakoduri commented Sep 25, 2024

Uh oh!

soumyakoduri commented Sep 30, 2024

Uh oh!

soumyakoduri commented Sep 30, 2024

Uh oh!

soumyakoduri commented Sep 30, 2024

Uh oh!

soumyakoduri commented Sep 30, 2024

Uh oh!

soumyakoduri commented Sep 30, 2024

soumyakoduri commented Aug 19, 2024 •

edited

Loading

mattbenjamin left a comment •

edited

Loading

soumyakoduri commented Oct 3, 2024 •

edited

Loading