needs to check for error with a block like this:

	if (retcode < 0) {
	  lease_cr->go_down();
	  drain_all();
	  return set_cr_error(retcode);
	}

yes addressed it.. thanks!
Also observed in the tests that by reading into sync_marker directly here, we loose the in-memory sync_marker updated at the end of the do{} loop in the incremental_sync(). So the rgw server is repeatedly trying to read remote data log from this sync_marker in an endless loop. So I am planning to check version before updating sync_marker.. Does it seem right?

  tmp_objv.clear();
  /* Reread data sync status to fech the latest marker and update objv */


  yield call(new RGWSimpleRadosReadCR<rgw_data_sync_marker>(sync_env->dpp, sync_env->store,
                                                         rgw_raw_obj(pool, status_oid),
                                                         &tmp_sync_marker, true, &tmp_objv));
    if (retcode < 0) {
      lease_cr->go_down();
      drain_all();
      return set_cr_error(retcode);
    }
    if (tmp_objv.read_version.ver > objv.read_version.ver) {
      sync_marker = tmp_sync_marker;
      objv = tmp_objv;
    }

<<<<

Also observed in the tests that by reading into sync_marker directly here, we loose the in-memory sync_marker updated at the end of the do{} loop in the incremental_sync().

hmm, i think i see what you mean. RGWDataIncSyncShardCR is using sync_marker.marker as a local variable to track its current position in the listing, separately from RGWDataSyncShardMarkerTrack::sync_marker whose updates are written to the sync status object

if RGWDataIncSyncShardCR bails out with an error, we'd retry this call to RGWDataSyncShardCR with the same sync_marker.marker as before. but we really should start over at the position recorded in our sync status object, so i think this RGWSimpleRadosReadCR is doing the right thing

can you tell why RGWDataIncSyncShardCR is failing? it should run forever unless the lease times out, so i guess that's why? if that's happening consistently, we wouldn't expect to make much progress

I think I found the issue here.. objv should be cleared before calling "RGWSimpleRadosReadCR" otherwise it may fail in the cls_version_check() in the read_op. This and retcode error check should fix the loop we had observed in the tests earlier.