i took this commit into #45417 with an additional change, can you please review?

Where do blank generations come from? In discussion w/Yuval, the topic of generation "reset" also came up--is that actually needed?

Where do blank generations come from?

empty generations come from 'data full sync', which walks through a listing of remote bucket metadata and tries to run sync on each. it doesn't get any information about log generations from that list of bucket metadata keys, so it passed an 'empty' generation number into bucket sync

in RGWSyncBucketCR (https://github.com/ceph/ceph/blob/wip-rgw-multisite-reshard/src/rgw/rgw_data_sync.cc#L5368-L5383) we treat an empty generation as a request for the 'current' generation of that bucket's sync status - just because that's the only generation that 'bucket incremental sync' can ever make progress on

In discussion w/Yuval the topic of generation "reset" also came up--is that actually needed?

not sure what this means

In discussion w/Yuval, the topic of generation "reset" also came up--is that actually needed?

@yuvalif was that related to bucket sync disable/enable? do you suspect there's a problem with empty gen vs. gen=0 there?

In discussion w/Yuval, the topic of generation "reset" also came up--is that actually needed?

@yuvalif was that related to bucket sync disable/enable? do you suspect there's a problem with empty gen vs. gen=0 there?

was wondering about the case where we have incremental sync with some gen going on, and then we disable+enable sync.
this results in a full sync with no generation. if we set the gen number of the full sync to zero, how would we know how to compare these entries with the existing incremental sync enties?

was wondering about the case where we have incremental sync with some gen going on, and then we disable+enable sync. this results in a full sync with no generation

let's be careful to differentiate between 'bucket full sync' and 'data full sync'. bucket sync disable/enable is cycling through the Incremental -> Init -> Full -> Incremental states of bucket sync, not data sync

remember this logic in the Init state that fetches the remote's latest log generation and markers, and records them as the position where Incremental should resume after Full. so bucket sync uses this sync status object to keep track of which generation it's on

@adamemerson unfortunately my approach in #45417 (comment) isn't going to work, so this PR may be the best we can do

my main concern with this one is that it allows us to spawn two concurrent syncs on the same thing (one with a real gen, and one with empty), now that they're tracked by separate cache entries

when the 'bucket sync cache' was added, it allowed us to detect racing calls to RGWDataSyncSingleEntryCR and avoid spawning sync on both. because of this, we were able to able remove the use of per-bucket-shard cls_locks from bucket sync

however, along with that change to remove locking, i did at least make sure that all the sync status writes were using cls_version, so i believe that racing bucket syncs should be okay in general. they may duplicate some work, but the first time they try to record their progress in the sync status object, one of them will fail with ECANCELED

so let's take this fix, and try to exercise these races to make sure this untested error path does what we expect?

cc @yuvalif @mattbenjamin

merged into wip-rgw-multisite-reshard, thanks @adamemerson