rgw/multisite: remove per-shard sync status object after incremental sync finishes#42459
Closed
smanjara wants to merge 107 commits intoceph:wip-rgw-multisite-reshardfrom
Closed
rgw/multisite: remove per-shard sync status object after incremental sync finishes#42459smanjara wants to merge 107 commits intoceph:wip-rgw-multisite-reshardfrom
smanjara wants to merge 107 commits intoceph:wip-rgw-multisite-reshardfrom
Conversation
Signed-off-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>
allows other code to spawn this coroutine without having the class definition Signed-off-by: Casey Bodley <cbodley@redhat.com>
RGWShardCollectCR was hard-coded to ignore ENOENT errors and print a 'failed to fetch log status' error message. this moves that logic into a handle_result() virtual function. it also exposes the member variables 'status' and 'max_concurrent' as protected, so they can be consulted or modified by overrides of handle_result() and spawn_next() Signed-off-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>
a coroutine to initialize a bucket for full sync using a new bucket-wide sync status object Signed-off-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>
full sync happens as the bucket level, so the shards will always start in StateIncrementalSync Signed-off-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>
renamed ListBucketShardCR to ListRemoteBucketCR and removed the shard-id parameter renamed BucketFullSyncShardMarkerTrack to BucketFullSyncMarkerTrack, which now updates the bucket-level rgw_bucket_sync_status renamed BucketShardFullSyncCR to BucketFullSyncCR BucketSyncCR now takes a bucket-wide lease during full sync Signed-off-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>
if metadata sync hasn't finished, the 'bucket checkpoint' commands may not find its bucket info Signed-off-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>
the ability to filter tests by attribute is provided by the nose.plugins.attrib plugin, which wasn't being loaded by default Signed-off-by: Casey Bodley <cbodley@redhat.com>
this backoff is triggered often by the per-bucket lease for full sync, and causes tests to fail with checkpoint timeouts Signed-off-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>
adds a backward-compatible binary encoding for error repo keys that can contain a generation number along with the bucket and shard Signed-off-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>
poll on rgw_read_bucket_full_sync_status() until full_status.incremental_gen catches up to the latest_gen we got from rgw_read_remote_bilog_info() Signed-off-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>
…revious generation Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>
rgw: add datalog entries after reshard
…ting Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>
…-fix-incremental rgw/multisite_reshard: pass the right generation number for bilog listing
Needed so we can get the incremental generation. Guard this behind a version check and return the original output if less than 2. Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
To bucket_index_layout_generation Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Fetch the current generation from remote peers and trim the minimum marker on the minimum generation. Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
From the REST interface and radosgw-admin. Assume Generation 0 if none provided and error if it doesn't exist. Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
A more generally applicable way of removing objects in coroutines. Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Only one generation per call. Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
the sync_pair is used as input to RGWBucketPipeSyncStatusManager::status_oid() to generate the per-shard sync status object names this sync status tracks incremental bucket sync, which reads changes from a source bucket's bilog shard, and copies objects from the remote source bucket to the local destination bucket this doesn't require sync to know anything about the destination bucket shards, so rgw_bucket_sync_pair_info and status_oid() now only track the the destination's rgw_bucket instead of rgw_bucket_shard Signed-off-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>
Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>
smanjara
commented
Jul 23, 2021
| } | ||
| } | ||
|
|
||
| yield { |
Contributor
Author
There was a problem hiding this comment.
@cbodley I'm not sure if this is the right place to add shard object removal. We could add this piece after we finish running sync_bucket_shard_cr in RGWRunBucketSourcesSyncCR() to make sure that the sync is complete for a particular shard before removal.
31 tasks
679ce4a to
5bbcc37
Compare
|
This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved |
3 tasks
Contributor
|
updated in #42999 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Signed-off-by: Shilpa Jagannath smanjara@redhat.com
Checklist
Show available Jenkins commands
jenkins retest this pleasejenkins test classic perfjenkins test crimson perfjenkins test signedjenkins test make checkjenkins test make check arm64jenkins test submodulesjenkins test dashboardjenkins test apijenkins test docsjenkins render docsjenkins test ceph-volume alljenkins test ceph-volume tox