[WIP]rgw: update bucket sync status after bucket shards finishes current gen#40605
Closed
smanjara wants to merge 82 commits intoceph:wip-rgw-multisite-reshardfrom
Closed
[WIP]rgw: update bucket sync status after bucket shards finishes current gen#40605smanjara wants to merge 82 commits intoceph:wip-rgw-multisite-reshardfrom
smanjara wants to merge 82 commits intoceph:wip-rgw-multisite-reshardfrom
Conversation
Signed-off-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>
allows other code to spawn this coroutine without having the class definition Signed-off-by: Casey Bodley <cbodley@redhat.com>
RGWShardCollectCR was hard-coded to ignore ENOENT errors and print a 'failed to fetch log status' error message. this moves that logic into a handle_result() virtual function. it also exposes the member variables 'status' and 'max_concurrent' as protected, so they can be consulted or modified by overrides of handle_result() and spawn_next() Signed-off-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>
a coroutine to initialize a bucket for full sync using a new bucket-wide sync status object Signed-off-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>
full sync happens as the bucket level, so the shards will always start in StateIncrementalSync Signed-off-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>
renamed ListBucketShardCR to ListRemoteBucketCR and removed the shard-id parameter renamed BucketFullSyncShardMarkerTrack to BucketFullSyncMarkerTrack, which now updates the bucket-level rgw_bucket_sync_status renamed BucketShardFullSyncCR to BucketFullSyncCR BucketSyncCR now takes a bucket-wide lease during full sync Signed-off-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>
if metadata sync hasn't finished, the 'bucket checkpoint' commands may not find its bucket info Signed-off-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>
the ability to filter tests by attribute is provided by the nose.plugins.attrib plugin, which wasn't being loaded by default Signed-off-by: Casey Bodley <cbodley@redhat.com>
this backoff is triggered often by the per-bucket lease for full sync, and causes tests to fail with checkpoint timeouts Signed-off-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>
adds a backward-compatible binary encoding for error repo keys that can contain a generation number along with the bucket and shard Signed-off-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>
cbodley
reviewed
Apr 15, 2021
e3f6d92 to
221a8d7
Compare
Contributor
Author
|
@cbodley addressed all comments. Please review. Thanks. |
cbodley
reviewed
Apr 22, 2021
4a0ae3e to
e916a63
Compare
Contributor
|
@smanjara @adamemerson what's the current plan for this one? Is this merging shortly? |
Contributor
Author
It needs another round of review by @cbodley |
cbodley
reviewed
Apr 30, 2021
if the old index is still referenced by an InIndex log layout, we can't call clean_index() to remove the index objects yet. log trimming will do that later, once the bilogs are no longer needed Signed-off-by: Casey Bodley <cbodley@redhat.com>
e916a63 to
5bff3af
Compare
cbodley
reviewed
May 3, 2021
Contributor
|
now that we're flagging the shards as |
Contributor
Author
done. Since the check for current gen is already present #39396, I have just added a check to see if the shard is done or not before spawning RGWSyncBucketShardCR |
cbodley
reviewed
May 4, 2021
f61a8e0 to
99ba8fc
Compare
cbodley
approved these changes
May 4, 2021
src/rgw/rgw_data_sync.cc
Outdated
| sync_pipe, bucket_status.state, | ||
| tn, progress)); | ||
| if (retcode < 0) { | ||
| return set_cr_error(retcode); |
Contributor
There was a problem hiding this comment.
it looks like the indentation is off for this whole block
Contributor
Author
There was a problem hiding this comment.
yes it should have been under Incremental sync block. thanks
Contributor
|
once you fix the indentation, can you please squash the commits? |
b6bca55 to
733eb58
Compare
Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>
733eb58 to
6d7bb1c
Compare
Contributor
Author
done |
36c2959 to
95355ab
Compare
|
This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved |
Contributor
|
rebased (fixed some dpp conflicts) and merged into wip-rgw-multisite-reshard. thanks! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Signed-off-by: Shilpa Jagannath smanjara@redhat.com
Checklist
Show available Jenkins commands
jenkins retest this pleasejenkins test classic perfjenkins test crimson perfjenkins test signedjenkins test make checkjenkins test make check arm64jenkins test submodulesjenkins test dashboardjenkins test apijenkins test docsjenkins render docsjenkins test ceph-volume alljenkins test ceph-volume tox