Skip to content

rgw: groundwork for supporting dynamic resharding in multisite environment#34096

Merged
cbodley merged 6 commits intoceph:masterfrom
smanjara:wip-dynamic-resharding
Apr 1, 2020
Merged

rgw: groundwork for supporting dynamic resharding in multisite environment#34096
cbodley merged 6 commits intoceph:masterfrom
smanjara:wip-dynamic-resharding

Conversation

@smanjara
Copy link
Contributor

Work related to introducing bucket index layout and replacing the existing RGWBucketInfo's sharding-related information with the layout.

Design doc: https://github.com/ceph/ceph/blob/master/src/doc/rgw/multisite-reshard.md#bucket-index-resharding

@cbodley
Copy link
Contributor

cbodley commented Mar 20, 2020

nice work! you're right that it's rather verbose to type out layout.current_index.layout.normal.num_shards. i'd eventually like to see some more refactoring so that less of this code relies on having a 'normal' layout - and at least this makes it easy to grep for current_index.layout.normal to find all of the places that do

cbodley and others added 3 commits March 30, 2020 16:31
Signed-off-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>
@smanjara smanjara force-pushed the wip-dynamic-resharding branch from 438fe7b to f9d0a81 Compare March 30, 2020 11:01
@cbodley
Copy link
Contributor

cbodley commented Mar 30, 2020

lets get the RGWBucketInfo changes tested/merged first, then follow up with the changes to reshard itself

Shilpa Jagannath added 2 commits March 31, 2020 16:43
…_layout header file

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>
… decoding

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>
@smanjara smanjara force-pushed the wip-dynamic-resharding branch from f9d0a81 to 2cb7ef5 Compare March 31, 2020 11:23
…ectly

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>
@smanjara smanjara force-pushed the wip-dynamic-resharding branch 2 times, most recently from 6021fbb to d6b1e6b Compare April 1, 2020 09:36
@smanjara
Copy link
Contributor Author

smanjara commented Apr 1, 2020

Test run:
http://pulpito.ceph.com/smanjara-2020-04-01_09:34:22-rgw-wip-dynamic-resharding-distro-basic-smithi/

@cbodley cbodley merged commit ba10480 into ceph:master Apr 1, 2020
@cbodley cbodley mentioned this pull request Apr 15, 2020
3 tasks
Copy link
Contributor

@cbodley cbodley left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

addressed in #34570

uint32_t hash_type;
JSONDecoder::decode_json("bi_shard_hash_type", hash_type, obj);
bucket_index_shard_hash_type = (uint8_t)hash_type;
layout.current_index.layout.normal.num_shards = (uint8_t)hash_type;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oops, this should be layout.normal.hash_type

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for catching this!

info.layout.current_index.layout.normal.num_shards = bucket_index_max_shards;
}
info.bucket_index_shard_hash_type = RGWBucketInfo::MOD;
info.layout.current_index.layout.normal.num_shards = RGWBucketInfo::MOD;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

layout.normal.hash_type = rgw::BucketHashType::Mod

// - value of 0 indicates there is no sharding (this is by default
// before this feature is implemented).
// - value of UINT32_T::MAX indicates this is a blind bucket.
uint32_t num_shards{0};
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The variable was removed, but the comments immediately before describing the variable were left behind, making reading the code confusing.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ack. will address in the currently open pr 35175

}

int num_source_shards = (bucket_info.num_shards > 0 ? bucket_info.num_shards : 1);
int num_source_shards = (bucket_info.layout.current_index.layout.normal.num_shards > 0 ? bucket_info.layout.current_index.layout.normal.num_shards : 1);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All these transformations inserting layout.current_index.layout.normal throughout makes the code harder to understand. It was already super-repetetive, but this makes it stand out even more. I therefore wonder if it should just be a function on bucket_info or bucket_info.layout -- a function that returns a value 1 or larger to indicate the actual # of shards.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this one too.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you, @smanjara!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks @ivancich. yeah, there's a bunch of new indirection happening here. part of it comes from the need to track a separate current_index layout and target_index during reshard. another part comes from the BucketIndexType abstraction, where only certain index types actually have a shard count - Indexless buckets would be an example of a type that doesn't, and Matt has also been exploring a tree-like index

so my long-term goal is to factor out most of the places where common code relies on a num_shards, so that they're constrained to code paths that a) have already checked the BucketIndexType, and b) are operating on a specific layout type like bucket_index_normal_layout& layout and have easy access to layout.num_shards

in the short-term, i do think a helper function is a good idea to cut down on the boilerplate. in later refactoring work, we can just grep for that function name to identify the places that we're making these unchecked assumptions about the index type. instead of a member function though, i'd recommend a free function like this:

// rgw_bucket_layout.h
inline uint32_t current_num_shards(const BucketLayout& layout) {
  return std::max(layout.current_index.layout.normal.num_shards, 1u);
}

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, this looks great! The true problem preceded these changes and has been around for a while, which is that the same logic was distributed (copy/pasted) throughout the code when it should have been centralized. The refactoring and its added indirection only made this issue pop out.

Thanks, @smanjara and @cbodley!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

helper function added in #35175

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants