rgw: groundwork for supporting dynamic resharding in multisite environment#34096
rgw: groundwork for supporting dynamic resharding in multisite environment#34096cbodley merged 6 commits intoceph:masterfrom
Conversation
|
nice work! you're right that it's rather verbose to type out |
Signed-off-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>
438fe7b to
f9d0a81
Compare
|
lets get the RGWBucketInfo changes tested/merged first, then follow up with the changes to reshard itself |
…_layout header file Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>
… decoding Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>
f9d0a81 to
2cb7ef5
Compare
…ectly Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>
6021fbb to
d6b1e6b
Compare
| uint32_t hash_type; | ||
| JSONDecoder::decode_json("bi_shard_hash_type", hash_type, obj); | ||
| bucket_index_shard_hash_type = (uint8_t)hash_type; | ||
| layout.current_index.layout.normal.num_shards = (uint8_t)hash_type; |
There was a problem hiding this comment.
oops, this should be layout.normal.hash_type
There was a problem hiding this comment.
thanks for catching this!
| info.layout.current_index.layout.normal.num_shards = bucket_index_max_shards; | ||
| } | ||
| info.bucket_index_shard_hash_type = RGWBucketInfo::MOD; | ||
| info.layout.current_index.layout.normal.num_shards = RGWBucketInfo::MOD; |
There was a problem hiding this comment.
layout.normal.hash_type = rgw::BucketHashType::Mod
| // - value of 0 indicates there is no sharding (this is by default | ||
| // before this feature is implemented). | ||
| // - value of UINT32_T::MAX indicates this is a blind bucket. | ||
| uint32_t num_shards{0}; |
There was a problem hiding this comment.
The variable was removed, but the comments immediately before describing the variable were left behind, making reading the code confusing.
There was a problem hiding this comment.
ack. will address in the currently open pr 35175
| } | ||
|
|
||
| int num_source_shards = (bucket_info.num_shards > 0 ? bucket_info.num_shards : 1); | ||
| int num_source_shards = (bucket_info.layout.current_index.layout.normal.num_shards > 0 ? bucket_info.layout.current_index.layout.normal.num_shards : 1); |
There was a problem hiding this comment.
All these transformations inserting layout.current_index.layout.normal throughout makes the code harder to understand. It was already super-repetetive, but this makes it stand out even more. I therefore wonder if it should just be a function on bucket_info or bucket_info.layout -- a function that returns a value 1 or larger to indicate the actual # of shards.
There was a problem hiding this comment.
thanks @ivancich. yeah, there's a bunch of new indirection happening here. part of it comes from the need to track a separate current_index layout and target_index during reshard. another part comes from the BucketIndexType abstraction, where only certain index types actually have a shard count - Indexless buckets would be an example of a type that doesn't, and Matt has also been exploring a tree-like index
so my long-term goal is to factor out most of the places where common code relies on a num_shards, so that they're constrained to code paths that a) have already checked the BucketIndexType, and b) are operating on a specific layout type like bucket_index_normal_layout& layout and have easy access to layout.num_shards
in the short-term, i do think a helper function is a good idea to cut down on the boilerplate. in later refactoring work, we can just grep for that function name to identify the places that we're making these unchecked assumptions about the index type. instead of a member function though, i'd recommend a free function like this:
// rgw_bucket_layout.h
inline uint32_t current_num_shards(const BucketLayout& layout) {
return std::max(layout.current_index.layout.normal.num_shards, 1u);
}There was a problem hiding this comment.
Yeah, this looks great! The true problem preceded these changes and has been around for a while, which is that the same logic was distributed (copy/pasted) throughout the code when it should have been centralized. The refactoring and its added indirection only made this issue pop out.
Work related to introducing bucket index layout and replacing the existing RGWBucketInfo's sharding-related information with the layout.
Design doc: https://github.com/ceph/ceph/blob/master/src/doc/rgw/multisite-reshard.md#bucket-index-resharding