There and back again.

De-scrambling this omelette took way longer than I originally anticipated, and scope of work grew way bigger than originally intended. There were a lot of interdependencies that needed to untangle. This started as an effort to move stuff out of RGWRados into modules. Turned quickly into refactoring of metadata handling. The original metadata code was doing some funny things in order to ensure that when writing metadata entries (e.g., bucket/user creation/modification/removal), the metadata log is updated. The different calls (e.g., RGWRados::put_bucket_instance_info(), etc.) would translate the entity id (e.g., bucket, user) into a metadata key, and then would funnel into RGWMetadataManager::put_entry() that will then convert that metadata key into raods {pool, oid} (by querying the specific metadata handler), update the metadata log, and write the corresponding system object. A generic metadata put would work similarly, but will go from RGWMetadataManagaer::put() into the corresponding handler put() method, which in turn will call the manager put_entry(). The otp handler called the manager mutate() instead of put_entry() as it wasn't doing a simple system object write, but called a cls_otp method.

New handling works different by trying to make sure that RGWMetadataManager doesn't need to know anything about the metadata layout on one hand, and on the other hand the different random utility functions (e.g., rgw_put_bucket_store_info()) don't call into the metadata manager directly or indirectly. This responsibility was pushed down to a set of svc (I'll avoid stating the badly named term 'service', should have been 'module'; if anyone wants to run a find and replace and change everything be my guest). There are the meta and meta_be svcs. The meta_be (backend) svc is the base svc for meta backend handling. Then there are different meta_be implementations: meta_be_sobj, meta_be_otp (that extends from sobj).

A meta_be svc has a backend type associated with it (e.g., SOBJ, OTP). Each can have different type of data that it needs in order to access and control the backend. The abstract RGWSI_MetaBackend::Context is being used to hold that data, and each backend type extends it. However, the meta_be svc is generic, and doesn't know anything about buckets and users. For each metadata type we need to create a meta-backend handler instance. The meta-backend handler can generate a type specific context that will be used by the specific backend. For the sobj backend there is a "module" that the handler holds and has a set of methods that can be used to translate between meta keys and the sobj {pool, oid}. Each metadata handler that wants to use the sobj backend implements a module instance that does these translations. At the moment things are explicit, and everything is created on initialization and ad-hoc. The metadata handlers at the moment know that they will be used with sobj (or otp) backends. In the future we can do things a bit differently in the way these things are constructed, but this structure should still hold.

Now that everything was pushed down below the metadata manager and metadata handlers, the utility functions don't need to call into them for writing certain metadata entries. However, there was a need to create new utility code that would use the new scheme. Note that when creating a user (for example), we don't just create the metadata entry, but there are a bunch of other objects that we create (the different indexes -- access_key, swift, email). These are also very specific to the backend that is being used. I created new base svcs for user, and for bucket, and created implementation svcs: user_rados, bucket_sobj. Note that the user implementation has rados specific calls, whereas bucket only does sobj specific stuff (which is still only rados, but in theory can be replaced).

It became clear to me that the svc layer is not where we really want to keep all the application apis. It should serve as a a lower abstraction layer for modular entities. One example for why this is needed is that calls that create a user's bucket, need to interact with both user svc and bucket svc. Having the two svcs know about each other would be problematic. Instead I created 'controls' (RGWBucketCtl, RGWUserCtl, RGWOTPCtl) that provide higher level apis and call themselves into the different svcs. The lower level svcs need to be provided with an appropriate backend context. Some of those don't really need that context, but I made sure they get it anyway -- mostly so that it's explicit that those calls might be backend dependent. Also, the context passed to these calls needed to be explicitly typed so that we don't make a mistake and accidentally pass a wrong context. This was a real issue when dealing with svc.bucket that has two separate backends: one for bucket, and one for bucket.instance.

I tried to avoid introducing new explicit dynamic allocations. This was a bit challenging, since the meta backends require having a context passed into them from the top level, but the top level doesn't need to know anything about the specific implementation. One way to do it was to leverage ceph::static_ptr<>, however, that required exposing all context implementations to the static_ptr<> definition, which I wasn't happy about. The solution I preferred was providing an abstract method on the backend that would get a lambda that receives a context pointer. The backends will then implement it by putting a context (of their type) on the stack, and calling that lambda. Hopefully that prevents dynamic allocations and is actually useful. The same was done for the backend handlers. With the latter there were cases where it was needed to pass two different context types (in the bucket + bucket.instance case).

Other notable things:

• A bunch of code was moved from RGWRados to a new 'cls' svc.
• A lot of old C calls that got RGWRados pointer as a param were removed, turned those into calls into the specific ctl/svc calls. There still a lot of these around.
• Changed a lot of calls that got RGWRados pointer and replaced it with the specific ctl or svc that they actually were using. Again, didn't go through everything, was doing it on a per-find basis.
• I removed a few REST management calls that were used to lock and unlock different multisite related logs. These were used in multisite v1. There might still be some around, but should also be removed in the future. I also removed the metadata lock functionality.
• I mostly removed most of the sync_type param that was used when putting a metadata object. This was maybe used in multisite v1, but I'm not sure.
• New svcs: bi (rados), bilog (rados), datalog (rados), mdlog (rados -- although not stated), quota
• ctl initaialization is done at rgw_service.cc after svc initialization, and the RGWCtl structure is similar to RGWService. RGWRados now also holds a ctl member, similar to svc.

TBD:

• There are still a few #warnings that need to be handled. The two main issues left behind are the metadata and chained cache that when I looked at I thought it needed some rework (looks like there are paths where we don't go to the metadata cache when dealing with bucket instance).

• Since code is more generic and avoids being backend specific, the new ctl methods don't accept RGWObjectCtx (which is sobj specific). However, by removing that we lose something, and there needs to be a (good) way to add it back.

• There is an unrelated includes cleanup work that snuck in and should be removed (and be sent separately).

• rebase

• lots of testing

• References tracker ticket

• Updates documentation if necessary

• Includes tests for new functionality or reproducer for bug