ceph-dencoder: RBD - Add missing types#52481
Conversation
|
jenkins test make check |
src/tools/ceph-dencoder/rbd_types.h
Outdated
| using namespace librbd::watch_notify; | ||
| TYPE_NOCOPY(NotifyMessage) | ||
| TYPE(ResponseMessage) | ||
| TYPE(AsyncRequestId) |
There was a problem hiding this comment.
Same here and below for using namespace cls::rbd;. I don't understand why these entries are being duplicated.
There was a problem hiding this comment.
the corpuse repository contain types without namespaces. that causing ceph-dencoder to output the type is not recognize (wihtout full namespace), i didn't want to remove the duplicate exist type names.
There was a problem hiding this comment.
the corpuse repository contain types without namespaces.
Still not following. For example, RBD has several different NotifyMessage types in different namespaces. How does that work?
that causing ceph-dencoder to output the type is not recognize (wihtout full namespace)
Do you have an example of such output?
There was a problem hiding this comment.
got it, so we will need from now to change the WRITE_CLASS_ENCODER so we will dump the type with the namespace, currently we have WRITE_CLASS_ENCODER(NotifyMessage) so it will be WRITE_CLASS_ENCODER(librbd::watch_notify::NotifyMessage).
we will lose the archived version checks, but we will make it right for the next versions.
There was a problem hiding this comment.
got it, so we will need from now to change the WRITE_CLASS_ENCODER so we will dump the type with the namespace, currently we have WRITE_CLASS_ENCODER(NotifyMessage) so it will be WRITE_CLASS_ENCODER(librbd::watch_notify::NotifyMessage).
My first instinct would be to change WRITE_CLASS_ENCODER macro to get the fully-qualified name of the type internally and use that for naming the dump file. But, if
we will lose the archived version checks, but we will make it right for the next versions.
is the case, that is probably too disruptive.
Hrm... I think NotifyMessage is unique in that we have five of them in different namespaces so as a (rather poor) workaround we might consider just giving each of them a unique name.
But fully-qualified names have been there in ceph-dencoder types.h and later rbd_types.h for years: see e.g. eb8d37a. Are you saying that it never worked?
Can you describe the breakage related to the object corpus in detail? Since three out of five NotifyMessage types happen to be registered with ceph-dencoder but their fully-qualified names get stripped, what actually made it to the corpus?
There was a problem hiding this comment.
@idryomov I added new column to your table if corpus has object:
| WRITE_CLASS macro in the code | TYPE macro in rbd_types.h or as noted | STATUS | NOTES | corpus entries |
|---|---|---|---|---|
| src/cls/journal/cls_journal_types.h:WRITE_CLASS_ENCODER(ObjectPosition); | TYPE(cls::journal::ObjectPosition) | in rgw_types.h -- why? | ||
| src/cls/journal/cls_journal_types.h:WRITE_CLASS_ENCODER(ObjectSetPosition); | TYPE(cls::journal::ObjectSetPosition) | in rgw_types.h -- why? | ||
| src/cls/journal/cls_journal_types.h:WRITE_CLASS_ENCODER(Client); | TYPE(cls::journal::Client) | in rgw_types.h -- why? | ||
| src/cls/journal/cls_journal_types.h:WRITE_CLASS_ENCODER(Tag); | TYPE(cls::journal::Tag) | in rgw_types.h -- why? | ||
| src/cls/lock/cls_lock_ops.h:WRITE_CLASS_ENCODER(cls_lock_lock_op); | TYPE(cls_lock_lock_op) | MATCH | in common_types.h | |
| src/cls/lock/cls_lock_ops.h:WRITE_CLASS_ENCODER(cls_lock_unlock_op); | TYPE(cls_lock_unlock_op) | MATCH | in common_types.h | |
| src/cls/lock/cls_lock_ops.h:WRITE_CLASS_ENCODER(cls_lock_break_op); | TYPE(cls_lock_break_op) | MATCH | in common_types.h | |
| src/cls/lock/cls_lock_ops.h:WRITE_CLASS_ENCODER(cls_lock_get_info_op); | TYPE(cls_lock_get_info_op) | MATCH | in common_types.h | |
| src/cls/lock/cls_lock_ops.h:WRITE_CLASS_ENCODER_FEATURES(cls_lock_get_info_reply); | TYPE_FEATUREFUL(cls_lock_get_info_reply) | MATCH | in common_types.h | |
| src/cls/lock/cls_lock_ops.h:WRITE_CLASS_ENCODER(cls_lock_list_locks_reply); | TYPE(cls_lock_list_locks_reply) | MATCH | in common_types.h | |
| src/cls/lock/cls_lock_ops.h:WRITE_CLASS_ENCODER(cls_lock_assert_op); | TYPE(cls_lock_assert_op) | MATCH | in common_types.h | |
| src/cls/lock/cls_lock_ops.h:WRITE_CLASS_ENCODER(cls_lock_set_cookie_op); | TYPE(cls_lock_set_cookie_op) | MATCH | in common_types.h | |
| src/cls/lock/cls_lock_types.h: WRITE_CLASS_ENCODER(locker_id_t); | TYPE(rados::cls::lock::locker_id_t) | in common_types.h | ||
| src/cls/lock/cls_lock_types.h: WRITE_CLASS_ENCODER_FEATURES(locker_info_t); | TYPE_FEATUREFUL(rados::cls::lock::locker_info_t) | in common_types.h | ||
| src/cls/lock/cls_lock_types.h: WRITE_CLASS_ENCODER_FEATURES(lock_info_t); | TYPE_FEATUREFUL(rados::cls::lock::lock_info_t) | in common_types.h | ||
| src/cls/rbd/cls_rbd.h:WRITE_CLASS_ENCODER_FEATURES(cls_rbd_parent); | TYPE_FEATUREFUL(cls_rbd_parent) | MATCH | ./objects/cls_rbd_parent | |
| src/cls/rbd/cls_rbd.h:WRITE_CLASS_ENCODER_FEATURES(cls_rbd_snap); | TYPE_FEATUREFUL(cls_rbd_snap) | MATCH | ./objects/cls_rbd_snap | |
| src/cls/rbd/cls_rbd_types.h:WRITE_CLASS_ENCODER(MirrorPeer); | TYPE(cls::rbd::MirrorPeer) | ./objects/cls::rbd::MirrorPeer | ||
| src/cls/rbd/cls_rbd_types.h:WRITE_CLASS_ENCODER(MirrorImage); | TYPE(cls::rbd::MirrorImage) | ./objects/cls::rbd::MirrorImage | ||
| src/cls/rbd/cls_rbd_types.h:WRITE_CLASS_ENCODER(MirrorImageSiteStatus); | TYPE(cls::rbd::MirrorImageSiteStatus) | ./objects/cls::rbd::MirrorImageSiteStatus | ||
| src/cls/rbd/cls_rbd_types.h:WRITE_CLASS_ENCODER_FEATURES(MirrorImageSiteStatusOnDisk); | TYPE_FEATUREFUL(cls::rbd::MirrorImageSiteStatusOnDisk) | ./objects/cls::rbd::MirrorImageSiteStatusOnDisk | ||
| src/cls/rbd/cls_rbd_types.h:WRITE_CLASS_ENCODER(MirrorImageStatus); | TYPE(cls::rbd::MirrorImageStatus) | ./objects/cls::rbd::MirrorImageStatus | ||
| src/cls/rbd/cls_rbd_types.h:WRITE_CLASS_ENCODER(ParentImageSpec); | TYPE(cls::rbd::ParentImageSpec) | ./objects/cls::rbd::ParentImageSpec | ||
| src/cls/rbd/cls_rbd_types.h:WRITE_CLASS_ENCODER(ChildImageSpec); | TYPE(cls::rbd::ChildImageSpec) | ./objects/cls::rbd::ChildImageSpec | ||
| src/cls/rbd/cls_rbd_types.h:WRITE_CLASS_ENCODER(GroupImageSpec); | TYPE(cls::rbd::GroupImageSpec) | ./objects/cls::rbd::GroupImageSpec | ||
| src/cls/rbd/cls_rbd_types.h:WRITE_CLASS_ENCODER(GroupImageStatus); | TYPE(cls::rbd::GroupImageStatus) | ./objects/cls::rbd::GroupImageStatus | ||
| src/cls/rbd/cls_rbd_types.h:WRITE_CLASS_ENCODER(GroupSpec); | TYPE(cls::rbd::GroupSpec) | ./objects/cls::rbd::GroupSpec | ||
| src/cls/rbd/cls_rbd_types.h:WRITE_CLASS_ENCODER(SnapshotNamespace); | TYPE(cls::rbd::SnapshotNamespace) | ./objects/cls::rbd::SnapshotNamespace | ||
| src/cls/rbd/cls_rbd_types.h:WRITE_CLASS_ENCODER(SnapshotInfo); | TYPE(cls::rbd::SnapshotInfo) | ./objects/cls::rbd::SnapshotInfo | ||
| src/cls/rbd/cls_rbd_types.h:WRITE_CLASS_ENCODER(ImageSnapshotSpec); | TYPE(cls::rbd::ImageSnapshotSpec) | ./objects/cls::rbd::ImageSnapshotSpec | ||
| src/cls/rbd/cls_rbd_types.h:WRITE_CLASS_ENCODER(GroupSnapshot); | TYPE(cls::rbd::GroupSnapshot) | ./objects/cls::rbd::GroupSnapshot | ||
| src/cls/rbd/cls_rbd_types.h:WRITE_CLASS_ENCODER(TrashImageSpec); | ||||
| src/cls/rbd/cls_rbd_types.h:WRITE_CLASS_ENCODER(MirrorImageMap); | TYPE(cls::rbd::MirrorImageMap) | ./objects/cls::rbd::MirrorImageMap | ||
| src/cls/rbd/cls_rbd_types.h:WRITE_CLASS_ENCODER(MigrationSpec); | TYPE(cls::rbd::MigrationSpec) | ./objects/cls::rbd::MigrationSpec | ||
| src/journal/Entry.h:WRITE_CLASS_ENCODER(journal::Entry); | TYPE(journal::Entry) | MATCH | in common_types.h | |
| src/librbd/WatchNotifyTypes.h:WRITE_CLASS_ENCODER(ClientId); | duplicate WRITE_CLASS_ENCODER for librbd::watcher::ClientId | |||
| src/librbd/WatchNotifyTypes.h:WRITE_CLASS_ENCODER(AsyncRequestId); | ||||
| src/librbd/WatchNotifyTypes.h:WRITE_CLASS_ENCODER(NotifyMessage); | TYPE_NOCOPY(librbd::watch_notify::NotifyMessage) | ./objects/librbd::watch_notify::NotifyMessage | ||
| src/librbd/WatchNotifyTypes.h:WRITE_CLASS_ENCODER(ResponseMessage); | TYPE(librbd::watch_notify::ResponseMessage) | ./objects/librbd::watch_notify::ResponseMessage | ||
| src/librbd/cache/pwl/Types.h:WRITE_CLASS_DENC(librbd::cache::pwl::WriteLogCacheEntry); | TYPE(librbd::cache::pwl::WriteLogCacheEntry) | MATCH | ||
| src/librbd/cache/pwl/Types.h:WRITE_CLASS_DENC(librbd::cache::pwl::WriteLogPoolRoot); | TYPE(librbd::cache::pwl::WriteLogPoolRoot) | MATCH | ||
| src/librbd/cache/pwl/ssd/Types.h:WRITE_CLASS_DENC(librbd::cache::pwl::ssd::SuperBlock); | TYPE(librbd::cache::pwl::ssd::SuperBlock) | MATCH | ||
| src/librbd/journal/Types.h:WRITE_CLASS_ENCODER(EventEntry); | TYPE(librbd::journal::EventEntry) | ./objects/librbd::journal::EventEntry | ||
| src/librbd/journal/Types.h:WRITE_CLASS_ENCODER(ClientData); | TYPE(librbd::journal::ClientData) | ./objects/librbd::journal::ClientData | ||
| src/librbd/journal/Types.h:WRITE_CLASS_ENCODER(TagData); | TYPE(librbd::journal::TagData) | ./objects/librbd::journal::TagData | ||
| src/librbd/mirror/snapshot/Types.h:WRITE_CLASS_ENCODER(ImageStateHeader); | ||||
| src/librbd/mirror/snapshot/Types.h:WRITE_CLASS_ENCODER(SnapState); | ||||
| src/librbd/mirror/snapshot/Types.h:WRITE_CLASS_ENCODER(ImageState); | ||||
| src/librbd/mirroring_watcher/Types.h:WRITE_CLASS_ENCODER(NotifyMessage); | TYPE(librbd::mirroring_watcher::NotifyMessage) | ./objects/librbd::mirroring_watcher::NotifyMessage | ||
| src/librbd/trash_watcher/Types.h:WRITE_CLASS_ENCODER(NotifyMessage); | TYPE(librbd::trash_watcher::NotifyMessage) | ./objects/librbd::trash_watcher::NotifyMessage | ||
| src/librbd/watcher/Types.h:WRITE_CLASS_ENCODER(ClientId); | ||||
| src/librbd/watcher/Types.h:WRITE_CLASS_ENCODER(NotifyResponse); | ||||
| src/rbd_replay/ActionTypes.h:WRITE_CLASS_ENCODER(Dependency); | TYPE(rbd_replay::action::Dependency) | ./objects/rbd_replay::action::Dependency | ||
| src/rbd_replay/ActionTypes.h:WRITE_CLASS_ENCODER(ActionEntry); | TYPE(rbd_replay::action::ActionEntry) | ./objects/rbd_replay::action::ActionEntry | ||
| src/tools/rbd_mirror/image_map/Types.h:WRITE_CLASS_ENCODER(PolicyData); | TYPE(rbd::mirror::image_map::PolicyData) | ./objects/rbd::mirror::image_map::PolicyData | ||
| src/tools/rbd_mirror/instance_watcher/Types.h:WRITE_CLASS_ENCODER(NotifyMessage); | ||||
| src/tools/rbd_mirror/instance_watcher/Types.h:WRITE_CLASS_ENCODER(NotifyAckPayload); | ||||
| src/tools/rbd_mirror/leader_watcher/Types.h:WRITE_CLASS_ENCODER(NotifyMessage); |
yes, most of the types doesn't have entries in corpus, I'll need to add the corpus script more test that will cause the corpus capture those types
There was a problem hiding this comment.
@idryomov if we want to match the TYPE and WRITE_CLASS_ENCODER, that means we need to update all the corpus directories as well (if we updating the TYPE)
There was a problem hiding this comment.
@idryomov if we want to match the TYPE and WRITE_CLASS_ENCODER, that means we need to update all the corpus directories as well (if we updating the TYPE)
It's been a while and the details are out of my head... IIRC the reason I put together this table is that you stated that the namespace part of the name doesn't make it to the corpus:
the corpuse repository contain types without namespaces. that causing ceph-dencoder to output the type is not recognize (wihtout full namespace), i didn't want to remove the duplicate exist type names.
But now, entries like ./objects/cls::rbd::MirrorPeer, ./objects/librbd::journal::TagData, etc in the column that you added today seem to contradict that -- the namespace part is clearly there.
So I'm not sure what you mean by "if we want to match the TYPE and WRITE_CLASS_ENCODER" (emphasis mine). Am I wrong in assuming that they need to match for the tests to work correctly?
There was a problem hiding this comment.
Am I wrong in assuming that they need to match for the tests to work correctly?
If I'm not wrong, then a further observation is that the intersection of MATCH status and an entry being present in the corpus produces only two entries: ./objects/cls_rbd_parent and ./objects/cls_rbd_snap.
- Are these two types the only RBD-related types that are not only stored in the corpus and also checked?
- What is the behavior for entries such as
./objects/cls::rbd::MirrorPeerwhich are stored in the corpus but their WRITE_CLASS_ENCODER and TYPE names don't match? Are they just skipped?
There was a problem hiding this comment.
Are they just skipped?
You may have answered this already in an earlier comment
[...] since the corpus script first check if the type is supported by ceph-dencoder, its ignore all the types that return unsupported.
but I want to get a confirmation to avoid misunderstanding.
Is there some message logged when this happens? If so, can you confirm that this message is produced for cls::rbd::MirrorPeer and other RBD-related types that have entries in the corpus except for cls_rbd_parent and cls_rbd_snap?
39d63d8 to
acc566e
Compare
6f7684d to
be5b64b
Compare
|
jenkins retest this please |
be5b64b to
9e957e7
Compare
574fe56 to
f3d4c42
Compare
…code-decode comparison Currently, ceph-dencoder lacks certain rbd types, preventing us from accurately checking the ceph corpus for encode-decode mismatches. This pull request aims to address this issue by adding the missing types to ceph-dencoder. To successfully incorporate these types into ceph-dencoder, we need to introduce the necessary `dump` and `generate_test_instances` functions that was missing in some types. These functions are essential for proper encode and decode of the added types. This PR will enhance the functionality of ceph-dencoder by including the missing types, enabling a comprehensive analysis of encode-decode consistency. With the addition of these types, we can ensure the robustness and correctness of the ceph corpus. This update will significantly contribute to improving the overall reliability and accuracy of ceph-dencoder. It allows for a more comprehensive assessment of the encode-decode behavior, leading to enhanced data integrity and stability within the ceph ecosystem. Fixes: https://tracker.ceph.com/issues/61788 Signed-off-by: Nitzan Mordechai <nmordech@redhat.com>
Signed-off-by: Nitzan Mordechai <nmordech@redhat.com>
f3d4c42 to
892672c
Compare
|
@idryomov i completed the changes for ceph-objects-corpus and now we are good to go. is there any other changes needed for that PR? |
What exactly do you mean here? I see "submodule: update ceph-object-corpus submodule" commit, but it doesn't seem to touch any submodules. In fact it appears to be completely empty. |
| TYPE(GroupSpec) | ||
| TYPE(ImageSnapshotSpec) | ||
| TYPE(SnapshotInfo) | ||
| TYPE(SnapshotNamespace) |
There was a problem hiding this comment.
We need to decide which way to go for types which are not matching or missing: introduce them either as fully qualified or not fully qualified and be consistent about it. This is actually an interesting question because I don't think we have ever intended for C++ namespaces to "leak" in this fashion. Regardless, I really doubt that both
TYPE(cls::rbd::SnapshotNamespace)
and
TYPE(SnapshotNamespace)
are needed and this goes for 15 other types above too.
The only type which has to be fully qualified is NotifyMessage because there are five of them in different namespaces.
|
This pull request has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs for another 30 days. |
|
This pull request has been automatically closed because there has been no activity for 90 days. Please feel free to reopen this pull request (or open a new one) if the proposed change is still appropriate. Thank you for your contribution! |
|
We talked about resurrecting this PR during the CDS RADOS session this Tuesday. |
|
This pull request has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs for another 30 days. |
|
This pull request has been automatically closed because there has been no activity for 90 days. Please feel free to reopen this pull request (or open a new one) if the proposed change is still appropriate. Thank you for your contribution! |
Currently, ceph-dencoder lacks certain rbd types, preventing us from accurately checking the ceph corpus for encode-decode mismatches. This pull request aims to address this issue by adding the missing types to ceph-dencoder.
To successfully incorporate these types into ceph-dencoder, we need to introduce the necessary
dumpandgenerate_test_instancesfunctions that was missing in some types. These functions are essential for proper encode and decode of the added types.This PR will enhance the functionality of ceph-dencoder by including the missing types, enabling a comprehensive analysis of encode-decode consistency. With the addition of these types, we can ensure the robustness and correctness of the ceph corpus.
This update will significantly contribute to improving the overall reliability and accuracy of ceph-dencoder. It allows for a more comprehensive assessment of the encode-decode behavior, leading to enhanced data integrity and stability within the ceph ecosystem.
Fixes: https://tracker.ceph.com/issues/61788
Contribution Guidelines
To sign and title your commits, please refer to Submitting Patches to Ceph.
If you are submitting a fix for a stable branch (e.g. "pacific"), please refer to Submitting Patches to Ceph - Backports for the proper workflow.
Checklist
Show available Jenkins commands
jenkins retest this pleasejenkins test classic perfjenkins test crimson perfjenkins test signedjenkins test make checkjenkins test make check arm64jenkins test submodulesjenkins test dashboardjenkins test dashboard cephadmjenkins test apijenkins test docsjenkins render docsjenkins test ceph-volume alljenkins test ceph-volume toxjenkins test windows