ceph-dencoder: MDS - Add missing types by NitzanMordhai · Pull Request #52623 · ceph/ceph

NitzanMordhai · 2023-07-25T11:06:37Z

…code-decode comparison

Currently, ceph-dencoder lacks certain mds types, preventing us from accurately checking the ceph corpus for encode-decode mismatches. This pull request aims to address this issue by adding the missing types to ceph-dencoder.

To successfully incorporate these types into ceph-dencoder, we need to introduce the necessary dump and generate_test_instances functions that was missing in some types. These functions are essential for proper encode and decode of the added types.

This PR will enhance the functionality of ceph-dencoder by including the missing types, enabling a comprehensive analysis of encode-decode consistency. With the addition of these types, we can ensure the robustness and correctness of the ceph corpus.

This update will significantly contribute to improving the overall reliability and accuracy of ceph-dencoder. It allows for a more comprehensive assessment of the encode-decode behavior, leading to enhanced data integrity and stability within the ceph ecosystem.

Fixes: https://tracker.ceph.com/issues/61788

Contribution Guidelines

To sign and title your commits, please refer to Submitting Patches to Ceph.
If you are submitting a fix for a stable branch (e.g. "pacific"), please refer to Submitting Patches to Ceph - Backports for the proper workflow.

Checklist

Tracker (select at least one)
- References tracker ticket
- Very recent bug; references commit where it was introduced
- New feature (ticket optional)
- Doc update (no ticket needed)
- Code cleanup (no ticket needed)
Component impact
- Affects Dashboard, opened tracker ticket
- Affects Orchestrator, opened tracker ticket
- No impact that needs to be tracked
Documentation (select at least one)
- Updates relevant documentation
- No doc update is appropriate
Tests (select at least one)
- Includes unit test(s)
- Includes integration test(s)
- Includes bug reproducer
- No tests

Show available Jenkins commands

jenkins retest this please
jenkins test classic perf
jenkins test crimson perf
jenkins test signed
jenkins test make check
jenkins test make check arm64
jenkins test submodules
jenkins test dashboard
jenkins test dashboard cephadm
jenkins test api
jenkins test docs
jenkins render docs
jenkins test ceph-volume all
jenkins test ceph-volume tox
jenkins test windows

vshankar · 2023-07-26T06:33:18Z

@NitzanMordhai Do you have test results (corpus runs as mentioned in the tracker) that now with this change doe snot throw out unsupported type warnings?

NitzanMordhai · 2023-07-27T06:09:48Z

@NitzanMordhai Do you have test results (corpus runs as mentioned in the tracker) that now with this change doe snot throw out unsupported type warnings?

@vshankar , I don't have such output since I didn't checked all PR changes for dencoder changes. I can run it separately for each PR, but you will need to find the mds types in that general list. Is it something you want to try?

vshankar · 2023-08-01T13:34:26Z

src/mds/Capability.cc

+  ls.back()->cap_id = 1;
+  ls.back()->issue_seq = 2;
+  ls.back()->mseq = 3;
+}


Capability::Import has a ctor that takes cap_id, issue_seq and mseq as arguments - why not use that?

Done, also "fixed" the Export to use the correct ctor

vshankar · 2023-08-01T13:38:56Z

src/messages/MMDSBeacon.h

+    f->close_section();
+  }
+
+  static void generate_test_instances(std::list<MDSHealthMetric*>& ls) {


Similar to my earlier comment about using the constructor.

vshankar · 2023-08-01T13:40:11Z

src/mgr/MDSPerfMetricTypes.h

  static constexpr bool featured = false;
  static constexpr bool need_contiguous = true;
-  static void bound_encode(const MDSPerfMetricKeyDescriptor& v, size_t& p) {
+  static void bound_encode(const MDSPerfMetricKeyDescriptor &v, size_t &p) {


unrelated changes?

yes, reverting

vshankar · 2023-08-01T13:40:58Z

src/mgr/MetricTypes.h

+  void dump(ceph::Formatter *f) const {
+    metric_report.dump(f);
+  }
+  static void generate_test_instances(std::list<MDSMetricPayload*>& ls) {


nit: MDSMetricPayload()

vshankar · 2023-08-01T13:41:08Z

src/mgr/MetricTypes.h

+    f->close_section();
+  }
+  static void generate_test_instances(std::list<OSDMetricPayload*>& ls) {
+    ls.push_back(new OSDMetricPayload);


nit: OSDMetricPayload()

github-actions · 2023-08-14T12:54:13Z

This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved

vshankar · 2023-08-14T16:50:36Z

Thanks @NitzanMordhai. Please rebase - I'll plan to run this through fs suite this week.

NitzanMordhai · 2023-08-16T05:03:48Z

Thanks @NitzanMordhai. Please rebase - I'll plan to run this through fs suite this week.

@vshankar done, but the make check will fail because the corpus PR (ceph/ceph-object-corpus#16) still under review.

vshankar · 2023-09-05T05:03:47Z

Thanks @NitzanMordhai. Please rebase - I'll plan to run this through fs suite this week.

@vshankar done, but the make check will fail because the corpus PR (ceph/ceph-object-corpus#16) still under review.

@NitzanMordhai This can be put to test now, yes?

vshankar · 2023-09-05T05:08:05Z

@NitzanMordhai could you also check the jenkins failures to see if a rebase is required?

NitzanMordhai · 2023-09-05T13:29:34Z

@NitzanMordhai could you also check the jenkins failures to see if a rebase is required?

Working on that, will ping you when its ready

vshankar · 2024-02-21T14:04:47Z

jenkins test make check

vshankar · 2024-02-26T06:52:08Z

Test runs in ~2h: https://shaman.ceph.com/builds/ceph/wip-vshankar-testing-20240226.064629/

vshankar · 2024-02-26T10:46:59Z

Test runs in ~2h: https://shaman.ceph.com/builds/ceph/wip-vshankar-testing-20240226.064629/

https://pulpito.ceph.com/vshankar-2024-02-26_10:07:12-fs-wip-vshankar-testing-20240226.064629-testing-default-smithi/

* refs/pull/52623/head: ceph-dencoder: MDS - Add missing types

github-actions · 2024-03-22T07:31:20Z

This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved

vshankar · 2024-04-10T11:51:24Z

@NitzanMordhai sorry, this has to be rebased (a largish PR got merged not so long ago).

Currently, ceph-dencoder lacks certain mds types, preventing us from accurately checking the ceph corpus for encode-decode mismatches. This pull request aims to address this issue by adding the missing types to ceph-dencoder. To successfully incorporate these types into ceph-dencoder, we need to introduce the necessary `dump` and `generate_test_instances` functions that was missing in some types. These functions are essential for proper encode and decode of the added types. This PR will enhance the functionality of ceph-dencoder by including the missing types, enabling a comprehensive analysis of encode-decode consistency. With the addition of these types, we can ensure the robustness and correctness of the ceph corpus. This update will significantly contribute to improving the overall reliability and accuracy of ceph-dencoder. It allows for a more comprehensive assessment of the encode-decode behavior, leading to enhanced data integrity and stability within the ceph ecosystem. Fixes: https://tracker.ceph.com/issues/61788 Signed-off-by: Nitzan Mordechai <nmordech@redhat.com>

NitzanMordhai · 2024-04-10T12:04:56Z

@NitzanMordhai sorry, this has to be rebased (a largish PR got merged not so long ago).

done

github-actions · 2024-06-09T13:01:50Z

This pull request has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs for another 30 days.
If you are a maintainer or core committer, please follow-up on this pull request to identify what steps should be taken by the author to move this proposed change forward.
If you are the author of this pull request, thank you for your proposed contribution. If you believe this change is still appropriate, please ensure that any feedback has been addressed and ask for a code review.

github-actions · 2024-07-09T14:01:21Z

This pull request has been automatically closed because there has been no activity for 90 days. Please feel free to reopen this pull request (or open a new one) if the proposed change is still appropriate. Thank you for your contribution!

rzarzynski · 2024-08-15T10:31:19Z

We talked about resurrecting this PR during the CDS RADOS session this Tuesday.
@vshankar: would you mind retaking a look?

vshankar · 2024-08-19T15:55:20Z

We talked about resurrecting this PR during the CDS RADOS session this Tuesday. @vshankar: would you mind retaking a look?

Of course. Thanks for the nudge @rzarzynski

vshankar · 2024-08-20T04:28:26Z

This PR is under test in https://tracker.ceph.com/issues/67626.

vshankar · 2024-08-28T13:49:59Z

https://tracker.ceph.com/projects/cephfs/wiki/Main#wip-vshankar-testing-20240820042758-debug

* refs/pull/52623/head: ceph-dencoder: MDS - Add missing types

vshankar · 2024-09-02T05:50:29Z

This change is good to merge. I'll have another look before merging since I has reviewed it a while back.

vshankar · 2024-09-17T11:22:06Z

jenkins test make check arm64

NitzanMordhai requested a review from rzarzynski July 25, 2023 11:06

NitzanMordhai requested a review from a team as a code owner July 25, 2023 11:06

github-actions bot added cephfs Ceph File System core mgr labels Jul 25, 2023

rzarzynski requested review from a team, gregsfortytwo and vshankar July 25, 2023 12:13

vshankar reviewed Aug 1, 2023

View reviewed changes

NitzanMordhai force-pushed the wip-nitzan-ceph-dencoder-mds-extend-type-available branch from 76374df to 0e09881 Compare August 3, 2023 05:43

NitzanMordhai requested a review from vshankar August 3, 2023 05:44

Matan-B mentioned this pull request Aug 3, 2023

ceph-dencoder: OSD - Add missing types #52482

Closed

14 tasks

NitzanMordhai changed the title ~~ceph-dencoder: Add missing mds types to ceph-dencoder for accurate en…~~ ceph-dencoder: MDS - Add missing types Aug 6, 2023

NitzanMordhai force-pushed the wip-nitzan-ceph-dencoder-mds-extend-type-available branch from 0e09881 to 91b860f Compare August 6, 2023 08:18

github-actions bot added the needs-rebase label Aug 14, 2023

NitzanMordhai force-pushed the wip-nitzan-ceph-dencoder-mds-extend-type-available branch from 91b860f to 5fd54c5 Compare August 16, 2023 05:02

github-actions bot removed the needs-rebase label Aug 16, 2023

NitzanMordhai force-pushed the wip-nitzan-ceph-dencoder-mds-extend-type-available branch from 5fd54c5 to 7564d23 Compare September 5, 2023 10:15

NitzanMordhai force-pushed the wip-nitzan-ceph-dencoder-mds-extend-type-available branch 4 times, most recently from 85198d4 to 6be3c76 Compare September 7, 2023 05:57

vshankar added a commit to vshankar/ceph that referenced this pull request Feb 29, 2024

Merge PR ceph#52623 into wip-vshankar-testing-20240226.064629

b78efeb

* refs/pull/52623/head: ceph-dencoder: MDS - Add missing types

github-actions bot added the needs-rebase label Mar 22, 2024

NitzanMordhai force-pushed the wip-nitzan-ceph-dencoder-mds-extend-type-available branch from f096253 to 09f3c87 Compare April 10, 2024 12:04

github-actions bot removed the needs-rebase label Apr 10, 2024

github-actions bot added the stale label Jun 9, 2024

github-actions bot closed this Jul 9, 2024

rzarzynski reopened this Aug 15, 2024

github-actions bot removed the stale label Aug 15, 2024

vshankar added wip-vshankar-testing2 and removed wip-vshankar-testing5 labels Aug 20, 2024

joscollin pushed a commit to joscollin/ceph that referenced this pull request Aug 29, 2024

Merge PR ceph#52623 into wip-vshankar-testing-20240820.042758-debug

6226ef4

* refs/pull/52623/head: ceph-dencoder: MDS - Add missing types

vshankar approved these changes Sep 17, 2024

View reviewed changes

vshankar merged commit 2d93ad5 into main Sep 30, 2024

vshankar deleted the wip-nitzan-ceph-dencoder-mds-extend-type-available branch September 30, 2024 06:58

vshankar removed the wip-vshankar-testing2 label Sep 30, 2024

Conversation

NitzanMordhai commented Jul 25, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Contribution Guidelines

Checklist

Uh oh!

vshankar commented Jul 26, 2023

Uh oh!

NitzanMordhai commented Jul 27, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Aug 14, 2023

Uh oh!

vshankar commented Aug 14, 2023

Uh oh!

NitzanMordhai commented Aug 16, 2023

Uh oh!

vshankar commented Sep 5, 2023

Uh oh!

vshankar commented Sep 5, 2023

Uh oh!

NitzanMordhai commented Sep 5, 2023

Uh oh!

vshankar commented Feb 21, 2024

Uh oh!

vshankar commented Feb 26, 2024

Uh oh!

vshankar commented Feb 26, 2024

Uh oh!

github-actions bot commented Mar 22, 2024

Uh oh!

vshankar commented Apr 10, 2024

Uh oh!

NitzanMordhai commented Apr 10, 2024

Uh oh!

github-actions bot commented Jun 9, 2024

Uh oh!

github-actions bot commented Jul 9, 2024

Uh oh!

rzarzynski commented Aug 15, 2024

Uh oh!

vshankar commented Aug 19, 2024

Uh oh!

vshankar commented Aug 20, 2024

Uh oh!

vshankar commented Aug 28, 2024

Uh oh!

vshankar commented Sep 2, 2024

Uh oh!

vshankar commented Sep 17, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

NitzanMordhai commented Jul 25, 2023 •

edited

Loading