doc: Add a generated reference of all mon commands. by sebastian-philipp · Pull Request #33886 · ceph/ceph

sebastian-philipp · 2020-03-11T13:59:13Z

We gather the MGR commands by importing all modules and then concat all Module.COMMAND lists.

Then we use the C preprocessor to parse the MonCommands.h and then pretend the output to be Python. Which actually works.

Then we generate a ReST document which contains all mon commands.

Fixes: sebastian-philipp/ceph-command-api#2

Signed-off-by: Sebastian Wagner sebastian.wagner@suse.com

Depends on

Checklist

References tracker ticket
Updates documentation if necessary
Includes tests for new functionality or reproducer for bug

Show available Jenkins commands

jenkins retest this please
jenkins test crimson perf
jenkins test signed
jenkins test make check
jenkins test make check arm64
jenkins test submodules
jenkins test dashboard
jenkins test dashboard backend
jenkins test docs
jenkins render docs
jenkins test ceph-volume all
jenkins test ceph-volume tox

sebastian-philipp · 2020-03-11T14:07:15Z

jenkins render docs

ceph-jenkins · 2020-03-11T14:27:10Z

Doc render available at http://docs.ceph.com/ceph-prs/33886/

sebastian-philipp · 2020-03-11T14:31:49Z

https://docs.ceph.com/ceph-prs/33886/api/mon_command_api/

sebastian-philipp · 2020-03-12T10:10:44Z

jenkins retest this please

sebastian-philipp · 2020-03-12T15:42:37Z

ImportError while loading conftest '/home/jenkins-build/build/workspace/ceph-pull-requests/src/pybind/mgr/dashboard/conftest.py'.
__init__.py:39: in <module>
    from .module import Module, StandbyModule  # noqa: F401
module.py:55: in <module>
    from .plugins import feature_toggles, debug  # noqa # pylint: disable=unused-import
plugins/feature_toggles.py:16: in <module>
    from ..controllers.cephfs import CephFS
controllers/cephfs.py:9: in <module>
    import cephfs
E   ModuleNotFoundError: No module named 'cephfs'

liewegas · 2020-03-16T14:52:20Z

This is great!

can we show args positionally instead of with --foo=bar syntax? i'm not even sure if the latte rworks for the n=N args
hide the ones with HIDDEN flag?

sebastian-philipp · 2020-03-16T18:20:08Z

This is great!

can we show args positionally instead of with --foo=bar syntax? i'm not even sure if the latte rworks for the n=N args

osd pool create doesn't work with positional arguments:

ceph osd pool create --pool=poolname --pg_num=1 --pgp_num=1 --pool_type=choice --erasure_code_profile=string --rule=string --expected_num_objects=1 --size=1 --pg_num_min=1 --autoscale_mode=choice --target_size_bytes=1 --target_size_ratio=0.0

because of:

ceph osd pool create poolname 1 1 choice string string 11 1 choice 1 0.0

hide the ones with HIDDEN flag?

sebastian-philipp · 2020-03-17T12:50:12Z

I'd like to depend on #32319 here, as I need @jan--f 's tox.ini enhancements.

sebastian-philipp · 2020-04-02T13:56:57Z

This is great!

can we show args positionally instead of with --foo=bar syntax? i'm not even sure if the latte rworks for the n=N args

Fixed for the simple commands

hide the ones with HIDDEN flag?

Fixed

sebastian-philipp · 2020-04-03T09:33:27Z

jenkins render docs

sebastian-philipp · 2020-04-03T09:33:49Z

jenkins test make check

ceph-jenkins · 2020-04-03T09:50:40Z

Doc render available at http://docs.ceph.com/ceph-prs/33886/

sebastian-philipp · 2020-04-08T11:56:08Z

jenkins test make check

sebastian-philipp · 2020-04-08T11:56:21Z

anyone wants to review this?

tchaikov · 2020-04-08T12:27:06Z

@sebastian-philipp i hate to say "sorry". but sorry for the latency. it's on my list.

sebastian-philipp · 2020-04-08T12:37:24Z

ImportError while loading conftest '/home/jenkins-build/build/workspace/ceph-pull-requests/src/pybind/mgr/dashboard/conftest.py'.
__init__.py:39: in <module>
    from .module import Module, StandbyModule  # noqa: F401
module.py:15: in <module>
    from mgr_module import MgrModule, MgrStandbyModule, Option, CLIWriteCommand
../mgr_module.py:14: in <module>
    import rados
E   ModuleNotFoundError: No module named 'rados'

LenzGr · 2020-04-14T11:05:59Z

@tchaikov with our move of the documentation to readthedocs.io, is this still feasible? Or would this be part of the locally hosted API documentation then?

tchaikov · 2020-04-14T13:05:29Z

@LenzGr yeah, probably the latter. as it'd be difficult to sphinxize following command

$vdir/bin/python $TOPDIR/doc/scripts/gen_mon_command_api.py > $TOPDIR/doc/api/mon_command_api.rst

if we cannot think of a way to do so, we will have to keep this part of document in docs.ceph.com. we could link the RTD document to it though.

zdover23

LGTM. proceed.

sebastian-philipp · 2020-04-24T13:56:06Z

make check:

==================================== ERRORS ====================================
____________________ ERROR collecting tests/test_health.py _____________________
ImportError while importing test module '/home/jenkins-build/build/workspace/ceph-pull-requests/src/pybind/mgr/insights/tests/test_health.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
__init__.py:7: in <module>
    from .module import Module
module.py:6: in <module>
    from mgr_module import MgrModule, CommandResult
../mgr_module.py:1: in <module>
    import ceph_module  # noqa
E   ModuleNotFoundError: No module named 'ceph_module'

This is fixed in #34738

LenzGr · 2020-04-30T09:59:02Z

Nice work. I wonder if it would be possible to add at least one hierarchy level here, to improve the document structure by putting e.g. all dashboard commands under a headline?

epuertat

Looks awesome! Kudos to you! Left a few suggestions.

Other question I had is about changes to stand-alone dashboard unit tests (don't they require anything else to be built or tested, just tox -e py3, right?).

admin/doc-requirements.txt

epuertat · 2020-04-30T11:37:50Z

doc/scripts/gen_mon_command_api.py

+    t = {
+        'CephInt': 'int',
+        'CephString': 'str',
+        'CephChoices': 'str',
+        'CephPgid': 'str',
+        'CephOsdName': 'str',
+        'CephPoolname': 'str',
+        'CephObjectname': 'str',
+        'CephUUID': 'str',
+        'CephEntityAddr': 'str',
+        'CephIPAddr': 'str',
+        'CephName': 'str',
+        'CephBool': 'bool',
+        'CephFloat': 'float',
+        'CephFilepath': 'str',
+    }


For #34840 I've been thinking how to get this. As all Ceph Types are subclasses of CephArgtype we could simply exploit this and add an @abstractproperty type to the CephArgtype:

class CephInt(CephArgtype): py_type = int ... CEPH_TYPE_TO_PY_TYPE = {CephType.__class__ : CephType.py_type for CephType in CephArgType.__subclassess__}

ok, for now, I'd like not to depend on ceph_argparse.py in the docs. the location is a bit awkward and it has no setup.py, which makes it impossible to add ceph_argparse in the requirements.txt.

I really want to move it to python-common first

epuertat · 2020-04-30T11:38:50Z

doc/scripts/gen_mon_command_api.py

+    bash_example = {
+        'CephInt': '1',
+        'CephString': 'string',
+        'CephChoices': 'choice',
+        'CephPgid': '0',
+        'CephOsdName': 'osd.0',
+        'CephPoolname': 'poolname',
+        'CephObjectname': 'objectname',
+        'CephUUID': 'uuid',
+        'CephEntityAddr': 'entityaddr',
+        'CephIPAddr': '0.0.0.0',
+        'CephName': 'name',
+        'CephBool': 'true',
+        'CephFloat': '0.0',
+        'CephFilepath': '/path/to/file',
+    }


And we can also add this metadata to there.

doc/scripts/gen_mon_command_api.py

src/pybind/mgr/dashboard/conftest.py

epuertat · 2020-04-30T12:10:21Z

Nice work. I wonder if it would be possible to add at least one hierarchy level here, to improve the document structure by putting e.g. all dashboard commands under a headline?

Yeah, I thought the same: what about nesting by 1st & 2nd tokens?

sebastian-philipp · 2020-04-30T12:13:47Z

Nice work. I wonder if it would be possible to add at least one hierarchy level here, to improve the document structure by putting e.g. all dashboard commands under a headline?

Yeah, I thought the same: what about nesting by 1st & 2nd tokens?

Not sure about it, This would generate a lot of single-entry pages. Still I can see the point for grouping the dashboard and orchestrator, but for all others as well?

We gather the MGR commands by importing all modules and then concat all `Module.COMMAND` lists. Then we use the C preprocessor to parse the MonCommands.h and then pretent to output to be Python. Which actually works. Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Generate a ReST document which contains all mon commands. Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

sebastian-philipp · 2020-05-13T09:18:56Z

http://pulpito.ceph.com/swagner-2020-05-13_07:42:35-rados-wip-swagner-testing-2020-05-12-1623-distro-basic-smithi/

sebastian-philipp added the documentation label Mar 11, 2020

sebastian-philipp requested review from epuertat and neha-ojha March 11, 2020 13:59

sebastian-philipp requested a review from a team as a code owner March 11, 2020 13:59

sebastian-philipp force-pushed the doc-mon-command-api branch 2 times, most recently from 5d2f005 to 981616b Compare March 11, 2020 14:04

sebastian-philipp requested a review from tchaikov March 12, 2020 09:16

sebastian-philipp force-pushed the doc-mon-command-api branch from bbad7cb to 319fb33 Compare April 2, 2020 13:56

sebastian-philipp requested a review from zdover23 April 2, 2020 13:58

sebastian-philipp added the dashboard label Apr 2, 2020

sebastian-philipp force-pushed the doc-mon-command-api branch from 319fb33 to acd4919 Compare April 3, 2020 10:33

sebastian-philipp added the needs-review label Apr 14, 2020

zdover23 approved these changes Apr 22, 2020

View reviewed changes

sebastian-philipp force-pushed the doc-mon-command-api branch from acd4919 to 05ed550 Compare April 24, 2020 09:28

sebastian-philipp mentioned this pull request Apr 24, 2020

pybind/mgr: Remove insights/tox.ini #34738

Merged

3 tasks

epuertat approved these changes Apr 30, 2020

View reviewed changes

sebastian-philipp removed the needs-review label May 4, 2020

sebastian-philipp force-pushed the doc-mon-command-api branch from 05ed550 to 0b69821 Compare May 5, 2020 10:40

sebastian-philipp added 3 commits May 5, 2020 12:51

doc/api: Add a generated reference of all mon commands.

4733080

Generate a ReST document which contains all mon commands. Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashboard: Add dashboard to mon_command reference docs

6b029c0

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

pybind/mgr: Add mgr/progress to mon_command reference

7378a0c

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

sebastian-philipp force-pushed the doc-mon-command-api branch from 0b69821 to 7378a0c Compare May 5, 2020 10:52

sebastian-philipp added needs-qa mgr wip-swagner-testing My Teuthology tests labels May 5, 2020

sebastian-philipp merged commit 79d88ad into ceph:master May 13, 2020

tchaikov mentioned this pull request Jan 6, 2021

mgr: use annotations for argdesc #38722

Merged

3 tasks

Conversation

sebastian-philipp commented Mar 11, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Depends on

Checklist

Uh oh!

sebastian-philipp commented Mar 11, 2020

Uh oh!

ceph-jenkins commented Mar 11, 2020

Uh oh!

sebastian-philipp commented Mar 11, 2020

Uh oh!

sebastian-philipp commented Mar 12, 2020

Uh oh!

sebastian-philipp commented Mar 12, 2020

Uh oh!

liewegas commented Mar 16, 2020

Uh oh!

sebastian-philipp commented Mar 16, 2020

Uh oh!

sebastian-philipp commented Mar 17, 2020

Uh oh!

sebastian-philipp commented Apr 2, 2020

Uh oh!

sebastian-philipp commented Apr 3, 2020

Uh oh!

sebastian-philipp commented Apr 3, 2020

Uh oh!

ceph-jenkins commented Apr 3, 2020

Uh oh!

sebastian-philipp commented Apr 8, 2020

Uh oh!

sebastian-philipp commented Apr 8, 2020

Uh oh!

tchaikov commented Apr 8, 2020

Uh oh!

sebastian-philipp commented Apr 8, 2020

Uh oh!

LenzGr commented Apr 14, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tchaikov commented Apr 14, 2020

Uh oh!

zdover23 left a comment

Choose a reason for hiding this comment

Uh oh!

sebastian-philipp commented Apr 24, 2020

Uh oh!

LenzGr commented Apr 30, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

epuertat left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

epuertat Apr 30, 2020

Choose a reason for hiding this comment

Uh oh!

sebastian-philipp May 4, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

epuertat Apr 30, 2020

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

epuertat commented Apr 30, 2020

Uh oh!

sebastian-philipp commented Apr 30, 2020

Uh oh!

sebastian-philipp commented May 13, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

sebastian-philipp commented Mar 11, 2020 •

edited

Loading

LenzGr commented Apr 14, 2020 •

edited

Loading

LenzGr commented Apr 30, 2020 •

edited

Loading

sebastian-philipp May 4, 2020 •

edited

Loading