Skip to content

mgr/orch: increase readability for yaml representation#35537

Merged
sebastian-philipp merged 8 commits intoceph:masterfrom
sebastian-philipp:cephadm-yaml-ordered-readable
Jun 25, 2020
Merged

mgr/orch: increase readability for yaml representation#35537
sebastian-philipp merged 8 commits intoceph:masterfrom
sebastian-philipp:cephadm-yaml-ordered-readable

Conversation

@sebastian-philipp
Copy link
Contributor

@sebastian-philipp sebastian-philipp commented Jun 11, 2020

ceph orch ls --format=yaml before:

block_db_size: null
block_wal_size: null
data_devices:
  all: false
  limit: null
  model: MC-55-44-XZ
  paths: []
  rotational: null
  size: null
  vendor: null
data_directories: null
db_devices:
  all: false
  limit: null
  model: SSD-123-foo
  paths: []
  rotational: null
  size: null
  vendor: null
db_slots: null
status:
  container_image_id: 74803e884bea289d2d2d3ebdf6d37cd560499e955595695b1390a89800f4e37a
  container_image_name: docker.io/ceph/daemon-base:latest-master-devel
  created: '2020-06-10T10:37:31.051288'
  last_refresh: '2020-06-10T10:57:40.715637'
  running: 1
  size: 1
encrypted: false
journal_devices: null
journal_size: null
objectstore: bluestore
osd_id_claims: {}
osds_per_device: null
placement:
  host_pattern: '*'
service_id: osd_spec_default
service_name: osd.osd_spec_default
service_type: osd
unmanaged: false
wal_devices:
  all: false
  limit: null
  model: NVME-QQQQ-987
  paths: []
  rotational: null
  size: null
  vendor: null
wal_slots: null

after:

service_type: osd
service_id: osd_spec_default
service_name: osd.osd_spec_default
placement:
  host_pattern: '*'
spec:
  data_devices:
    model: MC-55-44-XZ
  db_devices:
    model: SSD-123-foo
  objectstore: bluestore
  wal_devices:
    model: NVME-QQQQ-987
status:
  container_image_id: 74803e884bea289d2d2d3ebdf6d37cd560499e955595695b1390a89800f4e37a
  container_image_name: docker.io/ceph/daemon-base:latest-master-devel
  created: '2020-06-10T10:37:31.051288'
  last_refresh: '2020-06-10T10:57:40.715637'
  running: 1
  size: 1

Fixes: https://tracker.ceph.com/issues/45321

Checklist

  • References tracker ticket
  • Updates documentation if necessary
  • Includes tests for new functionality or reproducer for bug

Show available Jenkins commands
  • jenkins retest this please
  • jenkins test classic perf
  • jenkins test crimson perf
  • jenkins test signed
  • jenkins test make check
  • jenkins test make check arm64
  • jenkins test submodules
  • jenkins test dashboard
  • jenkins test dashboard backend
  • jenkins test docs
  • jenkins render docs
  • jenkins test ceph-volume all
  • jenkins test ceph-volume tox

@sebastian-philipp sebastian-philipp requested a review from a team as a code owner June 11, 2020 10:31
@sebastian-philipp sebastian-philipp changed the title orch: increase readability for yaml representation mgr/orch: increase readability for yaml representation Jun 11, 2020
@sebastian-philipp
Copy link
Contributor Author

2020-06-12T00:48:22.313 INFO:tasks.cephfs_test_runner:Starting test: test_yaml (tasks.cephadm_cases.test_cli.TestCephadmCLI)
2020-06-12T00:48:22.313 INFO:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early log 'Starting test tasks.cephadm_cases.test_cli.TestCephadmCLI.test_yaml'
2020-06-12T00:48:22.535 INFO:ceph.mon.a.smithi190.stdout:Jun 12 00:48:22 smithi190 bash[15877]: audit 2020-06-12T00:48:21.401286+0000 mon.a (mon.0) 513 : audit [DBG] from='client.? 172.21.15.190:0/3873381430' entity='client.admin' cmd=[{"prefix": "mgr dump", "fo
rmat": "json-pretty"}]: dispatch
2020-06-12T00:48:23.228 INFO:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early log 'Ended test tasks.cephadm_cases.test_cli.TestCephadmCLI.test_yaml'
2020-06-12T00:48:23.485 INFO:ceph.mon.a.smithi190.stdout:Jun 12 00:48:23 smithi190 bash[15877]: cluster 2020-06-12T00:48:22.205243+0000 mgr.a (mgr.14377) 3 : cluster [DBG] pgmap v4: 97 pgs: 97 active+clean; 30 KiB data, 1.1 MiB used, 265 GiB / 268 GiB avail
2020-06-12T00:48:23.485 INFO:ceph.mon.a.smithi190.stdout:Jun 12 00:48:23 smithi190 bash[15877]: cluster 2020-06-12T00:48:22.643451+0000 client.admin (client.?) 0 : cluster [INF] Starting test tasks.cephadm_cases.test_cli.TestCephadmCLI.test_yaml
2020-06-12T00:48:23.485 INFO:ceph.mon.a.smithi190.stdout:Jun 12 00:48:23 smithi190 bash[15877]: audit 2020-06-12T00:48:22.643542+0000 mon.a (mon.0) 514 : audit [INF] from='client.? 172.21.15.190:0/2012565865' entity='client.admin' cmd=[{"prefix": "log", "logtext
": ["Starting test tasks.cephadm_cases.test_cli.TestCephadmCLI.test_yaml"]}]: dispatch
2020-06-12T00:48:24.235 INFO:tasks.cephfs_test_runner:test_yaml (tasks.cephadm_cases.test_cli.TestCephadmCLI) ... ERROR
2020-06-12T00:48:24.236 INFO:tasks.cephfs_test_runner:
2020-06-12T00:48:24.236 INFO:tasks.cephfs_test_runner:======================================================================
2020-06-12T00:48:24.237 INFO:tasks.cephfs_test_runner:ERROR: test_yaml (tasks.cephadm_cases.test_cli.TestCephadmCLI)
2020-06-12T00:48:24.237 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2020-06-12T00:48:24.237 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2020-06-12T00:48:24.237 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-swagner3-testing-2020-06-11-1428/qa/tasks/cephadm_cases/test_cli.py", line 26, in test_yaml
2020-06-12T00:48:24.238 INFO:tasks.cephfs_test_runner:    out = self._orch_cmd(['device', 'ls' '--format', 'yaml'])
2020-06-12T00:48:24.238 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-swagner3-testing-2020-06-11-1428/qa/tasks/cephadm_cases/test_cli.py", line 13, in _orch_cmd
2020-06-12T00:48:24.238 INFO:tasks.cephfs_test_runner:    return self._cmd("orch", *args)
2020-06-12T00:48:24.239 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-swagner3-testing-2020-06-11-1428/qa/tasks/cephadm_cases/test_cli.py", line 10, in _cmd
2020-06-12T00:48:24.239 INFO:tasks.cephfs_test_runner:    return self.mgr_cluster.mon_manager.raw_cluster_cmd(*args)
2020-06-12T00:48:24.239 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-swagner3-testing-2020-06-11-1428/qa/tasks/ceph_manager.py", line 1363, in raw_cluster_cmd
2020-06-12T00:48:24.239 INFO:tasks.cephfs_test_runner:    stdout=BytesIO(),
2020-06-12T00:48:24.240 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 206, in run
2020-06-12T00:48:24.240 INFO:tasks.cephfs_test_runner:    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
2020-06-12T00:48:24.240 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 469, in run
2020-06-12T00:48:24.240 INFO:tasks.cephfs_test_runner:    cwd=cwd)
2020-06-12T00:48:24.241 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 67, in __init__
2020-06-12T00:48:24.241 INFO:tasks.cephfs_test_runner:    self.command = quote(args)
2020-06-12T00:48:24.241 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 258, in quote
2020-06-12T00:48:24.242 INFO:tasks.cephfs_test_runner:    return ' '.join(_quote(args))
2020-06-12T00:48:24.242 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 256, in _quote
2020-06-12T00:48:24.242 INFO:tasks.cephfs_test_runner:    yield pipes.quote(a)
2020-06-12T00:48:24.242 INFO:tasks.cephfs_test_runner:  File "/usr/lib/python3.6/shlex.py", line 314, in quote
2020-06-12T00:48:24.243 INFO:tasks.cephfs_test_runner:    if _find_unsafe(s) is None:
2020-06-12T00:48:24.243 INFO:tasks.cephfs_test_runner:TypeError: expected string or bytes-like object
2020-06-12T00:48:24.243 INFO:tasks.cephfs_test_runner:
2020-06-12T00:48:24.243 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2020-06-12T00:48:24.244 INFO:tasks.cephfs_test_runner:Ran 4 tests in 98.487s
2020-06-12T00:48:24.244 INFO:tasks.cephfs_test_runner:
2020-06-12T00:48:24.244 INFO:tasks.cephfs_test_runner:FAILED (errors=1)

@sebastian-philipp
Copy link
Contributor Author

out['version'] = self.version
out['status'] = self.status
out['status_desc'] = self.status_desc

Copy link
Contributor

@kshtsk kshtsk Jun 15, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe this form will be syntactically sweeter for you:

out = OrderedDict([
            ('hostname', self.hostname),
            ('container_id', self.container_id),
            ('container_image_id', self.container_image_id),
            ('container_image_name', self.container_image_name),
            ('daemon_id', self.daemon_id),
            ('daemon_type', self.daemon_type),
            ('version', self.version),
            ('status', self.status),
            ('status_desc', self.status_desc),
])

Comment on lines +1398 to +1344
empty = [k for k, v in out.items() if v is None]
for e in empty:
del out[e]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's confusing, why not just use regular for?

for k,v in out.items():
  if not v:
     del out[k]

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, as it will modify dictionary size during iteration.

Copy link
Contributor

@kshtsk kshtsk Jun 15, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, as it will modify dictionary size during iteration.

It should not be a problem, however, if you unsure, you can use only keys() and construct a list from it:

for k in list(out.keys()):
  if not out[k]:
     del out[k]

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, that's what I'm doing here.

@sebastian-philipp
Copy link
Contributor Author

@sebastian-philipp sebastian-philipp force-pushed the cephadm-yaml-ordered-readable branch 2 times, most recently from ba6f132 to 61032bb Compare June 17, 2020 08:44
@sebastian-philipp
Copy link
Contributor Author

@sebastian-philipp
Copy link
Contributor Author

#35644

@mgfritch
Copy link
Contributor

jenkins test make check

@sebastian-philipp
Copy link
Contributor Author

Minor nits. Otherwise looks good.

mind if I merge this as it is? otherwise I'll have to do another PR validation run for it.

@varshar16
Copy link
Contributor

Minor nits. Otherwise looks good.

mind if I merge this as it is? otherwise I'll have to do another PR validation run for it.

After you clarify on this change here:
#35537 (comment)

@sebastian-philipp sebastian-philipp force-pushed the cephadm-yaml-ordered-readable branch from 61032bb to 6515c3e Compare June 25, 2020 10:18
* Add test for new yaml representation

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
* Changes: An empty OSD Spec is now invalid.
* OSDSpec.validate() now fails, if service-id is empty

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
# Please do not modify those JSON values.

spec = ServiceSpec.from_json(spec_json)
# just some verification that we can sill read old octopus specs
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found a typo

Suggested change
# just some verification that we can sill read old octopus specs
# just some verification that we can still read old octopus specs

@sebastian-philipp sebastian-philipp added the wip-swagner-testing My Teuthology tests label Jun 25, 2020
@sebastian-philipp
Copy link
Contributor Author

@sebastian-philipp
Copy link
Contributor Author

jenkins test dashboard backend

@smithfarm
Copy link
Contributor

smithfarm commented Aug 14, 2020

this PR introduced a regression which was fixed by #36363

is that fix going to backported to octopus, too?

UPDATE: it was backported via #36558

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants