python-common: only validate host_pattern if present by jschmid1 · Pull Request #34860 · ceph/ceph

jschmid1 · 2020-04-30T13:06:53Z

Signed-off-by: Joshua Schmid jschmid@suse.de

Fixes: https://tracker.ceph.com/issues/45203

Show available Jenkins commands

jenkins retest this please
jenkins test classic perf
jenkins test crimson perf
jenkins test signed
jenkins test make check
jenkins test make check arm64
jenkins test submodules
jenkins test dashboard
jenkins test dashboard backend
jenkins test docs
jenkins render docs
jenkins test ceph-volume all
jenkins test ceph-volume tox

smithfarm · 2020-04-30T13:26:12Z

Note: @sebastian-philipp appears to have changed the scope of the tracker https://tracker.ceph.com/issues/45203 ... I guess this PR just fixes the originally reported bug, but doesn't implement the feature?

src/python-common/ceph/deployment/drive_group.py

sebastian-philipp · 2020-05-08T11:31:31Z

ping?

jschmid1 · 2020-05-12T08:43:32Z

I noticed that we don't have a uniformget_hosts return type across our orchestrators.

While we have:

cephadm:

self.cache.get_hosts() -> List[str]
goes via HostCache and looks for daemons hostnames
self._get_hosts() -> List[str]
goes via inventory
self.get_hosts() -> List[orchestrator.HostSpec]
gets information from list(inventory.all_specs()) very odd..
several instances where we get hosts via one of the above funcs and shove the return in HostPlacementSpec

rook:

self.get_hosts -> List[orchestrator.HostSpec]
gets this from rook_cluster.get_node_names()

test_orchestrator:

self.get_hosts -> List[orchestrator.HostSpec]
via the _inventory object

This probably needs to be cleaned up some day. I'd argue that this is out of scope of this patchset.

sebastian-philipp · 2020-05-12T08:48:28Z

Plus HostPlacementSpec is odd:

https://github.com/ceph/ceph/blob/master/src/pybind/mgr/cephadm/module.py#L1181-L1186

to get the hosts matching a label, you need to call

self._get_hosts(spec.placement.label)

to get the host matching a patterm, you need to call

spec.placement.pattern_matches_hosts(self.inventory.keys())

Would be great to have a unified

HostPlacementSpec.get_matching_hosts(hosts: List[HostSpec])

src/python-common/ceph/deployment/service_spec.py

src/pybind/mgr/rook/module.py

src/pybind/mgr/test_orchestrator/module.py

sebastian-philipp · 2020-05-13T07:44:48Z

I noticed that we don't have a uniformget_hosts return type across our orchestrators.

While we have:

cephadm:

self.cache.get_hosts() -> List[str]
goes via HostCache and looks for daemons hostnames

self._get_hosts() -> List[str]
goes via inventory

self.get_hosts() -> List[orchestrator.HostSpec]

self.get_hosts() -> Completion

gets information from list(inventory.all_specs()) very odd..

several instances where we get hosts via one of the above funcs and shove the return in HostPlacementSpec

rook:

self.get_hosts -> List[orchestrator.HostSpec]

self.get_hosts() -> Completion

gets this from rook_cluster.get_node_names()

test_orchestrator:

self.get_hosts -> List[orchestrator.HostSpec]

self.get_hosts() -> Completion

via the _inventory object

This probably needs to be cleaned up some day. I'd argue that this is out of scope of this patchset.

sebastian-philipp · 2020-05-13T09:15:05Z

=================================== FAILURES ===================================
_________________________ test_DriveGroup[test_input0] _________________________

test_input = [{'data_devices': {'paths': ['/dev/sda']}, 'placement': {'host_pattern': 'hostname'}, 'service_id': 'testing_drivegroup', 'service_type': 'osd'}]

    @pytest.mark.parametrize("test_input",
    [
        (
            [  # new style json
                {
                    'service_type': 'osd',
                    'service_id': 'testing_drivegroup',
                    'placement': {'host_pattern': 'hostname'},
                    'data_devices': {'paths': ['/dev/sda']}
                }
            ]
        ),
    ])
    def test_DriveGroup(test_input):
        dg = [DriveGroupSpec.from_json(inp) for inp in test_input][0]
>       assert dg.placement.pattern_matches_hosts(['hostname']) == ['hostname']
E       AttributeError: 'PlacementSpec' object has no attribute 'pattern_matches_hosts'

ceph/tests/test_drive_group.py:27: AttributeError
___________________________ test_drivegroup_pattern ____________________________

    def test_drivegroup_pattern():
        dg = DriveGroupSpec(PlacementSpec(host_pattern='node[1-3]'), data_devices=DeviceSelection(all=True))
>       assert dg.placement.pattern_matches_hosts(['node{}'.format(i) for i in range(10)]) == ['node1', 'node2', 'node3']
E       AttributeError: 'PlacementSpec' object has no attribute 'pattern_matches_hosts'

ceph/tests/test_drive_group.py:56: AttributeError

src/pybind/mgr/rook/module.py

Signed-off-by: Joshua Schmid <jschmid@suse.de>

jschmid1 · 2020-05-28T09:19:26Z

about to schedule another teuthology run for test_orchestrator ..

looks fine now: http://pulpito.ceph.com/jschmid-2020-05-28_12:06:59-rados:cephadm-wip-jschmid1-testing-2020-05-28-1115-distro-basic-smithi/

Signed-off-by: Joshua Schmid <jschmid@suse.de>

sebastian-philipp · 2020-06-02T12:49:00Z

http://pulpito.ceph.com/swagner-2020-06-02_12:46:10-rados:cephadm-wip-swagner3-testing-2020-06-02-1145-distro-basic-smithi/

jschmid1 · 2020-06-02T14:08:56Z

http://pulpito.ceph.com/swagner-2020-06-02_12:46:10-rados:cephadm-wip-swagner3-testing-2020-06-02-1145-distro-basic-smithi/

looks good :thumbs:

ceph#34860 broke Rook Integration tests (Rook orchestrator module) This fix the error that can be seen in: https://jenkins.rook.io/blue/rest/organizations/jenkins/pipelines/rook/pipelines/rook/branches/master/runs/2046/nodes/63/steps/121/log/?start=0 Signed-off-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

ceph#34860 broke Rook Integration tests (Rook orchestrator module) This fix the error that can be seen in: https://jenkins.rook.io/blue/rest/organizations/jenkins/pipelines/rook/pipelines/rook/branches/master/runs/2046/nodes/63/steps/121/log/?start=0 Signed-off-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com> (cherry picked from commit 9fd4b48)

ceph#34860 broke Rook Integration tests (Rook orchestrator module) This fix the error that can be seen in: https://jenkins.rook.io/blue/rest/organizations/jenkins/pipelines/rook/pipelines/rook/branches/master/runs/2046/nodes/63/steps/121/log/?start=0 Signed-off-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

ceph/ceph#34860 broke Rook Integration tests (Rook orchestrator module) This fix the error that can be seen in: https://jenkins.rook.io/blue/rest/organizations/jenkins/pipelines/rook/pipelines/rook/branches/master/runs/2046/nodes/63/steps/121/log/?start=0 Signed-off-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

jschmid1 added the cephadm label Apr 30, 2020

jschmid1 requested review from sebastian-philipp and smithfarm April 30, 2020 13:06

jschmid1 requested a review from a team as a code owner April 30, 2020 13:06

smithfarm added the feature label Apr 30, 2020

smithfarm added bug-fix and removed feature labels Apr 30, 2020

smithfarm mentioned this pull request Apr 30, 2020

ceph_salt_deployment: deploy OSDs from YAML (ServiceSpec) file SUSE/sesdev#203

Merged

15 tasks

sebastian-philipp suggested changes Apr 30, 2020

View reviewed changes

src/python-common/ceph/deployment/drive_group.py Show resolved Hide resolved

jschmid1 force-pushed the host_spec_fixes branch from 4898e0a to 79c89ec Compare May 12, 2020 08:26

jschmid1 added the needs-rebase label May 12, 2020

jschmid1 force-pushed the host_spec_fixes branch from 79c89ec to 0ec9442 Compare May 12, 2020 09:03

jschmid1 added needs-qa needs-review and removed needs-rebase labels May 12, 2020

sebastian-philipp suggested changes May 12, 2020

View reviewed changes

src/python-common/ceph/deployment/service_spec.py Show resolved Hide resolved

src/pybind/mgr/rook/module.py Outdated Show resolved Hide resolved

src/pybind/mgr/test_orchestrator/module.py Outdated Show resolved Hide resolved

sebastian-philipp removed needs-qa needs-review labels May 13, 2020

sebastian-philipp mentioned this pull request May 13, 2020

mgr/cephadm: make scheduling make sense #34633

Merged

6 tasks

jschmid1 force-pushed the host_spec_fixes branch from 1842909 to 8996a70 Compare May 18, 2020 09:49

jschmid1 requested a review from sebastian-philipp May 18, 2020 14:24

sebastian-philipp added the needs-qa label May 19, 2020

sebastian-philipp suggested changes May 19, 2020

View reviewed changes

src/pybind/mgr/rook/module.py Show resolved Hide resolved

sebastian-philipp added the wip-swagner-testing My Teuthology tests label May 19, 2020

sebastian-philipp removed the wip-swagner-testing My Teuthology tests label May 20, 2020

tchaikov removed the needs-qa label May 22, 2020

jschmid1 added the wip-jschmid1-testing label May 26, 2020

sebastian-philipp added the wip-swagner3-testing label May 27, 2020

jschmid1 force-pushed the host_spec_fixes branch 2 times, most recently from 8bc754e to 40fb411 Compare May 27, 2020 09:49

Joshua Schmid added 4 commits May 28, 2020 10:10

python-common: only validate host_pattern if present

3b77181

Signed-off-by: Joshua Schmid <jschmid@suse.de>

mgr/cephadm: placementspec->host matching adaption

0223185

Signed-off-by: Joshua Schmid <jschmid@suse.de>

mgr/rook: placementspec->host matching adaption

d488de9

Signed-off-by: Joshua Schmid <jschmid@suse.de>

mgr/test_orchestrator: placementspec->host matching adaption

646ff0f

Signed-off-by: Joshua Schmid <jschmid@suse.de>

jschmid1 force-pushed the host_spec_fixes branch from 014db53 to 3b28491 Compare May 28, 2020 09:15

jschmid1 requested a review from sebastian-philipp May 28, 2020 13:24

sebastian-philipp mentioned this pull request May 28, 2020

mgr/cephadm: cephadm bootstrap add --apply-spec <cluster.yaml> #34879

Merged

python-common: improve host matching funcs

d2aa054

Signed-off-by: Joshua Schmid <jschmid@suse.de>

jschmid1 force-pushed the host_spec_fixes branch from 3b28491 to d2aa054 Compare May 28, 2020 14:04

sebastian-philipp approved these changes May 28, 2020

View reviewed changes

sebastian-philipp removed the wip-swagner3-testing label May 28, 2020

jschmid1 removed the wip-jschmid1-testing label May 29, 2020

sebastian-philipp added the wip-swagner3-testing label Jun 2, 2020

sebastian-philipp mentioned this pull request Jun 2, 2020

Improve Ceph services spec definition fmount/tripleo-ceph#15

Merged

sebastian-philipp merged commit 66052c9 into ceph:master Jun 3, 2020

sebastian-philipp mentioned this pull request Jun 8, 2020

octopus: cephadm batch backport June (2) #35475

Merged

jmolmo mentioned this pull request Jul 3, 2020

mgr/rook: Fix error creating OSDs #35915

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

python-common: only validate host_pattern if present#34860

python-common: only validate host_pattern if present#34860
sebastian-philipp merged 5 commits intoceph:masterfrom
jschmid1:host_spec_fixes

jschmid1 commented Apr 30, 2020

Uh oh!

smithfarm commented Apr 30, 2020

Uh oh!

Uh oh!

sebastian-philipp commented May 8, 2020

Uh oh!

jschmid1 commented May 12, 2020

Uh oh!

sebastian-philipp commented May 12, 2020

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sebastian-philipp commented May 13, 2020

Uh oh!

sebastian-philipp commented May 13, 2020

Uh oh!

Uh oh!

jschmid1 commented May 28, 2020 •

edited

Loading

Uh oh!

sebastian-philipp commented Jun 2, 2020

Uh oh!

jschmid1 commented Jun 2, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

jschmid1 commented Apr 30, 2020

Uh oh!

smithfarm commented Apr 30, 2020

Uh oh!

Uh oh!

sebastian-philipp commented May 8, 2020

Uh oh!

jschmid1 commented May 12, 2020

Uh oh!

sebastian-philipp commented May 12, 2020

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sebastian-philipp commented May 13, 2020

Uh oh!

sebastian-philipp commented May 13, 2020

Uh oh!

Uh oh!

jschmid1 commented May 28, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sebastian-philipp commented Jun 2, 2020

Uh oh!

jschmid1 commented Jun 2, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

jschmid1 commented May 28, 2020 •

edited

Loading