qa: wait for daemons to come up via cephadm by batrick · Pull Request #39969 · ceph/ceph

batrick · 2021-03-09T21:55:44Z

Rather than waiting for a set amount of time.

Fixes: https://tracker.ceph.com/issues/49684
Signed-off-by: Patrick Donnelly pdonnell@redhat.com

Checklist

References tracker ticket
Updates documentation if necessary
Includes tests for new functionality or reproducer for bug

Show available Jenkins commands

jenkins retest this please
jenkins test classic perf
jenkins test crimson perf
jenkins test signed
jenkins test make check
jenkins test make check arm64
jenkins test submodules
jenkins test dashboard
jenkins test api
jenkins test docs
jenkins render docs
jenkins test ceph-volume all
jenkins test ceph-volume tox

qa/tasks/ceph_manager.py

batrick · 2021-03-11T15:00:52Z

@tchaikov yes sorry. I'm in the middle of fixing this PR after finding out qa: wait for daemons to come up via cephadm was not the only change necessary. I'll switch this to a draft.

batrick · 2021-03-12T04:18:12Z

https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20210312.020610

* refs/pull/39969/head: qa: wait for daemons to come up via cephadm qa: reduce dependence on teuthology role list for mds qa: improve usability of do_rados helper qa: remove get_lone_mds_id qa: simplify tests which stop MDS ranks qa: use mds_tell for getting mds status qa: remove shutdown lockdep not relevant to unit test qa: remove fallback for non-multifs ceph version qa: add Filesystem.reset helper

batrick · 2021-03-17T03:20:58Z

https://pulpito.ceph.com/pdonnell-2021-03-15_22:16:56-fs-wip-pdonnell-testing-20210315.182203-distro-basic-smithi/

and failure caused by PR: #39969

* refs/pull/39969/head: qa: wait for daemons to come up via cephadm qa: reduce dependence on teuthology role list for mds qa: improve usability of do_rados helper qa: remove get_lone_mds_id qa: simplify tests which stop MDS ranks qa: use mds_tell for getting mds status qa: remove shutdown lockdep not relevant to unit test qa: remove fallback for non-multifs ceph version qa: add Filesystem.reset helper

qa/tasks/ceph_manager.py

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

This code was for jewel (IIRC) or older. Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

This is part of a fleet of commits trying to remove unnecessary mds_stop calls in testing. Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds_asok is broken with cephadm currently due to [1]. [1] https://tracker.ceph.com/issues/48019#note-5 Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Instead of stopping MDS daemons and individually failing MDS daemons, just fail the ranks or the entire file system, where possible. Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

This helps remove a dependence on the list of mds_daemons from teuthology/cephadm. Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

It's not yet possible to completely remove the dependency on mds_ids/mds_daemons in the CephFS tests but this commit reduces it enough for most code paths to work with cephadm. The main change here is use of CephManager.do_rados, with some improvements. Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Rather than waiting for a set amount of time. Fixes: https://tracker.ceph.com/issues/49684 Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

batrick · 2021-03-26T22:10:43Z

https://pulpito.ceph.com/pdonnell-2021-03-24_23:26:35-fs-wip-pdonnell-testing-20210324.190252-distro-basic-smithi/

https://tracker.ceph.com/issues/49500
qa: "Assertion `cb_done' failed."
https://tracker.ceph.com/issues/50019
qa: mount failure with cephadm "probably no MDS server is up?"
https://tracker.ceph.com/issues/50020
qa: "RADOS object not found (Failed to operate read op for oid cephfs_mirror)"
https://tracker.ceph.com/issues/48773
qa: scrub does not complete
https://tracker.ceph.com/issues/45434
qa: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) failed
https://tracker.ceph.com/issues/48805
mds: "cluster [WRN] Scrub error on inode 0x1000000039d (/client.0/tmp/blogbench-1.0/src/blogtest_in) see mds.a log and damage ls output for details"
https://tracker.ceph.com/issues/48772
qa: pjd: not ok 9, 44, 80
https://tracker.ceph.com/issues/50021
qa: snaptest-git-ceph failure during mon thrashing
https://tracker.ceph.com/issues/48771
qa: iogen: workload fails to cause balancing
https://tracker.ceph.com/issues/50016
qa: test_damage: "RuntimeError: 2 mutations had unexpected outcomes"
https://tracker.ceph.com/issues/49466
qa: "Command failed on gibba030 with status 1: 'set -ex\nsudo dd of=/tmp/tmp.ZEeZBasJer'"

batrick · 2021-03-26T22:12:24Z

@tchaikov CephFS is good. Please merge when you're ready.

tchaikov · 2021-03-27T05:13:53Z

@batrick ack. will include this changeset in my next batch.

tchaikov · 2021-03-28T12:01:00Z

failures

dismissed

batrick added cephfs Ceph File System needs-review labels Mar 9, 2021

batrick requested a review from liewegas March 9, 2021 21:55

github-actions bot added the tests label Mar 9, 2021

liewegas approved these changes Mar 10, 2021

View reviewed changes

batrick force-pushed the i49684 branch from 8cfcec7 to af2552f Compare March 10, 2021 22:47

tchaikov added the wip-kefu-testing label Mar 11, 2021

tchaikov requested changes Mar 11, 2021

View reviewed changes

qa/tasks/ceph_manager.py Outdated Show resolved Hide resolved

tchaikov removed the wip-kefu-testing label Mar 11, 2021

batrick marked this pull request as draft March 11, 2021 15:00

batrick removed the needs-review label Mar 11, 2021

batrick force-pushed the i49684 branch 5 times, most recently from bbec76b to fe7b23c Compare March 11, 2021 21:28

batrick added the wip-pdonnell-testing label Mar 11, 2021

batrick force-pushed the i49684 branch 2 times, most recently from 0de84d8 to cb91555 Compare March 12, 2021 02:05

batrick force-pushed the i49684 branch 4 times, most recently from d820667 to b4e8232 Compare March 12, 2021 22:18

batrick force-pushed the i49684 branch from b4e8232 to b3ce2f3 Compare March 15, 2021 17:40

batrick marked this pull request as ready for review March 15, 2021 18:16

batrick added core needs-qa labels Mar 15, 2021

batrick mentioned this pull request Mar 17, 2021

qa: don't expect "file exists" error for "mkdir -p" #39847

Merged

batrick force-pushed the i49684 branch 2 times, most recently from 86a23fc to dd09ee4 Compare March 18, 2021 23:21

batrick force-pushed the i49684 branch from dd09ee4 to 1446b77 Compare March 19, 2021 01:47

tchaikov approved these changes Mar 19, 2021

View reviewed changes

tchaikov added the wip-kefu2-testing label Mar 19, 2021

tchaikov previously requested changes Mar 21, 2021

View reviewed changes

qa/tasks/ceph_manager.py Show resolved Hide resolved

tchaikov removed needs-qa wip-kefu2-testing labels Mar 21, 2021

batrick added 9 commits March 21, 2021 10:35

qa: add Filesystem.reset helper

eccf961

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: remove fallback for non-multifs ceph version

4c3c69b

This code was for jewel (IIRC) or older. Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: remove shutdown lockdep not relevant to unit test

fe85001

This is part of a fleet of commits trying to remove unnecessary mds_stop calls in testing. Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: use mds_tell for getting mds status

6ca48f6

mds_asok is broken with cephadm currently due to [1]. [1] https://tracker.ceph.com/issues/48019#note-5 Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: simplify tests which stop MDS ranks

0825d6a

Instead of stopping MDS daemons and individually failing MDS daemons, just fail the ranks or the entire file system, where possible. Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: remove get_lone_mds_id

8e45af0

This helps remove a dependence on the list of mds_daemons from teuthology/cephadm. Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: improve usability of do_rados helper

24bb1aa

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: wait for daemons to come up via cephadm

d06689e

Rather than waiting for a set amount of time. Fixes: https://tracker.ceph.com/issues/49684 Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

batrick force-pushed the i49684 branch from 1446b77 to d06689e Compare March 21, 2021 17:35

tchaikov added the needs-qa label Mar 21, 2021

tchaikov added the wip-kefu-testing label Mar 27, 2021

tchaikov merged commit 73925c4 into ceph:master Mar 28, 2021

batrick deleted the i49684 branch April 6, 2021 17:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

qa: wait for daemons to come up via cephadm#39969

qa: wait for daemons to come up via cephadm#39969
tchaikov merged 9 commits intoceph:masterfrom
batrick:i49684

batrick commented Mar 9, 2021

Uh oh!

Uh oh!

batrick commented Mar 11, 2021

Uh oh!

batrick commented Mar 12, 2021

Uh oh!

batrick commented Mar 17, 2021

Uh oh!

Uh oh!

batrick commented Mar 26, 2021

Uh oh!

batrick commented Mar 26, 2021

Uh oh!

tchaikov commented Mar 27, 2021

Uh oh!

tchaikov commented Mar 28, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

batrick commented Mar 9, 2021

Checklist

Uh oh!

Uh oh!

batrick commented Mar 11, 2021

Uh oh!

batrick commented Mar 12, 2021

Uh oh!

batrick commented Mar 17, 2021

Uh oh!

Uh oh!

batrick commented Mar 26, 2021

Uh oh!

batrick commented Mar 26, 2021

Uh oh!

tchaikov commented Mar 27, 2021

Uh oh!

tchaikov commented Mar 28, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants