qa: do not create rbd pool for CephFS QA by batrick · Pull Request #53037 · ceph/ceph

batrick · 2023-08-17T13:28:32Z

Fixes: https://tracker.ceph.com/issues/62482

Contribution Guidelines

To sign and title your commits, please refer to Submitting Patches to Ceph.
If you are submitting a fix for a stable branch (e.g. "pacific"), please refer to Submitting Patches to Ceph - Backports for the proper workflow.

Checklist

Tracker (select at least one)
- References tracker ticket
- Very recent bug; references commit where it was introduced
- New feature (ticket optional)
- Doc update (no ticket needed)
- Code cleanup (no ticket needed)
Component impact
- Affects Dashboard, opened tracker ticket
- Affects Orchestrator, opened tracker ticket
- No impact that needs to be tracked
Documentation (select at least one)
- Updates relevant documentation
- No doc update is appropriate
Tests (select at least one)
- Includes unit test(s)
- Includes integration test(s)
- Includes bug reproducer
- No tests

Show available Jenkins commands

jenkins retest this please
jenkins test classic perf
jenkins test crimson perf
jenkins test signed
jenkins test make check
jenkins test make check arm64
jenkins test submodules
jenkins test dashboard
jenkins test dashboard cephadm
jenkins test api
jenkins test docs
jenkins render docs
jenkins test ceph-volume all
jenkins test ceph-volume tox
jenkins test windows

batrick · 2023-08-17T13:33:59Z

https://pulpito.ceph.com/pdonnell-2023-08-17_18:28:28-fs:functional-main-distro-default-smithi/

batrick · 2023-08-17T18:29:01Z

jenkins test make check

batrick · 2023-08-17T18:29:06Z

jenkins test make check arm64

batrick · 2023-08-18T20:18:14Z

jenkins test make check

batrick · 2023-08-18T20:18:19Z

jenkins test make check arm64

dparmar18 · 2023-08-23T18:38:05Z

we had a similar PR that ignored this warning #53077 but i think NOT creating the rbd pool is much better solution

mchangir

This is good.
Although the config should've required explicit request of type of pools to be created rather than implicitly creating them and then requiring to explicitly disable their creation. This is so twisted.

vshankar · 2023-08-26T02:41:54Z

@batrick We could probably revert changes from #53077? Could you include that in this change?

This reverts commit b8bf0c6, reversing changes made to fe07f64. Silencing this health warning is unnecessary if we stop creating the rbd pool in CephFS testing. Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Fixes: https://tracker.ceph.com/issues/62482 Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

batrick · 2023-08-28T13:17:09Z

@vshankar done

batrick · 2023-08-29T00:54:27Z

jenkins test windows

* refs/pull/53037/head: qa: do not create rbd pool for CephFS QA Revert "Merge PR ceph#53077 into main"

vshankar · 2023-09-04T14:16:29Z

Tested with my mirroring changes: https://pulpito.ceph.com/?branch=wip-62072-3

idryomov · 2023-09-05T17:01:32Z

qa/cephfs/overrides/ignorelist_health.yaml

      - \(MDS_ALL_DOWN\)
      - \(MDS_UP_LESS_THAN_MAX\)
      - \(FS_INLINE_DATA_DEPRECATED\)
-      - \(POOL_APP_NOT_ENABLED\)


While I obviously agree with not creating an RBD pool for CephFS tests, I'm surprised to learn that @vshankar's run passed with POOL_APP_NOT_ENABLED dropped from the ignorelist. The issue is not (specific to) the RBD pool, rather it's the raising of a bogus health alert. See the recent thread on the dev list:

https://lists.ceph.io/hyperkitty/list/dev@ceph.io/thread/ZTDYC5HN677RR26EB4P6PORN6L2IFH4R/

The RBD pool in this case is just the messenger -- it's tagged with the RBD application immediately:

ceph/qa/tasks/ceph.py

Lines 429 to 440 in 636d2a4

if config.get('create_rbd_pool', True):

log.info('Creating RBD pool')

mon_remote.run(

args=['sudo', 'ceph', '--cluster', cluster_name,

'osd', 'pool', 'create', 'rbd', '8'])

mon_remote.run(

args=[

'sudo', 'ceph', '--cluster', cluster_name,

'osd', 'pool', 'application', 'enable',

'rbd', 'rbd', '--yes-i-really-mean-it'

],

check_status=False)

In the example job linked by @batrick in https://tracker.ceph.com/issues/62482, POOL_APP_NOT_ENABLED is raised for CephFS pools a few seconds after getting raised on the RBD pool:

2023-08-16T11:24:28.204+0000 7fc702096640 20 mgr.server operator() health checks: { "POOL_APP_NOT_ENABLED": { "severity": "HEALTH_WARN", "summary": { "message": "1 pool(s) do not have an application enabled", "count": 1 }, "detail": [ { "message": "application not enabled on pool 'rbd'" }, { "message": "use 'ceph osd pool application enable <pool-name> <app-name>', where <app-name> is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." } ] } }

2023-08-16T11:24:34.204+0000 7fc702096640 20 mgr.server operator() health checks: { "POOL_APP_NOT_ENABLED": { "severity": "HEALTH_WARN", "summary": { "message": "2 pool(s) do not have an application enabled", "count": 2 }, "detail": [ { "message": "application not enabled on pool 'cephfs_metadata'" }, { "message": "application not enabled on pool 'cephfs_data'" }, { "message": "use 'ceph osd pool application enable <pool-name> <app-name>', where <app-name> is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." } ] } }

I think there was an underlying RADOS change with a fix in-flight for that which maybe was merged when Venky do QA for this. Venky, do you have that link handy still?

and ya, this warning came back: https://pulpito.ceph.com/pdonnell-2023-09-05_22:23:53-fs:functional-main-distro-default-smithi/7389194/

I think there was an underlying RADOS change with a fix in-flight for that which maybe was merged when Venky do QA for this. Venky, do you have that link handy still?

Unfortunately, no. I had run a pretty minimal set of test for this and that's probably the reason no failures related to the warning showed up -- just got unlucky I guess?

I had run a pretty minimal set of test for this and that's probably the reason no failures related to the warning showed up -- just got unlucky I guess?

Yeah, this health alert became not only bogus but also racy...

batrick added cephfs Ceph File System needs-review labels Aug 17, 2023

batrick mentioned this pull request Aug 17, 2023

mds: drop locks and retry when lock set changes #52522

Merged

14 tasks

batrick requested a review from a team August 23, 2023 18:05

batrick mentioned this pull request Aug 23, 2023

qa: add POOL_APP_NOT_ENABLED to ignorelist for cephfs tests #53077

Merged

14 tasks

mchangir approved these changes Aug 25, 2023

View reviewed changes

vshankar approved these changes Aug 26, 2023

View reviewed changes

batrick added 2 commits August 28, 2023 09:16

Revert "Merge PR ceph#53077 into main"

af4558f

This reverts commit b8bf0c6, reversing changes made to fe07f64. Silencing this health warning is unnecessary if we stop creating the rbd pool in CephFS testing. Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: do not create rbd pool for CephFS QA

636d2a4

Fixes: https://tracker.ceph.com/issues/62482 Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

batrick force-pushed the i62482 branch from 7fee103 to 636d2a4 Compare August 28, 2023 13:16

vshankar approved these changes Aug 28, 2023

View reviewed changes

batrick added needs-qa wip-pdonnell-testing and removed needs-review labels Aug 29, 2023

batrick added a commit to batrick/ceph that referenced this pull request Aug 30, 2023

Merge PR ceph#53037 into wip-batrick-testing-20230830.165602

e4264ec

* refs/pull/53037/head: qa: do not create rbd pool for CephFS QA Revert "Merge PR ceph#53077 into main"

vshankar merged commit 27edb75 into ceph:main Sep 4, 2023

idryomov reviewed Sep 5, 2023

View reviewed changes

batrick deleted the i62482 branch September 5, 2023 18:48

vshankar mentioned this pull request Sep 8, 2023

qa/cephfs: add back POOL_APP_NOT_ENABLED to ignorelist (partial revert) #53331

Merged

vshankar mentioned this pull request Oct 6, 2023

quincy: qa: add POOL_APP_NOT_ENABLED to ignorelist for cephfs tests #53863

Merged

14 tasks

	if config.get('create_rbd_pool', True):
	log.info('Creating RBD pool')
	mon_remote.run(
	args=['sudo', 'ceph', '--cluster', cluster_name,
	'osd', 'pool', 'create', 'rbd', '8'])
	mon_remote.run(
	args=[
	'sudo', 'ceph', '--cluster', cluster_name,
	'osd', 'pool', 'application', 'enable',
	'rbd', 'rbd', '--yes-i-really-mean-it'
	],
	check_status=False)

Conversation

batrick commented Aug 17, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Contribution Guidelines

Checklist

Uh oh!

batrick commented Aug 17, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

batrick commented Aug 17, 2023

Uh oh!

batrick commented Aug 17, 2023

Uh oh!

batrick commented Aug 18, 2023

Uh oh!

batrick commented Aug 18, 2023

Uh oh!

dparmar18 commented Aug 23, 2023

Uh oh!

mchangir left a comment

Choose a reason for hiding this comment

Uh oh!

vshankar commented Aug 26, 2023

Uh oh!

batrick commented Aug 28, 2023

Uh oh!

batrick commented Aug 29, 2023

Uh oh!

vshankar commented Sep 4, 2023

Uh oh!

idryomov Sep 5, 2023

Choose a reason for hiding this comment

Uh oh!

batrick Sep 5, 2023

Choose a reason for hiding this comment

Uh oh!

batrick Sep 6, 2023

Choose a reason for hiding this comment

Uh oh!

vshankar Sep 8, 2023

Choose a reason for hiding this comment

Uh oh!

idryomov Sep 8, 2023

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

batrick commented Aug 17, 2023 •

edited

Loading

batrick commented Aug 17, 2023 •

edited

Loading