qa/cephfs: upgrade xfstests_dev.py to run more tests by rishabh-d-dave · Pull Request #45960 · ceph/ceph

rishabh-d-dave · 2022-04-19T12:42:06Z

Fixes: https://tracker.ceph.com/issues/18475

Checklist

Tracker (select at least one)
- References tracker ticket
- Very recent bug; references commit where it was introduced
- New feature (ticket optional)
- Doc update (no ticket needed)
- Code cleanup (no ticket needed)
Component impact
- Affects Dashboard, opened tracker ticket
- Affects Orchestrator, opened tracker ticket
- No impact that needs to be tracked
Documentation (select at least one)
- Updates relevant documentation
- No doc update is appropriate
Tests (select at least one)
- Includes unit test(s)
- Includes integration test(s)
- Includes bug reproducer
- No tests

Show available Jenkins commands

jenkins retest this please
jenkins test classic perf
jenkins test crimson perf
jenkins test signed
jenkins test make check
jenkins test make check arm64
jenkins test submodules
jenkins test dashboard
jenkins test dashboard cephadm
jenkins test api
jenkins test docs
jenkins render docs
jenkins test ceph-volume all
jenkins test ceph-volume tox
jenkins test windows

rishabh-d-dave · 2022-04-19T12:42:41Z

DNM because the PR needs to add tests that should be run regularly in nightlies.

EDIT -
Also DNM because it needs commits from PR #45950.

rishabh-d-dave · 2022-04-19T12:44:14Z

@gregsfortytwo @jtlayton @vshankar Which tests from xfstests-dev do we wish to run in nightlies? I'll add one more commit to this PR accordingly.

gregsfortytwo · 2022-04-25T13:43:10Z

@gregsfortytwo @jtlayton @vshankar Which tests from xfstests-dev do we wish to run in nightlies? I'll add one more commit to this PR accordingly.

I think we want to run all of them, unless there are some which fail on FUSE filesystems? But that should all be handled within the xfstests suite.

vshankar · 2022-04-25T14:20:16Z

@gregsfortytwo @jtlayton @vshankar Which tests from xfstests-dev do we wish to run in nightlies? I'll add one more commit to this PR accordingly.

generic/*

lxbsz · 2022-04-26T01:46:21Z

qa/tasks/cephfs/xfstests_dev.py

+        return p.returncode
+
+    def run_testfile(self, testdir, testfile):
+        return self.run_tests(f'{testdir}/{testfile}')


Why not fix the test_acls.py by using this new helper in this PR?

I intend to do it but I wanted to figure out first what more tests we want to run.

rishabh-d-dave · 2022-04-26T02:36:00Z

@gregsfortytwo @vshankar

@gregsfortytwo @jtlayton @vshankar Which tests from xfstests-dev do we wish to run in nightlies? I'll add one more commit to this PR accordingly.

generic/*

What about tests/ceph?

lxbsz · 2022-04-26T03:02:08Z

@gregsfortytwo @vshankar

@gregsfortytwo @jtlayton @vshankar Which tests from xfstests-dev do we wish to run in nightlies? I'll add one more commit to this PR accordingly.

generic/*

What about tests/ceph?

@jtlayton has one doc about the xfsdev test [1], this is what we usually will run.

[1] https://jtlayton.wordpress.com/2021/11/29/testing-the-linux-kernel-cephfs-client-with-xfstests/

rishabh-d-dave · 2022-04-26T03:27:48Z

@lxbsz

@jtlayton has one doc about the xfsdev test [1], this is what we usually will run.

[1] https://jtlayton.wordpress.com/2021/11/29/testing-the-linux-kernel-cephfs-client-with-xfstests/

Oh, awesome. Thanks!

vshankar · 2022-05-02T06:31:20Z

@rishabh-d-dave - right now, test_acls uses xfstests_dev, but we would want to run more xfstests not from test_acls by itself. Please do the changes w.r.t. that.

rishabh-d-dave · 2022-05-02T14:59:30Z

jenkins test make check

rishabh-d-dave · 2022-05-06T17:29:41Z

make check failure is valid, I'll fix it after these tests finish running - http://pulpito.front.sepia.ceph.com/rishabh-2022-05-06_15:47:39-fs-wip-rishabh-fs-auth-subcmd-distro-basic-smithi/

flake8 run-test: commands[0] | flake8 --select=F,E9 --exclude=venv,.tox
./tasks/cephfs/tests_from_xfstests-dev.py:3:1: F401 'io.BytesIO' imported but unused
ERROR: InvocationError for command /home/jenkins-build/build/workspace/ceph-pull-requests/qa/.tox/flake8

https://jenkins.ceph.com/job/ceph-pull-requests/94981/

rishabh-d-dave · 2022-05-07T04:15:39Z

Quoting #45960 (comment).

make check failure is valid, I'll fix it after these tests finish running - http://pulpito.front.sepia.ceph.com/rishabh-2022-05-06_15:47:39-fs-wip-rishabh-fs-auth-subcmd-distro-basic-smithi/
flake8 run-test: commands[0] | flake8 --select=F,E9 --exclude=venv,.tox
./tasks/cephfs/tests_from_xfstests-dev.py:3:1: F401 'io.BytesIO' imported but unused
ERROR: InvocationError for command /home/jenkins-build/build/workspace/ceph-pull-requests/qa/.tox/flake8
https://jenkins.ceph.com/job/ceph-pull-requests/94981/

Fixed now.

rishabh-d-dave · 2022-05-09T09:32:08Z

@gregsfortytwo @vshankar @lxbsz @jtlayton
On our teuthology test environment, 3 out of 682 tests from xfstests-dev's generic suite failed. Here's the link - http://pulpito.front.sepia.ceph.com/rishabh-2022-05-07_18:22:10-fs-wip-rishabh-fs-auth-subcmd-distro-basic-smithi/

Are any of these failures (copied below) already known? I'll start investigating into these failure and try fixing them otherwise.

vshankar · 2023-02-27T05:35:37Z

@rishabh-d-dave This would need a retest once #44240 is merged.

qa/tasks/cephfs/xfstests_dev.py

rishabh-d-dave · 2023-03-06T20:37:53Z

kernel: http://pulpito.front.sepia.ceph.com/rishabh-2023-03-03_17:30:49-fs:functional-wip-rishabh-fs-qa-workunit-quota.sh-4-distro-default-smithi/

fuse: http://pulpito.front.sepia.ceph.com/rishabh-2023-03-03_17:32:02-fs:functional-wip-rishabh-fs-qa-workunit-quota.sh-4-distro-default-smithi/

rishabh-d-dave · 2023-03-07T11:34:45Z

fuse: http://pulpito.front.sepia.ceph.com/rishabh-2023-03-06_20:45:05-fs:functional-wip-rishabh-fs-qa-workunit-quota.sh-4-distro-default-smithi/

kernel: http://pulpito.front.sepia.ceph.com/rishabh-2023-03-07_11:30:13-fs:functional-wip-rishabh-fs-qa-workunit-quota.sh-4-distro-default-smithi/

rishabh-d-dave · 2023-03-08T16:03:14Z

http://pulpito.front.sepia.ceph.com/rishabh-2023-03-08_14:29:19-fs:functional-wip-rishabh-fs-qa-workunit-quota.sh-4-distro-default-smithi/

EDIT -
Test ran fine.

New changes are working fine.
Log -

2023-03-08T16:08:47.655 INFO:tasks.cephfs.xfstests_dev:Command return value: 0
2023-03-08T16:08:47.655 INFO:tasks.cephfs.xfstests_dev:=========================================================Total number of tests failed = 4=========================================================
2023-03-08T16:08:47.656 DEBUG:teuthology.orchestra.run.smithi130:> sudo userdel --force --remove fsgqa
2023-03-08T16:08:48.583 DEBUG:teuthology.orchestra.run.smithi130:> sudo userdel --force --remove 123456-fsgqa
2023-03-08T16:08:49.379 DEBUG:teuthology.orchestra.run.smithi130:> sudo groupdel fsgqa

vshankar · 2023-03-08T16:47:24Z

@rishabh-d-dave you mentioned about ~40 failures with ceph-fuse. Could you link me the run for that?

rishabh-d-dave · 2023-03-08T17:13:22Z

@rishabh-d-dave you mentioned about ~40 failures with ceph-fuse. Could you link me the run for that?

It' in the comment just above. The log had double entries. So the total number of failures is actually 20. Following are tracker tickets created for these failures -

~~fuse: https://tracker.ceph.com/issues/58937~~
fuse: https://tracker.ceph.com/issues/58945
kernel: https://tracker.ceph.com/issues/58938

rishabh-d-dave · 2023-03-08T19:31:08Z

make check failure looks unrelated - https://jenkins.ceph.com/job/ceph-pull-requests/111838/. Running this Ceph CI job one more time.

rishabh-d-dave · 2023-03-08T19:31:52Z

jenkins test make check

rishabh-d-dave · 2023-03-08T19:32:00Z

kernel: http://pulpito.front.sepia.ceph.com/rishabh-2023-03-08_16:40:25-fs:functional-wip-rishabh-fs-qa-workunit-quota.sh-4-distro-default-smithi/
fuse: http://pulpito.front.sepia.ceph.com/rishabh-2023-03-08_17:17:52-fs:functional-wip-rishabh-fs-qa-workunit-quota.sh-4-distro-default-smithi/

rishabh-d-dave · 2023-03-08T19:32:57Z

kernel: http://pulpito.front.sepia.ceph.com/rishabh-2023-03-08_16:40:25-fs:functional-wip-rishabh-fs-qa-workunit-quota.sh-4-distro-default-smithi/
fuse: http://pulpito.front.sepia.ceph.com/rishabh-2023-03-08_17:17:52-fs:functional-wip-rishabh-fs-qa-workunit-quota.sh-4-distro-default-smithi/

vshankar · 2023-03-09T02:01:04Z

@rishabh-d-dave you mentioned about ~40 failures with ceph-fuse. Could you link me the run for that?

It' in the comment just above. The log had double entries. So the total number of failures is actually 20. Following are tracker tickets created for these failures -

fuse: https://tracker.ceph.com/issues/58937 kernel: https://tracker.ceph.com/issues/58938

I left a comment in tracker 58937. I might be misinterpreting the whole thing, but I cannot find a failing teuthology job with ceph-fuse.

qa/tasks/cephfs/xfstests_dev.py

lxbsz · 2023-03-09T02:27:35Z

@rishabh-d-dave you mentioned about ~40 failures with ceph-fuse. Could you link me the run for that?

It' in the comment just above. The log had double entries. So the total number of failures is actually 20. Following are tracker tickets created for these failures -
fuse: https://tracker.ceph.com/issues/58937 kernel: https://tracker.ceph.com/issues/58938

I left a comment in tracker 58937. I might be misinterpreting the whole thing, but I cannot find a failing teuthology job with ceph-fuse.

This PR doesn't fail it when there have failure tests. So the qa teuthology jobs will always pass.

@rishabh-d-dave IMO you can just skip those cases currently, which maybe buggy in xfstests-dev. And you can add then back one by one after you fixed them in future.

lxbsz · 2023-03-09T02:46:47Z

qa/tasks/cephfs/xfstests_dev.py

+        else:
+            try:
+                self.assertEqual(p.returncode, 0)
+                line = 'Passed all 0 tests'


Shouldn't this be Passed all ?

And then just do:

self.assertIn(line, stdout)

should be enough ??

As I remembered I never seen Passed all 0 tests yet.

I can't exactly recall when I saw Passed all 0 tests but it does occur sometimes.

This is a false check anyway.

If there is one test and it fails, the output will be:

... ------------------------------------------------ +---------------------------------------------dd-- fsx.3 : -d -N numops -l filelen -S 0 -x ----------------------------------------------- Ran: generic/075 Failures: generic/075 Failed 1 of 1 tests

If there have more than one tests, the output will be:

... ------------------------------------------------ +---------------------------------------------dd-- fsx.3 : -d -N numops -l filelen -S 0 -x ----------------------------------------------- generic/123 3s ... 4s Ran: generic/075 generic/123 Failures: generic/075 Failed 1 of 2 tests

If both fail it will be:

... ----------------------------------------------- fsx.3 : -A -d -N numops -l filelen -S 0 -x ------------------------------------------------ +--------------------------------------------dd--- Ran: generic/075 generic/112 Failures: generic/075 generic/112 Failed 2 of 2 tests

For group test it will be:

... Failures: generic/112 Failed 1 of 568 tests

Please see http://qa-proxy.ceph.com/teuthology/xiubli-2023-02-23_03:24:22-fs:fscrypt-wip-fscrypt-20230215-0834-distro-default-smithi/7185216/teuthology.log.

This is what happens normally. But in certain cases "" gets printed (I guess I saw it when ceph-fuse wasn't support and tests for acls was triggered.).

qa/tasks/cephfs/xfstests_dev.py

Signed-off-by: Rishabh Dave <ridave@redhat.com>

rishabh-d-dave · 2023-03-10T12:23:39Z

Optimizations, nothing more.

rishabh-d-dave · 2023-03-11T05:33:46Z

Tests are running as expected -

kclient: http://pulpito.front.sepia.ceph.com/rishabh-2023-03-10_12:26:35-fs:functional-wip-rishabh-fs-auth-subcmd-printauth-distro-default-smithi/
fuse: http://pulpito.front.sepia.ceph.com/rishabh-2023-03-10_12:26:41-fs:functional-wip-rishabh-fs-auth-subcmd-printauth-distro-default-smithi/

rishabh-d-dave · 2023-03-11T05:33:55Z

jenkins test make check arm64

Signed-off-by: Rishabh Dave <ridave@redhat.com>

lxbsz

LGTM.

rishabh-d-dave · 2023-03-13T07:07:33Z

jenkins test make check arm64

vshankar · 2023-03-14T04:38:31Z

@rishabh-d-dave Please create trackers for the test failures. Otherwise LGTM.

rishabh-d-dave · 2023-03-14T06:32:49Z

@vshankar

@rishabh-d-dave Please create trackers for the test failures. Otherwise LGTM.

The tracker for test failurs were created few days ago -

fuse: https://tracker.ceph.com/issues/58945
kernel: https://tracker.ceph.com/issues/58938

vshankar · 2023-03-14T06:56:09Z

@vshankar

@rishabh-d-dave Please create trackers for the test failures. Otherwise LGTM.

The tracker for test failurs were created few days ago -

fuse: https://tracker.ceph.com/issues/58945 kernel: https://tracker.ceph.com/issues/58938

Ah, since its assigned, I didn't see those dueing bug scrub.

vshankar · 2023-03-14T06:56:21Z

@vshankar

@rishabh-d-dave Please create trackers for the test failures. Otherwise LGTM.

The tracker for test failurs were created few days ago -
fuse: https://tracker.ceph.com/issues/58945 kernel: https://tracker.ceph.com/issues/58938

Ah, since its assigned, I didn't see those during bug scrub.

vshankar · 2023-03-14T08:54:23Z

@rishabh-d-dave you can go ahead and merge.

rishabh-d-dave · 2023-03-14T14:41:18Z

Tests ran fine.

kclient: http://pulpito.front.sepia.ceph.com/rishabh-2023-03-14_07:28:57-fs:functional-wip-rishabh-fs-auth-subcmd-printauth-distro-default-smithi

fuse: http://pulpito.front.sepia.ceph.com/rishabh-2023-03-14_07:29:12-fs:functional-wip-rishabh-fs-auth-subcmd-printauth-distro-default-smithi

The tests from xfstests-dev repo were executed partially so that code from this PR could be test in its entirety in lesser time. Here's the patch that was applied on top of this PR branch to reduce the duration -

diff --git a/qa/tasks/cephfs/xfstests_dev.py b/qa/tasks/cephfs/xfstests_dev.py
index 5de2ddb1568..541a6575842 100644
--- a/qa/tasks/cephfs/xfstests_dev.py
+++ b/qa/tasks/cephfs/xfstests_dev.py
@@ -359,8 +359,12 @@ class XFSTestsDev(CephFSTestCase):
 
         testfiles = [f for f in testfiles if f.isdigit()]
 
+        i = 0
         for testfile in testfiles:
+            if i > 130:
+                break
             self.run_testfile(testdir, testfile)
+            i += 1
 
         log.info('========================================================='
                  f'Total number of tests failed = {self.total_tests_failed}'

The failure of the tests above is not due to the patch on this PR. Tickets have been to investigate deeper into this -

fuse: https://tracker.ceph.com/issues/58945
kernel: https://tracker.ceph.com/issues/58938

rishabh-d-dave added cephfs Ceph File System DNM needs-review tests labels Apr 19, 2022

rishabh-d-dave requested a review from a team April 19, 2022 12:42

rishabh-d-dave force-pushed the xfstests-dev-run-more-tests branch 3 times, most recently from 5cda4c1 to 75c730d Compare April 25, 2022 08:13

lxbsz reviewed Apr 26, 2022

View reviewed changes

vshankar added wip-vshankar-testing and removed wip-vshankar-testing labels May 2, 2022

rishabh-d-dave force-pushed the xfstests-dev-run-more-tests branch 2 times, most recently from 4ab8fa9 to 796ec48 Compare May 7, 2022 04:04

rishabh-d-dave force-pushed the xfstests-dev-run-more-tests branch 4 times, most recently from f8f61f5 to 6a083e7 Compare May 7, 2022 18:21

rishabh-d-dave added the wip-rishabh-testing Rishabh's testing label label May 8, 2022

lxbsz reviewed Feb 28, 2023

View reviewed changes

qa/tasks/cephfs/xfstests_dev.py Outdated Show resolved Hide resolved

lxbsz reviewed Mar 9, 2023

View reviewed changes

qa/tasks/cephfs/xfstests_dev.py Show resolved Hide resolved

lxbsz reviewed Mar 9, 2023

View reviewed changes

qa/tasks/cephfs/xfstests_dev.py Outdated Show resolved Hide resolved

qa/cephfs: add helper methods to run tests from xfstests-dev

f43e109

Signed-off-by: Rishabh Dave <ridave@redhat.com>

qa/cephfs: run all generic tests from xfstests-dev

c4ce722

Signed-off-by: Rishabh Dave <ridave@redhat.com>

lxbsz approved these changes Mar 13, 2023

View reviewed changes

vshankar approved these changes Mar 14, 2023

View reviewed changes

Conversation

rishabh-d-dave commented Apr 19, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Uh oh!

rishabh-d-dave commented Apr 19, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rishabh-d-dave commented Apr 19, 2022

Uh oh!

gregsfortytwo commented Apr 25, 2022

Uh oh!

vshankar commented Apr 25, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lxbsz Apr 26, 2022

Choose a reason for hiding this comment

Uh oh!

rishabh-d-dave Apr 26, 2022

Choose a reason for hiding this comment

Uh oh!

rishabh-d-dave commented Apr 26, 2022

Uh oh!

lxbsz commented Apr 26, 2022

Uh oh!

rishabh-d-dave commented Apr 26, 2022

Uh oh!

vshankar commented May 2, 2022

Uh oh!

rishabh-d-dave commented May 2, 2022

Uh oh!

rishabh-d-dave commented May 6, 2022

Uh oh!

rishabh-d-dave commented May 7, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rishabh-d-dave commented May 9, 2022

Uh oh!

vshankar commented Feb 27, 2023

Uh oh!

Uh oh!

rishabh-d-dave commented Mar 6, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rishabh-d-dave commented Mar 7, 2023

Uh oh!

rishabh-d-dave commented Mar 8, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vshankar commented Mar 8, 2023

Uh oh!

rishabh-d-dave commented Mar 8, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rishabh-d-dave commented Mar 8, 2023

Uh oh!

rishabh-d-dave commented Mar 8, 2023

Uh oh!

rishabh-d-dave commented Mar 8, 2023

Uh oh!

rishabh-d-dave commented Mar 8, 2023

Uh oh!

vshankar commented Mar 9, 2023

Uh oh!

Uh oh!

lxbsz commented Mar 9, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lxbsz Mar 9, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lxbsz Mar 9, 2023

Choose a reason for hiding this comment

Uh oh!

rishabh-d-dave Mar 9, 2023

Choose a reason for hiding this comment

Uh oh!

lxbsz Mar 9, 2023

Choose a reason for hiding this comment

rishabh-d-dave commented Apr 19, 2022 •

edited

Loading

rishabh-d-dave commented Apr 19, 2022 •

edited

Loading

vshankar commented Apr 25, 2022 •

edited

Loading

rishabh-d-dave commented May 7, 2022 •

edited

Loading

rishabh-d-dave commented Mar 6, 2023 •

edited

Loading

rishabh-d-dave commented Mar 8, 2023 •

edited

Loading

rishabh-d-dave commented Mar 8, 2023 •

edited

Loading

lxbsz commented Mar 9, 2023 •

edited

Loading

lxbsz Mar 9, 2023 •

edited

Loading