cephfs_mirror, qa: fix test failure test_cephfs_mirror_cancel_mirroring_and_readd by joscollin · Pull Request #56118 · ceph/ceph

joscollin · 2024-03-11T10:52:51Z

Fixes: https://tracker.ceph.com/issues/64711
Fixes: https://tracker.ceph.com/issues/65115

Contribution Guidelines

To sign and title your commits, please refer to Submitting Patches to Ceph.
If you are submitting a fix for a stable branch (e.g. "quincy"), please refer to Submitting Patches to Ceph - Backports for the proper workflow.
When filling out the below checklist, you may click boxes directly in the GitHub web UI. When entering or editing the entire PR message in the GitHub web UI editor, you may also select a checklist item by adding an x between the brackets: [x]. Spaces and capitalization matter when checking off items this way.

Checklist

Tracker (select at least one)
- References tracker ticket
- Very recent bug; references commit where it was introduced
- New feature (ticket optional)
- Doc update (no ticket needed)
- Code cleanup (no ticket needed)
Component impact
- Affects Dashboard, opened tracker ticket
- Affects Orchestrator, opened tracker ticket
- No impact that needs to be tracked
Documentation (select at least one)
- Updates relevant documentation
- No doc update is appropriate
Tests (select at least one)
- Includes unit test(s)
- Includes integration test(s)
- Includes bug reproducer
- No tests

Show available Jenkins commands

jenkins retest this please
jenkins test classic perf
jenkins test crimson perf
jenkins test signed
jenkins test make check
jenkins test make check arm64
jenkins test submodules
jenkins test dashboard
jenkins test dashboard cephadm
jenkins test api
jenkins test docs
jenkins render docs
jenkins test ceph-volume all
jenkins test ceph-volume tox
jenkins test windows
jenkins test rook e2e

src/tools/cephfs_mirror/PeerReplayer.cc

qa/tasks/cephfs/test_mirroring.py

github-actions · 2024-03-14T11:46:05Z

This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved

joscollin · 2024-03-26T06:03:22Z

jenkins test make check arm64

vshankar · 2024-03-26T09:45:34Z

Thanks for update @joscollin. Could you run this through fs:mirror and share the results?

Suggest running with -N 10 to schedule more jobs.

joscollin · 2024-03-26T10:45:03Z

Thanks for update @joscollin. Could you run this through fs:mirror and share the results?

Suggest running with -N 10 to schedule more jobs.

The shaman build is in progress (main rebased). Will run the fs:mirror when it's ready.

I ran fs:mirror on the top of your failed sha: 717ce3c. They are all passed:
https://pulpito.ceph.com/jcollin-2024-03-26_03:56:22-fs:mirror-wip-jcollin-testing-25032024-distro-default-smithi/

joscollin · 2024-03-26T10:45:22Z

jenkins test make check arm64

joscollin · 2024-03-26T14:49:12Z

jenkins test make check

joscollin · 2024-03-27T04:55:03Z

Thanks for update @joscollin. Could you run this through fs:mirror and share the results?

Suggest running with -N 10 to schedule more jobs.

@vshankar
Test test_cephfs_mirror_cancel_mirroring_and_readd passed 25 times with no Bad file descriptor in the logs. The 1 failed job is unrelated.

Please check:
https://pulpito.ceph.com/jcollin-2024-03-27_00:57:31-fs:mirror-wip-jcollin-testing-26032024-distro-default-smithi/

src/tools/cephfs_mirror/PeerReplayer.cc

vshankar · 2024-04-01T05:15:38Z

@joscollin changed under qa look good.

joscollin · 2024-04-08T05:56:19Z

jenkins test make check arm64

* refs/pull/56118/head: qa: fix check_peer_snap_in_progress wait for 'syncing' cephfs_mirror: Fixes 'Bad file descriptor' showing during test_cephfs_mirror_cancel_mirroring_and_readd execution

vshankar

@joscollin PTAL as the build failure.

src/tools/cephfs_mirror/PeerReplayer.cc

joscollin · 2024-04-19T13:44:48Z

rebased

joscollin · 2024-08-14T00:56:01Z

jenkins test api

joscollin · 2024-08-14T00:56:08Z

jenkins test make check

joscollin · 2024-08-14T07:49:06Z

jenkins test make check arm64

joscollin · 2024-08-14T13:12:53Z

@vshankar The test passed on top of your failed branch: https://pulpito.ceph.com/jcollin-2024-08-14_10:58:02-fs:mirror-wip-vshankar-testing-20240806.162829-debug-distro-default-smithi/

joscollin · 2024-08-20T04:01:44Z

jenkins test make check arm64

qa/tasks/cephfs/test_mirroring.py

…_mirror_cancel_mirroring_and_readd execution Avoid stale fh by calling pre_sync_check_and_open_handles inside the loop Fixes: https://tracker.ceph.com/issues/65115 Signed-off-by: Jos Collin <jcollin@redhat.com>

* Make all replayer threads busy and then query for 'syncing' state instead of just fetching the current status. * Dropped 'current_syncing_snap' check, as it's not compulsory for this test. The actual intension is to make threads in 'syncing' status and 'current_syncing_snap' check is not necessary for that. * Drop 'snaps_deleted' metrics check in test_cephfs_mirror_cancel_mirroring_and_readd. test_cephfs_mirror_cancel_mirroring_and_readd primarily focusses on the synchronization of the newly added directory paths post removal of the previously added/syncing directory paths. So checking of 'snaps_deleted' metrics is unnecessary here. * Wait for more time to finish the new snapshot creations and the sync backoff. We need to wait for more time in test_cephfs_mirror_cancel_mirroring_and_readd, as the test makes all replayer threads busy. Fixes: https://tracker.ceph.com/issues/64711 Signed-off-by: Jos Collin <jcollin@redhat.com>

joscollin · 2024-08-25T00:43:50Z

jenkins test make check arm64

joscollin · 2024-08-27T06:10:48Z

@rishabh-d-dave @vshankar Please add this PR to your next QA batch.

joscollin · 2024-09-10T04:33:47Z

jenkins test make check arm64

vshankar · 2024-09-17T04:40:30Z

This PR is under test in https://tracker.ceph.com/issues/68092.

joscollin · 2024-09-23T10:34:24Z

This PR is under test in https://tracker.ceph.com/issues/68092.

@vshankar test_cephfs_mirror_cancel_mirroring_and_readd passed in https://tracker.ceph.com/issues/68092.

* refs/pull/56118/head: qa: make all replayer threads busy and query 'syncing' status cephfs_mirror: Fixes 'Bad file descriptor' showing during test_cephfs_mirror_cancel_mirroring_and_readd execution Reviewed-by: Rishabh Dave <ridave@redhat.com>

joscollin · 2024-09-25T03:31:52Z

https://tracker.ceph.com/issues/68187

vshankar

https://tracker.ceph.com/projects/cephfs/wiki/Main#wip-vshankar-testing-20240917043942-debug

Comment addressed.

* refs/pull/56118/head: qa: make all replayer threads busy and query 'syncing' status cephfs_mirror: Fixes 'Bad file descriptor' showing during test_cephfs_mirror_cancel_mirroring_and_readd execution Reviewed-by: Rishabh Dave <ridave@redhat.com>

joscollin · 2024-12-14T07:44:09Z

2024-07-11T20:19:57.022 INFO:tasks.cephfs_test_runner:FAIL: test_cephfs_mirror_incremental_sync (tasks.cephfs.test_mirroring.TestMirroring)
2024-07-11T20:19:57.022 INFO:tasks.cephfs_test_runner:Test incremental snapshot synchronization (based on mtime differences).
2024-07-11T20:19:57.022 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2024-07-11T20:19:57.022 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2024-07-11T20:19:57.022 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_167d9c833410f9c01f499ea86eaca54be8d33d31/qa/tasks/cephfs/test_mirroring.py", line 1158, in test_cephfs_mirror_incremental_sync
2024-07-11T20:19:57.022 INFO:tasks.cephfs_test_runner:    self.verify_snapshot(repo_path, 'snap_c')
2024-07-11T20:19:57.023 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_167d9c833410f9c01f499ea86eaca54be8d33d31/qa/tasks/cephfs/test_mirroring.py", line 248, in verify_snapshot
2024-07-11T20:19:57.023 INFO:tasks.cephfs_test_runner:    self.assertTrue(source_res == dest_res)
2024-07-11T20:19:57.023 INFO:tasks.cephfs_test_runner:AssertionError: False is not true
2024-07-11T20:19:57.023 INFO:tasks.cephfs_test_runner:
2024-07-11T20:19:57.023 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2024-07-11T20:19:57.023 INFO:tasks.cephfs_test_runner:Ran 9 tests in 2845.658s
2024-07-11T20:19:57.023 INFO:tasks.cephfs_test_runner:
2024-07-11T20:19:57.024 INFO:tasks.cephfs_test_runner:FAILED (failures=1)

Unexpected failure here.

Fixed by ced1f6a

github-actions bot added cephfs Ceph File System tests labels Mar 11, 2024

joscollin changed the title ~~cephfs_mirror: fix test failure: test_cephfs_mirror_cancel_mirroring_and_readd~~ cephfs_mirror: fix test failure test_cephfs_mirror_cancel_mirroring_and_readd Mar 11, 2024

joscollin force-pushed the wip-B64711-qa-failure_cancel-mirroring-and-readd branch from cd8ceea to 6923235 Compare March 11, 2024 12:01

joscollin marked this pull request as draft March 12, 2024 04:38

vshankar requested changes Mar 12, 2024

View reviewed changes

src/tools/cephfs_mirror/PeerReplayer.cc Outdated Show resolved Hide resolved

qa/tasks/cephfs/test_mirroring.py Outdated Show resolved Hide resolved

github-actions bot added the needs-rebase label Mar 14, 2024

joscollin force-pushed the wip-B64711-qa-failure_cancel-mirroring-and-readd branch from 6923235 to d532a11 Compare March 22, 2024 12:45

github-actions bot removed the needs-rebase label Mar 22, 2024

joscollin marked this pull request as ready for review March 22, 2024 12:45

joscollin force-pushed the wip-B64711-qa-failure_cancel-mirroring-and-readd branch from d532a11 to 26cd7bc Compare March 25, 2024 03:51

joscollin requested a review from vshankar March 25, 2024 14:20

joscollin force-pushed the wip-B64711-qa-failure_cancel-mirroring-and-readd branch from 26cd7bc to 7b283c7 Compare March 26, 2024 09:01

vshankar reviewed Apr 1, 2024

View reviewed changes

src/tools/cephfs_mirror/PeerReplayer.cc Outdated Show resolved Hide resolved

vshankar added the wip-vshankar-testing label Apr 8, 2024

vshankar requested changes Apr 8, 2024

View reviewed changes

src/tools/cephfs_mirror/PeerReplayer.cc Outdated Show resolved Hide resolved

vshankar removed the wip-vshankar-testing label Apr 8, 2024

joscollin force-pushed the wip-B64711-qa-failure_cancel-mirroring-and-readd branch from 7b283c7 to 41dcd81 Compare April 9, 2024 05:34

joscollin force-pushed the wip-B64711-qa-failure_cancel-mirroring-and-readd branch from 41dcd81 to ad05017 Compare April 19, 2024 13:44

joscollin requested a review from vshankar August 13, 2024 15:09

vshankar reviewed Aug 20, 2024

View reviewed changes

qa/tasks/cephfs/test_mirroring.py Outdated Show resolved Hide resolved

joscollin added 2 commits August 20, 2024 13:20

cephfs_mirror: Fixes 'Bad file descriptor' showing during test_cephfs…

bd78bdc

…_mirror_cancel_mirroring_and_readd execution Avoid stale fh by calling pre_sync_check_and_open_handles inside the loop Fixes: https://tracker.ceph.com/issues/65115 Signed-off-by: Jos Collin <jcollin@redhat.com>

joscollin force-pushed the wip-B64711-qa-failure_cancel-mirroring-and-readd branch from 3986c36 to db9825d Compare August 20, 2024 07:50

joscollin changed the title ~~cephfs_mirror: fix test failure test_cephfs_mirror_cancel_mirroring_and_readd~~ cephfs_mirror, qa: fix test failure test_cephfs_mirror_cancel_mirroring_and_readd Aug 27, 2024

joscollin added the needs-qa label Aug 27, 2024

vshankar added the wip-vshankar-testing4 label Aug 28, 2024

vshankar approved these changes Oct 7, 2024

View reviewed changes

vshankar merged commit 635b554 into ceph:main Oct 7, 2024

vshankar removed the wip-vshankar-testing4 label Oct 7, 2024

joscollin deleted the wip-B64711-qa-failure_cancel-mirroring-and-readd branch October 8, 2024 03:25

Conversation

joscollin commented Mar 11, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Contribution Guidelines

Checklist

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Mar 14, 2024

Uh oh!

joscollin commented Mar 26, 2024

Uh oh!

vshankar commented Mar 26, 2024

Uh oh!

joscollin commented Mar 26, 2024

Uh oh!

joscollin commented Mar 26, 2024

Uh oh!

joscollin commented Mar 26, 2024

Uh oh!

joscollin commented Mar 27, 2024

Uh oh!

Uh oh!

vshankar commented Apr 1, 2024

Uh oh!

joscollin commented Apr 8, 2024

Uh oh!

vshankar left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

joscollin commented Apr 19, 2024

Uh oh!

joscollin commented Aug 14, 2024

Uh oh!

joscollin commented Aug 14, 2024

Uh oh!

joscollin commented Aug 14, 2024

Uh oh!

joscollin commented Aug 14, 2024

Uh oh!

joscollin commented Aug 20, 2024

Uh oh!

Uh oh!

joscollin commented Aug 25, 2024

Uh oh!

joscollin commented Aug 27, 2024

Uh oh!

joscollin commented Sep 10, 2024

Uh oh!

vshankar commented Sep 17, 2024

Uh oh!

joscollin commented Sep 23, 2024

Uh oh!

joscollin commented Sep 25, 2024

Uh oh!

vshankar left a comment

Choose a reason for hiding this comment

Uh oh!

joscollin commented Dec 14, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

joscollin commented Mar 11, 2024 •

edited

Loading