cephfs-mirror: integrate blockdiff with mirror daemon by vshankar · Pull Request #62250 · ceph/ceph

vshankar · 2025-03-12T14:06:24Z

Depends on #61937

Contribution Guidelines

To sign and title your commits, please refer to Submitting Patches to Ceph.
If you are submitting a fix for a stable branch (e.g. "quincy"), please refer to Submitting Patches to Ceph - Backports for the proper workflow.
When filling out the below checklist, you may click boxes directly in the GitHub web UI. When entering or editing the entire PR message in the GitHub web UI editor, you may also select a checklist item by adding an x between the brackets: [x]. Spaces and capitalization matter when checking off items this way.

Checklist

Tracker (select at least one)
- References tracker ticket
- Very recent bug; references commit where it was introduced
- New feature (ticket optional)
- Doc update (no ticket needed)
- Code cleanup (no ticket needed)
Component impact
- Affects Dashboard, opened tracker ticket
- Affects Orchestrator, opened tracker ticket
- No impact that needs to be tracked
Documentation (select at least one)
- Updates relevant documentation
- No doc update is appropriate
Tests (select at least one)
- Includes unit test(s)
- Includes integration test(s)
- Includes bug reproducer
- No tests

Show available Jenkins commands

jenkins test classic perf Jenkins Job | Jenkins Job Definition
jenkins test crimson perf Jenkins Job | Jenkins Job Definition
jenkins test signed Jenkins Job | Jenkins Job Definition
jenkins test make check Jenkins Job | Jenkins Job Definition
jenkins test make check arm64 Jenkins Job | Jenkins Job Definition
jenkins test submodules Jenkins Job | Jenkins Job Definition
jenkins test dashboard Jenkins Job | Jenkins Job Definition
jenkins test dashboard cephadm Jenkins Job | Jenkins Job Definition
jenkins test api Jenkins Job | Jenkins Job Definition
jenkins test docs ReadTheDocs | Github Workflow Definition
jenkins test ceph-volume all Jenkins Jobs | Jenkins Jobs Definition
jenkins test windows Jenkins Job | Jenkins Job Definition
jenkins test rook e2e Jenkins Job | Jenkins Job Definition

* refs/pull/62250/head: separate stat add debug cephfs-mirror: integrate blockdiff API for regular file transfers mds: dout snapdiff snapid's before validation check cephfs-mirror: current sync mschanism uses sync mechanism subclass'ing cephfs-mirror: switch to remote dir_root when local prev snapshot is unavailable qa: add test for syncing already existing snapshots cephfs_mirror: avoid latest changes on the source fs to enable mirroring

vshankar · 2025-03-25T08:01:58Z

jenkins test make check

vshankar · 2025-03-25T08:02:03Z

jenkins test make check arm64

vshankar · 2025-03-25T10:26:03Z

Running one final batch of qa tests -- this mostly looks good. I will squash the commits once done.

vshankar · 2025-03-26T08:45:11Z

@joscollin Test run here: https://pulpito.ceph.com/vshankar-2025-03-26_05:53:19-fs:mirror-wip-vshankar-testing-20250325.045436-debug-testing-default-gibba/

The test failure is due to this change

diff --git a/qa/tasks/cephfs/test_mirroring.py b/qa/tasks/cephfs/test_mirroring.py
index 4d17a9a902f..a14f775aa0f 100644
--- a/qa/tasks/cephfs/test_mirroring.py
+++ b/qa/tasks/cephfs/test_mirroring.py
@@ -4,6 +4,7 @@ import errno
 import logging
 import random
 import time
+import unittest

 from io import StringIO
 from collections import deque
@@ -572,6 +573,7 @@ class TestMirroring(CephFSTestCase):

         self.disable_mirroring(self.primary_fs_name, self.primary_fs_id)

+    @unittest.skip("Temporary skip of this test")
     def test_cephfs_mirror_stats(self):
         log.debug('reconfigure client auth caps')
         self.get_ceph_cmd_result(

which I added since I was seeing a weird failure with test_cephfs_mirror_stats (OSDs crashing due to ENOSPC). I think that requires a qa fix, but apart from that, all other tests are running fine.

joscollin · 2025-03-26T09:03:18Z

@joscollin Test run here: https://pulpito.ceph.com/vshankar-2025-03-26_05:53:19-fs:mirror-wip-vshankar-testing-20250325.045436-debug-testing-default-gibba/

The test failure is due to this change
diff --git a/qa/tasks/cephfs/test_mirroring.py b/qa/tasks/cephfs/test_mirroring.py
index 4d17a9a902f..a14f775aa0f 100644
--- a/qa/tasks/cephfs/test_mirroring.py
+++ b/qa/tasks/cephfs/test_mirroring.py
@@ -4,6 +4,7 @@ import errno
 import logging
 import random
 import time
+import unittest

 from io import StringIO
 from collections import deque
@@ -572,6 +573,7 @@ class TestMirroring(CephFSTestCase):

         self.disable_mirroring(self.primary_fs_name, self.primary_fs_id)

+    @unittest.skip("Temporary skip of this test")
     def test_cephfs_mirror_stats(self):
         log.debug('reconfigure client auth caps')
         self.get_ceph_cmd_result(
which I added since I was seeing a weird failure with test_cephfs_mirror_stats (OSDs crashing due to ENOSPC). I think that requires a qa fix, but apart from that, all other tests are running fine.

If it was this one https://tracker.ceph.com/issues/69232, you need to rebase. I hope it's different one.

vshankar · 2025-03-26T09:27:24Z

@joscollin Test run here: https://pulpito.ceph.com/vshankar-2025-03-26_05:53:19-fs:mirror-wip-vshankar-testing-20250325.045436-debug-testing-default-gibba/
The test failure is due to this change
diff --git a/qa/tasks/cephfs/test_mirroring.py b/qa/tasks/cephfs/test_mirroring.py
index 4d17a9a902f..a14f775aa0f 100644
--- a/qa/tasks/cephfs/test_mirroring.py
+++ b/qa/tasks/cephfs/test_mirroring.py
@@ -4,6 +4,7 @@ import errno
 import logging
 import random
 import time
+import unittest

 from io import StringIO
 from collections import deque
@@ -572,6 +573,7 @@ class TestMirroring(CephFSTestCase):

         self.disable_mirroring(self.primary_fs_name, self.primary_fs_id)

+    @unittest.skip("Temporary skip of this test")
     def test_cephfs_mirror_stats(self):
         log.debug('reconfigure client auth caps')
         self.get_ceph_cmd_result(
which I added since I was seeing a weird failure with test_cephfs_mirror_stats (OSDs crashing due to ENOSPC). I think that requires a qa fix, but apart from that, all other tests are running fine.
If it was this one https://tracker.ceph.com/issues/69232, you need to rebase. I hope it's different one.

Not really. The test triggers as OSD assert due to ENOSPC.

joscollin

The mirroring seems working, but this branch shows the below error with all fs snapshot mirror commands. Please check.

2025-03-26T15:05:09.871+0530 7f76df6006c0 -1 asok(0x7f76d8000ba0) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: The UNIX domain socket path /home/jcollin/workspace/ceph/.worktrees/wip-integrate-blockdiff-with-mirror-daemon/build/asok/client.admin.73732.asok is too long! The maximum length on this system is 107

src/tools/cephfs_mirror/PeerReplayer.cc

vshankar · 2025-03-26T09:50:48Z

The mirroring seems working, but this branch shows the below error with all fs snapshot mirror commands. Please check.

This is think due to the branch name being too long and the other paths just added to it.

src/tools/cephfs_mirror/PeerReplayer.cc

src/tools/cephfs_mirror/PeerReplayer.h

vshankar · 2025-03-27T07:27:45Z

This PR is under test in https://tracker.ceph.com/issues/70685.

src/tools/cephfs_mirror/PeerReplayer.h

src/tools/cephfs_mirror/PeerReplayer.cc

vshankar · 2025-04-07T16:52:16Z

Thanks for the review @joscollin. I will have a look asap.

vshankar · 2025-05-06T11:24:48Z

jenkins retest this please

vshankar · 2025-05-07T08:16:44Z

Latest test run results: https://pulpito.ceph.com/vshankar-2025-05-06_14:13:12-fs-wip-vshankar-testing-20250506.112544-debug-testing-default-smithi/

vshankar · 2025-05-07T08:16:55Z

jenkins retest this please

joscollin · 2025-05-07T09:13:46Z

Latest test run results: https://pulpito.ceph.com/vshankar-2025-05-06_14:13:12-fs-wip-vshankar-testing-20250506.112544-debug-testing-default-smithi/

@vshankar
Why test_cephfs_mirror_stats passed in one of the runs? You could've include #63131.

vshankar · 2025-05-07T09:16:50Z

Latest test run results: https://pulpito.ceph.com/vshankar-2025-05-06_14:13:12-fs-wip-vshankar-testing-20250506.112544-debug-testing-default-smithi/

@vshankar Why test_cephfs_mirror_stats passed in one of the runs? You could've include #63131.

Because it's a racy test IMO. Also, #63131 isn't reviewed yet, so why to include in this test run?

joscollin · 2025-05-07T09:19:57Z

Latest test run results: https://pulpito.ceph.com/vshankar-2025-05-06_14:13:12-fs-wip-vshankar-testing-20250506.112544-debug-testing-default-smithi/

@vshankar Why test_cephfs_mirror_stats passed in one of the runs? You could've include #63131.

Because it's a racy test IMO. Also, #63131 isn't reviewed yet, so why to include in this test run?

ok, so sometimes it passes too :)

vshankar · 2025-05-07T10:17:33Z

jenkins test make check arm64

vshankar · 2025-05-07T10:18:10Z

jenkins test make check

vshankar · 2025-05-07T11:13:53Z

jenkins test make check arm64

This avoids considering latest changes from the source filesystem for the mirroring of already existing snapshots. Thus the destination filesystem and snapshots would be created based only on the source snapshots. The destination fs would be a replica of the last snapshot taken. Fixes: https://tracker.ceph.com/issues/68567 Signed-off-by: Jos Collin <jcollin@redhat.com>

Fixes: https://tracker.ceph.com/issues/68567 Signed-off-by: Jos Collin <jcollin@redhat.com>

Fixes: https://tracker.ceph.com/issues/69671 Signed-off-by: Jos Collin <jcollin@redhat.com> Signed-off-by: Venky Shankar <vshankar@redhat.com>

As a debug aid when the snapdiff request is erroring out. Signed-off-by: Venky Shankar <vshankar@redhat.com>

Fixes: http://tracker.ceph.com/issues/70225 Signed-off-by: Venky Shankar <vshankar@redhat.com>

Signed-off-by: Venky Shankar <vshankar@redhat.com>

vshankar · 2025-05-12T09:01:58Z

jenkins test api

vshankar · 2025-05-12T09:02:05Z

jenkins test windows

vshankar · 2025-05-12T13:00:06Z

https://tracker.ceph.com/projects/cephfs/wiki/QA_main_2025#wip-vshankar-testing-20250506112544-debug

vshankar added the cephfs Ceph File System label Mar 12, 2025

vshankar requested a review from a team March 12, 2025 14:06

github-actions bot added the tests label Mar 12, 2025

vshankar mentioned this pull request Mar 19, 2025

mds/client: file blockdiff support #61937

Merged

14 tasks

vshankar force-pushed the wip-integrate-blockdiff-with-mirror-daemon branch from 4a06189 to 565a218 Compare March 21, 2025 10:37

vshankar added the wip-vshankar-testing5 label Mar 21, 2025

vshankar force-pushed the wip-integrate-blockdiff-with-mirror-daemon branch 4 times, most recently from 7eac446 to 8d066af Compare March 25, 2025 04:53

joscollin reviewed Mar 26, 2025

View reviewed changes

src/tools/cephfs_mirror/PeerReplayer.cc Show resolved Hide resolved

joscollin requested changes Mar 26, 2025

View reviewed changes

src/tools/cephfs_mirror/PeerReplayer.cc Outdated Show resolved Hide resolved

vshankar force-pushed the wip-integrate-blockdiff-with-mirror-daemon branch from 8d066af to a381fb3 Compare March 27, 2025 05:26

vshankar changed the title ~~Wip integrate blockdiff with mirror daemon~~ cephfs-mirror: integrate blockdiff with mirror daemon Mar 27, 2025

vshankar requested a review from a team March 27, 2025 05:52

joscollin requested changes Mar 27, 2025

View reviewed changes

src/tools/cephfs_mirror/PeerReplayer.h Show resolved Hide resolved

vshankar mentioned this pull request Mar 28, 2025

cephfs_mirror: avoid latest changes on the source fs to enable mirroring #61539

Closed

14 tasks

joscollin requested changes Apr 2, 2025

View reviewed changes

src/tools/cephfs_mirror/PeerReplayer.h Show resolved Hide resolved

src/tools/cephfs_mirror/PeerReplayer.cc Show resolved Hide resolved

src/tools/cephfs_mirror/PeerReplayer.cc Show resolved Hide resolved

joscollin reviewed Apr 3, 2025

View reviewed changes

src/tools/cephfs_mirror/PeerReplayer.cc Show resolved Hide resolved

src/tools/cephfs_mirror/PeerReplayer.cc Show resolved Hide resolved

src/tools/cephfs_mirror/PeerReplayer.cc Show resolved Hide resolved

joscollin reviewed Apr 4, 2025

View reviewed changes

joscollin added needs-qa and removed orchestrator monitoring cephadm nfs nvmeof labels May 6, 2025

cbodley removed the request for review from a team May 7, 2025 15:09

joscollin and others added 6 commits May 9, 2025 03:07

qa: add test for syncing already existing snapshots

830626b

Fixes: https://tracker.ceph.com/issues/68567 Signed-off-by: Jos Collin <jcollin@redhat.com>

cephfs-mirror: current sync mechanism uses sync mechanism subclass'ing

d9ac431

Fixes: https://tracker.ceph.com/issues/69671 Signed-off-by: Jos Collin <jcollin@redhat.com> Signed-off-by: Venky Shankar <vshankar@redhat.com>

mds: dout snapdiff snapid's before validation check

ae9b035

As a debug aid when the snapdiff request is erroring out. Signed-off-by: Venky Shankar <vshankar@redhat.com>

cephfs-mirror: integrate blockdiff API for regular file transfers

19802ba

Fixes: http://tracker.ceph.com/issues/70225 Signed-off-by: Venky Shankar <vshankar@redhat.com>

qa/cephfs: increase data to be delay data sync by mirror daemon

ab566d5

Signed-off-by: Venky Shankar <vshankar@redhat.com>

vshankar force-pushed the wip-integrate-blockdiff-with-mirror-daemon branch from 24adf97 to ab566d5 Compare May 8, 2025 21:37

vshankar merged commit 53e3248 into ceph:main May 12, 2025
12 checks passed

vshankar removed the wip-vshankar-testing5 label May 12, 2025

joscollin mentioned this pull request May 29, 2025

reef: cephfs_mirror: use snapdiff api for incremental syncing #58985

Open

joscollin removed the needs-qa label Jul 28, 2025

Conversation

vshankar commented Mar 12, 2025 • edited by joscollin Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Contribution Guidelines

Checklist

Uh oh!

vshankar commented Mar 25, 2025

Uh oh!

vshankar commented Mar 25, 2025

Uh oh!

vshankar commented Mar 25, 2025

Uh oh!

vshankar commented Mar 26, 2025

Uh oh!

joscollin commented Mar 26, 2025

Uh oh!

vshankar commented Mar 26, 2025

Uh oh!

joscollin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

vshankar commented Mar 26, 2025

Uh oh!

Uh oh!

Uh oh!

vshankar commented Mar 27, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vshankar commented Apr 7, 2025

Uh oh!

vshankar commented May 6, 2025

Uh oh!

vshankar commented May 7, 2025

Uh oh!

vshankar commented May 7, 2025

Uh oh!

joscollin commented May 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vshankar commented May 7, 2025

Uh oh!

joscollin commented May 7, 2025

Uh oh!

vshankar commented May 7, 2025

Uh oh!

vshankar commented May 7, 2025

Uh oh!

vshankar commented May 7, 2025

Uh oh!

vshankar commented May 12, 2025

Uh oh!

vshankar commented May 12, 2025

Uh oh!

vshankar commented May 12, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

vshankar commented Mar 12, 2025 •

edited by joscollin

Loading

joscollin commented May 7, 2025 •

edited

Loading