Fixes: https://tracker.ceph.com/issues/69190
Signed-off-by: Md Mahamudur Rahaman Sajib mahamudur.sajib@croit.io

Concurrent Full Mirroring Design

Overview

The goal of this design is to improve the efficiency of directory mirroring at scale.
In the previous approach, concurrency was present but provided little benefit when:

• A single large directory was being mirrored, or
• The number of active directories being synced was small.

To solve this, the mirroring process is restructured around two dedicated task thread pools.

Thread Pools

1. FileSyncPool

• Responsible for file transfers.
• Files are leaf nodes of the directory tree → no dependencies.
• Each file transfer request is pushed into this pool’s queue and executed asynchronously.
• This achieves maximum concurrency for file transfers with no complexity.

2. DirScanPool

• Responsible for directory scanning and creation, directory I/O, file metadata I/O.

• Directory creation has dependencies:

            (1)
           /   \
        (2)     (3)
       /   \     /  \
   (4)    (5)(6)   (7)
  

In this tree, directory (2) must exist before (4) and (5) can be created.

• To respect ordering, tasks are defined as:

“Create a directory and push tasks for its sub-directories into the pool.”

• Example:
• Task 1 creates directory (1) and enqueues tasks for (2) and (3).
• One thread take task 2 creates directory (2) and enqueues tasks for (4) and (5).
• Another thread take task 3 creates directory (2) and enqueues tasks for (6) and (7).
• This results in a concurrent breadth-first traversal of the tree.

Queue Size Constraint

• In production, we manage hundreds of millions of directories (up to ~800M).
• Storing every directory as a task in the queue is infeasible.
• The queue size is therefore capped (e.g., 1e5 tasks).

Avoiding Deadlocks

A naïve wait–notify model can deadlock. Example:

• Queue size = 1, worker count = 1.
• While processing Task 1, the worker cannot enqueue Task 3 because the queue is full.
• Task 1 never finishes → deadlock.

Solution:

• If the queue is full, instead of enqueuing sub-directory tasks, the worker processes them inline with DFS.
• While DFS, always looking for chance to push subdirectory task into the queue to distribute I/O load between threads evenly
• This ensures continuous progress and avoids deadlocks.

Benefits

• Uniform Workload Distribution: Keeps all threads busy scanning directories.
• Continuous Throughput: Ensures FileSyncPool is constantly fed with file-transfer tasks.
• Scalability: Handles massive directory trees without blowing up memory.
• Robustness: Prevents deadlocks under high load.

Summary

This design achieves true concurrent mirroring by separating file transfers and directory operations into dedicated pools, applying dependency-aware task definitions, limiting queue size, and adopting an inline execution fallback when the queue is full.

Contribution Guidelines

• To sign and title your commits, please refer to Submitting Patches to Ceph.

• If you are submitting a fix for a stable branch (e.g. "quincy"), please refer to Submitting Patches to Ceph - Backports for the proper workflow.

• When filling out the below checklist, you may click boxes directly in the GitHub web UI. When entering or editing the entire PR message in the GitHub web UI editor, you may also select a checklist item by adding an x between the brackets: [x]. Spaces and capitalization matter when checking off items this way.

Checklist

• Tracker (select at least one)
  • References tracker ticket
  • Very recent bug; references commit where it was introduced
  • New feature (ticket optional)
  • Doc update (no ticket needed)
  • Code cleanup (no ticket needed)
• Component impact
  • Affects Dashboard, opened tracker ticket
  • Affects Orchestrator, opened tracker ticket
  • No impact that needs to be tracked
• Documentation (select at least one)
  • Updates relevant documentation
  • No doc update is appropriate
• Tests (select at least one)
  • Includes unit test(s)
  • Includes integration test(s)
  • Includes bug reproducer
  • No tests