Skip to content

deadlock in --no-traverse when all transfers skipped #8656

@ncw

Description

@ncw

Issue made from #8647 by @jeremy

What is the problem you are having with rclone?

When every source object already exists at the destination and --no-traverse is set, march stalled/deadlocked after 100 entries because dstChan lagged srcChan. Introduced in v1.70.0 with callback-based syncing.

I couldn't get a test case working reliably, but here's a script to reproduce:

#!/usr/bin/env bash
# Reproduces rclone v1.70+ deadlock with --no-traverse when all files exist
# Usage: ./repro.sh [rclone-binary-path]

# Use provided rclone binary or default to system rclone
RCLONE="${1:-rclone}"

echo "Using rclone: $RCLONE"
echo "Version: $($RCLONE version | head -1)"
echo

# Create test directories
src=$(mktemp -d)
dst=$(mktemp -d)

# Create 200 test files (enough to trigger the ~100 file deadlock)
echo "Creating 200 test files..."
for i in {1..200}; do
    echo "content$i" > "$src/file$i.txt"
done

# Initial copy to establish destination
echo "Initial copy to destination..."
$RCLONE copy "$src/" "$dst/" --quiet

# Create file list
filelist=$(mktemp)
ls "$src" > "$filelist"

# This command deadlocks in v1.70+ when all files exist at destination
echo "Testing --no-traverse with all files existing (10s timeout)..."
timeout 10s $RCLONE copy "$src/" "$dst/" \
    --no-traverse \
    --files-from "$filelist" \
    --ignore-existing \
    --progress

if [ $? -eq 124 ]; then
    echo -e "\n❌ FAIL: Command timed out (deadlock detected)"
    exit 1
else
    echo -e "\n✅ PASS: Command completed successfully"
fi

# Cleanup
rm -rf "$src" "$dst" "$filelist"

Results on 1.69.3, 1.70.2, and with this patch:

Using rclone: rclone
Version: rclone v1.69.3

Creating 200 test files...
Initial copy to destination...
Testing --no-traverse with all files existing (10s timeout)...
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Checks:               200 / 200, 100%
Elapsed time:         0.0s

✅ PASS: Command completed successfully
Using rclone: rclone
Version: rclone v1.70.2

Creating 200 test files...
Initial copy to destination...
Testing --no-traverse with all files existing (10s timeout)...
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Checks:                 0 / 0, -, Listed 200
Elapsed time:         9.5s
❌ FAIL: Command timed out (deadlock detected)
Using rclone: ./rclone
Version: rclone v1.71.0-DEV

Creating 200 test files...
Initial copy to destination...
Testing --no-traverse with all files existing (10s timeout)...
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Checks:               200 / 200, 100%, Listed 200
Elapsed time:         0.0s

✅ PASS: Command completed successfully

What is your rclone version (output from rclone version)

v1.70.2

The associated forum post URL from https://forum.rclone.org

How to use GitHub

  • Please use the 👍 reaction to show that you are affected by the same issue.
  • Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
  • Subscribe to receive notifications on status change and new comments.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions