Hi Rclone community,
I am reaching out for assistance with an issue we're experiencing while migrating our OpenStack ObjectStore to a newer cluster. We encounter problems specifically with containers/buckets that contain a large number of files (> 100,000 files). Despite extensive research through Rclone GitHub Issues, forums, and other resources, we haven't found a viable solution.
Problem Description
We are trying to run the following command to migrate our data:
rclone copy source:my-bucket destination:my-bucket --transfers 64 --size-only --server-side-across-configs --multi-thread-write-buffer-size 3Mi --multi-thread-cutoff 1Gi --multi-thread-chunk-size 512Mi --progress -vv
We have experimented with various other options as well, but each time, only a portion of the bucket is copied, and no errors are reported.
Notably, the copied portion has the exact number of objects and size as reported by rclone size source:my-bucket:
root@root#rclone about source:my-bucket
Used: 24.596 GiB
Objects: 71.233k
root@root# rclone size source:my-bucket
Total objects: 47.978k (47978)
Total size: 12.043 GiB (12930961798 Byte)
root@root# rclone size destination:my-bucket
Total objects: 47.978k (47978)
Total size: 12.043 GiB (12930961798 Byte)
The number of objects and size should match the output from rclone about source:my-bucket. This discrepancy leads us to suspect a potential bug or configuration issue.
Rclone Version
rclone v1.67.0
- os/version: ubuntu 20.04 (64 bit)
- os/kernel: 6.8.11-300.fc40.x86_64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.22.4
- go/linking: static
- go/tags: none
OS Details
NAME="Ubuntu"
VERSION="20.04.6 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.6 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
Cloud Storage System
OpenStack Swift - ObjectStore
Commands Tried
rclone copy/sync source:my-bucket destination:my-bucket --transfers 64 --size-only --server-side-across-configs --multi-thread-write-buffer-size 3Mi --multi-thread-cutoff 1Gi --multi-thread-chunk-size 512Mi --progress -vv
rclone about source:my-bucket
rclone size source:my-bucket
rclone size destination:my-bucket
I would greatly appreciate any guidance or solutions to resolve this issue. If more information is needed, I am happy to provide it promptly.
Thank you very much for your support!
Best regards,
Hai
Hi Rclone community,
I am reaching out for assistance with an issue we're experiencing while migrating our OpenStack ObjectStore to a newer cluster. We encounter problems specifically with containers/buckets that contain a large number of files (> 100,000 files). Despite extensive research through Rclone GitHub Issues, forums, and other resources, we haven't found a viable solution.
Problem Description
We are trying to run the following command to migrate our data:
We have experimented with various other options as well, but each time, only a portion of the bucket is copied, and no errors are reported.
Notably, the copied portion has the exact number of objects and size as reported by
rclone size source:my-bucket:The number of objects and size should match the output from
rclone about source:my-bucket. This discrepancy leads us to suspect a potential bug or configuration issue.Rclone Version
OS Details
Cloud Storage System
OpenStack Swift - ObjectStore
Commands Tried
I would greatly appreciate any guidance or solutions to resolve this issue. If more information is needed, I am happy to provide it promptly.
Thank you very much for your support!
Best regards,
Hai