add workarounds for Ceph RGW to Swift backend#5224
Conversation
|
This is related to my open PR ncw/swift#167 |
14b18e4 to
627011d
Compare
|
@ncw @jetpackdanger |
|
@ivandeex Yup, thanks! I hope to be able to return to this soon. We've still happily using rclone for our migrations and will be even happier when we don't have to worry about which binary we're using! :) |
4f52856 to
bdacbad
Compare
|
Is this ready for review? |
bdacbad to
a461cce
Compare
|
@ivandeex I think it's worth a first look , now that I've written a proper commit message! I've left two things unchecked because I'm not sure there's a reasonable way test this change, and I wasn't sure if these options were suitable for documenting in |
ncw
left a comment
There was a problem hiding this comment.
Sorry for the delay on this!
Let's get this merged :-)
It seems to be missing the config definitions but otherwise looks good!
a461cce to
de58d86
Compare
|
Yikes, I'm really not sure where the last two weeks went! Anyway, I've added the config definitions as requested, and the result of |
|
This has sat stale for a long time know. Is this workaround even still necessary at this point or has this been fixed in radosgw. Maybe you could confirm this @jetpackdanger |
Ceph's Swift API emulation does not fully confirm to the API spec. As a result, it sometimes returns fewer items in a container than the requested limit, which according to the spec should means that there are no more objects left in the container. (Note that python-swiftclient always fetches unless the current page is empty.) This commit adds a pair of new Swift backend settings to handle this. Set `fetch_until_empty_page` to true to always fetch another page of the container listing unless there are no items left. Alternatively, set `partial_page_fetch_threshold` to an integer percentage. In this case rclone will fetch a new page only when the current page is within this percentage of the limit. Swift API reference: https://docs.openstack.org/swift/latest/api/pagination.html PR against ncw/swift with research and discussion: ncw/swift#167 Fixes rclone#7924
|
This fix is still needed! Sorry I never merged it. I've rebased it on master and I'll get it merged. |
What is the purpose of this change?
Ceph's Swift API emulation does not fully confirm to the API spec.
As a result, it sometimes returns fewer items in a container than
the requested limit, which according to the spec should means
that there are no more objects left in the container. (Note that
python-swiftclient always fetches unless the current page is empty.)
This commit adds a pair of new Swift backend settings to handle this.
Set
fetch_until_empty_pageto true to always fetch anotherpage of the container listing unless there are no items left.
Alternatively, set
partial_page_fetch_thresholdto an integerpercentage. In this case rclone will fetch a new page only when
the current page is within this percentage of the limit.
Was the change discussed in an issue or in the forum before?
PR against ncw/swift with research and discussion: ncw/swift#167
Swift API reference: https://docs.openstack.org/swift/latest/api/pagination.html
Checklist