Skip to content

rgw/s3: support paginated ListBuckets#64742

Merged
cbodley merged 1 commit intoceph:mainfrom
cbodley:wip-72315
Aug 27, 2025
Merged

rgw/s3: support paginated ListBuckets#64742
cbodley merged 1 commit intoceph:mainfrom
cbodley:wip-72315

Conversation

@cbodley
Copy link
Contributor

@cbodley cbodley commented Jul 29, 2025

at some point, aws added pagination support to ListBuckets in the form of query params max-buckets and continuation-token

RGWListBuckets already supported pagination for the swift api, so we only need to hook up the s3 request/response formats to use it

Fixes: https://tracker.ceph.com/issues/72315

Show available Jenkins commands

@cbodley
Copy link
Contributor Author

cbodley commented Jul 29, 2025

test case in ceph/s3-tests#679

s->formatter->close_section();
s->formatter->close_section(); // Buckets
if (!marker.empty()) {
s->formatter->dump_string("ContinuationToken", marker);
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

note that the aws doc says "ContinuationToken is obfuscated and is not a real bucket."

but we make no effort to obfuscate here, like we don't for ListObjectsV2

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

when they say "obfuscated" they probably should mean "opaque"

Copy link
Member

@ivancich ivancich left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

at some point, aws added pagination support to ListBuckets[1] in the
form of query params max-buckets and continuation-token

RGWListBuckets already supported pagination for the swift api, so we
only need to hook up the s3 request/response formats to use it

Fixes: https://tracker.ceph.com/issues/72315

[1] https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListBuckets.html

Signed-off-by: Casey Bodley <cbodley@redhat.com>
s->formatter->close_section();
s->formatter->close_section(); // Buckets
if (!marker.empty()) {
s->formatter->dump_string("ContinuationToken", marker);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

when they say "obfuscated" they probably should mean "opaque"

@cbodley
Copy link
Contributor Author

cbodley commented Aug 21, 2025

qa https://pulpito.ceph.com/cbodley-2025-08-14_20:48:33-rgw-wip-72315-distro-default-smithi/
rerun https://pulpito.ceph.com/cbodley-2025-08-15_13:14:47-rgw-wip-72315-distro-default-smithi/

at least one verify job shows the new test case passing:

s3tests_boto3/functional/test_s3.py::test_list_buckets_paginated PASSED [ 59%]

@cbodley
Copy link
Contributor Author

cbodley commented Aug 21, 2025

jenkins test api

@cbodley
Copy link
Contributor Author

cbodley commented Aug 21, 2025

jenkins test windows

@cbodley
Copy link
Contributor Author

cbodley commented Aug 21, 2025

jenkins test make check arm64

@cbodley
Copy link
Contributor Author

cbodley commented Aug 21, 2025

jenkins test api

1 similar comment
@cbodley
Copy link
Contributor Author

cbodley commented Aug 25, 2025

jenkins test api

@cbodley
Copy link
Contributor Author

cbodley commented Aug 25, 2025

jenkins test make check arm64

@cbodley
Copy link
Contributor Author

cbodley commented Aug 26, 2025

https://jenkins.ceph.com/job/ceph-pull-requests-arm64/79833/

The following tests FAILED:
7 - run-tox-mgr-dashboard-py3 (Failed)
262 - unittest_rbd_mirror (SEGFAULT)

@cbodley
Copy link
Contributor Author

cbodley commented Aug 26, 2025

jenkins test make check arm64

@cbodley cbodley merged commit c531eec into ceph:main Aug 27, 2025
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants