Skip to content

qa/radosgw_admin: replace boto2 with boto3#67095

Merged
cbodley merged 6 commits intoceph:mainfrom
cbodley:wip-74591
Feb 26, 2026
Merged

qa/radosgw_admin: replace boto2 with boto3#67095
cbodley merged 6 commits intoceph:mainfrom
cbodley:wip-74591

Conversation

@cbodley
Copy link
Contributor

@cbodley cbodley commented Jan 27, 2026

Fixes: https://tracker.ceph.com/issues/74591

TODO:

  • radosgw_admin.py
  • radosgw_admin_rest.py
Show available Jenkins commands

You must only issue one Jenkins command per-comment. Jenkins does not understand
comments with more than one command.

@cbodley cbodley requested a review from smanjara February 12, 2026 17:45
@cbodley
Copy link
Contributor Author

cbodley commented Feb 13, 2026

successful radosgw_admin task in https://pulpito.ceph.com/cbodley-2026-02-12_21:25:01-rgw:singleton-main-distro-default-trial/

fails just after on radosgw_admin_rest

@cbodley
Copy link
Contributor Author

cbodley commented Feb 20, 2026

this is mostly working, but query params are now getting url-encoded for sigv4 auth but rgw doesn't url-decode

from https://qa-proxy.ceph.com/teuthology/cbodley-2026-02-20_15:11:21-rgw:singleton-main-distro-default-trial/60768/teuthology.log

2026-02-20T15:24:55.979 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: [('user', 'subuser'), 'rm'] {'subuser': 'foo:foo1'}
2026-02-20T15:24:55.982 INFO:tasks.radosgw_admin_rest: json result: {'Code': 'NoSuchSubUser', 'Message': '', 'RequestId': 'tx00000bc188ff270c581b9-0069987cc7-4217-default', 'HostId': '4217-default-default'}
2026-02-20T15:24:55.982 ERROR:teuthology.run_tasks:Saw exception from tasks.

this fails with NoSuchSubUser because the server sees this as QUERY_STRING=subuser=foo%3Afoo1. this will require a code fix

they were comparing the output of 'radosgw-admin policy' with the xml
output of boto2's get_acl() function, but boto3 doesn't expose the xml
response

'radosgw-admin policy' is just dumping the same output as our s3
GetObjectAcl api, so test coverage from s3-tests is sufficient for this

Signed-off-by: Casey Bodley <cbodley@redhat.com>
@cbodley cbodley force-pushed the wip-74591 branch 2 times, most recently from 5ada475 to 741c42f Compare February 20, 2026 17:29
the requestlog queue is only used for debugging, and relied on boto2
hooks that aren't worth rewriting in boto3

Signed-off-by: Casey Bodley <cbodley@redhat.com>
@cbodley
Copy link
Contributor Author

cbodley commented Feb 23, 2026

@cbodley cbodley marked this pull request as ready for review February 23, 2026 21:07
@cbodley cbodley requested a review from a team February 23, 2026 21:07
Fixes: https://tracker.ceph.com/issues/74591

Signed-off-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>
replace the boto2 connection with a boto3 client for bucket/object
creation

aws credentials are passed into rgwadmin_rest() as an instance of
botocore.auth.SigV4Auth

build_admin_request() now returns an instance of
botocore.awsrequest.AWSRequest instead of
boto.connection.AWSAuthConnection

Fixes: https://tracker.ceph.com/issues/74591

Signed-off-by: Casey Bodley <cbodley@redhat.com>
bucket.delete()
rl.log_and_clear("delete_bucket", bucket_name, user1)

# TESTCASE 'policy', 'bucket', 'policy', 'get bucket policy', 'returns S3 policy'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

where is this testcase in the new code?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removed with commit message:

qa/radosgw_admin: remove acl test cases

they were comparing the output of 'radosgw-admin policy' with the xml output of boto2's get_acl() function, but boto3 doesn't expose the xml response

'radosgw-admin policy' is just dumping the same output as our s3 GetObjectAcl api, so test coverage from s3-tests is sufficient for this

@yuvalif yuvalif self-requested a review February 24, 2026 14:29
@cbodley
Copy link
Contributor Author

cbodley commented Feb 24, 2026

seeing run-tox-qa fail with unrelated error:

tasks/rgw_multi/tools.py:52: error: Name "boto.s3.user.User" is not defined [name-defined]

raised #67486 for that

@cbodley
Copy link
Contributor Author

cbodley commented Feb 24, 2026

jenkins test make check

@cbodley
Copy link
Contributor Author

cbodley commented Feb 24, 2026

jenkins test make check arm64

@ivancich
Copy link
Member

jenkins test make check

@ivancich
Copy link
Member

jenkins test make check arm64

@cbodley
Copy link
Contributor Author

cbodley commented Feb 26, 2026

jenkins test make check

@cbodley
Copy link
Contributor Author

cbodley commented Feb 26, 2026

jenkins test make check arm64

@cbodley
Copy link
Contributor Author

cbodley commented Feb 26, 2026

Building remotely on 10.0.1.205+ip-10-0-1-205 (small installed-os-noble jammy sepia x86_64 focal noble aws amd64)ERROR: 10.0.1.205+ip-10-0-1-205 seems to be offline

@cbodley
Copy link
Contributor Author

cbodley commented Feb 26, 2026

jenkins test make check

@cbodley
Copy link
Contributor Author

cbodley commented Feb 26, 2026

The following tests FAILED:
34 - run-rbd-unit-tests-127.sh (Timeout)

@cbodley
Copy link
Contributor Author

cbodley commented Feb 26, 2026

jenkins test make check arm64

@cbodley
Copy link
Contributor Author

cbodley commented Feb 26, 2026

fatal: unable to access 'https://github.com/google/gtest-parallel.git/': Could not resolve host: github.com

@cbodley
Copy link
Contributor Author

cbodley commented Feb 26, 2026

jenkins test make check arm64

@cbodley
Copy link
Contributor Author

cbodley commented Feb 26, 2026

fatal: unable to access 'https://github.com/google/gtest-parallel.git/': Could not resolve host: github.com

@cbodley
Copy link
Contributor Author

cbodley commented Feb 26, 2026

jenkins test make check arm64

@cbodley cbodley merged commit a9b2f32 into ceph:main Feb 26, 2026
15 of 16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants