qa/rgw: run s3tests against keystone ec2 by cbodley · Pull Request #52813 · ceph/ceph

cbodley · 2023-08-04T16:34:07Z

allow keystone task's yaml to create ec2 credentials for its users
allow s3test task's yaml to point at keystone users instead of creating a local user with radosgw-admin
add an s3test task to the rgw/tempest suite

Fixes: https://tracker.ceph.com/issues/59424

Show available Jenkins commands

jenkins retest this please
jenkins test classic perf
jenkins test crimson perf
jenkins test signed
jenkins test make check
jenkins test make check arm64
jenkins test submodules
jenkins test dashboard
jenkins test dashboard cephadm
jenkins test api
jenkins test docs
jenkins render docs
jenkins test ceph-volume all
jenkins test ceph-volume tox
jenkins test windows

cbodley · 2023-08-04T16:35:09Z

cc @tobias-urdin, thanks for the pointers in #52673 (comment)

cbodley · 2023-08-04T16:37:52Z

it might make sense to rename this rgw/tempest suite to rgw/keystone, and run tempest and s3test tasks as 2 separate jobs

cbodley · 2023-08-04T16:44:17Z

qa/tasks/keystone.py

+    run_in_keystone_venv(ctx, client,
+        [ 'openstack', 'ec2', 'credentials', 'list', '--debug' ] +
+        os_auth_args(public_host, public_port)
+    return ( 'access', 'secret' ) # TODO


will need to see what this output looks like in order to parse it correctly

this is what the ec2 credentials create command returned:

RESP BODY: {"credential": {"user_id": "d41f3e2a147044eca92fa604951dccd9", "tenant_id": "3b5f6f00a3b84b4bb857c6e456dd2abf", "access": "ab19b2a007d7489fa433db6cc0e30333", "secret": "6e141ec00c5b41468d92cad54e181637", "trust_id": null, "links": {...}}}

tobias-urdin · 2023-08-05T14:40:33Z

this is great! will check it out next week, thanks

tobias-urdin · 2023-08-07T11:55:51Z

it might make sense to rename this rgw/tempest suite to rgw/keystone, and run tempest and s3test tasks as 2 separate jobs

LGTM, looking forward to some test results that it works. I think separating them might be a good idea

cbodley · 2023-08-08T18:04:27Z

qa/tasks/keystone.py

+    proc = run_in_keystone_venv(ctx, client,
+        ['openstack', 'ec2', 'credentials', 'list',
+         '--user', user, '--format', 'json', '--debug'] +
+        os_auth_args(public_host, public_port),
+        stdout=StringIO())
+    response = json.loads(proc.stdout.getvalue())


making some progress:

2023-08-08T16:12:06.337 INFO:teuthology.orchestra.run.smithi062.stdout:[ 2023-08-08T16:12:06.337 INFO:teuthology.orchestra.run.smithi062.stdout: { 2023-08-08T16:12:06.337 INFO:teuthology.orchestra.run.smithi062.stdout: "Access": "6351f6d729454264ba27a4a1832c3f8d", 2023-08-08T16:12:06.337 INFO:teuthology.orchestra.run.smithi062.stdout: "Secret": "651d8ede06934828bb5666b996d22410", 2023-08-08T16:12:06.337 INFO:teuthology.orchestra.run.smithi062.stdout: "Project ID": "5ce5ea6c00ae47fa9edb966b12b27686", 2023-08-08T16:12:06.338 INFO:teuthology.orchestra.run.smithi062.stdout: "User ID": "5c692f3ae22a417bac41acab291d81e1" 2023-08-08T16:12:06.338 INFO:teuthology.orchestra.run.smithi062.stdout: } 2023-08-08T16:12:06.338 INFO:teuthology.orchestra.run.smithi062.stdout:] 2023-08-08T16:12:06.469 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_cbodley_ceph_91cc18c91d8eafc57e15922762084e7051d083a6/qa/tasks/s3tests.py", line 114, in create_users access, secret = ctx.keystone.read_ec2_credentials(ctx, **keystone_user) File "/home/teuthworker/src/github.com_cbodley_ceph_91cc18c91d8eafc57e15922762084e7051d083a6/qa/tasks/keystone.py", line 435, in read_ec2_credentials response = json.loads(proc.stdout.getvalue()) AttributeError: 'NoneType' object has no attribute 'stdout'

not sure what it's complaining about now

response = json.loads(proc.stdout.getvalue()) AttributeError: 'list' object has no attribute 'stdout'

from http://qa-proxy.ceph.com/teuthology/cbodley-2023-08-08_18:09:50-rgw:tempest-main-distro-default-smithi/7363371/teuthology.log

Usually, that means the command (credentials list?) didn't have any output. Bad connection or crash, maybe?

got this part working:

2023-08-09T16:54:20.630 INFO:tasks.keystone:proc: [RemoteProcess(client=<paramiko.client.SSHClient object at 0x7f931a84b970>, args=['cd', '/home/ubuntu/cephtest/keystone', Raw('&&'), 'source', '.tox/venv/bin/activate', Raw('&&'), 'openstack', 'ec2', 'credentials', 'list', '--user', 's3tests-main', '--format', 'json', '--debug', '--os-username', 'admin', '--os-password', 'ADMIN', '--os-user-domain-id', 'default', '--os-project-name', 'admin', '--os-project-domain-id', 'default', '--os-identity-api-version', '3', '--os-auth-url', 'http://smithi177.front.sepia.ceph.com:5000/v3'], check_status=True, hostname='smithi177')] 2023-08-09T16:54:20.630 INFO:tasks.keystone:response: [{'Access': 'f0fdcaa1f1ac4b58ba6c46ab10885b49', 'Secret': '88f51ec220db44499c0dd6145342f7b9', 'Project ID': 'cf23225fc9af4c54a8a310dec694b53a', 'User ID': '4ee9cbac66ac45fc862a11a7adb5683b'}] 2023-08-09T16:54:20.630 DEBUG:tasks.s3tests:Using keystone user s3tests-main credentials (f0fdcaa1f1ac4b58ba6c46ab10885b49 : 88f51ec220db44499c0dd6145342f7b9) for foo.client.0 on client.0 2023-08-09T16:54:20.631 DEBUG:tasks.s3tests:Creating user bar.client.0 on client.0

cbodley · 2023-08-09T18:26:10Z

i got the s3tests running, but they all fail with Method Not Allowed - the s3 api isn't exposed due to frontend_prefix: /swift. that seems like a good reason to split the tasks

cbodley · 2023-08-09T20:24:33Z

last attempt was filled with InvalidAccessKeyId errors instead. hopefully adding rgw s3 auth use keystone: true will resolve those

cbodley · 2023-08-10T14:55:07Z

with rgw s3 auth use keystone enabled, tests now fail with AccessDenied. rgw log shows:

2023-08-09T21:10:58.836+0000 7f44682da640 2 req 15080010200567462358 0.323997051s s3:list_buckets s3 keystone: token validation ERROR: {"error":{"code":401,"message":"The request you have made requires authentication.","title":"Unauthorized"}}

tobias-urdin · 2023-08-10T17:31:41Z

with rgw s3 auth use keystone enabled, tests now fail with AccessDenied. rgw log shows:

2023-08-09T21:10:58.836+0000 7f44682da640 2 req 15080010200567462358 0.323997051s s3:list_buckets s3 keystone: token validation ERROR: {"error":{"code":401,"message":"The request you have made requires authentication.","title":"Unauthorized"}}

Just a quick guess would be that you need to assign the member role for the s3tests user in Keystone.

The below part, like it's done for rgwcrypt suite since we have rgw keystone accepted roles = admin,member, hopefully that should be it.

- keystone:
    client.0:
...snip...
      roles: [ name: member ]
      role-mappings:
        - name: member
          user: s3tests-main
          project: s3tests

cbodley · 2023-08-10T20:59:55Z

getting better! http://qa-proxy.ceph.com/teuthology/cbodley-2023-08-10_18:30:57-rgw:tempest-main-distro-default-smithi/7364897/teuthology.log

= 57 failed, 528 passed, 77 skipped, 117 deselected, 1 xfailed, 18 warnings in 687.05s (0:11:27) =

some acl tests failing because they expect the DisplayName to be Mr. foo.client.0 instead of s3tests

some encryption tests failing because they probably need extra config

when the s3tests task cleans up, it fails trying to delete the local user that we didn't create:

radosgw-admin -n client.0 user rm --uid foo.client.0 --purge-data --cluster ceph

cbodley · 2023-09-25T18:23:19Z

down to a single s3test failure in http://qa-proxy.ceph.com/teuthology/cbodley-2023-09-25_17:25:06-rgw:tempest-main-distro-default-smithi/7402007/teuthology.log: test_list_buckets_bad_auth gets InvalidAccessKeyId where it expects SignatureDoesNotMatch

tobias-urdin · 2023-09-25T19:45:14Z

test_list_buckets_bad_auth

That's awesome! That error is probably valid as well and the test needs to be changed to accommodate the difference in response when Keystone auth is used per https://github.com/ceph/ceph/blob/main/src/rgw/rgw_auth_keystone.cc#L448-L455

cbodley · 2023-09-25T19:52:54Z

rgw log from that failing test:

2023-09-25T17:57:58.766+0000 7f19e7847640  0 req 17314085603299168821 0.000000000s s3:list_buckets Secret string does not correctly sign payload, cache miss
2023-09-25T17:57:58.766+0000 7f19e7847640 20 req 17314085603299168821 0.000000000s s3:list_buckets found cached admin token
2023-09-25T17:57:58.766+0000 7f19e7847640 20 sending request to http://smithi083.front.sepia.ceph.com:5000/v3/s3tokens
2023-09-25T17:57:58.766+0000 7f19e7847640 20 register_request mgr=0x55e34cdaf440 req_data->id=1303, curl_handle=0x55e351ad5d20
2023-09-25T17:57:58.766+0000 7f19e7847640 20 WARNING: blocking http request
2023-09-25T17:57:58.766+0000 7f1a9d3c0640 20 link_request req_data=0x55e3517e6f00 req_data->id=1303, curl_handle=0x55e351ad5d20
2023-09-25T17:57:58.850+0000 7f19e7847640  2 req 17314085603299168821 0.083998606s s3:list_buckets s3 keystone: token validation ERROR: {"error":{"code":401,"message":"The request you have made requires authentication.","title":"Unauthorized"}}
":"172.21.15.83","user":"testx$tenanteduser.client.0","operation":"list_buckets","uri":"GET / HTTP/1.1","http_status":"200","error_code":"","bytes_sent":286,"bytes_received":0,"object_size":0,"total_time":123,"user_agent":"Boto3/1.24.96 Python/3.10.6 Linux/5.15.0-84-generic Botocore/1.27.96","referrer":"","trans_id":"tx000005cad476ef5a5d8be-006511ca25-1074-default","authentication_type":"Local","access_key_id":"NCTNZXHHCHGAZFMMCOEA","temp_url":false}9b8-44d2-87e2-4b0e54ad09d8.4214.229
2023-09-25T17:57:58.850+0000 7f19e7847640 20 req 17314085603299168821 0.083998606s s3:list_buckets rgw::auth::keystone::EC2Engine denied with reason=-1
...
2023-09-25T17:57:58.854+0000 7f19e7847640  5 req 17314085603299168821 0.087998547s s3:list_buckets error reading user info, uid=7f6fb8d9f45a4fdfb0e71907ca2ca1f0 can't authenticate
2023-09-25T17:57:58.854+0000 7f19e7847640 20 req 17314085603299168821 0.087998547s s3:list_buckets rgw::auth::s3::LocalEngine rejected with reason=-2028
2023-09-25T17:57:58.854+0000 7f19e7847640 20 req 17314085603299168821 0.087998547s s3:list_buckets rgw::auth::s3::AWSAuthStrategy rejected with reason=-2028
2023-09-25T17:57:58.854+0000 7f19e7847640  5 req 17314085603299168821 0.087998547s s3:list_buckets Failed the auth strategy, reason=-2028

because EC2Engine returns deny instead of reject, LocalEngine goes on to look for a local user with this access key and fails with ERR_INVALID_ACCESS_KEY

when looking up an access key in keystone's secret cache (https://github.com/ceph/ceph/blame/ad54514/src/rgw/rgw_auth_keystone.cc#L575-L582), we'll only use a cached secret if its signature matches the client's. but the test deliberately signs with the wrong secret key roflmao, so we never get to the point where we'd fail with SignatureDoesNotMatch

cbodley · 2023-09-26T16:23:03Z

because EC2Engine returns deny instead of reject, LocalEngine goes on to look for a local user with this access key and fails with ERR_INVALID_ACCESS_KEY

i opened #53680 with a proposed fix, and created https://tracker.ceph.com/issues/62989 to track the backports

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Fixes: https://tracker.ceph.com/issues/59424 Signed-off-by: Casey Bodley <cbodley@redhat.com>

tempest relies on `frontend_prefix: /swift` to serve the swift api from the root path s3tests relies on the same for s3. move the frontend_prefix override into the tempest task so it doesn't apply to s3tests Signed-off-by: Casey Bodley <cbodley@redhat.com>

cbodley · 2023-10-09T16:53:03Z

bundled with the fix from #53846, this passed qa in https://pulpito.ceph.com/cbodley-2023-10-06_14:43:00-rgw-wip-62989-again-distro-default-smithi/. we'll need to merge the fix first

cbodley · 2023-10-16T14:05:30Z

we'll need to merge the fix first

everything's passing now, this is ready for review

cbodley · 2023-11-06T15:43:11Z

passed qa after rerun in https://pulpito.ceph.com/cbodley-2023-11-03_18:34:54-rgw-wip-cbodley-testing-distro-default-smithi/

cbodley · 2023-11-06T15:43:46Z

@dang @alimaredia may i kindly request a final review/approval?

mdw-at-linuxbox · 2023-11-10T01:37:06Z

If I understand this right, the reason to bundle s3tests+ec2 & tempest together is because the tempest test sets up keystone. But - setting up keystone is fast, running s3tests or tempest is slow. I'd rather see this as 2 separate tests because that makes it easy to run just the s3 or tempest part alone.

cbodley · 2023-11-10T14:13:09Z

If I understand this right, the reason to bundle s3tests+ec2 & tempest together is because the tempest test sets up keystone. But - setting up keystone is fast, running s3tests or tempest is slow. I'd rather see this as 2 separate tests because that makes it easy to run just the s3 or tempest part alone.

thanks @mdw-at-linuxbox. it does run the s3tests as a separate job, because that requires radosgw to be configured for s3-at-root while tempest requires swift-at-root. you can use --filter to select one or the other

github-actions bot added rgw tests labels Aug 4, 2023

cbodley requested review from alimaredia and mdw-at-linuxbox August 4, 2023 16:34

cbodley force-pushed the wip-59424 branch from c23620b to adcb881 Compare August 4, 2023 16:42

cbodley commented Aug 4, 2023

View reviewed changes

cbodley force-pushed the wip-59424 branch 2 times, most recently from 1dd2e91 to 2b15abc Compare August 4, 2023 23:19

cbodley force-pushed the wip-59424 branch from 22f1c82 to 91cc18c Compare August 8, 2023 15:44

cbodley commented Aug 8, 2023

View reviewed changes

cbodley force-pushed the wip-59424 branch 3 times, most recently from 5292b60 to 0daef22 Compare August 9, 2023 16:26

cbodley force-pushed the wip-59424 branch 2 times, most recently from fd94e83 to d7d5733 Compare August 9, 2023 20:19

cbodley force-pushed the wip-59424 branch 2 times, most recently from faf86d7 to fa3d274 Compare September 25, 2023 17:24

cbodley mentioned this pull request Sep 26, 2023

rgw/keystone: EC2Engine uses reject() for ERR_SIGNATURE_NO_MATCH #53680

Merged

cbodley added 9 commits October 6, 2023 10:41

qa/rgw/tempest: rearrange tasks and overrides

6fa7167

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/keystone: merge overrides into config

f19d05d

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/keystone: extract os_auth_args to helper function

86b8b4b

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/keystone: config can create ec2 credentials

1456d12

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/keystone: expose function to read a user's ec2 creds

f9872e5

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/s3tests: create_users() takes all yaml config

ff590c5

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/s3tests: enable testing against keystone users with ec2

fa70c78

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw/tempest: run s3tests against keystone ec2

7b27f9d

Fixes: https://tracker.ceph.com/issues/59424 Signed-off-by: Casey Bodley <cbodley@redhat.com>

cbodley force-pushed the wip-59424 branch from fa3d274 to c19aab4 Compare October 6, 2023 14:41

cbodley marked this pull request as ready for review October 6, 2023 14:43

cbodley requested a review from a team as a code owner October 6, 2023 14:43

cbodley added needs-qa needs-review labels Oct 16, 2023

cbodley added the wip-cbodley-testing label Nov 1, 2023

cbodley added ready-to-merge and removed needs-qa wip-cbodley-testing labels Nov 6, 2023

dang approved these changes Nov 21, 2023

View reviewed changes

cbodley merged commit b0b050d into ceph:main Nov 21, 2023

cbodley deleted the wip-59424 branch November 21, 2023 13:34

cbodley mentioned this pull request Jan 15, 2025

reef: Revert "rgw/auth: Fix the return code returned by AuthStrategy," #61405

Merged

Conversation

cbodley commented Aug 4, 2023

Uh oh!

cbodley commented Aug 4, 2023

Uh oh!

cbodley commented Aug 4, 2023

Uh oh!

cbodley Aug 4, 2023

Choose a reason for hiding this comment

Uh oh!

cbodley Aug 4, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tobias-urdin commented Aug 5, 2023

Uh oh!

tobias-urdin commented Aug 7, 2023

Uh oh!

cbodley Aug 8, 2023

Choose a reason for hiding this comment

Uh oh!

cbodley Aug 8, 2023

Choose a reason for hiding this comment

Uh oh!

dang Aug 9, 2023

Choose a reason for hiding this comment

Uh oh!

cbodley Aug 9, 2023

Choose a reason for hiding this comment

Uh oh!

cbodley commented Aug 9, 2023

Uh oh!

cbodley commented Aug 9, 2023

Uh oh!

cbodley commented Aug 10, 2023

Uh oh!

tobias-urdin commented Aug 10, 2023

Uh oh!

cbodley commented Aug 10, 2023

Uh oh!

cbodley commented Sep 25, 2023

Uh oh!

tobias-urdin commented Sep 25, 2023

Uh oh!

cbodley commented Sep 25, 2023

Uh oh!

cbodley commented Sep 26, 2023

Uh oh!

cbodley commented Oct 9, 2023

Uh oh!

cbodley commented Oct 16, 2023

Uh oh!

cbodley commented Nov 6, 2023

Uh oh!

cbodley commented Nov 6, 2023

Uh oh!

mdw-at-linuxbox commented Nov 10, 2023

Uh oh!

cbodley commented Nov 10, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

cbodley Aug 4, 2023 •

edited

Loading