Ceph » rgw

on upstream main (ceph version 20.0.0-2805-g5b08bdd9),

rgw crashing while performing get-caller-identity with sts user

log snippet:

[cephuser@ceph-hsm-upstream-main-z1j23q-node6 ~]$ aws --endpoint-url http://10.0.67.54:80 --profile hsm sts assume-role --role-session-name "hsm_session1" --role-arn "arn:aws:iam:::role/hsm-role1" {
"Credentials": {
"AccessKeyId": "6u4JJIm6DTyQFvOeOwnN",
"SecretAccessKey": "CNXIFR3EE7KPURT8JG28UQDL39EQEJT00U2KG2OJ",
"SessionToken": "wOTtbmwUqlZXJuRC3QGey348R5U1qpsxpZ4gaoba9s1YXLQcUoVQQ3hOFQTmdQ4Tv86An7NvfFQddh7EA9xkXkWtq9240chw7lNF2y10GAOCpK+RU+49sFVoH4ZGYQmGSVEpUdOH/CzQAmIx8wxD4SDGXPR9Vh44tseYRLSDgR1YrRaEc4YcRIBfD85X6mKqPus3oImJi3bNT/n+8VMczDmGh8yqv3uv5f1KIn1X7p7xJpc3LofVJzwO4TrYUIsArejGNDyV24diymnyDjalFTQ59bGq5dOf8gyG0Kz5bvOUlvZjecLqa78tUs42htRKWXWyyblS/bcCs9mZE8bBkA==",
"Expiration": "2025-09-10T19:54:37.286253910Z"
},
"AssumedRoleUser": {
"Arn": "arn:aws:sts:::assumed-role/hsm-role1/hsm_session1"
},
"PackedPolicySize": 0
}
[cephuser@ceph-hsm-upstream-main-z1j23q-node6 ~]$
[cephuser@ceph-hsm-upstream-main-z1j23q-node6 ~]$ cat ~/.aws/credentials
[default]
aws_access_key_id = ac1abc1
aws_secret_access_key = ac1abc1
[sts_user1]
aws_access_key_id = 6u4JJIm6DTyQFvOeOwnN
aws_secret_access_key = CNXIFR3EE7KPURT8JG28UQDL39EQEJT00U2KG2OJ
aws_session_token = wOTtbmwUqlZXJuRC3QGey348R5U1qpsxpZ4gaoba9s1YXLQcUoVQQ3hOFQTmdQ4Tv86An7NvfFQddh7EA9xkXkWtq9240chw7lNF2y10GAOCpK+RU+49sFVoH4ZGYQmGSVEpUdOH/CzQAmIx8wxD4SDGXPR9Vh44tseYRLSDgR1YrRaEc4YcRIBfD85X6mKqPus3oImJi3bNT/n+8VMczDmGh8yqv3uv5f1KIn1X7p7xJpc3LofVJzwO4TrYUIsArejGNDyV24diymnyDjalFTQ59bGq5dOf8gyG0Kz5bvOUlvZjecLqa78tUs42htRKWXWyyblS/bcCs9mZE8bBkA==
[cephuser@ceph-hsm-upstream-main-z1j23q-node6 ~]$
[cephuser@ceph-hsm-upstream-main-z1j23q-node6 ~]$ aws --endpoint-url http://10.0.67.54:80 --profile sts_user1 sts get-caller-identity

Could not connect to the endpoint URL: "http://10.0.67.54:80/"
[cephuser@ceph-hsm-upstream-main-z1j23q-node6 ~]$

logs: https://docs.google.com/document/d/13z4h2jYLfVzMwQkz4oYoBY_JGfSc68ZriTRSrV6CJXo/edit?tab=t.lg392wn6zyiv#heading=h.md8d2cr8u14

rgw crash snippet found in rgw logs at debug level 20:

-9> 2025-09-10T19:06:36.437+0000 7f14916b6640  2 req 1718652682947875158 0.001000087s sts:get_caller_identity reading permissions
    -8> 2025-09-10T19:06:36.437+0000 7f14916b6640  2 req 1718652682947875158 0.001000087s sts:get_caller_identity init op
    -7> 2025-09-10T19:06:36.437+0000 7f14916b6640  2 req 1718652682947875158 0.001000087s sts:get_caller_identity verifying op mask
    -6> 2025-09-10T19:06:36.437+0000 7f14916b6640 20 req 1718652682947875158 0.001000087s sts:get_caller_identity required_mask= 0 user.op_mask=7
    -5> 2025-09-10T19:06:36.437+0000 7f14916b6640  2 req 1718652682947875158 0.001000087s sts:get_caller_identity verifying op permissions
    -4> 2025-09-10T19:06:36.437+0000 7f14916b6640  2 req 1718652682947875158 0.001000087s sts:get_caller_identity verifying op params
    -3> 2025-09-10T19:06:36.437+0000 7f14916b6640  2 req 1718652682947875158 0.001000087s sts:get_caller_identity pre-executing
    -2> 2025-09-10T19:06:36.437+0000 7f14916b6640  2 req 1718652682947875158 0.001000087s sts:get_caller_identity check rate limiting
    -1> 2025-09-10T19:06:36.437+0000 7f14916b6640  2 req 1718652682947875158 0.001000087s sts:get_caller_identity executing
     0> 2025-09-10T19:06:36.442+0000 7f14916b6640 -1 ** Caught signal (Aborted) *
 in thread 7f14916b6640 thread_name:io_context_pool

ceph version 20.3.0-2447-g6c001553 (6c001553ea4189ae5fe9bc0d0722348cbd31a3d1) tentacle (dev - RelWithDebInfo)
 1: /lib64/libc.so.6(+0x3fc30) [0x7f158d416c30]
 2: /lib64/libc.so.6(+0x8d02c) [0x7f158d46402c]
 3: raise()
 4: abort()
 5: /usr/bin/radosgw(+0x559468) [0x555e8a2fe468]
 6: /usr/bin/radosgw(+0x566667) [0x555e8a30b667]
 7: /usr/bin/radosgw(+0x57f206) [0x555e8a324206]
 8: (RGWSTSGetCallerIdentity::execute(optional_yield)+0x1a5) [0x555e8a44f945]
 9: (rgw_process_authenticated(RGWHandler_REST*, RGWOp*&, RGWRequest*, req_state*, optional_yield, rgw::sal::Driver*, bool)+0xaa1) [0x555e8a43df31]
 10: (process_request(RGWProcessEnv const&, RGWRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, RGWRestfulIO*, optional_yield, rgw::dmclock::Scheduler*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >, int*)+0xeed) [0x555e8a43f69d]
 11: /usr/bin/radosgw(+0x1157fd8) [0x555e8aefcfd8]
 12: /usr/bin/radosgw(+0x5cab5f) [0x555e8a36fb5f]
 13: /usr/bin/radosgw(+0x5b03d3) [0x555e8a3553d3]
 14: /usr/bin/radosgw(+0x11f62cf) [0x555e8af9b2cf]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
0/ 5 none