Ceph » rgw

summing up our discussion from slack - the failures seem to be specific to this particular branch. We can inspect the new commits in that branch to find the root cause. Otherwise we may need to bisect and find the changes causing transition/restore to fail.

from https://qa-proxy.ceph.com/teuthology/cbodley-2025-09-16_20:27:49-rgw-wip-63323-distro-default-gibba/8504100/teuthology.log

2025-09-16T20:55:12.747 INFO:teuthology.orchestra.run.gibba033.stdout:FAILED s3tests_boto3/functional/test_s3.py::test_restore_object_permanent - A...
2025-09-16T20:55:12.747 INFO:teuthology.orchestra.run.gibba033.stdout:FAILED s3tests_boto3/functional/test_s3.py::test_restore_noncur_obj - assert ...

While working on https://github.com/ceph/s3-tests/pull/686 , I found an issue with current `qa/tasks/rgw_cloudtier.py` - all the rgw services were not being restarted after configuring CLOUDTIER storage class. That is resulting in second RGW server (client.1) skipping/failing transition/restore requests. That may have resulted in spurious delayed transitions or errors. I am fixing it as part of https://github.com/ceph/ceph/pull/64933 . Hopefully the tests will stabilize now.

We also have https://tracker.ceph.com/issues/72877 opened to further improve checks in restore tests (if needed).

https://github.com/ceph/s3-tests/pull/686 has been merged, and the cloud-restore tests are now passing consistently. I’ll still keep the tracker open for a while longer to further confirm.

Thank you, @Soumya Koduri !

There was one more intermittent failure observed with "test_read_though" testcase. PRs https://github.com/ceph/ceph/pull/65926 & https://github.com/ceph/s3-tests/pull/701 fix the same.

Hi @Soumya Koduri , Could we get an update on the status of this tracker? QA runs are hitting this issue (e.g., https://qa-proxy.ceph.com/teuthology/anuchaithra-2025-11-13_06:25:18-rgw-wip-anrao4-testing-2025-11-12-1156-tentacle-distro-default-smithi/8600979/teuthology.log).

Hi Eric,

This bug is now fixed in main with the last PRs merged (mentioned in comment#8). However they need backport in tentacle (which is the reason for failure seen in the QE run above on tentacle branch).

I had raised https://github.com/ceph/ceph/pull/65830#issuecomment-3381527681 for the same. But QE need to pick my s3-tests branch linked to that PR .. https://github.com/soumyakoduri/s3-tests/commits/wip-skoduri-tentacle/

I already ran the restore tests and updated the PR https://github.com/ceph/ceph/pull/65830#issuecomment-3432162852 .. if no other failures are seen, I think this PR can be merged. I can then cherry-pick s3-tests commits as well to tentacle branch.

Is this in progress? I see it on tentacle and if there's a fix for it I'd like to cherry pick it.

I see the PR, I'll cherry-pick.

Never mind, I got confused by the tag. The tentacle I was testing just hadn't incorporated all of the fixes, it looks like.