I was thinking of getting rid of this file entirely because tests.py doesn't use it anymore after #67011, and the only method being used is append_query_arg(), which can be moved inside its user tests_es.py

I was thinking of getting rid of this file entirely because tests.py doesn't use it anymore after #67011, and the only method being used is append_query_arg(), which can be moved inside its user tests_es.py

good point. how does these new commits look?

• qa/rgw/multisite: move append_query_arg() to zone_es.py
• qa/rgw/multisite: remove unused assert_raises() and tools.py

i think something's busted with multisite tests on main.. i see a bunch of tests failing in https://qa-proxy.ceph.com/teuthology/cbodley-2026-02-24_19:39:04-rgw:multisite-wip-74591-distro-default-trial/68706/teuthology.log with:

An error occurred (AccessDenied) when calling the CreateBucket operation

and then segfaults on shutdown

i checked another recent run in https://pulpito.ceph.com/nithyab-2026-02-24_12:43:35-rgw-wip-nbalacha-lua-74219-distro-default-trial/ and see the same in https://qa-proxy.ceph.com/teuthology/nithyab-2026-02-24_12:43:35-rgw-wip-nbalacha-lua-74219-distro-default-trial/67966/teuthology.log

but neither rgw/multisite job in https://pulpito.ceph.com/cbodley-2026-02-24_19:39:04-rgw:multisite-wip-74591-distro-default-trial/ showed python errors, so i don't think there's a regression in this pr

An error occurred (AccessDenied) when calling the CreateBucket operation

oh, must be because #67083 hasn't merged yet

The following tests FAILED:
227 - unittest_peeringstate (Failed)

reported in https://tracker.ceph.com/issues/75144

jenkins test make check arm64

An error occurred (AccessDenied) when calling the CreateBucket operation

oh, must be because #67083 hasn't merged yet

yeah, #66203 is blocked as well. it's currently being testing in wip-anrao3-testing

and then segfaults on shutdown

i checked another recent run in https://pulpito.ceph.com/nithyab-2026-02-24_12:43:35-rgw-wip-nbalacha-lua-74219-distro-default-trial/ and see the same in https://qa-proxy.ceph.com/teuthology/nithyab-2026-02-24_12:43:35-rgw-wip-nbalacha-lua-74219-distro-default-trial/67966/teuthology.log

just before the shutdown here:

2026-02-24T13:27:31.910 INFO:tasks.rgw.c1.client.1.trial076.stdout: 10: (ceph::common::RefCountedObject::put() const+0x115) [0x7ff640c65d05]
2026-02-24T13:27:31.910 INFO:tasks.rgw.c1.client.1.trial076.stdout: 11: radosgw(+0x170764b) [0x55e4814fe64b]
2026-02-24T13:27:31.910 INFO:tasks.rgw.c1.client.1.trial076.stdout: 12: radosgw(+0x6e4dd1) [0x55e4804dbdd1]
2026-02-24T13:27:31.910 INFO:tasks.rgw.c1.client.1.trial076.stdout: 13: (rgw::AppMain::~AppMain()+0x4d3) [0x55e4804e93f3]

I see:

2026-02-24T13:27:31.908 INFO:tasks.rgw.c1.client.1.trial076.stdout:radosgw: ./obj-x86_64-linux-gnu/boost/include/boost/intrusive/list.hpp:1311
: boost::intrusive::list_impl<ValueTraits, SizeType, ConstantTimeSize, HeaderHolder>::iterator boost::intrusive::list_impl<ValueTraits, SizeTy
pe, ConstantTimeSize, HeaderHolder>::iterator_to(reference) [with ValueTraits = boost::intrusive::bhtraits<neorados::Notifier, boost::intrusiv
e::list_node_traits<void*>, boost::intrusive::safe_link, ceph::async::service_tag, 1>; SizeType = long unsigned int; bool ConstantTimeSize = t
rue; HeaderHolder = void; iterator = boost::intrusive::list_iterator<boost::intrusive::bhtraits<neorados::Notifier, boost::intrusive::list_nod
e_traits<void*>, boost::intrusive::safe_link, ceph::async::service_tag, 1>, false>; reference = neorados::Notifier&]: Assertion `!node_algorit
hms::inited(this->priv_value_traits().to_node_ptr(value))' failed.

following the class Notifier{} -> service_list_base_hook, it's using boost::intrusive::list_base_hook. during Notifier shutdown in https://github.com/ceph/ceph/blob/main/src/common/async/service.h#L49, we clear neoref and Client is destroyed. And then again, during the Notifier destructor call, we invoke svc.remove(), https://github.com/ceph/ceph/blob/main/src/common/async/service.h#L66. here we do check if the entries list is empty or not, if not empty we go ahead and remove the list entries. but there could be other entries still in the service list, so that condition alone might not be sufficient. we could be calling iterator_to() on a non-existent element here?

from the first line before the crash, it looks like we are using Intrusive hook with safe_link. should we check if the entry exists or not using is_linked() before trying to erase it from the list?

jenkins test make check

jenkins test make check arm64

jenkins test make check

jenkins test make check

and then segfaults on shutdown
i checked another recent run in https://pulpito.ceph.com/nithyab-2026-02-24_12:43:35-rgw-wip-nbalacha-lua-74219-distro-default-trial/ and see the same in https://qa-proxy.ceph.com/teuthology/nithyab-2026-02-24_12:43:35-rgw-wip-nbalacha-lua-74219-distro-default-trial/67966/teuthology.log

just before the shutdown here:

2026-02-24T13:27:31.910 INFO:tasks.rgw.c1.client.1.trial076.stdout: 10: (ceph::common::RefCountedObject::put() const+0x115) [0x7ff640c65d05]
2026-02-24T13:27:31.910 INFO:tasks.rgw.c1.client.1.trial076.stdout: 11: radosgw(+0x170764b) [0x55e4814fe64b]
2026-02-24T13:27:31.910 INFO:tasks.rgw.c1.client.1.trial076.stdout: 12: radosgw(+0x6e4dd1) [0x55e4804dbdd1]
2026-02-24T13:27:31.910 INFO:tasks.rgw.c1.client.1.trial076.stdout: 13: (rgw::AppMain::~AppMain()+0x4d3) [0x55e4804e93f3]

I see:

2026-02-24T13:27:31.908 INFO:tasks.rgw.c1.client.1.trial076.stdout:radosgw: ./obj-x86_64-linux-gnu/boost/include/boost/intrusive/list.hpp:1311
: boost::intrusive::list_impl<ValueTraits, SizeType, ConstantTimeSize, HeaderHolder>::iterator boost::intrusive::list_impl<ValueTraits, SizeTy
pe, ConstantTimeSize, HeaderHolder>::iterator_to(reference) [with ValueTraits = boost::intrusive::bhtraits<neorados::Notifier, boost::intrusiv
e::list_node_traits<void*>, boost::intrusive::safe_link, ceph::async::service_tag, 1>; SizeType = long unsigned int; bool ConstantTimeSize = t
rue; HeaderHolder = void; iterator = boost::intrusive::list_iterator<boost::intrusive::bhtraits<neorados::Notifier, boost::intrusive::list_nod
e_traits<void*>, boost::intrusive::safe_link, ceph::async::service_tag, 1>, false>; reference = neorados::Notifier&]: Assertion `!node_algorit
hms::inited(this->priv_value_traits().to_node_ptr(value))' failed.

following the class Notifier{} -> service_list_base_hook, it's using boost::intrusive::list_base_hook. during Notifier shutdown in https://github.com/ceph/ceph/blob/main/src/common/async/service.h#L49, we clear neoref and Client is destroyed. And then again, during the Notifier destructor call, we invoke svc.remove(), https://github.com/ceph/ceph/blob/main/src/common/async/service.h#L66. here we do check if the entries list is empty or not, if not empty we go ahead and remove the list entries. but there could be other entries still in the service list, so that condition alone might not be sufficient. we could be calling iterator_to() on a non-existent element here?

from the first line before the crash, it looks like we are using Intrusive hook with safe_link. should we check if the entry exists or not using is_linked() before trying to erase it from the list?

@cbodley is this segfault already fixed in some other PR or should I dig further into this?

@cbodley is this segfault already fixed in some other PR or should I dig further into this?

tracked in https://tracker.ceph.com/issues/75164 and discussed in https://ceph-storage.slack.com/archives/C05LPHSKVPG/p1771994911452119 where Seena and Adam both have patches to propose