Ceph » rgw

There is a tool that allows you to troubleshoot RGW requests that are blocked by dumping all requests waiting for a response from RADOS. The documentation describing how to use this tool is available here:

https://docs.ceph.com/en/latest/radosgw/troubleshooting/#blocked-radosgw-requests

However, this feature is currently broken — it returns an empty JSON list when queried.

The objecter_requests hook is initialized whenever a client creates a librados instance to communicate with RADOS.

The issue is that Objecter effectively supports only a single objecter_requests instance per process. In contrast, RGW currently initializes multiple librados clients:

• Config Store
• RADOS Store
• NeoRados
• RealmWatcher (this is itself a bug — it reuses the Config Store librados instance but calls init() again, resulting in a memory leak)

Since Objecter only supports a single client for request dumping, any subsequent calls to Objecter::init() are silently ignored. In our case, the first librados initialization occurs during Config Store setup. As a result, when we attempt to dump in-flight requests, we do not see RADOS requests related to the Rados Store(s3 requests), which are typically the ones of interest.

Looking at other Ceph components that use librados (such as RBD, MDS, and rbd-mirror), they each create a single librados client and reuse it throughout the process.

This raises the question:
Should RGW also be refactored to use a single librados client, or should Objecter be enhanced to support multiple librados clients per process?