it would probably make more sense for this throttle to be per-endpoint rather than global, wouldn't it?

the problem that was reported here: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/MGFY6Q5S7KC2BPHCKQCBM42DJ2O4WOH7/

was a coroutine stack overflow due to large number of open http requests.
this is not per endpoint, so, IMO, we should keep the counter global.

ok, so we want a global throttle to limit how much work we'll schedule at once. is there a reason this only applies to http endpoints though, and not others?

it exists in the kafka and amqp clients from the start :-)
https://github.com/ceph/ceph/blob/main/src/rgw/rgw_kafka.cc#L712

s_http_manager_inflight >= max_inflight since you made a local variable for it

can we document these options somewhere? either in a 'config' section in https://docs.ceph.com/en/latest/radosgw/notifications/ or a new 'bucket notification' section similar to https://docs.ceph.com/en/latest/radosgw/config-ref/#topic-persistency-settings

and note that when rendered that way, it only shows the long_desc

we have a confval directive that makes this much easier. for example, doc/radosgw/config-ref.rst just has:

.. confval:: rgw_frontends
.. confval:: rgw_data
.. confval:: rgw_enable_apis

which renders to https://docs.ceph.com/en/latest/radosgw/config-ref/#ceph-object-gateway-config-reference

this avoids having to duplicate the text between docs and config options

how do the upper levels handle EBUSY? process_entry() returns Failure back to process_queue(). will that just retry immediately?

if we keep returning EBUSY back to process_queue(), it will resend the cls_2pc_queue_list_entries() op every time and hammer the osd with reads. once we hit this limit, several process_queue() coroutines may end up doing this at once

i have a feeling that we'd be better off suspending the coroutine here until we're able to schedule this request. that's more complicated though

in case of persistent notifications we retry according to the retry configuration we have there.
we use: rgw_topic_persistency_sleep_duration to space out retries of a specific entry. but we are not looking into the result code.
we are treating all errors in the same way.

would be a nice enhancement to treat EBUSY differently. maybe use larger value for rgw_topic_persistency_sleep_duration?

in case of non-persistent notifications we reply EBUSY to the frontend, which probably convert that 503

looking at the use of rgw_topic_persistency_sleep_duration in https://github.com/ceph/ceph/blob/458d8862/src/rgw/driver/rados/rgw_notify.cc#L231-L246, that causes process_entry() to return Sleeping but doesn't cause process_queue() itself to sleep

the only sleep i see in process_queue() is for is_idle (https://github.com/ceph/ceph/blob/458d8862/src/rgw/driver/rados/rgw_notify.cc#L383-L389) which is false when the queue has entries to send/retry

so when process_entry() returns Sleeping, process_queue() will immediately loop back and resend cls_2pc_queue_list_entries(), then call process_entry() again with the same result. we don't want to keep sending those listing ops while we're throttled or waiting for retry

the sleep indication is per entry, and the queue can have many entries in different state.
i guess that one optimization is to check if all entries in the queue are in "sleep" state, we can put the queue processing into sleep state.

note, however, that this is a different issue. the RGW will be busy going through the queue, but will not send the notification to the HTTP server (which was causing the issue).