Proposal

The retry_on_http_429 field on queue_config allows Prometheus to retry requests when the server responds with the matching status code. On the other hand, for 5xx error codes, Prometheus will retry using the min_backoff/max_backoff parameters only, ignoring any Retry-After headers.

In cases that the remote endpoint responds with an informed header value, it will not be respected and an erroneous queue_configs can lead to overloading it.

Given that 5xx are understood to be retryable and can happen in different circumstances, I think it would be a good idea to allow Prometheus to respect the Retry-After headers on 5xx responses as a minimum time of waiting and start the min_backoff approach after this time has elapsed.

What do you think?