Proposed commit message

The input does not have a way to communicate back-pressure to clients, potentially leading to unconstrained growth in the publisher event queue and an OoM event. This change adds a mechanism to keep track of the total sum of in-flight message bytes from the client in order to allow the server to return a 503 HTTP status when the total is too large.

Note that this does not monitor the total memory in the queue as that would require a complete understanding of the allocations in the preparation of event values to be sent to the publisher, but rather uses the message length as a reasonable proxy.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Disruptive User Impact

Author's Checklist

• There is also the possibility of using the metrics to retain the in-flight bytes count. This may be worth doing if we actually want to use that gauge metric. If we do that, I'd also think that having a overage count to distinguish failure due to this from all others. What is here is the minimal change required to get the feature.

How to test this PR locally

Related issues

• Fixes x-pack/filebeat/input/http_endpoint: add back pressure mechanism #41764

Use cases

Screenshots

Logs