x-pack/filebeat/input/http_endpoint: fix handling of http_endpoint request exceeding memory limits#41765
Merged
efd6 merged 1 commit intoelastic:mainfrom Nov 28, 2024
Merged
Conversation
189160d to
f6527d5
Compare
Contributor
|
|
f6527d5 to
f24169b
Compare
Contributor
|
Pinging @elastic/security-service-integrations (Team:Security-Service Integrations) |
ac98f9f to
7c7251a
Compare
Contributor
|
This pull request is now in conflicts. Could you fix it? 🙏 |
7c7251a to
6f1cd44
Compare
e9148a8 to
76c7c77
Compare
…quest exceeding memory limits The input does not have a way to communicate back-pressure to clients, potentially leading to unconstrained growth in the publisher event queue and an OoM event. This change adds a mechanism to keep track of the total sum of in-flight message bytes from the client in order to allow the server to return a 503 HTTP status when the total is too large. Note that this does not monitor the total memory in the queue as that would require a complete understanding of the allocations in the preparation of event values to be sent to the publisher, but rather uses the message length as a reasonable proxy.
76c7c77 to
a7bf1ce
Compare
kcreddy
approved these changes
Nov 28, 2024
Contributor
kcreddy
left a comment
There was a problem hiding this comment.
Just a clarification. LGTM otherwise.
mergify bot
pushed a commit
that referenced
this pull request
Nov 28, 2024
…quest exceeding memory limits (#41765) The input does not have a way to communicate back-pressure to clients, potentially leading to unconstrained growth in the publisher event queue and an OoM event. This change adds a mechanism to keep track of the total sum of in-flight message bytes from the client in order to allow the server to return a 503 HTTP status when the total is too large. Note that this does not monitor the total memory in the queue as that would require a complete understanding of the allocations in the preparation of event values to be sent to the publisher, but rather uses the message length as a reasonable proxy. (cherry picked from commit 2ad3922)
7 tasks
mergify bot
pushed a commit
that referenced
this pull request
Nov 28, 2024
…quest exceeding memory limits (#41765) The input does not have a way to communicate back-pressure to clients, potentially leading to unconstrained growth in the publisher event queue and an OoM event. This change adds a mechanism to keep track of the total sum of in-flight message bytes from the client in order to allow the server to return a 503 HTTP status when the total is too large. Note that this does not monitor the total memory in the queue as that would require a complete understanding of the allocations in the preparation of event values to be sent to the publisher, but rather uses the message length as a reasonable proxy. (cherry picked from commit 2ad3922)
7 tasks
mergify bot
pushed a commit
that referenced
this pull request
Nov 28, 2024
…quest exceeding memory limits (#41765) The input does not have a way to communicate back-pressure to clients, potentially leading to unconstrained growth in the publisher event queue and an OoM event. This change adds a mechanism to keep track of the total sum of in-flight message bytes from the client in order to allow the server to return a 503 HTTP status when the total is too large. Note that this does not monitor the total memory in the queue as that would require a complete understanding of the allocations in the preparation of event values to be sent to the publisher, but rather uses the message length as a reasonable proxy. (cherry picked from commit 2ad3922)
7 tasks
efd6
added a commit
that referenced
this pull request
Nov 28, 2024
…ling of http_endpoint request exceeding memory limits (#41820) * x-pack/filebeat/input/http_endpoint: fix handling of http_endpoint request exceeding memory limits (#41765) The input does not have a way to communicate back-pressure to clients, potentially leading to unconstrained growth in the publisher event queue and an OoM event. This change adds a mechanism to keep track of the total sum of in-flight message bytes from the client in order to allow the server to return a 503 HTTP status when the total is too large. Note that this does not monitor the total memory in the queue as that would require a complete understanding of the allocations in the preparation of event values to be sent to the publisher, but rather uses the message length as a reasonable proxy. (cherry picked from commit 2ad3922) * remove irrelevant changelog entries --------- Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>
efd6
added a commit
that referenced
this pull request
Nov 28, 2024
…quest exceeding memory limits (#41765) (#41819) The input does not have a way to communicate back-pressure to clients, potentially leading to unconstrained growth in the publisher event queue and an OoM event. This change adds a mechanism to keep track of the total sum of in-flight message bytes from the client in order to allow the server to return a 503 HTTP status when the total is too large. Note that this does not monitor the total memory in the queue as that would require a complete understanding of the allocations in the preparation of event values to be sent to the publisher, but rather uses the message length as a reasonable proxy. (cherry picked from commit 2ad3922) Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>
efd6
added a commit
that referenced
this pull request
Nov 28, 2024
…ling of http_endpoint request exceeding memory limits (#41818) * x-pack/filebeat/input/http_endpoint: fix handling of http_endpoint request exceeding memory limits (#41765) The input does not have a way to communicate back-pressure to clients, potentially leading to unconstrained growth in the publisher event queue and an OoM event. This change adds a mechanism to keep track of the total sum of in-flight message bytes from the client in order to allow the server to return a 503 HTTP status when the total is too large. Note that this does not monitor the total memory in the queue as that would require a complete understanding of the allocations in the preparation of event values to be sent to the publisher, but rather uses the message length as a reasonable proxy. (cherry picked from commit 2ad3922) * remove irrelevant changelog entries --------- Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Proposed commit message
The input does not have a way to communicate back-pressure to clients, potentially leading to unconstrained growth in the publisher event queue and an OoM event. This change adds a mechanism to keep track of the total sum of in-flight message bytes from the client in order to allow the server to return a 503 HTTP status when the total is too large.
Note that this does not monitor the total memory in the queue as that would require a complete understanding of the allocations in the preparation of event values to be sent to the publisher, but rather uses the message length as a reasonable proxy.
Checklist
CHANGELOG.next.asciidocorCHANGELOG-developer.next.asciidoc.Disruptive User Impact
Author's Checklist
How to test this PR locally
Related issues
Use cases
Screenshots
Logs