Prevent date_histogram from OOMing#72081
Merged
nik9000 merged 5 commits intoelastic:masterfrom Apr 27, 2021
Merged
Conversation
Collaborator
|
Pinging @elastic/es-analytics-geo (Team:Analytics) |
Member
Author
|
Like #71758 this one has been here for a while, seems fairly rare, but has a fairly bad effect. |
This prevents the `date_histogram` from running out of memory allocating empty buckets when you set the interval to something tiny like `seconds` and aggregate over a very wide date range. Without this change we'd allocate memory very quickly and throw and out of memory error, taking down the node. With it we instead throw the standard "too many buckets" error. Relates to elastic#71758
Member
Author
|
Ouch, checkstyle. |
imotov
approved these changes
Apr 23, 2021
| "Tiny tiny tiny date_range": | ||
| - skip: | ||
| version: " - 7.99.99" | ||
| reason: fixed in 8.0 and being backported to 7.13.0 |
Contributor
There was a problem hiding this comment.
Probably too late for 7.13.0 now
| * this quickly in pathological cases and plenty large to keep the | ||
| * overhead minimal. | ||
| */ | ||
| int reportEmptyEvery = 10000; |
Member
Author
There was a problem hiding this comment.
Probably. I stuck it here so the comment above it could describe why it has the value it does. I'm not sure the right way to do that if its a constant without making it harder to read.
Contributor
There was a problem hiding this comment.
Can you at least make it final? My first reaction was "Why and where is he changing it?".
nik9000
added a commit
to nik9000/elasticsearch
that referenced
this pull request
Apr 27, 2021
This prevents the `date_histogram` from running out of memory allocating empty buckets when you set the interval to something tiny like `seconds` and aggregate over a very wide date range. Without this change we'd allocate memory very quickly and throw and out of memory error, taking down the node. With it we instead throw the standard "too many buckets" error. Relates to elastic#71758
nik9000
added a commit
that referenced
this pull request
Apr 28, 2021
This prevents the `date_histogram` from running out of memory allocating empty buckets when you set the interval to something tiny like `seconds` and aggregate over a very wide date range. Without this change we'd allocate memory very quickly and throw and out of memory error, taking down the node. With it we instead throw the standard "too many buckets" error. Relates to #71758
nik9000
added a commit
to nik9000/elasticsearch
that referenced
this pull request
Apr 28, 2021
Now that elastic#72081 has landed in the 7.x branch we can run its test in the backwards compatibility test suite.
nik9000
added a commit
that referenced
this pull request
Apr 28, 2021
Now that #72081 has landed in the 7.x branch we can run its test in the backwards compatibility test suite.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This prevents the
date_histogramfrom running out of memory allocatingempty buckets when you set the interval to something tiny like
secondsand aggregate over a very wide date range. Without this change we'd
allocate memory very quickly and throw and out of memory error, taking
down the node. With it we instead throw the standard "too many buckets"
error.
Relates to #71758