Limit memory consumption in test on overflow by andsel · Pull Request #17373 · elastic/logstash

andsel · 2025-03-20T10:08:37Z

Release notes

[rn:skip]

What does this PR do?

Updates only test code to be able to run a test that consumes big memory if:

the physical memory is bigger than the requested Java heap
JDK version is greater than or equal to 21.

The reason to limit the JDK version is that on 16GB machine the G1GC is more efficient than the one on previous JDKs and so let complete the test with 10GB heap, while in JDK 17 it consistently fails with OOM error.

Why is it important/What is the impact to the user?

N/A

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
~~[ ] I have made corresponding changes to the documentation~~
~~[ ] I have made corresponding change to the default configuration files (and/or docker env variables)~~
I have added tests that prove my fix is effective or that my feature works

Author's Checklist

tested on ubuntu 2024 16GB RAM VM with JDK 11, 17 and 21

How to test this PR locally

failing exhaustive: https://buildkite.com/elastic/logstash-exhaustive-tests-pipeline/builds/1564
success exhaustive: ~~https://buildkite.com/elastic/logstash-exhaustive-tests-pipeline/builds/1569~~ https://buildkite.com/elastic/logstash-exhaustive-tests-pipeline/builds/1594

Related issues

Relates Added test to verify the int overflow happen #17353

…p alive a reference to a 2GB Java String

mergify · 2025-03-20T10:09:15Z

This pull request does not have a backport label. Could you fix it @andsel? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit.
backport-8.x is the label to automatically backport to the 8.x branch.
If no backport is necessary, please add the backport-skip label

…to execute it

andsel · 2025-03-25T07:50:01Z

Why the test requires 10GB of JVM heap?

To explain why the test needs 10GB while it just allocate 2GB string it's needed to see all the allocations that are done.

With 4GB the test is not able to allocate space for the RubyString at https://github.com/elastic/logstash/pull/17373/files#diff-6c455e621a523fb4c88050efc394a576e69e0c0ef293b93ced25d8c4b6cadab4R161

Stack trace 4GB

org.logstash.common.BufferedTokenizerExtWithSizeLimitTest > givenTooLongInputExtractDoesntOverflow FAILED
    java.lang.OutOfMemoryError: Java heap space
        at java.base/java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:71)
        at java.base/java.nio.ByteBuffer.allocate(ByteBuffer.java:391)
        at java.base/java.nio.charset.CharsetEncoder.encode(CharsetEncoder.java:814)
        at java.base/java.nio.charset.Charset.encode(Charset.java:904)
        at org.jruby.RubyEncoding.doEncodeUTF8(RubyEncoding.java:223)
        at org.jruby.RubyString.encodeBytelist(RubyString.java:6820)
        at org.jruby.RubyString.<init>(RubyString.java:403)
        at org.jruby.RubyString.newString(RubyString.java:483)
        at org.jruby.Ruby.newString(Ruby.java:3516)
        at org.logstash.common.BufferedTokenizerExtWithSizeLimitTest.generateString(BufferedTokenizerExtWithSizeLimitTest.java:161)
        at org.logstash.common.BufferedTokenizerExtWithSizeLimitTest.givenTooLongInputExtractDoesntOverflow(BufferedTokenizerExtWithSizeLimitTest.java:147)

With 6GB the extract method is not able to reserve the space to convert the RubyString to Java String (this is a second copy)

Stack trace 6GB

org.logstash.common.BufferedTokenizerExtWithSizeLimitTest > givenTooLongInputExtractDoesntOverflow FAILED
    java.lang.OutOfMemoryError: Java heap space
        at java.base/java.nio.HeapCharBuffer.<init>(HeapCharBuffer.java:71)
        at java.base/java.nio.CharBuffer.allocate(CharBuffer.java:391)
        at java.base/java.nio.charset.CharsetDecoder.decode(CharsetDecoder.java:814)
        at java.base/java.nio.charset.Charset.decode(Charset.java:867)
        at org.jruby.RubyEncoding.decodeUTF8(RubyEncoding.java:308)
        at org.jruby.runtime.Helpers.decodeByteList(Helpers.java:2875)
        at org.jruby.RubyString.decodeString(RubyString.java:830)
        at org.jruby.RubyString.toString(RubyString.java:821)
        at java.base/java.lang.String.valueOf(String.java:4465)
        at java.base/java.lang.StringBuilder.append(StringBuilder.java:173)
        at org.logstash.common.BufferedTokenizerExt.extract(BufferedTokenizerExt.java:137)
        at org.logstash.common.BufferedTokenizerExtWithSizeLimitTest.givenTooLongInputExtractDoesntOverflow(BufferedTokenizerExtWithSizeLimitTest.java:148)

With 8GB the extract method is still not able to reserve the space to convert the RubyString to Java String (this is a second copy) and maybe during the conversion to UTF8 more inner copies are kept.

Stack trace 8GB

org.logstash.common.BufferedTokenizerExtWithSizeLimitTest > givenTooLongInputExtractDoesntOverflow FAILED
    java.lang.OutOfMemoryError: Java heap space
        at java.base/java.lang.StringUTF16.compress(StringUTF16.java:161)
        at java.base/java.lang.String.<init>(String.java:4768)
        at java.base/java.lang.String.<init>(String.java:303)
        at java.base/java.nio.HeapCharBuffer.toString(HeapCharBuffer.java:716)
        at java.base/java.nio.CharBuffer.toString(CharBuffer.java:1900)
        at org.jruby.RubyEncoding.decodeUTF8(RubyEncoding.java:308)
        at org.jruby.runtime.Helpers.decodeByteList(Helpers.java:2875)
        at org.jruby.RubyString.decodeString(RubyString.java:830)
        at org.jruby.RubyString.toString(RubyString.java:821)
        at java.base/java.lang.String.valueOf(String.java:4465)
        at java.base/java.lang.StringBuilder.append(StringBuilder.java:173)
        at org.logstash.common.BufferedTokenizerExt.extract(BufferedTokenizerExt.java:137)
        at org.logstash.common.BufferedTokenizerExtWithSizeLimitTest.givenTooLongInputExtractDoesntOverflow(BufferedTokenizerExtWithSizeLimitTest.java:148)

…ow test

elastic-sonarqube · 2025-03-25T08:06:25Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarQube

elasticmachine · 2025-03-25T08:22:56Z

💚 Build Succeeded

Buildkite Build
Commit: 4b3f827

History

💚 Build #2688 succeeded cff2dba
💚 Build #2683 succeeded 8c7b3f4
💚 Build #2676 succeeded 930a397
💚 Build #2674 succeeded f54c199
💚 Build #2673 succeeded b60ca78

cc @andsel

mashhurs

lgtm~

Updates only test code to be able to run a test that consumes big memory if: - the physical memory is bigger than the requested Java heap - JDK version is greater than or equal to 21. The reason to limit the JDK version is that on 16GB machine the G1GC is more efficient than the one on previous JDKs and so let complete the test with 10GB heap, while in JDK 17 it consistently fails with OOM error. (cherry picked from commit 075fdb4)

…17415) Updates only test code to be able to run a test that consumes big memory if: - the physical memory is bigger than the requested Java heap - JDK version is greater than or equal to 21. The reason to limit the JDK version is that on 16GB machine the G1GC is more efficient than the one on previous JDKs and so let complete the test with 10GB heap, while in JDK 17 it consistently fails with OOM error. (cherry picked from commit 075fdb4) Co-authored-by: Andrea Selva <selva.andre@gmail.com>

…#17414) Updates only test code to be able to run a test that consumes big memory if: - the physical memory is bigger than the requested Java heap - JDK version is greater than or equal to 21. The reason to limit the JDK version is that on 16GB machine the G1GC is more efficient than the one on previous JDKs and so let complete the test with 10GB heap, while in JDK 17 it consistently fails with OOM error. (cherry picked from commit 075fdb4) Co-authored-by: Andrea Selva <selva.andre@gmail.com>

…#17413) Updates only test code to be able to run a test that consumes big memory if: - the physical memory is bigger than the requested Java heap - JDK version is greater than or equal to 21. The reason to limit the JDK version is that on 16GB machine the G1GC is more efficient than the one on previous JDKs and so let complete the test with 10GB heap, while in JDK 17 it consistently fails with OOM error. (cherry picked from commit 075fdb4) Co-authored-by: Andrea Selva <selva.andre@gmail.com>

Updates only test code to be able to run a test that consumes big memory if: - the physical memory is bigger than the requested Java heap - JDK version is greater than or equal to 21. The reason to limit the JDK version is that on 16GB machine the G1GC is more efficient than the one on previous JDKs and so let complete the test with 10GB heap, while in JDK 17 it consistently fails with OOM error. (cherry picked from commit 075fdb4)

…17411) Updates only test code to be able to run a test that consumes big memory if: - the physical memory is bigger than the requested Java heap - JDK version is greater than or equal to 21. The reason to limit the JDK version is that on 16GB machine the G1GC is more efficient than the one on previous JDKs and so let complete the test with 10GB heap, while in JDK 17 it consistently fails with OOM error. (cherry picked from commit 075fdb4) Co-authored-by: Andrea Selva <selva.andre@gmail.com>

…#17412) Updates only test code to be able to run a test that consumes big memory if: - the physical memory is bigger than the requested Java heap - JDK version is greater than or equal to 21. The reason to limit the JDK version is that on 16GB machine the G1GC is more efficient than the one on previous JDKs and so let complete the test with 10GB heap, while in JDK 17 it consistently fails with OOM error. (cherry picked from commit 075fdb4) Co-authored-by: Andrea Selva <selva.andre@gmail.com>

Moved creation of RubyString inside generateString, this avoid to kee…

b60ca78

…p alive a reference to a 2GB Java String

andsel added 4 commits March 20, 2025 11:21

Ignore test if not enough physical memory is available

6b69872

Cleanup

f54c199

Println physicalVM memory

930a397

Ignore the test if JDK is not at least 21 that has proven to be able …

cff2dba

…to execute it

andsel force-pushed the fix/limit_heap_consumption_in_test branch from 8c7b3f4 to cff2dba Compare March 24, 2025 15:03

andsel self-assigned this Mar 24, 2025

andsel marked this pull request as ready for review March 24, 2025 16:31

Added link to the motivation why 10GB is needed to run the int overfl…

4b3f827

…ow test

mashhurs approved these changes Mar 25, 2025

View reviewed changes

andsel merged commit 075fdb4 into elastic:main Mar 26, 2025
9 checks passed

andsel added backport-active-9 Automated backport with mergify to all the active 9.[0-9]+ branches backport-active-8 Automated backport with mergify to all the active 8.[0-9]+ branches labels Mar 26, 2025

mergify bot mentioned this pull request Mar 26, 2025

[8.x] Limit memory consumption in test on overflow (backport #17373) #17411

Merged

4 tasks

This was referenced Mar 26, 2025

[8.18] Limit memory consumption in test on overflow (backport #17373) #17412

Merged

[8.17] Limit memory consumption in test on overflow (backport #17373) #17413

Merged

mergify bot mentioned this pull request Mar 26, 2025

[8.16] Limit memory consumption in test on overflow (backport #17373) #17414

Merged

4 tasks

mergify bot mentioned this pull request Mar 26, 2025

[9.0] Limit memory consumption in test on overflow (backport #17373) #17415

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Limit memory consumption in test on overflow#17373

Limit memory consumption in test on overflow#17373
andsel merged 6 commits intoelastic:mainfrom
andsel:fix/limit_heap_consumption_in_test

andsel commented Mar 20, 2025 •

edited

Loading

Uh oh!

mergify bot commented Mar 20, 2025

Uh oh!

andsel commented Mar 25, 2025

Uh oh!

elastic-sonarqube bot commented Mar 25, 2025

Uh oh!

elasticmachine commented Mar 25, 2025

Uh oh!

mashhurs left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

andsel commented Mar 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Release notes

What does this PR do?

Why is it important/What is the impact to the user?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Uh oh!

mergify bot commented Mar 20, 2025

Uh oh!

andsel commented Mar 25, 2025

Why the test requires 10GB of JVM heap?

Uh oh!

elastic-sonarqube bot commented Mar 25, 2025

Quality Gate passed

Uh oh!

elasticmachine commented Mar 25, 2025

💚 Build Succeeded

History

Uh oh!

mashhurs left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

andsel commented Mar 20, 2025 •

edited

Loading