./gradlew ":test:external-modules:test-esql-heap-attack:javaRestTest" --tests "org.elasticsearch.xpack.esql.heap_attack.HeapAttackSubqueryIT.testGiantTextFieldInSubqueryIntermediateResultsWithSort" -Dtests.seed=905A1F67BF1059DD -Dtests.locale=lkt-Latn-US -Dtests.timezone=Asia/Vientiane -Druntime.java=21 -Dtests.fips.enabled=true
Class Name | Shallow Heap | Retained Heap | Percentage
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| | |
org.elasticsearch.compute.operator.topn.TopNOperator @ 0xe8e3b6b8 | 96 | 229,562,024 | 54.29%
org.elasticsearch.common.util.PageCacheRecycler @ 0xe2fe1a58 | 24 | 48,855,048 | 11.55%
org.elasticsearch.common.util.concurrent.EsExecutors$EsThread @ 0xe34f96a0 elasticsearch[test-cluster-0][esql_worker][T#4] Thread| 112 | 36,304,760 | 8.59%
org.elasticsearch.index.mapper.BlockSourceReader$BytesRefs @ 0xe8e3b570 | 40 | 15,728,768 | 3.72%
jdk.internal.loader.ClassLoaders$AppClassLoader @ 0xe0007930 JNI Global | 96 | 3,429,280 | 0.81%
class java.time.zone.ZoneRulesProvider @ 0xe2b1ea58 System Class | 16 | 3,182,792 | 0.75%
org.elasticsearch.bootstrap.Elasticsearch @ 0xe48f6678 | 32 | 2,769,896 | 0.66%
jdk.internal.loader.Loader @ 0xe0021ac8 | 104 | 2,030,104 | 0.48%
java.util.zip.ZipFile$Source @ 0xe14ca430 | 80 | 1,984,976 | 0.47%
org.elasticsearch.ingest.IngestService @ 0xe3fb0460 | 72 | 1,601,816 | 0.38%
org.elasticsearch.painless.PainlessScriptEngine @ 0xe33d7be8 | 24 | 1,558,208 | 0.37%
class java.lang.invoke.MethodType @ 0xe0a5a908 System Class | 48 | 1,445,456 | 0.34%
java.util.zip.ZipFile$Source @ 0xe0078be0 | 80 | 1,315,088 | 0.31%
jdk.internal.loader.Loader @ 0xe282f050 | 104 | 1,260,224 | 0.30%
org.elasticsearch.gateway.GatewayMetaState$LucenePersistedState @ 0xe6440c78 | 40 | 1,095,728 | 0.26%
org.apache.http.conn.util.PublicSuffixMatcher @ 0xe2dbb778 | 24 | 884,480 | 0.21%
java.net.URLClassLoader @ 0xe23d2178 | 88 | 854,600 | 0.20%
jdk.internal.loader.Loader @ 0xe247b540 | 104 | 777,488 | 0.18%
org.bouncycastle.jsse.provider.ContextData @ 0xe2d10ed8 | 48 | 669,344 | 0.16%
jdk.internal.loader.Loader @ 0xe21aa248 | 104 | 668,472 | 0.16%
class sun.util.calendar.ZoneInfoFile @ 0xe00ebe78 System Class | 112 | 660,616 | 0.16%
org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager @ 0xe4513430 | 32 | 653,872 | 0.15%
org.elasticsearch.xpack.enrich.action.TransportEnrichReindexAction @ 0xe6269150 | 48 | 653,824 | 0.15%
org.elasticsearch.reindex.ReindexSslConfig @ 0xe46ddd20 | 24 | 652,104 | 0.15%
jdk.internal.loader.Loader @ 0xe233a220 | 104 | 621,216 | 0.15%
java.util.zip.ZipFile$Source @ 0xe00a2450 | 80 | 603,128 | 0.14%
jdk.internal.loader.Loader @ 0xe19279b0 | 104 | 598,992 | 0.14%
java.util.zip.ZipFile$Source @ 0xe1eb68c8 | 80 | 572,264 | 0.14%
Total: 28 of 161,531 entries; 161,503 more | 5,080,352 | 422,821,400 | 100.00%
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Seems particularly to affect the FIPS jobs, although it is also failing a reasonable amount in the non-FIPS tests.
CI Link
https://gradle-enterprise.elastic.co/s/xqq6uq6f2m7hk
Repro line
./gradlew ":test:external-modules:test-esql-heap-attack:javaRestTest" --tests "org.elasticsearch.xpack.esql.heap_attack.HeapAttackSubqueryIT.testGiantTextFieldInSubqueryIntermediateResultsWithSort" -Dtests.seed=905A1F67BF1059DD -Dtests.locale=lkt-Latn-US -Dtests.timezone=Asia/Vientiane -Druntime.java=21 -Dtests.fips.enabled=true
Does it reproduce?
No
Applicable branches
main
Failure history
https://es-delivery-stats.elastic.dev/app/r/s/Gn2wz
Failure excerpt
A node is going OOM and most of the heap is in the TopNOperator
Seems particularly to affect the FIPS jobs, although it is also failing a reasonable amount in the non-FIPS tests.