Skip to content

Incomplete remote response handler after transport close in integration tests #86249

@martijnvg

Description

@martijnvg

CI Link

https://gradle-enterprise.elastic.co/s/ma2rjwewae6hi

Repro line

./gradlew ':server:internalClusterTest' --tests "org.elasticsearch.action.termvectors.MultiTermVectorsIT" -Dtests.seed=D2D36E011E6F103B -Dtests.locale=en-US -Dtests.timezone=UTC -Druntime.java=17

Does it reproduce?

No

Applicable branches

master

Failure history

No response

Failure excerpt

I don't think the failed test suite (MultiTermVectorsIT) has anything to do with this failure.

Apr 28, 2022 7:07:07 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks |  
-- | --
  | 2> SEVERE: 73 threads leaked from SUITE scope at org.elasticsearch.action.termvectors.MultiTermVectorsIT: |  
  | 2>    1) Thread[id=308, name=elasticsearch[node_s3][generic][T#1], state=WAITING, group=TGRP-MultiTermVectorsIT] |  
  | 2>         at java.base@17.0.2/jdk.internal.misc.Unsafe.park(Native Method) |  
  | 2>         at java.base@17.0.2/java.util.concurrent.locks.LockSupport.park(LockSupport.java:341) |  
  | 2>         at java.base@17.0.2/java.util.concurrent.LinkedTransferQueue$Node.block(LinkedTransferQueue.java:470) |  
  | 2>         at java.base@17.0.2/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3463) |  
  | 2>         at java.base@17.0.2/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3434) |  
  | 2>         at java.base@17.0.2/java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:669) |  
  | 2>         at java.base@17.0.2/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:616) |  
  | 2>         at java.base@17.0.2/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1286) |  
  | 2>         at java.base@17.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1062) |  
  | 2>         at java.base@17.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122) |  
  | 2>         at java.base@17.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) |  
  | 2>         at java.base@17.0.2/java.lang.Thread.run(Thread.java:833)

And anther relevant snippet:

java.lang.RuntimeException: file handle leaks: [FileChannel(/dev/shm/elastic+elasticsearch+main+intake+multijob+part1/server/build/testrun/internalClusterTest/temp/org.elasticsearch.action.termvectors.MultiTermVectorsIT_D2D36E011E6F103B-001/tempDir-002/node_s2/_state/write.lock), FileChannel(/dev/shm/elastic+elasticsearch+main+intake+multijob+part1/server/build/testrun/internalClusterTest/temp/org.elasticsearch.action.termvectors.MultiTermVectorsIT_D2D36E011E6F103B-001/tempDir-002/node_s0/_state/write.lock), FileChannel(/dev/shm/elastic+elasticsearch+main+intake+multijob+part1/server/build/testrun/internalClusterTest/temp/org.elasticsearch.action.termvectors.MultiTermVectorsIT_D2D36E011E6F103B-001/tempDir-002/node_s1/node.lock), FileChannel(/dev/shm/elastic+elasticsearch+main+intake+multijob+part1/server/build/testrun/internalClusterTest/temp/org.elasticsearch.action.termvectors.MultiTermVectorsIT_D2D36E011E6F103B-001/tempDir-002/node_s2/_state/_m.cfs), FileChannel(/dev/shm/elastic+elasticsearch+main+intake+multijob+part1/server/build/testrun/internalClusterTest/temp/org.elasticsearch.action.termvectors.MultiTermVectorsIT_D2D36E011E6F103B-001/tempDir-002/node_s1/_state/write.lock), FileChannel(/dev/shm/elastic+elasticsearch+main+intake+multijob+part1/server/build/testrun/internalClusterTest/temp/org.elasticsearch.action.termvectors.MultiTermVectorsIT_D2D36E011E6F103B-001/tempDir-002/node_s3/node.lock), FileChannel(/dev/shm/elastic+elasticsearch+main+intake+multijob+part1/server/build/testrun/internalClusterTest/temp/org.elasticsearch.action.termvectors.MultiTermVectorsIT_D2D36E011E6F103B-001/tempDir-002/node_s2/node.lock), FileChannel(/dev/shm/elastic+elasticsearch+main+intake+multijob+part1/server/build/testrun/internalClusterTest/temp/org.elasticsearch.action.termvectors.MultiTermVectorsIT_D2D36E011E6F103B-001/tempDir-002/node_s0/node.lock)] |  
-- | --
  | at org.apache.lucene.tests.mockfile.LeakFS.onClose(LeakFS.java:63) |  
  | at org.apache.lucene.tests.mockfile.FilterFileSystem.close(FilterFileSystem.java:70) |  
  | at org.apache.lucene.tests.mockfile.FilterFileSystem.close(FilterFileSystem.java:71) |  
  | at org.apache.lucene.tests.util.TestRuleTemporaryFilesCleanup.afterAlways(TestRuleTemporaryFilesCleanup.java:223) |  
  | at com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.afterAlways(TestRuleAdapter.java:31) |  
  | at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43) |  
  | at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) |  
  | at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) |  
  | at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) |  
  | at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44) |  
  | at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60) |  
  | at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47) |  
  | at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) |  
  | at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:375) |  
  | at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:831) |  
  | at java.base/java.lang.Thread.run(Thread.java:833) |  
  |   |  
  | Caused by: |  
  | java.lang.Exception |  
  | at org.apache.lucene.tests.mockfile.LeakFS.onOpen(LeakFS.java:46) |  
  | at org.apache.lucene.tests.mockfile.HandleTrackingFS.callOpenHook(HandleTrackingFS.java:82) |  
  | at org.apache.lucene.tests.mockfile.HandleTrackingFS.newFileChannel(HandleTrackingFS.java:202) |  
  | at org.apache.lucene.tests.mockfile.HandleTrackingFS.newFileChannel(HandleTrackingFS.java:171) |  
  | at java.base/java.nio.channels.FileChannel.open(FileChannel.java:298) |  
  | at java.base/java.nio.channels.FileChannel.open(FileChannel.java:357) |  
  | at org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(NativeFSLockFactory.java:112) |  
  | at org.apache.lucene.store.FSLockFactory.obtainLock(FSLockFactory.java:43) |  
  | at org.apache.lucene.store.BaseDirectory.obtainLock(BaseDirectory.java:44) |  
  | at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:947) |  
  | at org.elasticsearch.gateway.PersistedClusterStateService.createIndexWriter(PersistedClusterStateService.java:254) |  
  | at org.elasticsearch.gateway.PersistedClusterStateService.createWriter(PersistedClusterStateService.java:216) |  
  | at org.elasticsearch.gateway.GatewayMetaState$LucenePersistedState.<init>(GatewayMetaState.java:448) |  
  | at org.elasticsearch.gateway.GatewayMetaState.start(GatewayMetaState.java:131) |  
  | at org.elasticsearch.node.Node.start(Node.java:1167) |  
  | at org.elasticsearch.test.InternalTestCluster$NodeAndClient.startNode(InternalTestCluster.java:982) |  
  | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) |  
  | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) |  
  | at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:714) |  
  | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) |  
  | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) |  
  | ... 1 more

Link to logs: https://gradle-enterprise.elastic.co/s/ma2rjwewae6hi/console-log#L9010

Metadata

Metadata

Assignees

Labels

:Distributed/NetworkHttp and internode communication implementations>test-failureTriaged test failures from CITeam:DistributedMeta label for distributed team.

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions