-
Notifications
You must be signed in to change notification settings - Fork 25.8k
[CI] IndexShardTests testIndexCheckOnStartup failure: AssertionError #90144
Copy link
Copy link
Closed
Labels
:Distributed/DistributedA catch all label for anything in the Distributed Area. Please avoid if you can.A catch all label for anything in the Distributed Area. Please avoid if you can.>test-failureTriaged test failures from CITriaged test failures from CITeam:DistributedMeta label for distributed team.Meta label for distributed team.
Description
CI Link
https://gradle-enterprise.elastic.co/s/t6gxc3xhkugpm
Repro line
./gradlew ':server:test' --tests "org.elasticsearch.index.shard.IndexShardTests.testIndexCheckOnStartup" -Dtests.seed=C312D6F77D4FB3D2 -Dtests.locale=ar -Dtests.timezone=Asia/Harbin -Druntime.java=18
Does it reproduce?
Yes
Applicable branches
main
Failure history
No response
Failure excerpt
This one looks like a seed-dependent edge case.
org.elasticsearch.index.shard.IndexShardTests > testIndexCheckOnStartup FAILED
java.lang.AssertionError: bloom_filter=BloomFilter[field=_id, startFilePointer=58, bloomFilterSize=191], max_docs=19
at __randomizedtesting.SeedInfo.seed([C312D6F77D4FB3D2:C2B4035E77C9710A]:0)
at org.elasticsearch.index.codec.bloomfilter.ES85BloomFilterPostingsFormat$FieldsReader.<init>(ES85BloomFilterPostingsFormat.java:309)
at org.elasticsearch.index.codec.bloomfilter.ES85BloomFilterPostingsFormat.fieldsProducer(ES85BloomFilterPostingsFormat.java:104)
at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.<init>(PerFieldPostingsFormat.java:329)
at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.fieldsProducer(PerFieldPostingsFormat.java:391)
at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:118)
at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:91)
at org.apache.lucene.index.CheckIndex.testSegment(CheckIndex.java:928)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:714)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:552)
at org.apache.lucene.tests.util.TestUtil.checkIndex(TestUtil.java:343)
at org.apache.lucene.tests.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:909)
at org.apache.lucene.store.FilterDirectory.close(FilterDirectory.java:111)
at org.elasticsearch.index.shard.IndexShardTests.testIndexCheckOnStartup(IndexShardTests.java:3472)
1> [2022-09-20T03:06:35,180][INFO ][o.e.i.s.IndexShardTests ] [testIndexCheckOnStartup] before test
1> [2022-09-20T03:06:40,594][INFO ][o.e.t.CorruptionUtils ] [testIndexCheckOnStartup] Corrupting file -- flipping at position 603 from ffffffbe to ffffffbf file: _0.cfs
1> [2022-09-20T03:06:40,596][INFO ][o.e.t.CorruptionUtils ] [testIndexCheckOnStartup] Checksum before: [310573513] after: [4230669878] checksum value after corruption: 310573513] file: _0.cfs length: 2403
1> [2022-09-20T03:06:40,691][WARN ][o.e.i.s.IndexShard ] [testIndexCheckOnStartup] [index][0] performing expensive diagnostic checks during shard startup [index.shard.check_on_startup=checksum]; these checks should only be enabled temporarily, you must remove this index setting as soon as possible
1> [2022-09-20T03:06:40,720][INFO ][o.e.i.s.IndexShard ] [testIndexCheckOnStartup] [index][0] check index [ok]: checksum check passed on [_0.cfe]
1> [2022-09-20T03:06:40,721][INFO ][o.e.i.s.IndexShard ] [testIndexCheckOnStartup] [index][0] check index [ok]: checksum check passed on [_0.si]
1> [2022-09-20T03:06:40,724][WARN ][o.e.i.s.IndexShard ] [testIndexCheckOnStartup] [index][0] check index [failure]: checksum failed on [_0.cfs]
1> org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=1282f9c9 actual=68a1558d (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(MMapIndexInput(path="/Users/wbrafford/work/repos/elasticsearch/server/build/testrun/test/temp/org.elasticsearch.index.shard.IndexShardTests_C312D6F77D4FB3D2-003/tempDir-002/indices/_na_/0/index/_0.cfs"))))
1> at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:440) ~[lucene-core-9.4.0-snapshot-923a9f800ae.jar:9.4.0-snapshot-923a9f800ae 923a9f800aef4f376eb1978c02e94ca6bacc5a5a - jenkins - 2022-08-16 19:10:52]
1> at org.apache.lucene.codecs.CodecUtil.checksumEntireFile(CodecUtil.java:619) ~[lucene-core-9.4.0-snapshot-923a9f800ae.jar:9.4.0-snapshot-923a9f800ae 923a9f800aef4f376eb1978c02e94ca6bacc5a5a - jenkins - 2022-08-16 19:10:52]
1> at org.elasticsearch.index.store.Store.checkIntegrity(Store.java:542) ~[main/:?]
1> at org.elasticsearch.index.shard.IndexShard.doCheckIndex(IndexShard.java:2924) ~[main/:?]
1> at org.elasticsearch.index.shard.IndexShard.checkIndex(IndexShard.java:2895) ~[main/:?]
1> at org.elasticsearch.index.shard.IndexShard.maybeCheckIndex(IndexShard.java:2885) ~[main/:?]
1> at org.elasticsearch.index.shard.IndexShard.openEngineAndRecoverFromTranslog(IndexShard.java:1885) ~[main/:?]
1> at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:463) ~[main/:?]
1> at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:90) ~[main/:?]
1> at org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:462) ~[main/:?]
1> at org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:88) ~[main/:?]
1> at org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:2248) ~[main/:?]
1> at org.elasticsearch.index.shard.IndexShardTestCase.recoverFromStore(IndexShardTestCase.java:962) ~[framework-8.5.0-SNAPSHOT.jar:?]
1> at org.elasticsearch.index.shard.IndexShardTestCase.recoverShardFromStore(IndexShardTestCase.java:628) ~[framework-8.5.0-SNAPSHOT.jar:?]
1> at org.elasticsearch.index.shard.IndexShardTestCase.newStartedShard(IndexShardTestCase.java:589) ~[framework-8.5.0-SNAPSHOT.jar:?]
1> at org.elasticsearch.index.shard.IndexShardTests.lambda$testIndexCheckOnStartup$79(IndexShardTests.java:3456) ~[test/:?]
1> at org.apache.lucene.tests.util.LuceneTestCase._expectThrows(LuceneTestCase.java:3012) ~[lucene-test-framework-9.4.0-snapshot-923a9f800ae.jar:9.4.0-snapshot-923a9f800ae 923a9f800aef4f376eb1978c02e94ca6bacc5a5a - jenkins - 2022-08-16 19:10:52]
1> at org.apache.lucene.tests.util.LuceneTestCase.expectThrows(LuceneTestCase.java:2849) ~[lucene-test-framework-9.4.0-snapshot-923a9f800ae.jar:9.4.0-snapshot-923a9f800ae 923a9f800aef4f376eb1978c02e94ca6bacc5a5a - jenkins - 2022-08-16 19:10:52]
1> at org.apache.lucene.tests.util.LuceneTestCase.expectThrows(LuceneTestCase.java:2840) ~[lucene-test-framework-9.4.0-snapshot-923a9f800ae.jar:9.4.0-snapshot-923a9f800ae 923a9f800aef4f376eb1978c02e94ca6bacc5a5a - jenkins - 2022-08-16 19:10:52]
1> at org.elasticsearch.index.shard.IndexShardTests.testIndexCheckOnStartup(IndexShardTests.java:3454) ~[test/:?]
1> at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104) ~[?:?]
1> at java.lang.reflect.Method.invoke(Method.java:577) ~[?:?]
1> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758) ~[randomizedtesting-runner-2.8.0.jar:?]
1> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946) ~[randomizedtesting-runner-2.8.0.jar:?]
1> at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982) ~[randomizedtesting-runner-2.8.0.jar:?]
1> at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996) ~[randomizedtesting-runner-2.8.0.jar:?]
1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.8.0.jar:?]
1> at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:44) ~[lucene-test-framework-9.4.0-snapshot-923a9f800ae.jar:9.4.0-snapshot-923a9f800ae 923a9f800aef4f376eb1978c02e94ca6bacc5a5a - jenkins - 2022-08-16 19:10:52]
1> at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) ~[lucene-test-framework-9.4.0-snapshot-923a9f800ae.jar:9.4.0-snapshot-923a9f800ae 923a9f800aef4f376eb1978c02e94ca6bacc5a5a - jenkins - 2022-08-16 19:10:52]
1> at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45) ~[lucene-test-framework-9.4.0-snapshot-923a9f800ae.jar:9.4.0-snapshot-923a9f800ae 923a9f800aef4f376eb1978c02e94ca6bacc5a5a - jenkins - 2022-08-16 19:10:52]
1> at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60) ~[lucene-test-framework-9.4.0-snapshot-923a9f800ae.jar:9.4.0-snapshot-923a9f800ae 923a9f800aef4f376eb1978c02e94ca6bacc5a5a - jenkins - 2022-08-16 19:10:52]
1> at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44) ~[lucene-test-framework-9.4.0-snapshot-923a9f800ae.jar:9.4.0-snapshot-923a9f800ae 923a9f800aef4f376eb1978c02e94ca6bacc5a5a - jenkins - 2022-08-16 19:10:52]
1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.8.0.jar:?]
1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390) ~[randomizedtesting-runner-2.8.0.jar:?]
1> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843) ~[randomizedtesting-runner-2.8.0.jar:?]
1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490) ~[randomizedtesting-runner-2.8.0.jar:?]
1> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955) ~[randomizedtesting-runner-2.8.0.jar:?]
1> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840) ~[randomizedtesting-runner-2.8.0.jar:?]
1> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891) ~[randomizedtesting-runner-2.8.0.jar:?]
1> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902) ~[randomizedtesting-runner-2.8.0.jar:?]
1> at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) ~[lucene-test-framework-9.4.0-snapshot-923a9f800ae.jar:9.4.0-snapshot-923a9f800ae 923a9f800aef4f376eb1978c02e94ca6bacc5a5a - jenkins - 2022-08-16 19:10:52]
1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.8.0.jar:?]
1> at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38) ~[lucene-test-framework-9.4.0-snapshot-923a9f800ae.jar:9.4.0-snapshot-923a9f800ae 923a9f800aef4f376eb1978c02e94ca6bacc5a5a - jenkins - 2022-08-16 19:10:52]
1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) ~[randomizedtesting-runner-2.8.0.jar:?]
1> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) ~[randomizedtesting-runner-2.8.0.jar:?]
1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.8.0.jar:?]
1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.8.0.jar:?]
1> at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) ~[lucene-test-framework-9.4.0-snapshot-923a9f800ae.jar:9.4.0-snapshot-923a9f800ae 923a9f800aef4f376eb1978c02e94ca6bacc5a5a - jenkins - 2022-08-16 19:10:52]
1> at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) ~ 1> [lucene-test-framework-9.4.0-snapshot-923a9f800ae.jar:9.4.0-snapshot-923a9f800ae 923a9f800aef4f376eb1978c02e94ca6bacc5a5a - jenkins - 2022-08-16 19:10:52]
1> at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44) ~[lucene-test-framework-9.4.0-snapshot-923a9f800ae.jar:9.4.0-snapshot-923a9f800ae 923a9f800aef4f376eb1978c02e94ca6bacc5a5a - jenkins - 2022-08-16 19:10:52]
1> at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60) ~[lucene-test-framework-9.4.0-snapshot-923a9f800ae.jar:9.4.0-snapshot-923a9f800ae 923a9f800aef4f376eb1978c02e94ca6bacc5a5a - jenkins - 2022-08-16 19:10:52]
1> at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47) ~[lucene-test-framework-9.4.0-snapshot-923a9f800ae.jar:9.4.0-snapshot-923a9f800ae 923a9f800aef4f376eb1978c02e94ca6bacc5a5a - jenkins - 2022-08-16 19:10:52]
1> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.8.0.jar:?]
1> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390) ~[randomizedtesting-runner-2.8.0.jar:?]
1> at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850) ~[randomizedtesting-runner-2.8.0.jar:?]
1> at java.lang.Thread.run(Thread.java:833) ~[?:?]
1> [2022-09-20T03:06:40,749][INFO ][o.e.i.s.IndexShard ] [testIndexCheckOnStartup] [index][0] check index [ok]: checksum check passed on [segments_3]
1> [2022-09-20T03:06:40,824][INFO ][o.e.i.s.IndexShardTests ] [testIndexCheckOnStartup] after test
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
:Distributed/DistributedA catch all label for anything in the Distributed Area. Please avoid if you can.A catch all label for anything in the Distributed Area. Please avoid if you can.>test-failureTriaged test failures from CITriaged test failures from CITeam:DistributedMeta label for distributed team.Meta label for distributed team.
Type
Fields
Give feedbackNo fields configured for issues without a type.