Skip to content

[CI] IndexShardTests testIndexCheckOnStartup failure: AssertionError #90144

@williamrandolph

Description

@williamrandolph

CI Link

https://gradle-enterprise.elastic.co/s/t6gxc3xhkugpm

Repro line

./gradlew ':server:test' --tests "org.elasticsearch.index.shard.IndexShardTests.testIndexCheckOnStartup" -Dtests.seed=C312D6F77D4FB3D2 -Dtests.locale=ar -Dtests.timezone=Asia/Harbin -Druntime.java=18

Does it reproduce?

Yes

Applicable branches

main

Failure history

No response

Failure excerpt

This one looks like a seed-dependent edge case.

org.elasticsearch.index.shard.IndexShardTests > testIndexCheckOnStartup FAILED
    java.lang.AssertionError: bloom_filter=BloomFilter[field=_id, startFilePointer=58, bloomFilterSize=191], max_docs=19
        at __randomizedtesting.SeedInfo.seed([C312D6F77D4FB3D2:C2B4035E77C9710A]:0)
        at org.elasticsearch.index.codec.bloomfilter.ES85BloomFilterPostingsFormat$FieldsReader.<init>(ES85BloomFilterPostingsFormat.java:309)
        at org.elasticsearch.index.codec.bloomfilter.ES85BloomFilterPostingsFormat.fieldsProducer(ES85BloomFilterPostingsFormat.java:104)
        at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.<init>(PerFieldPostingsFormat.java:329)
        at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.fieldsProducer(PerFieldPostingsFormat.java:391)
        at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:118)
        at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:91)
        at org.apache.lucene.index.CheckIndex.testSegment(CheckIndex.java:928)
        at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:714)
        at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:552)
        at org.apache.lucene.tests.util.TestUtil.checkIndex(TestUtil.java:343)
        at org.apache.lucene.tests.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:909)
        at org.apache.lucene.store.FilterDirectory.close(FilterDirectory.java:111)
        at org.elasticsearch.index.shard.IndexShardTests.testIndexCheckOnStartup(IndexShardTests.java:3472)

  1> [2022-09-20T03:06:35,180][INFO ][o.e.i.s.IndexShardTests  ] [testIndexCheckOnStartup] before test
  1> [2022-09-20T03:06:40,594][INFO ][o.e.t.CorruptionUtils    ] [testIndexCheckOnStartup] Corrupting file --  flipping at position 603 from ffffffbe to ffffffbf file: _0.cfs
  1> [2022-09-20T03:06:40,596][INFO ][o.e.t.CorruptionUtils    ] [testIndexCheckOnStartup] Checksum before: [310573513] after: [4230669878] checksum value after corruption: 310573513] file: _0.cfs length: 2403
  1> [2022-09-20T03:06:40,691][WARN ][o.e.i.s.IndexShard       ] [testIndexCheckOnStartup] [index][0] performing expensive diagnostic checks during shard startup [index.shard.check_on_startup=checksum]; these checks should only be enabled temporarily, you must remove this index setting as soon as possible
  1> [2022-09-20T03:06:40,720][INFO ][o.e.i.s.IndexShard       ] [testIndexCheckOnStartup] [index][0] check index [ok]: checksum check passed on [_0.cfe]
  1> [2022-09-20T03:06:40,721][INFO ][o.e.i.s.IndexShard       ] [testIndexCheckOnStartup] [index][0] check index [ok]: checksum check passed on [_0.si]
  1> [2022-09-20T03:06:40,724][WARN ][o.e.i.s.IndexShard       ] [testIndexCheckOnStartup] [index][0] check index [failure]: checksum failed on [_0.cfs]
  1> org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=1282f9c9 actual=68a1558d (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(MMapIndexInput(path="/Users/wbrafford/work/repos/elasticsearch/server/build/testrun/test/temp/org.elasticsearch.index.shard.IndexShardTests_C312D6F77D4FB3D2-003/tempDir-002/indices/_na_/0/index/_0.cfs"))))
  1>    at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:440) ~[lucene-core-9.4.0-snapshot-923a9f800ae.jar:9.4.0-snapshot-923a9f800ae 923a9f800aef4f376eb1978c02e94ca6bacc5a5a - jenkins - 2022-08-16 19:10:52]
  1>    at org.apache.lucene.codecs.CodecUtil.checksumEntireFile(CodecUtil.java:619) ~[lucene-core-9.4.0-snapshot-923a9f800ae.jar:9.4.0-snapshot-923a9f800ae 923a9f800aef4f376eb1978c02e94ca6bacc5a5a - jenkins - 2022-08-16 19:10:52]
  1>    at org.elasticsearch.index.store.Store.checkIntegrity(Store.java:542) ~[main/:?]
  1>    at org.elasticsearch.index.shard.IndexShard.doCheckIndex(IndexShard.java:2924) ~[main/:?]
  1>    at org.elasticsearch.index.shard.IndexShard.checkIndex(IndexShard.java:2895) ~[main/:?]
  1>    at org.elasticsearch.index.shard.IndexShard.maybeCheckIndex(IndexShard.java:2885) ~[main/:?]
  1>    at org.elasticsearch.index.shard.IndexShard.openEngineAndRecoverFromTranslog(IndexShard.java:1885) ~[main/:?]
  1>    at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:463) ~[main/:?]
  1>    at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:90) ~[main/:?]
  1>    at org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:462) ~[main/:?]
  1>    at org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:88) ~[main/:?]
  1>    at org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:2248) ~[main/:?]
  1>    at org.elasticsearch.index.shard.IndexShardTestCase.recoverFromStore(IndexShardTestCase.java:962) ~[framework-8.5.0-SNAPSHOT.jar:?]
  1>    at org.elasticsearch.index.shard.IndexShardTestCase.recoverShardFromStore(IndexShardTestCase.java:628) ~[framework-8.5.0-SNAPSHOT.jar:?]
  1>    at org.elasticsearch.index.shard.IndexShardTestCase.newStartedShard(IndexShardTestCase.java:589) ~[framework-8.5.0-SNAPSHOT.jar:?]
  1>    at org.elasticsearch.index.shard.IndexShardTests.lambda$testIndexCheckOnStartup$79(IndexShardTests.java:3456) ~[test/:?]
  1>    at org.apache.lucene.tests.util.LuceneTestCase._expectThrows(LuceneTestCase.java:3012) ~[lucene-test-framework-9.4.0-snapshot-923a9f800ae.jar:9.4.0-snapshot-923a9f800ae 923a9f800aef4f376eb1978c02e94ca6bacc5a5a - jenkins - 2022-08-16 19:10:52]
  1>    at org.apache.lucene.tests.util.LuceneTestCase.expectThrows(LuceneTestCase.java:2849) ~[lucene-test-framework-9.4.0-snapshot-923a9f800ae.jar:9.4.0-snapshot-923a9f800ae 923a9f800aef4f376eb1978c02e94ca6bacc5a5a - jenkins - 2022-08-16 19:10:52]
  1>    at org.apache.lucene.tests.util.LuceneTestCase.expectThrows(LuceneTestCase.java:2840) ~[lucene-test-framework-9.4.0-snapshot-923a9f800ae.jar:9.4.0-snapshot-923a9f800ae 923a9f800aef4f376eb1978c02e94ca6bacc5a5a - jenkins - 2022-08-16 19:10:52]
  1>    at org.elasticsearch.index.shard.IndexShardTests.testIndexCheckOnStartup(IndexShardTests.java:3454) ~[test/:?]
  1>    at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104) ~[?:?]
  1>    at java.lang.reflect.Method.invoke(Method.java:577) ~[?:?]
  1>    at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758) ~[randomizedtesting-runner-2.8.0.jar:?]
  1>    at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946) ~[randomizedtesting-runner-2.8.0.jar:?]
  1>    at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982) ~[randomizedtesting-runner-2.8.0.jar:?]
  1>    at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996) ~[randomizedtesting-runner-2.8.0.jar:?]
  1>    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.8.0.jar:?]
  1>    at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:44) ~[lucene-test-framework-9.4.0-snapshot-923a9f800ae.jar:9.4.0-snapshot-923a9f800ae 923a9f800aef4f376eb1978c02e94ca6bacc5a5a - jenkins - 2022-08-16 19:10:52]
  1>    at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) ~[lucene-test-framework-9.4.0-snapshot-923a9f800ae.jar:9.4.0-snapshot-923a9f800ae 923a9f800aef4f376eb1978c02e94ca6bacc5a5a - jenkins - 2022-08-16 19:10:52]
  1>    at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45) ~[lucene-test-framework-9.4.0-snapshot-923a9f800ae.jar:9.4.0-snapshot-923a9f800ae 923a9f800aef4f376eb1978c02e94ca6bacc5a5a - jenkins - 2022-08-16 19:10:52]
  1>    at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60) ~[lucene-test-framework-9.4.0-snapshot-923a9f800ae.jar:9.4.0-snapshot-923a9f800ae 923a9f800aef4f376eb1978c02e94ca6bacc5a5a - jenkins - 2022-08-16 19:10:52]
  1>    at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44) ~[lucene-test-framework-9.4.0-snapshot-923a9f800ae.jar:9.4.0-snapshot-923a9f800ae 923a9f800aef4f376eb1978c02e94ca6bacc5a5a - jenkins - 2022-08-16 19:10:52]
  1>    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.8.0.jar:?]
  1>    at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390) ~[randomizedtesting-runner-2.8.0.jar:?]
  1>    at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843) ~[randomizedtesting-runner-2.8.0.jar:?]
  1>    at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490) ~[randomizedtesting-runner-2.8.0.jar:?]
  1>    at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955) ~[randomizedtesting-runner-2.8.0.jar:?]
  1>    at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840) ~[randomizedtesting-runner-2.8.0.jar:?]
  1>    at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891) ~[randomizedtesting-runner-2.8.0.jar:?]
  1>    at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902) ~[randomizedtesting-runner-2.8.0.jar:?]
  1>    at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) ~[lucene-test-framework-9.4.0-snapshot-923a9f800ae.jar:9.4.0-snapshot-923a9f800ae 923a9f800aef4f376eb1978c02e94ca6bacc5a5a - jenkins - 2022-08-16 19:10:52]
  1>    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.8.0.jar:?]
  1>    at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38) ~[lucene-test-framework-9.4.0-snapshot-923a9f800ae.jar:9.4.0-snapshot-923a9f800ae 923a9f800aef4f376eb1978c02e94ca6bacc5a5a - jenkins - 2022-08-16 19:10:52]
  1>    at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) ~[randomizedtesting-runner-2.8.0.jar:?]
  1>    at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) ~[randomizedtesting-runner-2.8.0.jar:?]
  1>    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.8.0.jar:?]
  1>    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.8.0.jar:?]
  1>    at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) ~[lucene-test-framework-9.4.0-snapshot-923a9f800ae.jar:9.4.0-snapshot-923a9f800ae 923a9f800aef4f376eb1978c02e94ca6bacc5a5a - jenkins - 2022-08-16 19:10:52]
  1>    at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) ~  1> [lucene-test-framework-9.4.0-snapshot-923a9f800ae.jar:9.4.0-snapshot-923a9f800ae 923a9f800aef4f376eb1978c02e94ca6bacc5a5a - jenkins - 2022-08-16 19:10:52]
  1>    at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44) ~[lucene-test-framework-9.4.0-snapshot-923a9f800ae.jar:9.4.0-snapshot-923a9f800ae 923a9f800aef4f376eb1978c02e94ca6bacc5a5a - jenkins - 2022-08-16 19:10:52]
  1>    at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60) ~[lucene-test-framework-9.4.0-snapshot-923a9f800ae.jar:9.4.0-snapshot-923a9f800ae 923a9f800aef4f376eb1978c02e94ca6bacc5a5a - jenkins - 2022-08-16 19:10:52]
  1>    at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47) ~[lucene-test-framework-9.4.0-snapshot-923a9f800ae.jar:9.4.0-snapshot-923a9f800ae 923a9f800aef4f376eb1978c02e94ca6bacc5a5a - jenkins - 2022-08-16 19:10:52]
  1>    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.8.0.jar:?]
  1>    at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390) ~[randomizedtesting-runner-2.8.0.jar:?]
  1>    at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850) ~[randomizedtesting-runner-2.8.0.jar:?]
  1>    at java.lang.Thread.run(Thread.java:833) ~[?:?]
  1> [2022-09-20T03:06:40,749][INFO ][o.e.i.s.IndexShard       ] [testIndexCheckOnStartup] [index][0] check index [ok]: checksum check passed on [segments_3]
  1> [2022-09-20T03:06:40,824][INFO ][o.e.i.s.IndexShardTests  ] [testIndexCheckOnStartup] after test

Metadata

Metadata

Labels

:Distributed/DistributedA catch all label for anything in the Distributed Area. Please avoid if you can.>test-failureTriaged test failures from CITeam:DistributedMeta label for distributed team.

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions