[BUG] org.opensearch.indices.store.IndicesStoreIntegrationIT.testShardActiveElseWhere is flaky #12788
Description
Describe the bug
The test case org.opensearch.indices.store.IndicesStoreIntegrationIT.testShardActiveElseWhere {p0={"cluster.remote_store.state.enabled":"true"}} is flaky:
java.lang.AssertionError: shard [test][4] is not locked
at __randomizedtesting.SeedInfo.seed([67FB6E95E72E6169]:0)
at org.opensearch.env.NodeEnvironment.deleteShardDirectoryUnderLock(NodeEnvironment.java:587)
at org.opensearch.indices.IndicesService.deleteShardStore(IndicesService.java:1245)
at org.opensearch.index.IndexService.onShardClose(IndexService.java:678)
at org.opensearch.index.IndexService$StoreCloseListener.accept(IndexService.java:801)
at org.opensearch.index.IndexService$StoreCloseListener.accept(IndexService.java:788)
at org.opensearch.index.store.Store.closeInternal(Store.java:573)
at org.opensearch.index.store.Store$1.closeInternal(Store.java:194)
at org.opensearch.common.util.concurrent.AbstractRefCounted.decRef(AbstractRefCounted.java:78)
at org.opensearch.index.store.Store.decRef(Store.java:546)
at org.opensearch.index.engine.InternalEngine.refresh(InternalEngine.java:1777)
at org.opensearch.index.engine.InternalEngine.maybeRefresh(InternalEngine.java:1753)
at org.opensearch.index.shard.IndexShard.scheduledRefresh(IndexShard.java:4498)
at org.opensearch.index.IndexService.maybeRefreshEngine(IndexService.java:1026)
at org.opensearch.index.IndexService$AsyncRefreshTask.runInternal(IndexService.java:1170)
at org.opensearch.common.util.concurrent.AbstractAsyncTask.run(AbstractAsyncTask.java:159)
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:854)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)
Related component
Indexing
To Reproduce
./gradlew ':server:internalClusterTest' --tests "org.opensearch.indices.store.IndicesStoreIntegrationIT" -Dtests.method="testShardActiveElseWhere {p0={"cluster.remote_store.state.enabled":"true"}}" -Dtests.seed=67FB6E95E72E6169
Expected behavior
The test must always pass
Additional Details
Plugins
Standard
Screenshots
If applicable, add screenshots to help explain your problem.
Host/Environment (please complete the following information):
- CI
Additional context