logs: https://www.google.com/url?q=https%3A%2F%2Felasticsearch-ci.elastic.co%2Fjob%2Felastic%2Belasticsearch%2B6.x%2Bmultijob-unix-compatibility%2Fos%3Damazon%2F42%2Fconsole&sa=D&sntz=1&usg=AFQjCNFFG4hg9naqoVYMqXTq8OD-lszrlw

It is reproducible with:

./gradlew :server:test -Dtests.seed=F13528ED31F73FE7 -Dtests.class=org.elasticsearch.index.shard.RemoveCorruptedShardDataCommandTests -Dtests.method="testCorruptedIndex" -Dtests.security.manager=true -Dtests.locale=es-CO -Dtests.timezone=Canada/Central -Dcompiler.java=11 -Druntime.java=8

The reason:
corrupting segments file, when Lucene reads it, it throws generic IOException at https://github.com/apache/lucene-solr/blob/1d85cd783863f75cea133fb9c452302214165a4d/lucene/core/src/java/org/apache/lucene/store/DataInput.java#L141 and that's why ES would not consider it as an index corruption and does not catch it at https://github.com/elastic/elasticsearch/blob/master/server/src/main/java/org/elasticsearch/index/store/Store.java#L188 and therefore would not create a corruption marker (while RemoveCorruptedShardDataCommand relies on its presence)

It would be better if Lucene throws more specific exception rather generic one.

update: https://issues.apache.org/jira/browse/LUCENE-8525 to address this problem