Prevent index closing while snapshot is restoring the indices #16321
Description
On 1.7.1. This may be related to #15432 , but it is unclear in #15432 what the cause was. So I am filing a separate ticket on this.
In short, it looks like when snapshot restore is running, if the indices are closed while it is operating on a shard, it will leave the snapshot restore request in a STARTED state.
"restore" : {
"snapshots" : [ {
"snapshot" : "snapshot_name",
"repository" : "repository_name",
"state" : "STARTED",
And shards that are in INIT state as reported by the restore request:
{
"index" : "pricing_2015121502",
"shard" : 1,
"state" : "INIT"
}
So that when the end user tries to kick off another restore, it will fail:
failed to restore snapshot
org.elasticsearch.snapshots.ConcurrentSnapshotExecutionException: [repository_name:snapshot_name] Restore process is already running in this cluster
at org.elasticsearch.snapshots.RestoreService$1.execute(RestoreService.java:174)
at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:374)
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:196)
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Because only 1 snapshot restore request can be run at any point in time.
It will be a good idea for us to implement a check to prevent users from closing indices while the restore operation is working on those shards which will help prevent this type of issue.
Metadata
Metadata
Assignees
Type
Fields
Give feedbackNo fields configured for issues without a type.