Retrying an ILM action that is an AsyncActionStep does not work due to execution model changes #35397
Description
When an index is stuck in the error state of a step which is an AsyncActionStep, like so:
PUT _ilm/policy/bad
{
"policy": {
"phases": {
"warm": {
"min_age": "5s",
"actions": {
"shrink": {
"number_of_shards": 13
}
}
}
}
}
}
PUT /foo
{
"settings": {
"index.number_of_shards": 2,
"index.lifecycle.name": "bad"
}
}With error:
{
"indices" : {
"foo" : {
"index" : "foo",
"managed" : true,
"policy" : "bad",
"lifecycle_date" : "2018-11-08T22:47:45.865Z",
"lifecycle_date_millis" : 1541717265865,
"phase" : "warm",
"phase_time" : "2018-11-08T22:47:52.601Z",
"phase_time_millis" : 1541717272601,
"action" : "shrink",
"action_time" : "2018-11-08T22:47:52.601Z",
"action_time_millis" : 1541717272601,
"step" : "ERROR",
"step_time" : "2018-11-08T22:47:52.688Z",
"step_time_millis" : 1541717272688,
"failed_step" : "shrink",
"step_info" : {
"type" : "illegal_argument_exception",
"reason" : "the number of target shards [13] must be less that the number of source shards [2]"
},
"phase_execution" : {
"policy" : "bad",
"phase_definition" : {
"min_age" : "5s",
"actions" : {
"shrink" : {
"number_of_shards" : 13
}
}
},
"version" : 5,
"modified_date" : "2018-11-08T22:47:44.230Z",
"modified_date_in_millis" : 1541717264230
}
}
}
}When I correct the policy and issue a POST /foo/_ilm/retry...
The now warm/shrink/shrink step never executes, because we only execute AsyncActionSteps when they are transitioned into.
The 'retry' API should perform the correct steps to retry any type of operation.