Index is stuck in the `downsample` ILM action 

### Elasticsearch Version

8.6.0

### Installed Plugins

_No response_

### Java Version

_bundled_

### OS Version

Elasticsearch Service (ESS)

### Problem Description

Users may accidentally configure an ILM policy with the `downsample` [action](https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-downsample.html) where the ILM policy is attached to a regular index or a data stream (but not a [time series data stream (TSDS)](https://www.elastic.co/guide/en/elasticsearch/reference/current/tsds.html)).

In this situation, the regular index or backing index will remain stuck in the `downsample` ILM action (even after updating the ILM policy to remove the `downsample` action).

### Steps to Reproduce

1. Create a hot/warm deployment version 8.6.0 in ESS.

2. Create an ILM policy with a `downsample` action:
```
PUT _ilm/policy/my-lifecycle-policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_docs": "2"
          },
          "downsample": {
            "fixed_interval": "1m"
          }
        }
      },
      "warm": {
        "min_age": "1m",
        "actions": {
          "allocate": {
            "number_of_replicas": 0
          }
        }
      },
      "delete": {
        "min_age": "2m",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}
```

3. Update the `indices.lifecycle.poll_interval` for this current issue recreation:

```
PUT _cluster/settings
{
  "persistent": {
    "indices.lifecycle.poll_interval": "10s"
  }
}
```

4. Create a component template for mappings:
```
PUT _component_template/my-mappings
{
  "template": {
    "mappings": {
      "properties": {
        "@timestamp": {
          "type": "date",
          "format": "date_optional_time||epoch_millis"
        },
        "message": {
          "type": "wildcard"
        }
      }
    }
  },
  "_meta": {
    "description": "Mappings for @timestamp and message fields",
    "my-custom-meta-field": "More arbitrary metadata"
  }
}
```

5. Create a component template for index settings:
```
PUT _component_template/my-settings
{
  "template": {
    "settings": {
      "index.lifecycle.name": "my-lifecycle-policy"
    }
  },
  "_meta": {
    "description": "Settings for ILM",
    "my-custom-meta-field": "More arbitrary metadata"
  }
}
```

6. Create an index template:
```
PUT _index_template/my-index-template
{
  "index_patterns": ["my-data-stream*"],
  "data_stream": { },
  "composed_of": [ "my-mappings", "my-settings" ],
  "priority": 500,
  "_meta": {
    "description": "Template for my time series data",
    "my-custom-meta-field": "More arbitrary metadata"
  }
}
```

7. Create the data stream:
```
PUT _data_stream/my-data-stream
```

8. Add some documents:
```
POST my-data-stream/_doc
{
  "@timestamp": "2023-01-22T00:00:01.000Z",
  "message": "document 1"
}

POST my-data-stream/_doc
{
  "@timestamp": "2023-01-22T00:00:02.000Z",
  "message": "document 2"
}

POST my-data-stream/_doc
{
  "@timestamp": "2023-01-22T00:00:03.000Z",
  "message": "document 3"
}
```

9. Verify that the documents are stored:
```
GET my-data-stream/_search
```

10. Verify that a new backing index has been created  (i.e `.ds-my-data-stream-2023.01.22-000002`) (will be created after the next poll interval)
```
GET _cat/indices?expand_wildcards=all
```

11. Check the ILM explain results of the first index (i.e `.ds-my-data-stream-2023.01.22-000001`) and observe that the index is stuck in the `downsample` action with the following exception:


```
Rollup requires setting [index.mode=time_series] for index [.ds-my-data-stream-2023.01.22-000001]
```

```
GET .ds-my-data-stream-2023.01.22-000001/_ilm/explain
```

```
{
  "indices": {
    ".ds-my-data-stream-2023.01.22-000001": {
      "index": ".ds-my-data-stream-2023.01.22-000001",
      "managed": true,
      "policy": "my-lifecycle-policy",
      "index_creation_date_millis": 1674363304040,
      "time_since_index_creation": "2.04m",
      "lifecycle_date_millis": 1674363319364,
      "age": "1.78m",
      "phase": "hot",
      "phase_time_millis": 1674363419410,
      "action": "downsample",
      "action_time_millis": 1674363319564,
      "step": "ERROR",
      "step_time_millis": 1674363419410,
      "failed_step": "rollup",
      "is_auto_retryable_error": true,
      "failed_step_retry_count": 9,
      "step_info": {
        "type": "exception",
        "reason": "Rollup requires setting [index.mode=time_series] for index [.ds-my-data-stream-2023.01.22-000001]"
      },
      "phase_execution": {
        "policy": "my-lifecycle-policy",
        "phase_definition": {
          "min_age": "0ms",
          "actions": {
            "downsample": {
              "fixed_interval": "1m"
            },
            "rollover": {
              "max_docs": 2
            }
          }
        },
        "version": 1,
        "modified_date_in_millis": 1674363301001
      }
    }
  }
}
```

12. Update the ILM policy to remove the `downsample` action:

```
PUT _ilm/policy/my-lifecycle-policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_docs": "2"
          }
        }
      },
      "warm": {
        "min_age": "1m",
        "actions": {
          "allocate": {
            "number_of_replicas": 0
          }
        }
      },
      "delete": {
        "min_age": "2m",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}
```

13. Check the ILM explain results of index `.ds-my-data-stream-2023.01.22-000001` and observe that the index remains stuck in the `downsample` action, despite the ILM policy update:

```
{
  "indices": {
    ".ds-my-data-stream-2023.01.22-000001": {
      "index": ".ds-my-data-stream-2023.01.22-000001",
      "managed": true,
      "policy": "my-lifecycle-policy",
      "index_creation_date_millis": 1674363304040,
      "time_since_index_creation": "2.79m",
      "lifecycle_date_millis": 1674363319364,
      "age": "2.54m",
      "phase": "hot",
      "phase_time_millis": 1674363439419,
      "action": "downsample",
      "action_time_millis": 1674363319564,
      "step": "ERROR",
      "step_time_millis": 1674363469445,
      "failed_step": "rollup",
      "is_auto_retryable_error": true,
      "failed_step_retry_count": 14,
      "step_info": {
        "type": "exception",
        "reason": "Rollup requires setting [index.mode=time_series] for index [.ds-my-data-stream-2023.01.22-000001]"
      },
      "phase_execution": {
        "policy": "my-lifecycle-policy",
        "phase_definition": {
          "min_age": "0ms",
          "actions": {
            "downsample": {
              "fixed_interval": "1m"
            },
            "rollover": {
              "max_docs": 2
            }
          }
        },
        "version": 1,
        "modified_date_in_millis": 1674363301001
      }
    }
  }
}
```

### Logs (if relevant)

_No response_

### Workaround

I am not aware of any workarounds for now. To be discussed with @elastic/es-data-management .

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Index is stuck in the `downsample` ILM action #93123

Elasticsearch Version

Installed Plugins

Java Version

OS Version

Problem Description

Steps to Reproduce

Logs (if relevant)

Workaround

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Index is stuck in the downsample ILM action #93123

Description

Elasticsearch Version

Installed Plugins

Java Version

OS Version

Problem Description

Steps to Reproduce

Logs (if relevant)

Workaround

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions

Index is stuck in the `downsample` ILM action #93123