Simplify ILM Policy solution for managing lifecycle of rollup indices #70334
Description
Currently, we require that if a user wants to manage the lifecycle of a rollup index created within an ILM Policy (within a Datastream, for example), that this index have a separate policy. Maintaining policies is not easy, and this makes it even more difficult. For this reason, we should re-explore solutions to this problem.
# Example ILM Policy with Rollup Action and new policy: "my-rollup-ilm-policy"
PUT _ilm/policy/my-policy
{
"policy": {
"phases": {
"cold": {
"actions": {
"rollup": {
"config": {
"groups": {
"date_histogram": {
"field": "@timestamp",
"calendar_interval": "1y"
}
},
"metrics": [
{
"field": "my-numeric-field",
"metrics": [
"avg"
]
}
]
},
"rollup_policy": "my-rollup-ilm-policy"
}
}
}
}
}
}
One solution is to limit the actions that work for rollup indices. For example, maybe a rollup index must share the same lifecycle/policy as the original index for all phases except the delete phase. This would allow rollup indices to be deleted at a later time.
The policy for this may look like:
PUT _ilm/policy/my-policy
{
"policy": {
"phases": {
"cold": {
"actions": {
"rollup": {
"config": {
"groups": {
"date_histogram": {
"field": "@timestamp",
"calendar_interval": "1y"
}
},
"metrics": [
{
"field": "my-numeric-field",
"metrics": [
"avg"
]
}
]
}
}
}
}
},
"delete": {
"after": "1d",
"after_rollup": "1y",
"actions": { "delete": {} }
}
}
}
Potentially a separate after parameter just for the rollup index created by this policy can be used. Thus managing both original and rollup with minimal modification and no additional policy to maintain.
This is just one option, may come up with others.