Index Templates v1 are deprecated in Elasticsearch (elastic/elasticsearch#71307). In order to ensure forwards compatility with Elasticsearch 8, we should update Beats to move to the v2 API with template component support. Support is already available in libbeat, but we still default to the v1 API.
ILM and alias setup can be somewhat fragile at times. Deleting/updating the wrong resource at the wrong time can leave users with unmanaged indices. Moving to data streams streams already in 7.x will help improve robustness in resource setup and management.
Moving to data streams will not change the indexing strategy, naming, or versioning of Beats. We would just change which/how resources are setup:
- Index template will be updated to generate data streams instead of indices. We will add
"data_stream": { } to the generated template if data streams are enabled.
- Do not generate first ILM index if data streams are enabled
- Do not set up Alias if data streams are enabled
- Create data stream when
<beatname> setup is used (So we can create writer roles with reduced permissions). API: PUT _data_stream/<name>.
The data stream naming schema will follow the current index/alias naming schema. There are no changes required on the kibana index mapping setup.
Beats will default to v2 templates and data streams if Elasticsearch versions is > 7.13. Users should still be able to opt-out, though.
From an users point of view, this might be a breaking change. If alias settings are explicitely configured we will report a deprecation warning with details/link explaining to users that they should update their settings. In order to not surprise users with wrongly-named unexpected resources, we either have to fail or fall back to old school ILM+Alias generation.
Index Templates v1 are deprecated in Elasticsearch (elastic/elasticsearch#71307). In order to ensure forwards compatility with Elasticsearch 8, we should update Beats to move to the v2 API with template component support. Support is already available in libbeat, but we still default to the v1 API.
ILM and alias setup can be somewhat fragile at times. Deleting/updating the wrong resource at the wrong time can leave users with unmanaged indices. Moving to data streams streams already in 7.x will help improve robustness in resource setup and management.
Moving to data streams will not change the indexing strategy, naming, or versioning of Beats. We would just change which/how resources are setup:
"data_stream": { }to the generated template if data streams are enabled.<beatname> setupis used (So we can create writer roles with reduced permissions). API:PUT _data_stream/<name>.The data stream naming schema will follow the current index/alias naming schema. There are no changes required on the kibana index mapping setup.
Beats will default to v2 templates and data streams if Elasticsearch versions is > 7.13. Users should still be able to opt-out, though.
From an users point of view, this might be a breaking change. If alias settings are explicitely configured we will report a deprecation warning with details/link explaining to users that they should update their settings. In order to not surprise users with wrongly-named unexpected resources, we either have to fail or fall back to old school ILM+Alias generation.