Load index templates v2 by default when talking to ES 7.16 or ES 8.x#28538
Conversation
|
Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane) |
💔 Build Failed
Expand to view the summary
Build stats
Test stats 🧪
Steps errors
Expand to view the steps failures
|
ruflin
left a comment
There was a problem hiding this comment.
The change LGTM, CI does not seem to be too happy. The nice part about this change is that even though it is in theory a breaking change, I expect very few users will even notice. Some users will notice that try to ship data to <7.8 but these have the option to turn on legacy again if my understanding here is correct.
@nimarezainia Heads up on this slightly breaking change for 7.16 but it needs to be done to have a smooth migration to 8.
CHANGELOG.next.asciidoc
Outdated
| - Fix panic with inline SSL when the certificate or key were small than 256 bytes. {pull}23820[23820] | ||
| - Skip add_kubernetes_metadata processor when kubernetes metadata are already present {pull}27689[27689] | ||
| - Enable IMDSv2 support for `add_cloud_metadata` processor on AWS. {issue}22101[22101] {pull}28285[28285] | ||
| - Load index templates v2 by default when talking to ES 7.16 or ES 8.x {pull}28538[28538] |
There was a problem hiding this comment.
Can we add a note here that it also means template can only be loaded for Elasticsearch versions newer 7.8 (I think that is when the index templates were introduced).
|
@kvch I have moved this PR's base to 7.16 |
…ty-libeat-index-templates-7.x
|
jenkins run tests |
|
Lets assume we have a user using Metricbeat 7.15 shipping data to Elasticsearch 7.5. The user never modified the template settings and because of this uses the defaults. Now the user upgrades to 7.16 and metricbeat tries to setup the template. But it will fail because 7.5 does not support this type of templates yet. The user will have to modify the setting to |
…lastic#28538) ## What does this PR do? From now on Beats is going to load the new index templates to Elasticsearch 7.x and 8.x. The PR consists of a few test fixes, and I had found an issue in templates, that I will port to master. ## Why is it important? This way we can be forward compatible with Elasticsearch 8.x where legacy templates are no longer available.
|
Is there any reason why new templates couldn't also be marked as a data stream? |
|
ILM works slightly differently when using data streams. Data streams provide a name to access all of your data and the backing indices are managed by ILM. In 7.16 and before Beats loads and alias and uses it as a write index alias. We needed a bit more time to address those changes. Furthermore, we wanted to somewhat ease the transition from composable indices to data streams. If you want to use data streams in 7.16, you can just convert the alias loaded by Beats to a data stream by following the instructions of this section: https://www.elastic.co/guide/en/elasticsearch/reference/master/set-up-a-data-stream.html#convert-index-alias-to-data-stream Do not forget to disable ILM loading afterwards. Beats has no trouble sending to data streams. Just be careful to only use "create" operation when you are sending events (this is the default in Beats), and always have a |
What does this PR do?
From now on Beats is going to load the new index templates to Elasticsearch 7.x and 8.x.
The PR consists of a few test fixes, and I had found an issue in templates, that I will port to master.
Why is it important?
This way we can be forward compatible with Elasticsearch 8.x where legacy templates are no longer available.
Checklist
- [ ] I have commented my code, particularly in hard-to-understand areasCHANGELOG.next.asciidocorCHANGELOG-developer.next.asciidoc.Related issues
Partially addresses #25018