Today, Fleet is responsible for both updating the index templates and making sure these updates are applied to the corresponding data streams.
The way how Fleet computes the diff between the current state of a data stream and how it determines whether to update or roll over data streams is complex, flawed (see elastic/kibana#158653), brittle (for example elastic/kibana#158654, elastic/kibana#157869), and shouldn't be the responsibility of Fleet as it's a more general data management problem.
Borrowing ideas from Infrastructure as Code, there should be a way to declare the desired end state of new and existing data streams that match a given pattern and ES is responsible to determine what it needs to do to reach that state.
This docs page explains the different steps user need to do to modify a data stream: https://www.elastic.co/guide/en/elasticsearch/reference/current/modify-data-streams.html
The goal is that there's a single way to update a data stream so that users (including Fleet) don't need to determine whether or not they need to trigger a rollover and don't need to manually create a diff to update each data stream to be consistent with its index template.
A possible solution for this could be to add a new parameter to the put index template and component template APIs (something like apply: true). This will determine all matching data streams, and apply the changes needed so that the data stream's mappings and settings are consistent with the index template. It will also automatically determine whether the change requires a roll over of the data stream (for example, if a static index setting is changed or when an existing field mapping is changed) or whether updating the data stream is sufficient (for example, when only dynamic index settings are changed or when only adding fields to the mappings), to avoid creating unnecessary backing indices.
Today, Fleet is responsible for both updating the index templates and making sure these updates are applied to the corresponding data streams.
The way how Fleet computes the diff between the current state of a data stream and how it determines whether to update or roll over data streams is complex, flawed (see elastic/kibana#158653), brittle (for example elastic/kibana#158654, elastic/kibana#157869), and shouldn't be the responsibility of Fleet as it's a more general data management problem.
Borrowing ideas from Infrastructure as Code, there should be a way to declare the desired end state of new and existing data streams that match a given pattern and ES is responsible to determine what it needs to do to reach that state.
This docs page explains the different steps user need to do to modify a data stream: https://www.elastic.co/guide/en/elasticsearch/reference/current/modify-data-streams.html
The goal is that there's a single way to update a data stream so that users (including Fleet) don't need to determine whether or not they need to trigger a rollover and don't need to manually create a diff to update each data stream to be consistent with its index template.
A possible solution for this could be to add a new parameter to the put index template and component template APIs (something like
apply: true). This will determine all matching data streams, and apply the changes needed so that the data stream's mappings and settings are consistent with the index template. It will also automatically determine whether the change requires a roll over of the data stream (for example, if a static index setting is changed or when an existing field mapping is changed) or whether updating the data stream is sufficient (for example, when only dynamic index settings are changed or when only adding fields to the mappings), to avoid creating unnecessary backing indices.