Parent ticket: #101016
Summary
We need to fix the indexing implementation in rule_registry plugin in such a way so it can provide backwards compatibility for alerts-as-data indices in the future. Alerts-as-data schema can evolve, and we might need to introduce breaking changes into it. These breaking changes should be handled in one way or another, for example by:
- applying field aliases to old indices
- applying runtime fields to old indices
- reindexing old indices
Background of the problem
The current Alerts as Data implementation uses the same (well, similar) method that Fleet does for upgrading packages for "migrating" the alerts data indices. However, the approach that Fleet packages use for upgrading indices isn't appropriate to use for Alerts as Data because we want to maintain backward compatibility for all alerts data and the Fleet package upgrade approach doesn't provide backward compatibility and it's only focused on allowing data to be ingested in the new format. Additionally, the index-naming scheme without an index alias doesn't leave us with the option of using a reindex operation in the future.
Issues:
- Issue 1: non-additive changes do not trigger rollover. The Fleet package upgrade algorithm relies on Elasticsearch throwing an incompatible mappings error to trigger a rollover into a new index for upgrades. However, this does not occur for all non-additives changes, including renaming fields.
- Issue 2: Field aliases and runtime fields may not provide all backward compatibility. At this time, we don't know all of the types of breaking changes we will need to make to the Alerts data indices. Additionally, the Fleet package upgrade approach doesn't provide any backward compatibility right now using runtime fields or field aliases, so we don't know for certain how far it will get us. As a result, there is the potential that we will need to reindex the Alerts data indices. If we need to perform a reindex operation, we'll want to use an index alias that is updated to point at the indices in the new data format and exclude the data that is in the old format. However, the existing index-naming scheme in the design document should be adjusted to make the reindex possible. Currently, we will be creating indices with the following pattern:
.alerts-{registrationContext}.{datasetSuffix}-{namespace}-{leftPaddedIncrementingNumber}. This will likely result in users creating Dashboards, Visualization, etc. against index patterns like the following: .alerts-security.alerts-*, .alerts-*. If we discover that we need to reindex the data and perform a migration, it will require us to create a new index and copy all of the data from the original indices into the new indices while performing a transformation. If we were to add a version to the index-naming pattern to designate that we have a new version, there's a high risk that the index-patterns that users have been using will match both the old and the new indices. For example, if we were to add the version right after .alerts- in the index-name, this would result in us having a .alerts-security.alerts-default-000001 index and a .alerts-v2-security.alerts.default-000001 index, and the .alerts-* index-pattern would match both.
Solution for 7.15
We need to separate naming schemes for index aliases and concrete indices. This will allow us to implement reindexing in the future.
Aliases will be used for reading and writing data:
.alerts-{registrationContext}.{datasetSuffix}-{namespace}
Concrete indices will have a different naming scheme:
.internal.alerts-{registrationContext}.{datasetSuffix}-{namespace}-000001
How this enables reindexing: after reindexing a new (target) index will be created; we will be able to add the alias to it, and remove the alias from the old (source) index.
Also, we need to use versioning for index templates and documents being written into those indices. This will help us apply field aliases, runtime fields etc to the old indices when we need it.
Our decisions previously discussed:
- Use the current Kibana version for the versioning described above.
- Don’t include the version in the concrete index name by default (exception - it will be included after reindexing).
- Naming for concrete backing indices:
.internal.alerts-security.alerts-default-000001-r8.0.0
- prefix:
.internal.alerts
- ilm suffix:
000002
- reindexing suffix:
r8.0.0, not included by default, only included into a target index name after reindexing
- We will include the version into the index template metadata:
_meta.kibana.version = 7.16.0- will be included into both the index template
_meta and mappings._meta
- We will include the version into the document fields:
kibana.version = 7.16.0- it will use the version field type
Sub-tasks
Brandon's concerns regarding the index upgrade/rollover logic (#109276 (comment)) will be addressed as part of the parent ticket (#101016).
Parent ticket: #101016
Summary
We need to fix the indexing implementation in
rule_registryplugin in such a way so it can provide backwards compatibility for alerts-as-data indices in the future. Alerts-as-data schema can evolve, and we might need to introduce breaking changes into it. These breaking changes should be handled in one way or another, for example by:Background of the problem
The current Alerts as Data implementation uses the same (well, similar) method that Fleet does for upgrading packages for "migrating" the alerts data indices. However, the approach that Fleet packages use for upgrading indices isn't appropriate to use for Alerts as Data because we want to maintain backward compatibility for all alerts data and the Fleet package upgrade approach doesn't provide backward compatibility and it's only focused on allowing data to be ingested in the new format. Additionally, the index-naming scheme without an index alias doesn't leave us with the option of using a reindex operation in the future.
Issues:
.alerts-{registrationContext}.{datasetSuffix}-{namespace}-{leftPaddedIncrementingNumber}. This will likely result in users creating Dashboards, Visualization, etc. against index patterns like the following:.alerts-security.alerts-*,.alerts-*. If we discover that we need to reindex the data and perform a migration, it will require us to create a new index and copy all of the data from the original indices into the new indices while performing a transformation. If we were to add a version to the index-naming pattern to designate that we have a new version, there's a high risk that the index-patterns that users have been using will match both the old and the new indices. For example, if we were to add the version right after.alerts-in the index-name, this would result in us having a.alerts-security.alerts-default-000001index and a.alerts-v2-security.alerts.default-000001index, and the.alerts-*index-pattern would match both.Solution for 7.15
We need to separate naming schemes for index aliases and concrete indices. This will allow us to implement reindexing in the future.
Aliases will be used for reading and writing data:
Concrete indices will have a different naming scheme:
How this enables reindexing: after reindexing a new (target) index will be created; we will be able to add the alias to it, and remove the alias from the old (source) index.
Also, we need to use versioning for index templates and documents being written into those indices. This will help us apply field aliases, runtime fields etc to the old indices when we need it.
Our decisions previously discussed:
.internal.alerts-security.alerts-default-000001-r8.0.0.internal.alerts000002r8.0.0, not included by default, only included into a target index name after reindexing_meta.kibana.version=7.16.0_metaandmappings._metakibana.version=7.16.0Sub-tasks
.internal.alertsprefix.Brandon's concerns regarding the index upgrade/rollover logic (#109276 (comment)) will be addressed as part of the parent ticket (#101016).