Summary

Changes by rule type:

• KQL
  • Runtime fields in source documents will be evaluated against value list exceptions
• Threshold
  • Timestamp for generated alert is now extracted from fields instead of _source, so if the timestamp override is a runtime field then @timestamp will be correctly populated
  • threshold_result.value now uses the bucket key rather than extracting the value from _source - so if the key is a runtime field it will be correctly populated. This also fixes a bug where if the field being aggregated on was an array of values then threshold_result.value would contain the full array rather than the single value from the array that was actually being used as the key
• ML
  • Runtime fields in source documents will be evaluated against value list exceptions
• Threat match
  • Runtime fields can now be used in the indicator match indices
  • Runtime fields in source documents will be evaluated against value list exceptions
• EQL
  • No changes as runtime fields are not accessible in the EQL search API yet (https://github.com/elastic/dev/issues/1600)

Remaining dependencies on _source

• Building the alert document that goes into .siem-signals
  • We rely on the parent and ancestor information for signals on signals and this info is only available in _source - fields flattens arrays of objects which makes it impossible to reconstruct the original array of objects
  • Merging _source and fields does not work for arrays of objects because not all objects in the array have the same set of fields. For example,

[{
  field1: 1
},
{
  field1: 2,
  field2: 3
}]

would flatten to

fields: {
  field1: [1, 2]
  field2: [3]
}

If field2 is overwritten by a runtime field, then we don't have enough information to merge the field2 from fields with the array of objects from _source.

It's also possible to have runtime fields that can't be represented by a JSON structure at all.

fields: {
  field1: [1],
  field1.sub_field: [2]
}

This situation is possible to create with field1 coming from the _source of a document and field1.sub_field defined as a runtime field, however, these fields and values can't be represented as a single object.

• One solution would be to only merge fields from fields that don't conflict with existing fields in _source
  • However, this may not store all runtime fields
• Another solution would be to ask the ES team for the ability to retrieve only runtime fields from fields and store those in their own section of the alert document
  • Maybe possible to identify the runtime fields by comparing with _source, but it's much easier if ES can specify whether each field is runtime in the response
• Hybrid approach
  • Get runtime fields from fields response and then merge the runtime fields that do NOT conflict into _source
  • Store all runtime fields in a separate section as well for completeness

Checklist

Delete any items that are not applicable to this PR.

• Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
• Documentation was added for features that require explanation or tutorials
• Unit or functional tests were updated or added to match the most common scenarios
• Any UI touched in this PR is usable by keyboard only (learn more about keyboard accessibility)
• Any UI touched in this PR does not create any new axe failures (run axe in browser: FF, Chrome)
• If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the docker list
• This renders correctly on smaller devices using a responsive layout. (You can test this in your browser)
• This was checked for cross-browser compatibility

For maintainers

• This was checked for breaking API changes and was labeled appropriately