Skip to content

[aws, blacklens, github, microsoft_sentinel] Conflicting event.kind values in constant_keyword fields #17561

@andrewkroh

Description

@andrewkroh

The following data streams define event.kind as a constant_keyword field but attempt to set it to conflicting values within their ingest pipelines. This configuration will cause indexing errors because constant_keyword fields must have a single, immutable value for all documents in the index.

Summary of Issues

  • Fixed Value Conflict: The field definition explicitly sets a constant value (e.g., alert), but the ingest pipeline sets it to a different value (e.g., pipeline_error) during error handling or other conditional logic.
  • Multiple Value Conflict: The field definition is constant_keyword but does not specify a value (meaning the first indexed document sets the constant for the index). However, the pipeline sets the field to multiple different values (e.g., state normally, but pipeline_error on failure), ensuring that one of those cases will fail to index.

Affected Data Streams

Package Data Stream Field Definition Conflicting Pipeline Processor
aws securityhub_findings_full_posture fields/ecs.yml:4 (value: unset) default.yml:18 (sets state)
default.yml:2668 (sets pipeline_error)
blacklens alerts fields/ecs.yml:2 (value: alert) default.yml:99 (sets pipeline_error)
github code_scanning fields/ecs.yml:2 (value: alert) default.yml:287 (sets pipeline_error)
github dependabot fields/ecs.yml:2 (value: alert) default.yml:318 (sets pipeline_error)
github issues fields/ecs.yml:2 (value: event) default.yml:250 (sets pipeline_error)
github secret_scanning fields/ecs.yml:2 (value: alert) default.yml:300 (sets pipeline_error)
microsoft_sentinel alert fields/ecs.yml:2 (value: alert) default.yml:457 (sets pipeline_error)
microsoft_sentinel event fields/ecs.yml:2 (value: alert) default.yml:451 (sets pipeline_error)
microsoft_sentinel incident fields/ecs.yml:2 (value: alert) default.yml:455 (sets pipeline_error)

NOTE: websphere_application_server also uses contstant_keyword for event.kind, but does not set it to multiple values (only metric).

Detection

Details

Execute this sqlite query against the pre-build database from go-package-spec.

SELECT p.name                                                         AS package_name,
       ds.dir_name                                                    AS data_stream_name,
       f.name                                                         AS field_name,
       f.value                                                        AS defined_constant,
       GROUP_CONCAT(DISTINCT
                    json_extract(pro.attributes, '$.value'))          AS pipeline_values
FROM data_streams ds
         JOIN packages p ON ds.packages_id = p.id
         JOIN data_stream_fields dsf ON ds.id = dsf.data_stream_id
         JOIN fields f ON dsf.field_id = f.id
         JOIN ingest_pipelines ip ON ds.id = ip.data_streams_id
         JOIN ingest_processors pro ON ip.id = pro.ingest_pipelines_id
WHERE f.type = 'constant_keyword'
  AND pro.type IN ('set', 'append')
  AND json_extract(pro.attributes, '$.field') = f.name
GROUP BY ds.id, f.id
HAVING
-- Conflict 1: Pipeline sets a value different from the defined constant
    (defined_constant IS NOT NULL AND pipeline_values != defined_constant)
-- Conflict 2: Pipeline sets multiple distinct values for a constant field (even if defined_constant is null)
    OR (COUNT(DISTINCT json_extract(pro.attributes, '$.value')) > 1);
package_name data_stream_name field_name defined_constant pipeline_values
blacklens alerts event.kind alert pipeline_error
azure platformlogs event.dataset azure.platformlogs azure.springcloudlogs
aws securityhub_findings_full_posture event.kind null state,pipeline_error
github issues event.kind event pipeline_error
github secret_scanning event.kind alert pipeline_error
github code_scanning event.kind alert pipeline_error
github dependabot event.kind alert pipeline_error
microsoft_sentinel alert event.kind alert pipeline_error
microsoft_sentinel event event.kind alert pipeline_error
microsoft_sentinel incident event.kind alert pipeline_error

NOTE: azure.platformlogs has this problem with the event.dataset field, but it is resetting event.dataset before rerouting so it is a benign issue.

Metadata

Metadata

Labels

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions