[9.2] [Security Solution] [HDQ]: integration-based targeting and descriptor versioning (#258418)#260876
Merged
kibanamachine merged 1 commit intoelastic:9.2from Apr 2, 2026
Merged
Conversation
… versioning (elastic#258418) ## Summary ### Changes - DHQ descriptor versioning: each query descriptor now includes an optional version field (defaulting to 1). Old v1 descriptors are fully unaffected. - Integration-based targeting (v2): v2 descriptors replace the index field with an integrations field: a comma-separated list of regex patterns matched against Fleet-installed packages. The executor runs one query per matched integration, using the datastream indices resolved for that integration. - Datastream type filtering (v2): an optional datastreamTypes field (comma-separated regexes) narrows which datastreams are included for the matched integration. - Skip reasons: when a v2+ query cannot execute, they are skipped and reported with one of three reasons: integration_not_installed, datastreams_not_matched, or parse_failure. - EBT schema update: stats events gain descriptorVersion, status, and integration fields (matched patterns, resolved indices, integration name/version) to allow downstream consumers to distinguish v1 from v2 executions. - Permission check simplification: removed the indices.exists pre-check from checkPermissions; privilege checking alone is sufficient, and `exists` can lead to FP for indices patterns. ### Query descriptor V2 example ```yaml version: 2 id: "endpoint-process-stats" name: "Endpoint process event counts" integrations: "endpoint" datastreamTypes: "logs" type: "DSL" query: | { "aggs": { "by_action": { "terms": { "field": "event.action", "size": 20 } } }, "size": 0 } scheduleCron: "1h" enabled: true filterlist: "by_action.key": keep "by_action.doc_count": keep ``` ### Stats EBT document examples ```json { "name": "endpoint-process-stats", "traceId": "419cc487-3b17-48f0-bf7e-4e953ed9f050", "started": "2026-03-18T10:00:00.000Z", "finished": "2026-03-18T10:00:00.842Z", "descriptorVersion": 2, "status": "success", "passed": true, "numDocs": 1, "fieldNames": [ "by_action.key", "by_action.doc_count" ], "integration": { "name": "endpoint", "version": "8.14.2", "indices": [ "logs-endpoint.events.process-default" ] }, "circuitBreakers": { ... } } ``` ```json { "name": "endpoint-process-stats", "traceId": "b2c3d4e5-...", "started": "2026-03-18T10:00:00.000Z", "finished": "2026-03-18T10:00:00.012Z", "descriptorVersion": 2, "status": "skipped", "skipReason": "integration_not_installed", "passed": false, "numDocs": 0, "fieldNames": [] } ``` ```json { "name": "endpoint-process-stats", "traceId": "c3d4e5f6-...", "started": "2026-03-18T10:00:00.000Z", "finished": "2026-03-18T10:00:00.018Z", "descriptorVersion": 2, "status": "skipped", "skipReason": "datastreams_not_matched", "passed": false, "numDocs": 0, "fieldNames": [], "integration": { "name": "endpoint", "version": "8.14.2", "indices": [] } } ``` (cherry picked from commit c8ee39d)
8 tasks
| type: 'keyword', | ||
| _meta: { | ||
| optional: true, | ||
| description: |
Contributor
There was a problem hiding this comment.
🟢 Low event_based/events.ts:1219
The skipReason description lists 'unknown_version' as a valid value, but the SkipReason type defines the valid values as 'datastreams_not_matched', 'integration_not_installed', 'parse_failure', 'fleet_unavailable', and 'unsupported_query'. The description is inconsistent with the type definition and omits several valid skip reasons.
🤖 Copy this AI Prompt to have your agent fix this:
In file x-pack/solutions/security/plugins/security_solution/server/lib/telemetry/event_based/events.ts around line 1219:
The `skipReason` description lists `'unknown_version'` as a valid value, but the `SkipReason` type defines the valid values as `'datastreams_not_matched'`, `'integration_not_installed'`, `'parse_failure'`, `'fleet_unavailable'`, and `'unsupported_query'`. The description is inconsistent with the type definition and omits several valid skip reasons.
Evidence trail:
x-pack/solutions/security/plugins/security_solution/server/lib/telemetry/event_based/events.ts lines 1219-1221 (description says 'datastreams_not_matched', 'integration_not_installed' or 'unknown_version')
x-pack/solutions/security/plugins/security_solution/server/lib/telemetry/diagnostic/health_diagnostic_service.types.ts lines 159-164 (SkipReason type defines 'datastreams_not_matched' | 'integration_not_installed' | 'parse_failure' | 'fleet_unavailable' | 'unsupported_query')
Contributor
💚 Build Succeeded
Metrics [docs]Unknown metric groupsESLint disabled line counts
Total ESLint disabled count
cc @szaffarano |
szaffarano
added a commit
that referenced
this pull request
Apr 3, 2026
…criptor versioning (#258418) (#261112) # Backport This will backport the following commits from `main` to `8.19`: - [[Security Solution] [HDQ]: integration-based targeting and descriptor versioning (#258418)](#258418) <!--- Backport version: 11.0.1 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Sebastián Zaffarano","email":"sebastian.zaffarano@elastic.co"},"sourceCommit":{"committedDate":"2026-04-02T08:06:06Z","message":"[Security Solution] [HDQ]: integration-based targeting and descriptor versioning (#258418)\n\n## Summary\n\n### Changes\n- DHQ descriptor versioning: each query descriptor now includes an\noptional version field (defaulting to 1). Old v1 descriptors are fully\nunaffected.\n- Integration-based targeting (v2): v2 descriptors replace the index\nfield with an integrations field: a comma-separated list of regex\npatterns matched against Fleet-installed packages. The executor runs one\nquery per matched integration, using the datastream indices resolved for\nthat integration.\n- Datastream type filtering (v2): an optional datastreamTypes field\n(comma-separated regexes) narrows which datastreams are included for the\nmatched integration.\n- Skip reasons: when a v2+ query cannot execute, they are skipped and\nreported with one of three reasons: integration_not_installed,\ndatastreams_not_matched, or parse_failure.\n- EBT schema update: stats events gain descriptorVersion, status, and\nintegration fields (matched patterns, resolved indices, integration\nname/version) to allow downstream consumers to distinguish v1 from v2\nexecutions.\n- Permission check simplification: removed the indices.exists pre-check\nfrom checkPermissions; privilege checking alone is sufficient, and\n`exists` can lead to FP for indices patterns.\n\n### Query descriptor V2 example\n\n```yaml\nversion: 2\nid: \"endpoint-process-stats\"\nname: \"Endpoint process event counts\"\nintegrations: \"endpoint\"\ndatastreamTypes: \"logs\"\ntype: \"DSL\"\nquery: |\n {\n \"aggs\": { \"by_action\": { \"terms\": { \"field\": \"event.action\", \"size\": 20 } } },\n \"size\": 0\n }\nscheduleCron: \"1h\"\nenabled: true\nfilterlist:\n \"by_action.key\": keep\n \"by_action.doc_count\": keep\n```\n\n### Stats EBT document examples\n\n```json\n{\n \"name\": \"endpoint-process-stats\",\n \"traceId\": \"419cc487-3b17-48f0-bf7e-4e953ed9f050\",\n \"started\": \"2026-03-18T10:00:00.000Z\",\n \"finished\": \"2026-03-18T10:00:00.842Z\",\n \"descriptorVersion\": 2,\n \"status\": \"success\",\n \"passed\": true,\n \"numDocs\": 1,\n \"fieldNames\": [\n \"by_action.key\",\n \"by_action.doc_count\"\n ],\n \"integration\": {\n \"name\": \"endpoint\",\n \"version\": \"8.14.2\",\n \"indices\": [\n \"logs-endpoint.events.process-default\"\n ]\n },\n \"circuitBreakers\": { ...\n }\n}\n```\n\n```json\n{\n \"name\": \"endpoint-process-stats\",\n \"traceId\": \"b2c3d4e5-...\",\n \"started\": \"2026-03-18T10:00:00.000Z\",\n \"finished\": \"2026-03-18T10:00:00.012Z\",\n \"descriptorVersion\": 2,\n \"status\": \"skipped\",\n \"skipReason\": \"integration_not_installed\",\n \"passed\": false,\n \"numDocs\": 0,\n \"fieldNames\": []\n}\n```\n\n```json\n{\n \"name\": \"endpoint-process-stats\",\n \"traceId\": \"c3d4e5f6-...\",\n \"started\": \"2026-03-18T10:00:00.000Z\",\n \"finished\": \"2026-03-18T10:00:00.018Z\",\n \"descriptorVersion\": 2,\n \"status\": \"skipped\",\n \"skipReason\": \"datastreams_not_matched\",\n \"passed\": false,\n \"numDocs\": 0,\n \"fieldNames\": [],\n \"integration\": {\n \"name\": \"endpoint\",\n \"version\": \"8.14.2\",\n \"indices\": []\n }\n}\n```","sha":"c8ee39d5f2f726c49573f9e3cf7464d756467b53","branchLabelMapping":{"^v9.4.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","ci:build-cloud-image","ci:build-serverless-image","backport:version","v9.4.0","v9.3.3","v9.2.8","v8.19.14"],"title":"[Security Solution] [HDQ]: integration-based targeting and descriptor versioning","number":258418,"url":"https://github.com/elastic/kibana/pull/258418","mergeCommit":{"message":"[Security Solution] [HDQ]: integration-based targeting and descriptor versioning (#258418)\n\n## Summary\n\n### Changes\n- DHQ descriptor versioning: each query descriptor now includes an\noptional version field (defaulting to 1). Old v1 descriptors are fully\nunaffected.\n- Integration-based targeting (v2): v2 descriptors replace the index\nfield with an integrations field: a comma-separated list of regex\npatterns matched against Fleet-installed packages. The executor runs one\nquery per matched integration, using the datastream indices resolved for\nthat integration.\n- Datastream type filtering (v2): an optional datastreamTypes field\n(comma-separated regexes) narrows which datastreams are included for the\nmatched integration.\n- Skip reasons: when a v2+ query cannot execute, they are skipped and\nreported with one of three reasons: integration_not_installed,\ndatastreams_not_matched, or parse_failure.\n- EBT schema update: stats events gain descriptorVersion, status, and\nintegration fields (matched patterns, resolved indices, integration\nname/version) to allow downstream consumers to distinguish v1 from v2\nexecutions.\n- Permission check simplification: removed the indices.exists pre-check\nfrom checkPermissions; privilege checking alone is sufficient, and\n`exists` can lead to FP for indices patterns.\n\n### Query descriptor V2 example\n\n```yaml\nversion: 2\nid: \"endpoint-process-stats\"\nname: \"Endpoint process event counts\"\nintegrations: \"endpoint\"\ndatastreamTypes: \"logs\"\ntype: \"DSL\"\nquery: |\n {\n \"aggs\": { \"by_action\": { \"terms\": { \"field\": \"event.action\", \"size\": 20 } } },\n \"size\": 0\n }\nscheduleCron: \"1h\"\nenabled: true\nfilterlist:\n \"by_action.key\": keep\n \"by_action.doc_count\": keep\n```\n\n### Stats EBT document examples\n\n```json\n{\n \"name\": \"endpoint-process-stats\",\n \"traceId\": \"419cc487-3b17-48f0-bf7e-4e953ed9f050\",\n \"started\": \"2026-03-18T10:00:00.000Z\",\n \"finished\": \"2026-03-18T10:00:00.842Z\",\n \"descriptorVersion\": 2,\n \"status\": \"success\",\n \"passed\": true,\n \"numDocs\": 1,\n \"fieldNames\": [\n \"by_action.key\",\n \"by_action.doc_count\"\n ],\n \"integration\": {\n \"name\": \"endpoint\",\n \"version\": \"8.14.2\",\n \"indices\": [\n \"logs-endpoint.events.process-default\"\n ]\n },\n \"circuitBreakers\": { ...\n }\n}\n```\n\n```json\n{\n \"name\": \"endpoint-process-stats\",\n \"traceId\": \"b2c3d4e5-...\",\n \"started\": \"2026-03-18T10:00:00.000Z\",\n \"finished\": \"2026-03-18T10:00:00.012Z\",\n \"descriptorVersion\": 2,\n \"status\": \"skipped\",\n \"skipReason\": \"integration_not_installed\",\n \"passed\": false,\n \"numDocs\": 0,\n \"fieldNames\": []\n}\n```\n\n```json\n{\n \"name\": \"endpoint-process-stats\",\n \"traceId\": \"c3d4e5f6-...\",\n \"started\": \"2026-03-18T10:00:00.000Z\",\n \"finished\": \"2026-03-18T10:00:00.018Z\",\n \"descriptorVersion\": 2,\n \"status\": \"skipped\",\n \"skipReason\": \"datastreams_not_matched\",\n \"passed\": false,\n \"numDocs\": 0,\n \"fieldNames\": [],\n \"integration\": {\n \"name\": \"endpoint\",\n \"version\": \"8.14.2\",\n \"indices\": []\n }\n}\n```","sha":"c8ee39d5f2f726c49573f9e3cf7464d756467b53"}},"sourceBranch":"main","suggestedTargetBranches":["8.19"],"targetPullRequestStates":[{"branch":"main","label":"v9.4.0","branchLabelMappingKey":"^v9.4.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/258418","number":258418,"mergeCommit":{"message":"[Security Solution] [HDQ]: integration-based targeting and descriptor versioning (#258418)\n\n## Summary\n\n### Changes\n- DHQ descriptor versioning: each query descriptor now includes an\noptional version field (defaulting to 1). Old v1 descriptors are fully\nunaffected.\n- Integration-based targeting (v2): v2 descriptors replace the index\nfield with an integrations field: a comma-separated list of regex\npatterns matched against Fleet-installed packages. The executor runs one\nquery per matched integration, using the datastream indices resolved for\nthat integration.\n- Datastream type filtering (v2): an optional datastreamTypes field\n(comma-separated regexes) narrows which datastreams are included for the\nmatched integration.\n- Skip reasons: when a v2+ query cannot execute, they are skipped and\nreported with one of three reasons: integration_not_installed,\ndatastreams_not_matched, or parse_failure.\n- EBT schema update: stats events gain descriptorVersion, status, and\nintegration fields (matched patterns, resolved indices, integration\nname/version) to allow downstream consumers to distinguish v1 from v2\nexecutions.\n- Permission check simplification: removed the indices.exists pre-check\nfrom checkPermissions; privilege checking alone is sufficient, and\n`exists` can lead to FP for indices patterns.\n\n### Query descriptor V2 example\n\n```yaml\nversion: 2\nid: \"endpoint-process-stats\"\nname: \"Endpoint process event counts\"\nintegrations: \"endpoint\"\ndatastreamTypes: \"logs\"\ntype: \"DSL\"\nquery: |\n {\n \"aggs\": { \"by_action\": { \"terms\": { \"field\": \"event.action\", \"size\": 20 } } },\n \"size\": 0\n }\nscheduleCron: \"1h\"\nenabled: true\nfilterlist:\n \"by_action.key\": keep\n \"by_action.doc_count\": keep\n```\n\n### Stats EBT document examples\n\n```json\n{\n \"name\": \"endpoint-process-stats\",\n \"traceId\": \"419cc487-3b17-48f0-bf7e-4e953ed9f050\",\n \"started\": \"2026-03-18T10:00:00.000Z\",\n \"finished\": \"2026-03-18T10:00:00.842Z\",\n \"descriptorVersion\": 2,\n \"status\": \"success\",\n \"passed\": true,\n \"numDocs\": 1,\n \"fieldNames\": [\n \"by_action.key\",\n \"by_action.doc_count\"\n ],\n \"integration\": {\n \"name\": \"endpoint\",\n \"version\": \"8.14.2\",\n \"indices\": [\n \"logs-endpoint.events.process-default\"\n ]\n },\n \"circuitBreakers\": { ...\n }\n}\n```\n\n```json\n{\n \"name\": \"endpoint-process-stats\",\n \"traceId\": \"b2c3d4e5-...\",\n \"started\": \"2026-03-18T10:00:00.000Z\",\n \"finished\": \"2026-03-18T10:00:00.012Z\",\n \"descriptorVersion\": 2,\n \"status\": \"skipped\",\n \"skipReason\": \"integration_not_installed\",\n \"passed\": false,\n \"numDocs\": 0,\n \"fieldNames\": []\n}\n```\n\n```json\n{\n \"name\": \"endpoint-process-stats\",\n \"traceId\": \"c3d4e5f6-...\",\n \"started\": \"2026-03-18T10:00:00.000Z\",\n \"finished\": \"2026-03-18T10:00:00.018Z\",\n \"descriptorVersion\": 2,\n \"status\": \"skipped\",\n \"skipReason\": \"datastreams_not_matched\",\n \"passed\": false,\n \"numDocs\": 0,\n \"fieldNames\": [],\n \"integration\": {\n \"name\": \"endpoint\",\n \"version\": \"8.14.2\",\n \"indices\": []\n }\n}\n```","sha":"c8ee39d5f2f726c49573f9e3cf7464d756467b53"}},{"branch":"9.3","label":"v9.3.3","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"url":"https://github.com/elastic/kibana/pull/260877","number":260877,"state":"MERGED","mergeCommit":{"sha":"a33daa282e101a644cc7cad0b654c381111f3547","message":"[9.3] [Security Solution] [HDQ]: integration-based targeting and descriptor versioning (#258418) (#260877)\n\n# Backport\n\nThis will backport the following commits from `main` to `9.3`:\n- [[Security Solution] [HDQ]: integration-based targeting and descriptor\nversioning (#258418)](https://github.com/elastic/kibana/pull/258418)\n\n\n\n### Questions ?\nPlease refer to the [Backport tool\ndocumentation](https://github.com/sorenlouv/backport)\n\n\n\nCo-authored-by: Sebastián Zaffarano <sebastian.zaffarano@elastic.co>"}},{"branch":"9.2","label":"v9.2.8","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"url":"https://github.com/elastic/kibana/pull/260876","number":260876,"state":"MERGED","mergeCommit":{"sha":"2484ea8af037aecc848b80cdf39f66b62eb7b5a0","message":"[9.2] [Security Solution] [HDQ]: integration-based targeting and descriptor versioning (#258418) (#260876)\n\n# Backport\n\nThis will backport the following commits from `main` to `9.2`:\n- [[Security Solution] [HDQ]: integration-based targeting and descriptor\nversioning (#258418)](https://github.com/elastic/kibana/pull/258418)\n\n\n\n### Questions ?\nPlease refer to the [Backport tool\ndocumentation](https://github.com/sorenlouv/backport)\n\n\n\nCo-authored-by: Sebastián Zaffarano <sebastian.zaffarano@elastic.co>"}},{"branch":"8.19","label":"v8.19.14","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT-->
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backport
This will backport the following commits from
mainto9.2:Questions ?
Please refer to the Backport tool documentation