[9.3] [Security Solution] [HDQ]: integration-based targeting and descriptor versioning (#258418)#260877
Merged
kibanamachine merged 1 commit intoelastic:9.3from Apr 2, 2026
Merged
Conversation
… versioning (elastic#258418) ## Summary ### Changes - DHQ descriptor versioning: each query descriptor now includes an optional version field (defaulting to 1). Old v1 descriptors are fully unaffected. - Integration-based targeting (v2): v2 descriptors replace the index field with an integrations field: a comma-separated list of regex patterns matched against Fleet-installed packages. The executor runs one query per matched integration, using the datastream indices resolved for that integration. - Datastream type filtering (v2): an optional datastreamTypes field (comma-separated regexes) narrows which datastreams are included for the matched integration. - Skip reasons: when a v2+ query cannot execute, they are skipped and reported with one of three reasons: integration_not_installed, datastreams_not_matched, or parse_failure. - EBT schema update: stats events gain descriptorVersion, status, and integration fields (matched patterns, resolved indices, integration name/version) to allow downstream consumers to distinguish v1 from v2 executions. - Permission check simplification: removed the indices.exists pre-check from checkPermissions; privilege checking alone is sufficient, and `exists` can lead to FP for indices patterns. ### Query descriptor V2 example ```yaml version: 2 id: "endpoint-process-stats" name: "Endpoint process event counts" integrations: "endpoint" datastreamTypes: "logs" type: "DSL" query: | { "aggs": { "by_action": { "terms": { "field": "event.action", "size": 20 } } }, "size": 0 } scheduleCron: "1h" enabled: true filterlist: "by_action.key": keep "by_action.doc_count": keep ``` ### Stats EBT document examples ```json { "name": "endpoint-process-stats", "traceId": "419cc487-3b17-48f0-bf7e-4e953ed9f050", "started": "2026-03-18T10:00:00.000Z", "finished": "2026-03-18T10:00:00.842Z", "descriptorVersion": 2, "status": "success", "passed": true, "numDocs": 1, "fieldNames": [ "by_action.key", "by_action.doc_count" ], "integration": { "name": "endpoint", "version": "8.14.2", "indices": [ "logs-endpoint.events.process-default" ] }, "circuitBreakers": { ... } } ``` ```json { "name": "endpoint-process-stats", "traceId": "b2c3d4e5-...", "started": "2026-03-18T10:00:00.000Z", "finished": "2026-03-18T10:00:00.012Z", "descriptorVersion": 2, "status": "skipped", "skipReason": "integration_not_installed", "passed": false, "numDocs": 0, "fieldNames": [] } ``` ```json { "name": "endpoint-process-stats", "traceId": "c3d4e5f6-...", "started": "2026-03-18T10:00:00.000Z", "finished": "2026-03-18T10:00:00.018Z", "descriptorVersion": 2, "status": "skipped", "skipReason": "datastreams_not_matched", "passed": false, "numDocs": 0, "fieldNames": [], "integration": { "name": "endpoint", "version": "8.14.2", "indices": [] } } ``` (cherry picked from commit c8ee39d)
8 tasks
Comment on lines
+82
to
+85
| const { integrations: patterns, datastreamTypes: typePatterns } = query; | ||
| if (!patterns) { | ||
| return []; | ||
| } |
Contributor
There was a problem hiding this comment.
🟢 Low diagnostic/health_diagnostic_integration_resolver.ts:82
In resolveV2, when query.integrations is undefined, null, or an empty array, the method returns []. This causes the query to disappear from the output entirely instead of being tracked as skipped. Since resolve uses flatMap, returning [] silently drops the query with no record of why it was excluded.
- if (!patterns) {
- return [];
+ if (!patterns || patterns.length === 0) {
+ return [{ kind: 'skipped', query, reason: 'missing_integrations' } as SkippedQuery];🤖 Copy this AI Prompt to have your agent fix this:
In file x-pack/solutions/security/plugins/security_solution/server/lib/telemetry/diagnostic/health_diagnostic_integration_resolver.ts around lines 82-85:
In `resolveV2`, when `query.integrations` is undefined, null, or an empty array, the method returns `[]`. This causes the query to disappear from the output entirely instead of being tracked as skipped. Since `resolve` uses `flatMap`, returning `[]` silently drops the query with no record of why it was excluded.
Evidence trail:
x-pack/solutions/security/plugins/security_solution/server/lib/telemetry/diagnostic/health_diagnostic_integration_resolver.ts lines 46, 73-80, 93-96, 109-112 at REVIEWED_COMMIT. The `resolve` method uses `flatMap` (line 46). In `resolveV2`, line 78-80 returns `[]` when `patterns` (!query.integrations) is falsy, while other skip scenarios (lines 59-61, 93-96, 109-112) return SkippedQuery objects with reasons.
Comment on lines
139
to
155
| @@ -155,73 +155,72 @@ export class CircuitBreakingQueryExecutorImpl implements CircuitBreakingQueryExe | |||
| } | |||
Contributor
There was a problem hiding this comment.
🟢 Low diagnostic/health_diagnostic_receiver.ts:139
When has_all_requested is false, the thrown PermissionError with message "Missing read privileges" is immediately caught by the outer catch block and re-wrapped as PermissionError: Error checking privileges: PermissionError: Missing read privileges. This buries the specific failure reason under redundant wrapping. Consider checking whether e is already a PermissionError in the catch block and re-throwing it unchanged.
private async checkPermissions(index: Indices) {
try {
const res = await this.client.security.hasPrivileges({
index: [
{
names: index,
privileges: ['read'],
},
],
});
if (!res.has_all_requested) {
throw new PermissionError('Missing read privileges');
}
} catch (e) {
+ if (e instanceof PermissionError) {
+ throw e;
+ }
throw new PermissionError(`Error checking privileges: ${e}`);
}
}🤖 Copy this AI Prompt to have your agent fix this:
In file x-pack/solutions/security/plugins/security_solution/server/lib/telemetry/diagnostic/health_diagnostic_receiver.ts around lines 139-155:
When `has_all_requested` is false, the thrown `PermissionError` with message "Missing read privileges" is immediately caught by the outer `catch` block and re-wrapped as `PermissionError: Error checking privileges: PermissionError: Missing read privileges`. This buries the specific failure reason under redundant wrapping. Consider checking whether `e` is already a `PermissionError` in the catch block and re-throwing it unchanged.
Evidence trail:
x-pack/solutions/security/plugins/security_solution/server/lib/telemetry/diagnostic/health_diagnostic_receiver.ts lines 140-155 at REVIEWED_COMMIT. The try block throws `PermissionError('Missing read privileges')` at line 149 when `has_all_requested` is false. The catch block at lines 151-153 catches this error and re-throws it as `PermissionError(`Error checking privileges: ${e}`)`, creating double-wrapped error messages.
szaffarano
added a commit
that referenced
this pull request
Apr 3, 2026
…criptor versioning (#258418) (#261112) # Backport This will backport the following commits from `main` to `8.19`: - [[Security Solution] [HDQ]: integration-based targeting and descriptor versioning (#258418)](#258418) <!--- Backport version: 11.0.1 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Sebastián Zaffarano","email":"sebastian.zaffarano@elastic.co"},"sourceCommit":{"committedDate":"2026-04-02T08:06:06Z","message":"[Security Solution] [HDQ]: integration-based targeting and descriptor versioning (#258418)\n\n## Summary\n\n### Changes\n- DHQ descriptor versioning: each query descriptor now includes an\noptional version field (defaulting to 1). Old v1 descriptors are fully\nunaffected.\n- Integration-based targeting (v2): v2 descriptors replace the index\nfield with an integrations field: a comma-separated list of regex\npatterns matched against Fleet-installed packages. The executor runs one\nquery per matched integration, using the datastream indices resolved for\nthat integration.\n- Datastream type filtering (v2): an optional datastreamTypes field\n(comma-separated regexes) narrows which datastreams are included for the\nmatched integration.\n- Skip reasons: when a v2+ query cannot execute, they are skipped and\nreported with one of three reasons: integration_not_installed,\ndatastreams_not_matched, or parse_failure.\n- EBT schema update: stats events gain descriptorVersion, status, and\nintegration fields (matched patterns, resolved indices, integration\nname/version) to allow downstream consumers to distinguish v1 from v2\nexecutions.\n- Permission check simplification: removed the indices.exists pre-check\nfrom checkPermissions; privilege checking alone is sufficient, and\n`exists` can lead to FP for indices patterns.\n\n### Query descriptor V2 example\n\n```yaml\nversion: 2\nid: \"endpoint-process-stats\"\nname: \"Endpoint process event counts\"\nintegrations: \"endpoint\"\ndatastreamTypes: \"logs\"\ntype: \"DSL\"\nquery: |\n {\n \"aggs\": { \"by_action\": { \"terms\": { \"field\": \"event.action\", \"size\": 20 } } },\n \"size\": 0\n }\nscheduleCron: \"1h\"\nenabled: true\nfilterlist:\n \"by_action.key\": keep\n \"by_action.doc_count\": keep\n```\n\n### Stats EBT document examples\n\n```json\n{\n \"name\": \"endpoint-process-stats\",\n \"traceId\": \"419cc487-3b17-48f0-bf7e-4e953ed9f050\",\n \"started\": \"2026-03-18T10:00:00.000Z\",\n \"finished\": \"2026-03-18T10:00:00.842Z\",\n \"descriptorVersion\": 2,\n \"status\": \"success\",\n \"passed\": true,\n \"numDocs\": 1,\n \"fieldNames\": [\n \"by_action.key\",\n \"by_action.doc_count\"\n ],\n \"integration\": {\n \"name\": \"endpoint\",\n \"version\": \"8.14.2\",\n \"indices\": [\n \"logs-endpoint.events.process-default\"\n ]\n },\n \"circuitBreakers\": { ...\n }\n}\n```\n\n```json\n{\n \"name\": \"endpoint-process-stats\",\n \"traceId\": \"b2c3d4e5-...\",\n \"started\": \"2026-03-18T10:00:00.000Z\",\n \"finished\": \"2026-03-18T10:00:00.012Z\",\n \"descriptorVersion\": 2,\n \"status\": \"skipped\",\n \"skipReason\": \"integration_not_installed\",\n \"passed\": false,\n \"numDocs\": 0,\n \"fieldNames\": []\n}\n```\n\n```json\n{\n \"name\": \"endpoint-process-stats\",\n \"traceId\": \"c3d4e5f6-...\",\n \"started\": \"2026-03-18T10:00:00.000Z\",\n \"finished\": \"2026-03-18T10:00:00.018Z\",\n \"descriptorVersion\": 2,\n \"status\": \"skipped\",\n \"skipReason\": \"datastreams_not_matched\",\n \"passed\": false,\n \"numDocs\": 0,\n \"fieldNames\": [],\n \"integration\": {\n \"name\": \"endpoint\",\n \"version\": \"8.14.2\",\n \"indices\": []\n }\n}\n```","sha":"c8ee39d5f2f726c49573f9e3cf7464d756467b53","branchLabelMapping":{"^v9.4.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","ci:build-cloud-image","ci:build-serverless-image","backport:version","v9.4.0","v9.3.3","v9.2.8","v8.19.14"],"title":"[Security Solution] [HDQ]: integration-based targeting and descriptor versioning","number":258418,"url":"https://github.com/elastic/kibana/pull/258418","mergeCommit":{"message":"[Security Solution] [HDQ]: integration-based targeting and descriptor versioning (#258418)\n\n## Summary\n\n### Changes\n- DHQ descriptor versioning: each query descriptor now includes an\noptional version field (defaulting to 1). Old v1 descriptors are fully\nunaffected.\n- Integration-based targeting (v2): v2 descriptors replace the index\nfield with an integrations field: a comma-separated list of regex\npatterns matched against Fleet-installed packages. The executor runs one\nquery per matched integration, using the datastream indices resolved for\nthat integration.\n- Datastream type filtering (v2): an optional datastreamTypes field\n(comma-separated regexes) narrows which datastreams are included for the\nmatched integration.\n- Skip reasons: when a v2+ query cannot execute, they are skipped and\nreported with one of three reasons: integration_not_installed,\ndatastreams_not_matched, or parse_failure.\n- EBT schema update: stats events gain descriptorVersion, status, and\nintegration fields (matched patterns, resolved indices, integration\nname/version) to allow downstream consumers to distinguish v1 from v2\nexecutions.\n- Permission check simplification: removed the indices.exists pre-check\nfrom checkPermissions; privilege checking alone is sufficient, and\n`exists` can lead to FP for indices patterns.\n\n### Query descriptor V2 example\n\n```yaml\nversion: 2\nid: \"endpoint-process-stats\"\nname: \"Endpoint process event counts\"\nintegrations: \"endpoint\"\ndatastreamTypes: \"logs\"\ntype: \"DSL\"\nquery: |\n {\n \"aggs\": { \"by_action\": { \"terms\": { \"field\": \"event.action\", \"size\": 20 } } },\n \"size\": 0\n }\nscheduleCron: \"1h\"\nenabled: true\nfilterlist:\n \"by_action.key\": keep\n \"by_action.doc_count\": keep\n```\n\n### Stats EBT document examples\n\n```json\n{\n \"name\": \"endpoint-process-stats\",\n \"traceId\": \"419cc487-3b17-48f0-bf7e-4e953ed9f050\",\n \"started\": \"2026-03-18T10:00:00.000Z\",\n \"finished\": \"2026-03-18T10:00:00.842Z\",\n \"descriptorVersion\": 2,\n \"status\": \"success\",\n \"passed\": true,\n \"numDocs\": 1,\n \"fieldNames\": [\n \"by_action.key\",\n \"by_action.doc_count\"\n ],\n \"integration\": {\n \"name\": \"endpoint\",\n \"version\": \"8.14.2\",\n \"indices\": [\n \"logs-endpoint.events.process-default\"\n ]\n },\n \"circuitBreakers\": { ...\n }\n}\n```\n\n```json\n{\n \"name\": \"endpoint-process-stats\",\n \"traceId\": \"b2c3d4e5-...\",\n \"started\": \"2026-03-18T10:00:00.000Z\",\n \"finished\": \"2026-03-18T10:00:00.012Z\",\n \"descriptorVersion\": 2,\n \"status\": \"skipped\",\n \"skipReason\": \"integration_not_installed\",\n \"passed\": false,\n \"numDocs\": 0,\n \"fieldNames\": []\n}\n```\n\n```json\n{\n \"name\": \"endpoint-process-stats\",\n \"traceId\": \"c3d4e5f6-...\",\n \"started\": \"2026-03-18T10:00:00.000Z\",\n \"finished\": \"2026-03-18T10:00:00.018Z\",\n \"descriptorVersion\": 2,\n \"status\": \"skipped\",\n \"skipReason\": \"datastreams_not_matched\",\n \"passed\": false,\n \"numDocs\": 0,\n \"fieldNames\": [],\n \"integration\": {\n \"name\": \"endpoint\",\n \"version\": \"8.14.2\",\n \"indices\": []\n }\n}\n```","sha":"c8ee39d5f2f726c49573f9e3cf7464d756467b53"}},"sourceBranch":"main","suggestedTargetBranches":["8.19"],"targetPullRequestStates":[{"branch":"main","label":"v9.4.0","branchLabelMappingKey":"^v9.4.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/258418","number":258418,"mergeCommit":{"message":"[Security Solution] [HDQ]: integration-based targeting and descriptor versioning (#258418)\n\n## Summary\n\n### Changes\n- DHQ descriptor versioning: each query descriptor now includes an\noptional version field (defaulting to 1). Old v1 descriptors are fully\nunaffected.\n- Integration-based targeting (v2): v2 descriptors replace the index\nfield with an integrations field: a comma-separated list of regex\npatterns matched against Fleet-installed packages. The executor runs one\nquery per matched integration, using the datastream indices resolved for\nthat integration.\n- Datastream type filtering (v2): an optional datastreamTypes field\n(comma-separated regexes) narrows which datastreams are included for the\nmatched integration.\n- Skip reasons: when a v2+ query cannot execute, they are skipped and\nreported with one of three reasons: integration_not_installed,\ndatastreams_not_matched, or parse_failure.\n- EBT schema update: stats events gain descriptorVersion, status, and\nintegration fields (matched patterns, resolved indices, integration\nname/version) to allow downstream consumers to distinguish v1 from v2\nexecutions.\n- Permission check simplification: removed the indices.exists pre-check\nfrom checkPermissions; privilege checking alone is sufficient, and\n`exists` can lead to FP for indices patterns.\n\n### Query descriptor V2 example\n\n```yaml\nversion: 2\nid: \"endpoint-process-stats\"\nname: \"Endpoint process event counts\"\nintegrations: \"endpoint\"\ndatastreamTypes: \"logs\"\ntype: \"DSL\"\nquery: |\n {\n \"aggs\": { \"by_action\": { \"terms\": { \"field\": \"event.action\", \"size\": 20 } } },\n \"size\": 0\n }\nscheduleCron: \"1h\"\nenabled: true\nfilterlist:\n \"by_action.key\": keep\n \"by_action.doc_count\": keep\n```\n\n### Stats EBT document examples\n\n```json\n{\n \"name\": \"endpoint-process-stats\",\n \"traceId\": \"419cc487-3b17-48f0-bf7e-4e953ed9f050\",\n \"started\": \"2026-03-18T10:00:00.000Z\",\n \"finished\": \"2026-03-18T10:00:00.842Z\",\n \"descriptorVersion\": 2,\n \"status\": \"success\",\n \"passed\": true,\n \"numDocs\": 1,\n \"fieldNames\": [\n \"by_action.key\",\n \"by_action.doc_count\"\n ],\n \"integration\": {\n \"name\": \"endpoint\",\n \"version\": \"8.14.2\",\n \"indices\": [\n \"logs-endpoint.events.process-default\"\n ]\n },\n \"circuitBreakers\": { ...\n }\n}\n```\n\n```json\n{\n \"name\": \"endpoint-process-stats\",\n \"traceId\": \"b2c3d4e5-...\",\n \"started\": \"2026-03-18T10:00:00.000Z\",\n \"finished\": \"2026-03-18T10:00:00.012Z\",\n \"descriptorVersion\": 2,\n \"status\": \"skipped\",\n \"skipReason\": \"integration_not_installed\",\n \"passed\": false,\n \"numDocs\": 0,\n \"fieldNames\": []\n}\n```\n\n```json\n{\n \"name\": \"endpoint-process-stats\",\n \"traceId\": \"c3d4e5f6-...\",\n \"started\": \"2026-03-18T10:00:00.000Z\",\n \"finished\": \"2026-03-18T10:00:00.018Z\",\n \"descriptorVersion\": 2,\n \"status\": \"skipped\",\n \"skipReason\": \"datastreams_not_matched\",\n \"passed\": false,\n \"numDocs\": 0,\n \"fieldNames\": [],\n \"integration\": {\n \"name\": \"endpoint\",\n \"version\": \"8.14.2\",\n \"indices\": []\n }\n}\n```","sha":"c8ee39d5f2f726c49573f9e3cf7464d756467b53"}},{"branch":"9.3","label":"v9.3.3","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"url":"https://github.com/elastic/kibana/pull/260877","number":260877,"state":"MERGED","mergeCommit":{"sha":"a33daa282e101a644cc7cad0b654c381111f3547","message":"[9.3] [Security Solution] [HDQ]: integration-based targeting and descriptor versioning (#258418) (#260877)\n\n# Backport\n\nThis will backport the following commits from `main` to `9.3`:\n- [[Security Solution] [HDQ]: integration-based targeting and descriptor\nversioning (#258418)](https://github.com/elastic/kibana/pull/258418)\n\n\n\n### Questions ?\nPlease refer to the [Backport tool\ndocumentation](https://github.com/sorenlouv/backport)\n\n\n\nCo-authored-by: Sebastián Zaffarano <sebastian.zaffarano@elastic.co>"}},{"branch":"9.2","label":"v9.2.8","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"url":"https://github.com/elastic/kibana/pull/260876","number":260876,"state":"MERGED","mergeCommit":{"sha":"2484ea8af037aecc848b80cdf39f66b62eb7b5a0","message":"[9.2] [Security Solution] [HDQ]: integration-based targeting and descriptor versioning (#258418) (#260876)\n\n# Backport\n\nThis will backport the following commits from `main` to `9.2`:\n- [[Security Solution] [HDQ]: integration-based targeting and descriptor\nversioning (#258418)](https://github.com/elastic/kibana/pull/258418)\n\n\n\n### Questions ?\nPlease refer to the [Backport tool\ndocumentation](https://github.com/sorenlouv/backport)\n\n\n\nCo-authored-by: Sebastián Zaffarano <sebastian.zaffarano@elastic.co>"}},{"branch":"8.19","label":"v8.19.14","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT-->
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backport
This will backport the following commits from
mainto9.3:Questions ?
Please refer to the Backport tool documentation