Skip to content

[Security Solution] [HDQ]: integration-based targeting and descriptor versioning#258418

Merged
szaffarano merged 34 commits intoelastic:mainfrom
szaffarano:szaffarano/dhq-integrations
Apr 2, 2026
Merged

[Security Solution] [HDQ]: integration-based targeting and descriptor versioning#258418
szaffarano merged 34 commits intoelastic:mainfrom
szaffarano:szaffarano/dhq-integrations

Conversation

@szaffarano
Copy link
Copy Markdown
Contributor

@szaffarano szaffarano commented Mar 18, 2026

Summary

Changes

  • DHQ descriptor versioning: each query descriptor now includes an optional version field (defaulting to 1). Old v1 descriptors are fully unaffected.
  • Integration-based targeting (v2): v2 descriptors replace the index field with an integrations field: a comma-separated list of regex patterns matched against Fleet-installed packages. The executor runs one query per matched integration, using the datastream indices resolved for that integration.
  • Datastream type filtering (v2): an optional datastreamTypes field (comma-separated regexes) narrows which datastreams are included for the matched integration.
  • Skip reasons: when a v2+ query cannot execute, they are skipped and reported with one of three reasons: integration_not_installed, datastreams_not_matched, or parse_failure.
  • EBT schema update: stats events gain descriptorVersion, status, and integration fields (matched patterns, resolved indices, integration name/version) to allow downstream consumers to distinguish v1 from v2 executions.
  • Permission check simplification: removed the indices.exists pre-check from checkPermissions; privilege checking alone is sufficient, and exists can lead to FP for indices patterns.

Query descriptor V2 example

version: 2
id: "endpoint-process-stats"
name: "Endpoint process event counts"
integrations: "endpoint"
datastreamTypes: "logs"
type: "DSL"
query: |
  {
    "aggs": { "by_action": { "terms": { "field": "event.action", "size": 20 } } },
    "size": 0
  }
scheduleCron: "1h"
enabled: true
filterlist:
  "by_action.key": keep
  "by_action.doc_count": keep

Stats EBT document examples

{
  "name": "endpoint-process-stats",
  "traceId": "419cc487-3b17-48f0-bf7e-4e953ed9f050",
  "started": "2026-03-18T10:00:00.000Z",
  "finished": "2026-03-18T10:00:00.842Z",
  "descriptorVersion": 2,
  "status": "success",
  "passed": true,
  "numDocs": 1,
  "fieldNames": [
    "by_action.key",
    "by_action.doc_count"
  ],
  "integration": {
    "name": "endpoint",
    "version": "8.14.2",
    "indices": [
      "logs-endpoint.events.process-default"
    ]
  },
  "circuitBreakers": { ...
  }
}
{
  "name": "endpoint-process-stats",
  "traceId": "b2c3d4e5-...",
  "started": "2026-03-18T10:00:00.000Z",
  "finished": "2026-03-18T10:00:00.012Z",
  "descriptorVersion": 2,
  "status": "skipped",
  "skipReason": "integration_not_installed",
  "passed": false,
  "numDocs": 0,
  "fieldNames": []
}
{
  "name": "endpoint-process-stats",
  "traceId": "c3d4e5f6-...",
  "started": "2026-03-18T10:00:00.000Z",
  "finished": "2026-03-18T10:00:00.018Z",
  "descriptorVersion": 2,
  "status": "skipped",
  "skipReason": "datastreams_not_matched",
  "passed": false,
  "numDocs": 0,
  "fieldNames": [],
  "integration": {
    "name": "endpoint",
    "version": "8.14.2",
    "indices": []
  }
}

Checklist

Check the PR satisfies following conditions.

Reviewers should verify this PR satisfies this list as well.

  • Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
  • Documentation was added for features that require explanation or tutorials
  • Unit or functional tests were updated or added to match the most common scenarios
  • If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the docker list
  • This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The release_note:breaking label should be applied in these situations.
  • Flaky Test Runner was used on any tests changed
  • The PR description includes the appropriate Release Notes section, and the correct release_note:* label is applied per the guidelines
  • Review the backport guidelines and apply applicable backport:* labels.

@szaffarano szaffarano self-assigned this Mar 18, 2026
@szaffarano szaffarano requested review from a team as code owners March 18, 2026 16:51
@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai bot commented Apr 1, 2026

✅ Actions performed

Full review triggered.

1 similar comment
@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai bot commented Apr 1, 2026

✅ Actions performed

Full review triggered.

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

♻️ Duplicate comments (1)
x-pack/solutions/security/plugins/security_solution/server/lib/telemetry/diagnostic/health_diagnostic_receiver.ts (1)

91-104: ⚠️ Potential issue | 🟠 Major

Run ES|QL once against the resolved source set.

Line 289 can return multiple indices for one executable query, but Line 93 immediately fans that set back out into separate ES|QL requests. That breaks aggregation semantics: STATS/COUNT are computed per index instead of once across the full integration/tier-filtered source set, so valid multi-datastream queries emit duplicate or partial rows.

Also applies to: 252-290

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In
`@x-pack/solutions/security/plugins/security_solution/server/lib/telemetry/diagnostic/health_diagnostic_receiver.ts`
around lines 91 - 104, The current code fans out the resolved indices from
indicesFor(executableQuery) into separate ES|QL requests which causes
aggregation functions (e.g., STATS/COUNT) to run per-index; instead, after
checkPermissions(originalIndices) and inside the mergeMap that receives indices,
join the indices into a single source (e.g., indices.join(',')) and build one
esqlQuery (preserve the regex.test(query.query) branch so if query.query already
contains FROM you use it as-is); then call this.client.helpers.esql once with
that combined esqlQuery and abortSignal, call toRecords(), and map resp.records
to T as before—remove the indices.map(...) fan-out so the ES|QL runs once across
the full resolved source set.
🧹 Nitpick comments (2)
x-pack/solutions/security/plugins/security_solution/server/lib/telemetry/diagnostic/health_diagnostic_integration_resolver.ts (2)

86-95: Regex from variable input—controlled source but consider validation.

Static analysis flags potential ReDoS at lines 88 and 108. The patterns originate from internal artifact YAML (controlled), but a defense-in-depth approach would limit pattern complexity or add a timeout. The try/catch mitigates invalid regex but not catastrophic backtracking.

Low risk given the controlled source—acceptable to defer.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In
`@x-pack/solutions/security/plugins/security_solution/server/lib/telemetry/diagnostic/health_diagnostic_integration_resolver.ts`
around lines 86 - 95, The current creation of RegExp from variable patterns in
the matched calculation (installedPackages.filter + patterns.some creating new
RegExp(`^${pattern}$`)) risks ReDoS from complex patterns; add a lightweight
validation/sanitization step before constructing the RegExp: reject or escape
patterns that exceed a safe length or contain known catastrophic constructs
(e.g., nested quantifiers, excessive consecutive wildcards, unbounded
backtracking patterns), and log a warn via this.logger.warn when rejecting a
pattern, falling back to a safe literal match or skip; apply the same validation
where patterns are used elsewhere (referenced usage around line ~108) so only
validated patterns are passed into new RegExp.

78-85: Empty result when integrations is undefined.

If a v2 query somehow reaches resolveV2 with patterns undefined, it silently returns []. The type invariant is "enforced by parser," but a defensive skip result would improve observability.

Optional defensive tweak
   private resolveV2(
     query: HealthDiagnosticQueryV2,
     installedPackages: InstalledPackage[]
   ): ResolvedQuery[] {
     const { integrations: patterns, datastreamTypes: typePatterns } = query;
     if (!patterns) {
-      return [];
+      this.logger.warn('v2 query reached resolveV2 without integrations', { queryName: query.name });
+      return [{ kind: 'skipped', query, reason: 'parse_failure' }];
     }
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In
`@x-pack/solutions/security/plugins/security_solution/server/lib/telemetry/diagnostic/health_diagnostic_integration_resolver.ts`
around lines 78 - 85, resolveV2 currently returns [] silently when
query.integrations (patterns) is undefined; add a defensive observability action
before returning: detect when patterns is falsy in resolveV2 and emit a warning
(using the class's existing logger, e.g., this.logger.warn or the telemetry
logger available in this module) that includes the received
HealthDiagnosticQueryV2 payload and a short message indicating integrations was
missing, then return the empty array; if no logger exists in this class,
add/accept one (or use the module-level logger) so callers/ops can see these
malformed queries.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In
`@x-pack/solutions/security/plugins/security_solution/server/lib/telemetry/diagnostic/health_diagnostic_query_parser.ts`:
- Around line 36-46: The current parseV1 (and the analogous parseV2/other parse
functions) only perform shallow presence/type checks then cast raw to
HealthDiagnosticQueryV1/V2, allowing invalid enum values or wrong plain-object
shapes through; update parseV1 and parseV2 to fully validate enumerated fields
(e.g., validate 'type' is one of the allowed diagnostic types), ensure
'integrations' is a string[]/array when present, verify 'filterlist' is a plain
object (not an array) and that its entries/actions are valid enums/values before
performing the cast, and return a ParseFailureQuery on invalid shapes instead of
casting; use the existing assert helpers as a starting point but add explicit
enum/shape checks for symbols like parseV1, parseV2, 'type', 'integrations', and
'filterlist'.

---

Duplicate comments:
In
`@x-pack/solutions/security/plugins/security_solution/server/lib/telemetry/diagnostic/health_diagnostic_receiver.ts`:
- Around line 91-104: The current code fans out the resolved indices from
indicesFor(executableQuery) into separate ES|QL requests which causes
aggregation functions (e.g., STATS/COUNT) to run per-index; instead, after
checkPermissions(originalIndices) and inside the mergeMap that receives indices,
join the indices into a single source (e.g., indices.join(',')) and build one
esqlQuery (preserve the regex.test(query.query) branch so if query.query already
contains FROM you use it as-is); then call this.client.helpers.esql once with
that combined esqlQuery and abortSignal, call toRecords(), and map resp.records
to T as before—remove the indices.map(...) fan-out so the ES|QL runs once across
the full resolved source set.

---

Nitpick comments:
In
`@x-pack/solutions/security/plugins/security_solution/server/lib/telemetry/diagnostic/health_diagnostic_integration_resolver.ts`:
- Around line 86-95: The current creation of RegExp from variable patterns in
the matched calculation (installedPackages.filter + patterns.some creating new
RegExp(`^${pattern}$`)) risks ReDoS from complex patterns; add a lightweight
validation/sanitization step before constructing the RegExp: reject or escape
patterns that exceed a safe length or contain known catastrophic constructs
(e.g., nested quantifiers, excessive consecutive wildcards, unbounded
backtracking patterns), and log a warn via this.logger.warn when rejecting a
pattern, falling back to a safe literal match or skip; apply the same validation
where patterns are used elsewhere (referenced usage around line ~108) so only
validated patterns are passed into new RegExp.
- Around line 78-85: resolveV2 currently returns [] silently when
query.integrations (patterns) is undefined; add a defensive observability action
before returning: detect when patterns is falsy in resolveV2 and emit a warning
(using the class's existing logger, e.g., this.logger.warn or the telemetry
logger available in this module) that includes the received
HealthDiagnosticQueryV2 payload and a short message indicating integrations was
missing, then return the empty array; if no logger exists in this class,
add/accept one (or use the module-level logger) so callers/ops can see these
malformed queries.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Organization UI (inherited)

Review profile: CHILL

Plan: Pro

Run ID: eb178385-54ad-4d7c-b6f0-9bf4fc85f107

📥 Commits

Reviewing files that changed from the base of the PR and between 2db60b8 and c7e0f26.

📒 Files selected for processing (14)
  • x-pack/solutions/security/plugins/security_solution/server/lib/telemetry/diagnostic/__mocks__/index.ts
  • x-pack/solutions/security/plugins/security_solution/server/lib/telemetry/diagnostic/health_diagnostic_integration_resolver.test.ts
  • x-pack/solutions/security/plugins/security_solution/server/lib/telemetry/diagnostic/health_diagnostic_integration_resolver.ts
  • x-pack/solutions/security/plugins/security_solution/server/lib/telemetry/diagnostic/health_diagnostic_query_parser.test.ts
  • x-pack/solutions/security/plugins/security_solution/server/lib/telemetry/diagnostic/health_diagnostic_query_parser.ts
  • x-pack/solutions/security/plugins/security_solution/server/lib/telemetry/diagnostic/health_diagnostic_receiver.test.ts
  • x-pack/solutions/security/plugins/security_solution/server/lib/telemetry/diagnostic/health_diagnostic_receiver.ts
  • x-pack/solutions/security/plugins/security_solution/server/lib/telemetry/diagnostic/health_diagnostic_receiver.types.ts
  • x-pack/solutions/security/plugins/security_solution/server/lib/telemetry/diagnostic/health_diagnostic_service.test.ts
  • x-pack/solutions/security/plugins/security_solution/server/lib/telemetry/diagnostic/health_diagnostic_service.ts
  • x-pack/solutions/security/plugins/security_solution/server/lib/telemetry/diagnostic/health_diagnostic_service.types.ts
  • x-pack/solutions/security/plugins/security_solution/server/lib/telemetry/diagnostic/health_diagnostic_utils.ts
  • x-pack/solutions/security/plugins/security_solution/server/lib/telemetry/event_based/events.ts
  • x-pack/solutions/security/plugins/security_solution/server/plugin.ts

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

♻️ Duplicate comments (2)
x-pack/solutions/security/plugins/security_solution/server/lib/telemetry/diagnostic/health_diagnostic_query_parser.ts (1)

36-95: ⚠️ Potential issue | 🟠 Major

Validate runnable descriptor shape before casting.

These guards still let malformed descriptors through as HealthDiagnosticQueryV1/V2: whitespace-only index / integrations, arbitrary type values, filterlist: [], and invalid filter actions all survive the cast. That means bad artifacts fail later in search() / applyFilterlist() instead of going down the intended ParseFailureQuery path.

Also applies to: 106-116

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In
`@x-pack/solutions/security/plugins/security_solution/server/lib/telemetry/diagnostic/health_diagnostic_query_parser.ts`
around lines 36 - 95, parseV1 and parseV2 currently assert presence but still
cast malformed descriptors into HealthDiagnosticQueryV1/V2; tighten validation
by rejecting whitespace-only strings and validating enumerated fields and
structured objects before returning: in parseV1 ensure index and type are
non-empty trimmed strings and validate filterlist is a non-empty object with
valid filters/actions (use or add a validateFilterlist function and call it
before casting), and in parseV2 similarly reject whitespace-only
integrations/index, validate datastreamTypes when present, ensure type is an
allowed value, and validate filterlist contents; update both parseV1 and parseV2
to throw a ParseFailure-style Error when these stricter checks fail (also apply
the same stricter checks used in the later parsing logic referenced around
datastreamTypes/integrations).
x-pack/solutions/security/plugins/security_solution/server/lib/telemetry/diagnostic/health_diagnostic_receiver.ts (1)

91-104: ⚠️ Potential issue | 🟠 Major

Run the ES|QL query once for the whole resolved source set.

This still fans one descriptor out over every entry returned by indicesFor(). That changes ES|QL semantics as soon as one integration/datastream resolves to multiple indices: STATS, SORT ... LIMIT, and similar operators are computed per index instead of per integration, so the result rows and numDocs become wrong. Build a single FROM index1,index2,... | ... query here (or execute the inline-FROM form once).

Patch sketch
     return from(this.checkPermissions(originalIndices)).pipe(
       mergeMap(() => from(this.indicesFor(executableQuery))),
-      mergeMap((indices) =>
-        from(
-          indices.map((index) => {
-            const esqlQuery = regex.test(query.query)
-              ? query.query
-              : `FROM ${index} | ${query.query}`;
-            return from(
-              this.client.helpers.esql({ query: esqlQuery }, { signal: abortSignal }).toRecords()
-            ).pipe(mergeMap((resp) => resp.records.map((r) => r as T)));
-          })
-        ).pipe(mergeMap((obs) => obs))
-      )
+      mergeMap((indices) => {
+        if (indices.length === 0) {
+          return EMPTY;
+        }
+        const esqlQuery = regex.test(query.query)
+          ? query.query
+          : `FROM ${indices.join(',')} | ${query.query}`;
+        return from(
+          this.client.helpers.esql({ query: esqlQuery }, { signal: abortSignal }).toRecords()
+        ).pipe(mergeMap((resp) => resp.records.map((r) => r as T)));
+      })
     );
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In
`@x-pack/solutions/security/plugins/security_solution/server/lib/telemetry/diagnostic/health_diagnostic_receiver.ts`
around lines 91 - 104, The current code in health_diagnostic_receiver runs
client.helpers.esql once per index by mapping over indices from
indicesFor(executableQuery), which breaks ES|QL semantics for multi-index
integrations; change the logic in the mergeMap that consumes indices to build a
single comma-separated index list (e.g., const indicesList = indices.join(','))
and then run this.client.helpers.esql exactly once with either the original
query.query if it already contains a FROM or with `FROM ${indicesList} |
${query.query}`; keep using the existing abortSignal and toRecords(), then map
resp.records to T as before (replacing the per-index indices.map(...) block with
a single esql invocation).
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Duplicate comments:
In
`@x-pack/solutions/security/plugins/security_solution/server/lib/telemetry/diagnostic/health_diagnostic_query_parser.ts`:
- Around line 36-95: parseV1 and parseV2 currently assert presence but still
cast malformed descriptors into HealthDiagnosticQueryV1/V2; tighten validation
by rejecting whitespace-only strings and validating enumerated fields and
structured objects before returning: in parseV1 ensure index and type are
non-empty trimmed strings and validate filterlist is a non-empty object with
valid filters/actions (use or add a validateFilterlist function and call it
before casting), and in parseV2 similarly reject whitespace-only
integrations/index, validate datastreamTypes when present, ensure type is an
allowed value, and validate filterlist contents; update both parseV1 and parseV2
to throw a ParseFailure-style Error when these stricter checks fail (also apply
the same stricter checks used in the later parsing logic referenced around
datastreamTypes/integrations).

In
`@x-pack/solutions/security/plugins/security_solution/server/lib/telemetry/diagnostic/health_diagnostic_receiver.ts`:
- Around line 91-104: The current code in health_diagnostic_receiver runs
client.helpers.esql once per index by mapping over indices from
indicesFor(executableQuery), which breaks ES|QL semantics for multi-index
integrations; change the logic in the mergeMap that consumes indices to build a
single comma-separated index list (e.g., const indicesList = indices.join(','))
and then run this.client.helpers.esql exactly once with either the original
query.query if it already contains a FROM or with `FROM ${indicesList} |
${query.query}`; keep using the existing abortSignal and toRecords(), then map
resp.records to T as before (replacing the per-index indices.map(...) block with
a single esql invocation).

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Organization UI (inherited)

Review profile: CHILL

Plan: Pro

Run ID: 306de971-1e58-4005-83cc-b9a4036db3a9

📥 Commits

Reviewing files that changed from the base of the PR and between 2db60b8 and c7e0f26.

📒 Files selected for processing (14)
  • x-pack/solutions/security/plugins/security_solution/server/lib/telemetry/diagnostic/__mocks__/index.ts
  • x-pack/solutions/security/plugins/security_solution/server/lib/telemetry/diagnostic/health_diagnostic_integration_resolver.test.ts
  • x-pack/solutions/security/plugins/security_solution/server/lib/telemetry/diagnostic/health_diagnostic_integration_resolver.ts
  • x-pack/solutions/security/plugins/security_solution/server/lib/telemetry/diagnostic/health_diagnostic_query_parser.test.ts
  • x-pack/solutions/security/plugins/security_solution/server/lib/telemetry/diagnostic/health_diagnostic_query_parser.ts
  • x-pack/solutions/security/plugins/security_solution/server/lib/telemetry/diagnostic/health_diagnostic_receiver.test.ts
  • x-pack/solutions/security/plugins/security_solution/server/lib/telemetry/diagnostic/health_diagnostic_receiver.ts
  • x-pack/solutions/security/plugins/security_solution/server/lib/telemetry/diagnostic/health_diagnostic_receiver.types.ts
  • x-pack/solutions/security/plugins/security_solution/server/lib/telemetry/diagnostic/health_diagnostic_service.test.ts
  • x-pack/solutions/security/plugins/security_solution/server/lib/telemetry/diagnostic/health_diagnostic_service.ts
  • x-pack/solutions/security/plugins/security_solution/server/lib/telemetry/diagnostic/health_diagnostic_service.types.ts
  • x-pack/solutions/security/plugins/security_solution/server/lib/telemetry/diagnostic/health_diagnostic_utils.ts
  • x-pack/solutions/security/plugins/security_solution/server/lib/telemetry/event_based/events.ts
  • x-pack/solutions/security/plugins/security_solution/server/plugin.ts

@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Apr 2, 2026

💛 Build succeeded, but was flaky

  • Buildkite Build
  • Commit: 1544465
  • Kibana Serverless Image: docker.elastic.co/kibana-ci/kibana-serverless:pr-258418-1544465b5fdc

Failed CI Steps

Test Failures

  • [job] [logs] FTR Configs #98 / Agent Builder sidebar Sidebar Error Handling shows an error message and allows the user to retry

Metrics [docs]

Unknown metric groups

ESLint disabled line counts

id before after diff
securitySolution 746 747 +1

Total ESLint disabled count

id before after diff
securitySolution 851 852 +1

History

cc @szaffarano

@szaffarano szaffarano merged commit c8ee39d into elastic:main Apr 2, 2026
22 checks passed
@kibanamachine
Copy link
Copy Markdown
Contributor

Starting backport for target branches: 8.19, 9.2, 9.3

https://github.com/elastic/kibana/actions/runs/23890645227

kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Apr 2, 2026
… versioning (elastic#258418)

## Summary

### Changes
- DHQ descriptor versioning: each query descriptor now includes an
optional version field (defaulting to 1). Old v1 descriptors are fully
unaffected.
- Integration-based targeting (v2): v2 descriptors replace the index
field with an integrations field: a comma-separated list of regex
patterns matched against Fleet-installed packages. The executor runs one
query per matched integration, using the datastream indices resolved for
that integration.
- Datastream type filtering (v2): an optional datastreamTypes field
(comma-separated regexes) narrows which datastreams are included for the
matched integration.
- Skip reasons: when a v2+ query cannot execute, they are skipped and
reported with one of three reasons: integration_not_installed,
datastreams_not_matched, or parse_failure.
- EBT schema update: stats events gain descriptorVersion, status, and
integration fields (matched patterns, resolved indices, integration
name/version) to allow downstream consumers to distinguish v1 from v2
executions.
- Permission check simplification: removed the indices.exists pre-check
from checkPermissions; privilege checking alone is sufficient, and
`exists` can lead to FP for indices patterns.

### Query descriptor V2 example

```yaml
version: 2
id: "endpoint-process-stats"
name: "Endpoint process event counts"
integrations: "endpoint"
datastreamTypes: "logs"
type: "DSL"
query: |
  {
    "aggs": { "by_action": { "terms": { "field": "event.action", "size": 20 } } },
    "size": 0
  }
scheduleCron: "1h"
enabled: true
filterlist:
  "by_action.key": keep
  "by_action.doc_count": keep
```

### Stats EBT document examples

```json
{
  "name": "endpoint-process-stats",
  "traceId": "419cc487-3b17-48f0-bf7e-4e953ed9f050",
  "started": "2026-03-18T10:00:00.000Z",
  "finished": "2026-03-18T10:00:00.842Z",
  "descriptorVersion": 2,
  "status": "success",
  "passed": true,
  "numDocs": 1,
  "fieldNames": [
    "by_action.key",
    "by_action.doc_count"
  ],
  "integration": {
    "name": "endpoint",
    "version": "8.14.2",
    "indices": [
      "logs-endpoint.events.process-default"
    ]
  },
  "circuitBreakers": { ...
  }
}
```

```json
{
  "name": "endpoint-process-stats",
  "traceId": "b2c3d4e5-...",
  "started": "2026-03-18T10:00:00.000Z",
  "finished": "2026-03-18T10:00:00.012Z",
  "descriptorVersion": 2,
  "status": "skipped",
  "skipReason": "integration_not_installed",
  "passed": false,
  "numDocs": 0,
  "fieldNames": []
}
```

```json
{
  "name": "endpoint-process-stats",
  "traceId": "c3d4e5f6-...",
  "started": "2026-03-18T10:00:00.000Z",
  "finished": "2026-03-18T10:00:00.018Z",
  "descriptorVersion": 2,
  "status": "skipped",
  "skipReason": "datastreams_not_matched",
  "passed": false,
  "numDocs": 0,
  "fieldNames": [],
  "integration": {
    "name": "endpoint",
    "version": "8.14.2",
    "indices": []
  }
}
```

(cherry picked from commit c8ee39d)
kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Apr 2, 2026
… versioning (elastic#258418)

## Summary

### Changes
- DHQ descriptor versioning: each query descriptor now includes an
optional version field (defaulting to 1). Old v1 descriptors are fully
unaffected.
- Integration-based targeting (v2): v2 descriptors replace the index
field with an integrations field: a comma-separated list of regex
patterns matched against Fleet-installed packages. The executor runs one
query per matched integration, using the datastream indices resolved for
that integration.
- Datastream type filtering (v2): an optional datastreamTypes field
(comma-separated regexes) narrows which datastreams are included for the
matched integration.
- Skip reasons: when a v2+ query cannot execute, they are skipped and
reported with one of three reasons: integration_not_installed,
datastreams_not_matched, or parse_failure.
- EBT schema update: stats events gain descriptorVersion, status, and
integration fields (matched patterns, resolved indices, integration
name/version) to allow downstream consumers to distinguish v1 from v2
executions.
- Permission check simplification: removed the indices.exists pre-check
from checkPermissions; privilege checking alone is sufficient, and
`exists` can lead to FP for indices patterns.

### Query descriptor V2 example

```yaml
version: 2
id: "endpoint-process-stats"
name: "Endpoint process event counts"
integrations: "endpoint"
datastreamTypes: "logs"
type: "DSL"
query: |
  {
    "aggs": { "by_action": { "terms": { "field": "event.action", "size": 20 } } },
    "size": 0
  }
scheduleCron: "1h"
enabled: true
filterlist:
  "by_action.key": keep
  "by_action.doc_count": keep
```

### Stats EBT document examples

```json
{
  "name": "endpoint-process-stats",
  "traceId": "419cc487-3b17-48f0-bf7e-4e953ed9f050",
  "started": "2026-03-18T10:00:00.000Z",
  "finished": "2026-03-18T10:00:00.842Z",
  "descriptorVersion": 2,
  "status": "success",
  "passed": true,
  "numDocs": 1,
  "fieldNames": [
    "by_action.key",
    "by_action.doc_count"
  ],
  "integration": {
    "name": "endpoint",
    "version": "8.14.2",
    "indices": [
      "logs-endpoint.events.process-default"
    ]
  },
  "circuitBreakers": { ...
  }
}
```

```json
{
  "name": "endpoint-process-stats",
  "traceId": "b2c3d4e5-...",
  "started": "2026-03-18T10:00:00.000Z",
  "finished": "2026-03-18T10:00:00.012Z",
  "descriptorVersion": 2,
  "status": "skipped",
  "skipReason": "integration_not_installed",
  "passed": false,
  "numDocs": 0,
  "fieldNames": []
}
```

```json
{
  "name": "endpoint-process-stats",
  "traceId": "c3d4e5f6-...",
  "started": "2026-03-18T10:00:00.000Z",
  "finished": "2026-03-18T10:00:00.018Z",
  "descriptorVersion": 2,
  "status": "skipped",
  "skipReason": "datastreams_not_matched",
  "passed": false,
  "numDocs": 0,
  "fieldNames": [],
  "integration": {
    "name": "endpoint",
    "version": "8.14.2",
    "indices": []
  }
}
```

(cherry picked from commit c8ee39d)
@kibanamachine
Copy link
Copy Markdown
Contributor

💔 Some backports could not be created

Status Branch Result
8.19 Backport failed because of merge conflicts

You might need to backport the following PRs to 8.19:
- [Diagnostic Queries] Only apply ILM phases filter on ECH (#260254)
9.2
9.3

Note: Successful backport PRs will be merged automatically after passing CI.

Manual backport

To create the backport manually run:

node scripts/backport --pr 258418

Questions ?

Please refer to the Backport tool documentation

kibanamachine added a commit that referenced this pull request Apr 2, 2026
…riptor versioning (#258418) (#260877)

# Backport

This will backport the following commits from `main` to `9.3`:
- [[Security Solution] [HDQ]: integration-based targeting and descriptor
versioning (#258418)](#258418)

<!--- Backport version: 9.6.6 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sorenlouv/backport)

<!--BACKPORT [{"author":{"name":"Sebastián
Zaffarano","email":"sebastian.zaffarano@elastic.co"},"sourceCommit":{"committedDate":"2026-04-02T08:06:06Z","message":"[Security
Solution] [HDQ]: integration-based targeting and descriptor versioning
(#258418)\n\n## Summary\n\n### Changes\n- DHQ descriptor versioning:
each query descriptor now includes an\noptional version field
(defaulting to 1). Old v1 descriptors are fully\nunaffected.\n-
Integration-based targeting (v2): v2 descriptors replace the
index\nfield with an integrations field: a comma-separated list of
regex\npatterns matched against Fleet-installed packages. The executor
runs one\nquery per matched integration, using the datastream indices
resolved for\nthat integration.\n- Datastream type filtering (v2): an
optional datastreamTypes field\n(comma-separated regexes) narrows which
datastreams are included for the\nmatched integration.\n- Skip reasons:
when a v2+ query cannot execute, they are skipped and\nreported with one
of three reasons: integration_not_installed,\ndatastreams_not_matched,
or parse_failure.\n- EBT schema update: stats events gain
descriptorVersion, status, and\nintegration fields (matched patterns,
resolved indices, integration\nname/version) to allow downstream
consumers to distinguish v1 from v2\nexecutions.\n- Permission check
simplification: removed the indices.exists pre-check\nfrom
checkPermissions; privilege checking alone is sufficient, and\n`exists`
can lead to FP for indices patterns.\n\n### Query descriptor V2
example\n\n```yaml\nversion: 2\nid: \"endpoint-process-stats\"\nname:
\"Endpoint process event counts\"\nintegrations:
\"endpoint\"\ndatastreamTypes: \"logs\"\ntype: \"DSL\"\nquery: |\n {\n
\"aggs\": { \"by_action\": { \"terms\": { \"field\": \"event.action\",
\"size\": 20 } } },\n \"size\": 0\n }\nscheduleCron: \"1h\"\nenabled:
true\nfilterlist:\n \"by_action.key\": keep\n \"by_action.doc_count\":
keep\n```\n\n### Stats EBT document examples\n\n```json\n{\n \"name\":
\"endpoint-process-stats\",\n \"traceId\":
\"419cc487-3b17-48f0-bf7e-4e953ed9f050\",\n \"started\":
\"2026-03-18T10:00:00.000Z\",\n \"finished\":
\"2026-03-18T10:00:00.842Z\",\n \"descriptorVersion\": 2,\n \"status\":
\"success\",\n \"passed\": true,\n \"numDocs\": 1,\n \"fieldNames\": [\n
\"by_action.key\",\n \"by_action.doc_count\"\n ],\n \"integration\": {\n
\"name\": \"endpoint\",\n \"version\": \"8.14.2\",\n \"indices\": [\n
\"logs-endpoint.events.process-default\"\n ]\n },\n \"circuitBreakers\":
{ ...\n }\n}\n```\n\n```json\n{\n \"name\":
\"endpoint-process-stats\",\n \"traceId\": \"b2c3d4e5-...\",\n
\"started\": \"2026-03-18T10:00:00.000Z\",\n \"finished\":
\"2026-03-18T10:00:00.012Z\",\n \"descriptorVersion\": 2,\n \"status\":
\"skipped\",\n \"skipReason\": \"integration_not_installed\",\n
\"passed\": false,\n \"numDocs\": 0,\n \"fieldNames\":
[]\n}\n```\n\n```json\n{\n \"name\": \"endpoint-process-stats\",\n
\"traceId\": \"c3d4e5f6-...\",\n \"started\":
\"2026-03-18T10:00:00.000Z\",\n \"finished\":
\"2026-03-18T10:00:00.018Z\",\n \"descriptorVersion\": 2,\n \"status\":
\"skipped\",\n \"skipReason\": \"datastreams_not_matched\",\n
\"passed\": false,\n \"numDocs\": 0,\n \"fieldNames\": [],\n
\"integration\": {\n \"name\": \"endpoint\",\n \"version\":
\"8.14.2\",\n \"indices\": []\n
}\n}\n```","sha":"c8ee39d5f2f726c49573f9e3cf7464d756467b53","branchLabelMapping":{"^v9.4.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","ci:build-cloud-image","ci:build-serverless-image","backport:version","v9.4.0","v9.3.3","v9.2.8","v8.19.14"],"title":"[Security
Solution] [HDQ]: integration-based targeting and descriptor
versioning","number":258418,"url":"https://github.com/elastic/kibana/pull/258418","mergeCommit":{"message":"[Security
Solution] [HDQ]: integration-based targeting and descriptor versioning
(#258418)\n\n## Summary\n\n### Changes\n- DHQ descriptor versioning:
each query descriptor now includes an\noptional version field
(defaulting to 1). Old v1 descriptors are fully\nunaffected.\n-
Integration-based targeting (v2): v2 descriptors replace the
index\nfield with an integrations field: a comma-separated list of
regex\npatterns matched against Fleet-installed packages. The executor
runs one\nquery per matched integration, using the datastream indices
resolved for\nthat integration.\n- Datastream type filtering (v2): an
optional datastreamTypes field\n(comma-separated regexes) narrows which
datastreams are included for the\nmatched integration.\n- Skip reasons:
when a v2+ query cannot execute, they are skipped and\nreported with one
of three reasons: integration_not_installed,\ndatastreams_not_matched,
or parse_failure.\n- EBT schema update: stats events gain
descriptorVersion, status, and\nintegration fields (matched patterns,
resolved indices, integration\nname/version) to allow downstream
consumers to distinguish v1 from v2\nexecutions.\n- Permission check
simplification: removed the indices.exists pre-check\nfrom
checkPermissions; privilege checking alone is sufficient, and\n`exists`
can lead to FP for indices patterns.\n\n### Query descriptor V2
example\n\n```yaml\nversion: 2\nid: \"endpoint-process-stats\"\nname:
\"Endpoint process event counts\"\nintegrations:
\"endpoint\"\ndatastreamTypes: \"logs\"\ntype: \"DSL\"\nquery: |\n {\n
\"aggs\": { \"by_action\": { \"terms\": { \"field\": \"event.action\",
\"size\": 20 } } },\n \"size\": 0\n }\nscheduleCron: \"1h\"\nenabled:
true\nfilterlist:\n \"by_action.key\": keep\n \"by_action.doc_count\":
keep\n```\n\n### Stats EBT document examples\n\n```json\n{\n \"name\":
\"endpoint-process-stats\",\n \"traceId\":
\"419cc487-3b17-48f0-bf7e-4e953ed9f050\",\n \"started\":
\"2026-03-18T10:00:00.000Z\",\n \"finished\":
\"2026-03-18T10:00:00.842Z\",\n \"descriptorVersion\": 2,\n \"status\":
\"success\",\n \"passed\": true,\n \"numDocs\": 1,\n \"fieldNames\": [\n
\"by_action.key\",\n \"by_action.doc_count\"\n ],\n \"integration\": {\n
\"name\": \"endpoint\",\n \"version\": \"8.14.2\",\n \"indices\": [\n
\"logs-endpoint.events.process-default\"\n ]\n },\n \"circuitBreakers\":
{ ...\n }\n}\n```\n\n```json\n{\n \"name\":
\"endpoint-process-stats\",\n \"traceId\": \"b2c3d4e5-...\",\n
\"started\": \"2026-03-18T10:00:00.000Z\",\n \"finished\":
\"2026-03-18T10:00:00.012Z\",\n \"descriptorVersion\": 2,\n \"status\":
\"skipped\",\n \"skipReason\": \"integration_not_installed\",\n
\"passed\": false,\n \"numDocs\": 0,\n \"fieldNames\":
[]\n}\n```\n\n```json\n{\n \"name\": \"endpoint-process-stats\",\n
\"traceId\": \"c3d4e5f6-...\",\n \"started\":
\"2026-03-18T10:00:00.000Z\",\n \"finished\":
\"2026-03-18T10:00:00.018Z\",\n \"descriptorVersion\": 2,\n \"status\":
\"skipped\",\n \"skipReason\": \"datastreams_not_matched\",\n
\"passed\": false,\n \"numDocs\": 0,\n \"fieldNames\": [],\n
\"integration\": {\n \"name\": \"endpoint\",\n \"version\":
\"8.14.2\",\n \"indices\": []\n
}\n}\n```","sha":"c8ee39d5f2f726c49573f9e3cf7464d756467b53"}},"sourceBranch":"main","suggestedTargetBranches":["9.3","9.2","8.19"],"targetPullRequestStates":[{"branch":"main","label":"v9.4.0","branchLabelMappingKey":"^v9.4.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/258418","number":258418,"mergeCommit":{"message":"[Security
Solution] [HDQ]: integration-based targeting and descriptor versioning
(#258418)\n\n## Summary\n\n### Changes\n- DHQ descriptor versioning:
each query descriptor now includes an\noptional version field
(defaulting to 1). Old v1 descriptors are fully\nunaffected.\n-
Integration-based targeting (v2): v2 descriptors replace the
index\nfield with an integrations field: a comma-separated list of
regex\npatterns matched against Fleet-installed packages. The executor
runs one\nquery per matched integration, using the datastream indices
resolved for\nthat integration.\n- Datastream type filtering (v2): an
optional datastreamTypes field\n(comma-separated regexes) narrows which
datastreams are included for the\nmatched integration.\n- Skip reasons:
when a v2+ query cannot execute, they are skipped and\nreported with one
of three reasons: integration_not_installed,\ndatastreams_not_matched,
or parse_failure.\n- EBT schema update: stats events gain
descriptorVersion, status, and\nintegration fields (matched patterns,
resolved indices, integration\nname/version) to allow downstream
consumers to distinguish v1 from v2\nexecutions.\n- Permission check
simplification: removed the indices.exists pre-check\nfrom
checkPermissions; privilege checking alone is sufficient, and\n`exists`
can lead to FP for indices patterns.\n\n### Query descriptor V2
example\n\n```yaml\nversion: 2\nid: \"endpoint-process-stats\"\nname:
\"Endpoint process event counts\"\nintegrations:
\"endpoint\"\ndatastreamTypes: \"logs\"\ntype: \"DSL\"\nquery: |\n {\n
\"aggs\": { \"by_action\": { \"terms\": { \"field\": \"event.action\",
\"size\": 20 } } },\n \"size\": 0\n }\nscheduleCron: \"1h\"\nenabled:
true\nfilterlist:\n \"by_action.key\": keep\n \"by_action.doc_count\":
keep\n```\n\n### Stats EBT document examples\n\n```json\n{\n \"name\":
\"endpoint-process-stats\",\n \"traceId\":
\"419cc487-3b17-48f0-bf7e-4e953ed9f050\",\n \"started\":
\"2026-03-18T10:00:00.000Z\",\n \"finished\":
\"2026-03-18T10:00:00.842Z\",\n \"descriptorVersion\": 2,\n \"status\":
\"success\",\n \"passed\": true,\n \"numDocs\": 1,\n \"fieldNames\": [\n
\"by_action.key\",\n \"by_action.doc_count\"\n ],\n \"integration\": {\n
\"name\": \"endpoint\",\n \"version\": \"8.14.2\",\n \"indices\": [\n
\"logs-endpoint.events.process-default\"\n ]\n },\n \"circuitBreakers\":
{ ...\n }\n}\n```\n\n```json\n{\n \"name\":
\"endpoint-process-stats\",\n \"traceId\": \"b2c3d4e5-...\",\n
\"started\": \"2026-03-18T10:00:00.000Z\",\n \"finished\":
\"2026-03-18T10:00:00.012Z\",\n \"descriptorVersion\": 2,\n \"status\":
\"skipped\",\n \"skipReason\": \"integration_not_installed\",\n
\"passed\": false,\n \"numDocs\": 0,\n \"fieldNames\":
[]\n}\n```\n\n```json\n{\n \"name\": \"endpoint-process-stats\",\n
\"traceId\": \"c3d4e5f6-...\",\n \"started\":
\"2026-03-18T10:00:00.000Z\",\n \"finished\":
\"2026-03-18T10:00:00.018Z\",\n \"descriptorVersion\": 2,\n \"status\":
\"skipped\",\n \"skipReason\": \"datastreams_not_matched\",\n
\"passed\": false,\n \"numDocs\": 0,\n \"fieldNames\": [],\n
\"integration\": {\n \"name\": \"endpoint\",\n \"version\":
\"8.14.2\",\n \"indices\": []\n
}\n}\n```","sha":"c8ee39d5f2f726c49573f9e3cf7464d756467b53"}},{"branch":"9.3","label":"v9.3.3","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"9.2","label":"v9.2.8","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.19","label":"v8.19.14","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->

Co-authored-by: Sebastián Zaffarano <sebastian.zaffarano@elastic.co>
kibanamachine added a commit that referenced this pull request Apr 2, 2026
…riptor versioning (#258418) (#260876)

# Backport

This will backport the following commits from `main` to `9.2`:
- [[Security Solution] [HDQ]: integration-based targeting and descriptor
versioning (#258418)](#258418)

<!--- Backport version: 9.6.6 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sorenlouv/backport)

<!--BACKPORT [{"author":{"name":"Sebastián
Zaffarano","email":"sebastian.zaffarano@elastic.co"},"sourceCommit":{"committedDate":"2026-04-02T08:06:06Z","message":"[Security
Solution] [HDQ]: integration-based targeting and descriptor versioning
(#258418)\n\n## Summary\n\n### Changes\n- DHQ descriptor versioning:
each query descriptor now includes an\noptional version field
(defaulting to 1). Old v1 descriptors are fully\nunaffected.\n-
Integration-based targeting (v2): v2 descriptors replace the
index\nfield with an integrations field: a comma-separated list of
regex\npatterns matched against Fleet-installed packages. The executor
runs one\nquery per matched integration, using the datastream indices
resolved for\nthat integration.\n- Datastream type filtering (v2): an
optional datastreamTypes field\n(comma-separated regexes) narrows which
datastreams are included for the\nmatched integration.\n- Skip reasons:
when a v2+ query cannot execute, they are skipped and\nreported with one
of three reasons: integration_not_installed,\ndatastreams_not_matched,
or parse_failure.\n- EBT schema update: stats events gain
descriptorVersion, status, and\nintegration fields (matched patterns,
resolved indices, integration\nname/version) to allow downstream
consumers to distinguish v1 from v2\nexecutions.\n- Permission check
simplification: removed the indices.exists pre-check\nfrom
checkPermissions; privilege checking alone is sufficient, and\n`exists`
can lead to FP for indices patterns.\n\n### Query descriptor V2
example\n\n```yaml\nversion: 2\nid: \"endpoint-process-stats\"\nname:
\"Endpoint process event counts\"\nintegrations:
\"endpoint\"\ndatastreamTypes: \"logs\"\ntype: \"DSL\"\nquery: |\n {\n
\"aggs\": { \"by_action\": { \"terms\": { \"field\": \"event.action\",
\"size\": 20 } } },\n \"size\": 0\n }\nscheduleCron: \"1h\"\nenabled:
true\nfilterlist:\n \"by_action.key\": keep\n \"by_action.doc_count\":
keep\n```\n\n### Stats EBT document examples\n\n```json\n{\n \"name\":
\"endpoint-process-stats\",\n \"traceId\":
\"419cc487-3b17-48f0-bf7e-4e953ed9f050\",\n \"started\":
\"2026-03-18T10:00:00.000Z\",\n \"finished\":
\"2026-03-18T10:00:00.842Z\",\n \"descriptorVersion\": 2,\n \"status\":
\"success\",\n \"passed\": true,\n \"numDocs\": 1,\n \"fieldNames\": [\n
\"by_action.key\",\n \"by_action.doc_count\"\n ],\n \"integration\": {\n
\"name\": \"endpoint\",\n \"version\": \"8.14.2\",\n \"indices\": [\n
\"logs-endpoint.events.process-default\"\n ]\n },\n \"circuitBreakers\":
{ ...\n }\n}\n```\n\n```json\n{\n \"name\":
\"endpoint-process-stats\",\n \"traceId\": \"b2c3d4e5-...\",\n
\"started\": \"2026-03-18T10:00:00.000Z\",\n \"finished\":
\"2026-03-18T10:00:00.012Z\",\n \"descriptorVersion\": 2,\n \"status\":
\"skipped\",\n \"skipReason\": \"integration_not_installed\",\n
\"passed\": false,\n \"numDocs\": 0,\n \"fieldNames\":
[]\n}\n```\n\n```json\n{\n \"name\": \"endpoint-process-stats\",\n
\"traceId\": \"c3d4e5f6-...\",\n \"started\":
\"2026-03-18T10:00:00.000Z\",\n \"finished\":
\"2026-03-18T10:00:00.018Z\",\n \"descriptorVersion\": 2,\n \"status\":
\"skipped\",\n \"skipReason\": \"datastreams_not_matched\",\n
\"passed\": false,\n \"numDocs\": 0,\n \"fieldNames\": [],\n
\"integration\": {\n \"name\": \"endpoint\",\n \"version\":
\"8.14.2\",\n \"indices\": []\n
}\n}\n```","sha":"c8ee39d5f2f726c49573f9e3cf7464d756467b53","branchLabelMapping":{"^v9.4.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","ci:build-cloud-image","ci:build-serverless-image","backport:version","v9.4.0","v9.3.3","v9.2.8","v8.19.14"],"title":"[Security
Solution] [HDQ]: integration-based targeting and descriptor
versioning","number":258418,"url":"https://github.com/elastic/kibana/pull/258418","mergeCommit":{"message":"[Security
Solution] [HDQ]: integration-based targeting and descriptor versioning
(#258418)\n\n## Summary\n\n### Changes\n- DHQ descriptor versioning:
each query descriptor now includes an\noptional version field
(defaulting to 1). Old v1 descriptors are fully\nunaffected.\n-
Integration-based targeting (v2): v2 descriptors replace the
index\nfield with an integrations field: a comma-separated list of
regex\npatterns matched against Fleet-installed packages. The executor
runs one\nquery per matched integration, using the datastream indices
resolved for\nthat integration.\n- Datastream type filtering (v2): an
optional datastreamTypes field\n(comma-separated regexes) narrows which
datastreams are included for the\nmatched integration.\n- Skip reasons:
when a v2+ query cannot execute, they are skipped and\nreported with one
of three reasons: integration_not_installed,\ndatastreams_not_matched,
or parse_failure.\n- EBT schema update: stats events gain
descriptorVersion, status, and\nintegration fields (matched patterns,
resolved indices, integration\nname/version) to allow downstream
consumers to distinguish v1 from v2\nexecutions.\n- Permission check
simplification: removed the indices.exists pre-check\nfrom
checkPermissions; privilege checking alone is sufficient, and\n`exists`
can lead to FP for indices patterns.\n\n### Query descriptor V2
example\n\n```yaml\nversion: 2\nid: \"endpoint-process-stats\"\nname:
\"Endpoint process event counts\"\nintegrations:
\"endpoint\"\ndatastreamTypes: \"logs\"\ntype: \"DSL\"\nquery: |\n {\n
\"aggs\": { \"by_action\": { \"terms\": { \"field\": \"event.action\",
\"size\": 20 } } },\n \"size\": 0\n }\nscheduleCron: \"1h\"\nenabled:
true\nfilterlist:\n \"by_action.key\": keep\n \"by_action.doc_count\":
keep\n```\n\n### Stats EBT document examples\n\n```json\n{\n \"name\":
\"endpoint-process-stats\",\n \"traceId\":
\"419cc487-3b17-48f0-bf7e-4e953ed9f050\",\n \"started\":
\"2026-03-18T10:00:00.000Z\",\n \"finished\":
\"2026-03-18T10:00:00.842Z\",\n \"descriptorVersion\": 2,\n \"status\":
\"success\",\n \"passed\": true,\n \"numDocs\": 1,\n \"fieldNames\": [\n
\"by_action.key\",\n \"by_action.doc_count\"\n ],\n \"integration\": {\n
\"name\": \"endpoint\",\n \"version\": \"8.14.2\",\n \"indices\": [\n
\"logs-endpoint.events.process-default\"\n ]\n },\n \"circuitBreakers\":
{ ...\n }\n}\n```\n\n```json\n{\n \"name\":
\"endpoint-process-stats\",\n \"traceId\": \"b2c3d4e5-...\",\n
\"started\": \"2026-03-18T10:00:00.000Z\",\n \"finished\":
\"2026-03-18T10:00:00.012Z\",\n \"descriptorVersion\": 2,\n \"status\":
\"skipped\",\n \"skipReason\": \"integration_not_installed\",\n
\"passed\": false,\n \"numDocs\": 0,\n \"fieldNames\":
[]\n}\n```\n\n```json\n{\n \"name\": \"endpoint-process-stats\",\n
\"traceId\": \"c3d4e5f6-...\",\n \"started\":
\"2026-03-18T10:00:00.000Z\",\n \"finished\":
\"2026-03-18T10:00:00.018Z\",\n \"descriptorVersion\": 2,\n \"status\":
\"skipped\",\n \"skipReason\": \"datastreams_not_matched\",\n
\"passed\": false,\n \"numDocs\": 0,\n \"fieldNames\": [],\n
\"integration\": {\n \"name\": \"endpoint\",\n \"version\":
\"8.14.2\",\n \"indices\": []\n
}\n}\n```","sha":"c8ee39d5f2f726c49573f9e3cf7464d756467b53"}},"sourceBranch":"main","suggestedTargetBranches":["9.3","9.2","8.19"],"targetPullRequestStates":[{"branch":"main","label":"v9.4.0","branchLabelMappingKey":"^v9.4.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/258418","number":258418,"mergeCommit":{"message":"[Security
Solution] [HDQ]: integration-based targeting and descriptor versioning
(#258418)\n\n## Summary\n\n### Changes\n- DHQ descriptor versioning:
each query descriptor now includes an\noptional version field
(defaulting to 1). Old v1 descriptors are fully\nunaffected.\n-
Integration-based targeting (v2): v2 descriptors replace the
index\nfield with an integrations field: a comma-separated list of
regex\npatterns matched against Fleet-installed packages. The executor
runs one\nquery per matched integration, using the datastream indices
resolved for\nthat integration.\n- Datastream type filtering (v2): an
optional datastreamTypes field\n(comma-separated regexes) narrows which
datastreams are included for the\nmatched integration.\n- Skip reasons:
when a v2+ query cannot execute, they are skipped and\nreported with one
of three reasons: integration_not_installed,\ndatastreams_not_matched,
or parse_failure.\n- EBT schema update: stats events gain
descriptorVersion, status, and\nintegration fields (matched patterns,
resolved indices, integration\nname/version) to allow downstream
consumers to distinguish v1 from v2\nexecutions.\n- Permission check
simplification: removed the indices.exists pre-check\nfrom
checkPermissions; privilege checking alone is sufficient, and\n`exists`
can lead to FP for indices patterns.\n\n### Query descriptor V2
example\n\n```yaml\nversion: 2\nid: \"endpoint-process-stats\"\nname:
\"Endpoint process event counts\"\nintegrations:
\"endpoint\"\ndatastreamTypes: \"logs\"\ntype: \"DSL\"\nquery: |\n {\n
\"aggs\": { \"by_action\": { \"terms\": { \"field\": \"event.action\",
\"size\": 20 } } },\n \"size\": 0\n }\nscheduleCron: \"1h\"\nenabled:
true\nfilterlist:\n \"by_action.key\": keep\n \"by_action.doc_count\":
keep\n```\n\n### Stats EBT document examples\n\n```json\n{\n \"name\":
\"endpoint-process-stats\",\n \"traceId\":
\"419cc487-3b17-48f0-bf7e-4e953ed9f050\",\n \"started\":
\"2026-03-18T10:00:00.000Z\",\n \"finished\":
\"2026-03-18T10:00:00.842Z\",\n \"descriptorVersion\": 2,\n \"status\":
\"success\",\n \"passed\": true,\n \"numDocs\": 1,\n \"fieldNames\": [\n
\"by_action.key\",\n \"by_action.doc_count\"\n ],\n \"integration\": {\n
\"name\": \"endpoint\",\n \"version\": \"8.14.2\",\n \"indices\": [\n
\"logs-endpoint.events.process-default\"\n ]\n },\n \"circuitBreakers\":
{ ...\n }\n}\n```\n\n```json\n{\n \"name\":
\"endpoint-process-stats\",\n \"traceId\": \"b2c3d4e5-...\",\n
\"started\": \"2026-03-18T10:00:00.000Z\",\n \"finished\":
\"2026-03-18T10:00:00.012Z\",\n \"descriptorVersion\": 2,\n \"status\":
\"skipped\",\n \"skipReason\": \"integration_not_installed\",\n
\"passed\": false,\n \"numDocs\": 0,\n \"fieldNames\":
[]\n}\n```\n\n```json\n{\n \"name\": \"endpoint-process-stats\",\n
\"traceId\": \"c3d4e5f6-...\",\n \"started\":
\"2026-03-18T10:00:00.000Z\",\n \"finished\":
\"2026-03-18T10:00:00.018Z\",\n \"descriptorVersion\": 2,\n \"status\":
\"skipped\",\n \"skipReason\": \"datastreams_not_matched\",\n
\"passed\": false,\n \"numDocs\": 0,\n \"fieldNames\": [],\n
\"integration\": {\n \"name\": \"endpoint\",\n \"version\":
\"8.14.2\",\n \"indices\": []\n
}\n}\n```","sha":"c8ee39d5f2f726c49573f9e3cf7464d756467b53"}},{"branch":"9.3","label":"v9.3.3","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"9.2","label":"v9.2.8","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.19","label":"v8.19.14","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->

Co-authored-by: Sebastián Zaffarano <sebastian.zaffarano@elastic.co>
mbondyra added a commit to mbondyra/kibana that referenced this pull request Apr 2, 2026
…heck

* commit 'af66aadafa7470ca8ba3e3edd3793bde81fa4596': (31 commits)
  [Scout] Update test config manifests (elastic#260850)
  [SLO]: register alerts schema embeddable (elastic#256570)
  [Discover][Flyout] Update overview fields table with new prop headerVisibility set to false (elastic#260692)
  [AiButton/Security] Migrate ai-related buttons to use custom styles (elastic#259847)
  [One Workflow] Fix connector step icons falling back to generic plugs in YAML editor (elastic#260785)
  [Agent Builder] Dashboard skill: Guard against editing non-ESQL based panels (elastic#260714)
  Security quality gate Cypress cleanup - Periodic Pipeline (elastic#260820)
  [Search] Deprecate search indices in favour of index management (elastic#260210)
  Upgrade dependency @elastic/charts to v71.4.0 (elastic#260593)
  [Security Solution] [HDQ]: integration-based targeting and descriptor versioning (elastic#258418)
  docs(saved-objects): consolidate docs and document scoped vs system client (elastic#260743)
  Fix observability UIAM config and add CPS observability variant (elastic#260485)
  [Security Solution] Add "matched_indices_count" rule execution metric (elastic#259938)
  [SigEvents] Add callout with working promote action. (elastic#260433)
  [Alerting V2] Episode table actions (elastic#260195)
  [Automatic Migration] Add ability to skip Reference Set step in QRadar upload workflow (elastic#259959)
  [Rules] KQL-to-DSL conversion without data view produces incorrect queries for keyword fields for Metric threshold rule (elastic#260046)
  Update dependency lightningcss to v1.32.0 (main) (elastic#259017)
  Update postcss (main) (elastic#255420)
  Migrate server-side apm.addLabels to OTel dual-write helpers (elastic#259619)
  ...
paulinashakirova pushed a commit to paulinashakirova/kibana that referenced this pull request Apr 2, 2026
… versioning (elastic#258418)

## Summary

### Changes
- DHQ descriptor versioning: each query descriptor now includes an
optional version field (defaulting to 1). Old v1 descriptors are fully
unaffected.
- Integration-based targeting (v2): v2 descriptors replace the index
field with an integrations field: a comma-separated list of regex
patterns matched against Fleet-installed packages. The executor runs one
query per matched integration, using the datastream indices resolved for
that integration.
- Datastream type filtering (v2): an optional datastreamTypes field
(comma-separated regexes) narrows which datastreams are included for the
matched integration.
- Skip reasons: when a v2+ query cannot execute, they are skipped and
reported with one of three reasons: integration_not_installed,
datastreams_not_matched, or parse_failure.
- EBT schema update: stats events gain descriptorVersion, status, and
integration fields (matched patterns, resolved indices, integration
name/version) to allow downstream consumers to distinguish v1 from v2
executions.
- Permission check simplification: removed the indices.exists pre-check
from checkPermissions; privilege checking alone is sufficient, and
`exists` can lead to FP for indices patterns.

### Query descriptor V2 example

```yaml
version: 2
id: "endpoint-process-stats"
name: "Endpoint process event counts"
integrations: "endpoint"
datastreamTypes: "logs"
type: "DSL"
query: |
  {
    "aggs": { "by_action": { "terms": { "field": "event.action", "size": 20 } } },
    "size": 0
  }
scheduleCron: "1h"
enabled: true
filterlist:
  "by_action.key": keep
  "by_action.doc_count": keep
```

### Stats EBT document examples

```json
{
  "name": "endpoint-process-stats",
  "traceId": "419cc487-3b17-48f0-bf7e-4e953ed9f050",
  "started": "2026-03-18T10:00:00.000Z",
  "finished": "2026-03-18T10:00:00.842Z",
  "descriptorVersion": 2,
  "status": "success",
  "passed": true,
  "numDocs": 1,
  "fieldNames": [
    "by_action.key",
    "by_action.doc_count"
  ],
  "integration": {
    "name": "endpoint",
    "version": "8.14.2",
    "indices": [
      "logs-endpoint.events.process-default"
    ]
  },
  "circuitBreakers": { ...
  }
}
```

```json
{
  "name": "endpoint-process-stats",
  "traceId": "b2c3d4e5-...",
  "started": "2026-03-18T10:00:00.000Z",
  "finished": "2026-03-18T10:00:00.012Z",
  "descriptorVersion": 2,
  "status": "skipped",
  "skipReason": "integration_not_installed",
  "passed": false,
  "numDocs": 0,
  "fieldNames": []
}
```

```json
{
  "name": "endpoint-process-stats",
  "traceId": "c3d4e5f6-...",
  "started": "2026-03-18T10:00:00.000Z",
  "finished": "2026-03-18T10:00:00.018Z",
  "descriptorVersion": 2,
  "status": "skipped",
  "skipReason": "datastreams_not_matched",
  "passed": false,
  "numDocs": 0,
  "fieldNames": [],
  "integration": {
    "name": "endpoint",
    "version": "8.14.2",
    "indices": []
  }
}
```
szaffarano added a commit to szaffarano/kibana that referenced this pull request Apr 3, 2026
… versioning (elastic#258418)

## Summary

### Changes
- DHQ descriptor versioning: each query descriptor now includes an
optional version field (defaulting to 1). Old v1 descriptors are fully
unaffected.
- Integration-based targeting (v2): v2 descriptors replace the index
field with an integrations field: a comma-separated list of regex
patterns matched against Fleet-installed packages. The executor runs one
query per matched integration, using the datastream indices resolved for
that integration.
- Datastream type filtering (v2): an optional datastreamTypes field
(comma-separated regexes) narrows which datastreams are included for the
matched integration.
- Skip reasons: when a v2+ query cannot execute, they are skipped and
reported with one of three reasons: integration_not_installed,
datastreams_not_matched, or parse_failure.
- EBT schema update: stats events gain descriptorVersion, status, and
integration fields (matched patterns, resolved indices, integration
name/version) to allow downstream consumers to distinguish v1 from v2
executions.
- Permission check simplification: removed the indices.exists pre-check
from checkPermissions; privilege checking alone is sufficient, and
`exists` can lead to FP for indices patterns.

### Query descriptor V2 example

```yaml
version: 2
id: "endpoint-process-stats"
name: "Endpoint process event counts"
integrations: "endpoint"
datastreamTypes: "logs"
type: "DSL"
query: |
  {
    "aggs": { "by_action": { "terms": { "field": "event.action", "size": 20 } } },
    "size": 0
  }
scheduleCron: "1h"
enabled: true
filterlist:
  "by_action.key": keep
  "by_action.doc_count": keep
```

### Stats EBT document examples

```json
{
  "name": "endpoint-process-stats",
  "traceId": "419cc487-3b17-48f0-bf7e-4e953ed9f050",
  "started": "2026-03-18T10:00:00.000Z",
  "finished": "2026-03-18T10:00:00.842Z",
  "descriptorVersion": 2,
  "status": "success",
  "passed": true,
  "numDocs": 1,
  "fieldNames": [
    "by_action.key",
    "by_action.doc_count"
  ],
  "integration": {
    "name": "endpoint",
    "version": "8.14.2",
    "indices": [
      "logs-endpoint.events.process-default"
    ]
  },
  "circuitBreakers": { ...
  }
}
```

```json
{
  "name": "endpoint-process-stats",
  "traceId": "b2c3d4e5-...",
  "started": "2026-03-18T10:00:00.000Z",
  "finished": "2026-03-18T10:00:00.012Z",
  "descriptorVersion": 2,
  "status": "skipped",
  "skipReason": "integration_not_installed",
  "passed": false,
  "numDocs": 0,
  "fieldNames": []
}
```

```json
{
  "name": "endpoint-process-stats",
  "traceId": "c3d4e5f6-...",
  "started": "2026-03-18T10:00:00.000Z",
  "finished": "2026-03-18T10:00:00.018Z",
  "descriptorVersion": 2,
  "status": "skipped",
  "skipReason": "datastreams_not_matched",
  "passed": false,
  "numDocs": 0,
  "fieldNames": [],
  "integration": {
    "name": "endpoint",
    "version": "8.14.2",
    "indices": []
  }
}
```

(cherry picked from commit c8ee39d)
@szaffarano
Copy link
Copy Markdown
Contributor Author

💚 All backports created successfully

Status Branch Result
8.19

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

szaffarano added a commit that referenced this pull request Apr 3, 2026
…criptor versioning (#258418) (#261112)

# Backport

This will backport the following commits from `main` to `8.19`:
- [[Security Solution] [HDQ]: integration-based targeting and descriptor
versioning (#258418)](#258418)

<!--- Backport version: 11.0.1 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sorenlouv/backport)

<!--BACKPORT [{"author":{"name":"Sebastián
Zaffarano","email":"sebastian.zaffarano@elastic.co"},"sourceCommit":{"committedDate":"2026-04-02T08:06:06Z","message":"[Security
Solution] [HDQ]: integration-based targeting and descriptor versioning
(#258418)\n\n## Summary\n\n### Changes\n- DHQ descriptor versioning:
each query descriptor now includes an\noptional version field
(defaulting to 1). Old v1 descriptors are fully\nunaffected.\n-
Integration-based targeting (v2): v2 descriptors replace the
index\nfield with an integrations field: a comma-separated list of
regex\npatterns matched against Fleet-installed packages. The executor
runs one\nquery per matched integration, using the datastream indices
resolved for\nthat integration.\n- Datastream type filtering (v2): an
optional datastreamTypes field\n(comma-separated regexes) narrows which
datastreams are included for the\nmatched integration.\n- Skip reasons:
when a v2+ query cannot execute, they are skipped and\nreported with one
of three reasons: integration_not_installed,\ndatastreams_not_matched,
or parse_failure.\n- EBT schema update: stats events gain
descriptorVersion, status, and\nintegration fields (matched patterns,
resolved indices, integration\nname/version) to allow downstream
consumers to distinguish v1 from v2\nexecutions.\n- Permission check
simplification: removed the indices.exists pre-check\nfrom
checkPermissions; privilege checking alone is sufficient, and\n`exists`
can lead to FP for indices patterns.\n\n### Query descriptor V2
example\n\n```yaml\nversion: 2\nid: \"endpoint-process-stats\"\nname:
\"Endpoint process event counts\"\nintegrations:
\"endpoint\"\ndatastreamTypes: \"logs\"\ntype: \"DSL\"\nquery: |\n {\n
\"aggs\": { \"by_action\": { \"terms\": { \"field\": \"event.action\",
\"size\": 20 } } },\n \"size\": 0\n }\nscheduleCron: \"1h\"\nenabled:
true\nfilterlist:\n \"by_action.key\": keep\n \"by_action.doc_count\":
keep\n```\n\n### Stats EBT document examples\n\n```json\n{\n \"name\":
\"endpoint-process-stats\",\n \"traceId\":
\"419cc487-3b17-48f0-bf7e-4e953ed9f050\",\n \"started\":
\"2026-03-18T10:00:00.000Z\",\n \"finished\":
\"2026-03-18T10:00:00.842Z\",\n \"descriptorVersion\": 2,\n \"status\":
\"success\",\n \"passed\": true,\n \"numDocs\": 1,\n \"fieldNames\": [\n
\"by_action.key\",\n \"by_action.doc_count\"\n ],\n \"integration\": {\n
\"name\": \"endpoint\",\n \"version\": \"8.14.2\",\n \"indices\": [\n
\"logs-endpoint.events.process-default\"\n ]\n },\n \"circuitBreakers\":
{ ...\n }\n}\n```\n\n```json\n{\n \"name\":
\"endpoint-process-stats\",\n \"traceId\": \"b2c3d4e5-...\",\n
\"started\": \"2026-03-18T10:00:00.000Z\",\n \"finished\":
\"2026-03-18T10:00:00.012Z\",\n \"descriptorVersion\": 2,\n \"status\":
\"skipped\",\n \"skipReason\": \"integration_not_installed\",\n
\"passed\": false,\n \"numDocs\": 0,\n \"fieldNames\":
[]\n}\n```\n\n```json\n{\n \"name\": \"endpoint-process-stats\",\n
\"traceId\": \"c3d4e5f6-...\",\n \"started\":
\"2026-03-18T10:00:00.000Z\",\n \"finished\":
\"2026-03-18T10:00:00.018Z\",\n \"descriptorVersion\": 2,\n \"status\":
\"skipped\",\n \"skipReason\": \"datastreams_not_matched\",\n
\"passed\": false,\n \"numDocs\": 0,\n \"fieldNames\": [],\n
\"integration\": {\n \"name\": \"endpoint\",\n \"version\":
\"8.14.2\",\n \"indices\": []\n
}\n}\n```","sha":"c8ee39d5f2f726c49573f9e3cf7464d756467b53","branchLabelMapping":{"^v9.4.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","ci:build-cloud-image","ci:build-serverless-image","backport:version","v9.4.0","v9.3.3","v9.2.8","v8.19.14"],"title":"[Security
Solution] [HDQ]: integration-based targeting and descriptor
versioning","number":258418,"url":"https://github.com/elastic/kibana/pull/258418","mergeCommit":{"message":"[Security
Solution] [HDQ]: integration-based targeting and descriptor versioning
(#258418)\n\n## Summary\n\n### Changes\n- DHQ descriptor versioning:
each query descriptor now includes an\noptional version field
(defaulting to 1). Old v1 descriptors are fully\nunaffected.\n-
Integration-based targeting (v2): v2 descriptors replace the
index\nfield with an integrations field: a comma-separated list of
regex\npatterns matched against Fleet-installed packages. The executor
runs one\nquery per matched integration, using the datastream indices
resolved for\nthat integration.\n- Datastream type filtering (v2): an
optional datastreamTypes field\n(comma-separated regexes) narrows which
datastreams are included for the\nmatched integration.\n- Skip reasons:
when a v2+ query cannot execute, they are skipped and\nreported with one
of three reasons: integration_not_installed,\ndatastreams_not_matched,
or parse_failure.\n- EBT schema update: stats events gain
descriptorVersion, status, and\nintegration fields (matched patterns,
resolved indices, integration\nname/version) to allow downstream
consumers to distinguish v1 from v2\nexecutions.\n- Permission check
simplification: removed the indices.exists pre-check\nfrom
checkPermissions; privilege checking alone is sufficient, and\n`exists`
can lead to FP for indices patterns.\n\n### Query descriptor V2
example\n\n```yaml\nversion: 2\nid: \"endpoint-process-stats\"\nname:
\"Endpoint process event counts\"\nintegrations:
\"endpoint\"\ndatastreamTypes: \"logs\"\ntype: \"DSL\"\nquery: |\n {\n
\"aggs\": { \"by_action\": { \"terms\": { \"field\": \"event.action\",
\"size\": 20 } } },\n \"size\": 0\n }\nscheduleCron: \"1h\"\nenabled:
true\nfilterlist:\n \"by_action.key\": keep\n \"by_action.doc_count\":
keep\n```\n\n### Stats EBT document examples\n\n```json\n{\n \"name\":
\"endpoint-process-stats\",\n \"traceId\":
\"419cc487-3b17-48f0-bf7e-4e953ed9f050\",\n \"started\":
\"2026-03-18T10:00:00.000Z\",\n \"finished\":
\"2026-03-18T10:00:00.842Z\",\n \"descriptorVersion\": 2,\n \"status\":
\"success\",\n \"passed\": true,\n \"numDocs\": 1,\n \"fieldNames\": [\n
\"by_action.key\",\n \"by_action.doc_count\"\n ],\n \"integration\": {\n
\"name\": \"endpoint\",\n \"version\": \"8.14.2\",\n \"indices\": [\n
\"logs-endpoint.events.process-default\"\n ]\n },\n \"circuitBreakers\":
{ ...\n }\n}\n```\n\n```json\n{\n \"name\":
\"endpoint-process-stats\",\n \"traceId\": \"b2c3d4e5-...\",\n
\"started\": \"2026-03-18T10:00:00.000Z\",\n \"finished\":
\"2026-03-18T10:00:00.012Z\",\n \"descriptorVersion\": 2,\n \"status\":
\"skipped\",\n \"skipReason\": \"integration_not_installed\",\n
\"passed\": false,\n \"numDocs\": 0,\n \"fieldNames\":
[]\n}\n```\n\n```json\n{\n \"name\": \"endpoint-process-stats\",\n
\"traceId\": \"c3d4e5f6-...\",\n \"started\":
\"2026-03-18T10:00:00.000Z\",\n \"finished\":
\"2026-03-18T10:00:00.018Z\",\n \"descriptorVersion\": 2,\n \"status\":
\"skipped\",\n \"skipReason\": \"datastreams_not_matched\",\n
\"passed\": false,\n \"numDocs\": 0,\n \"fieldNames\": [],\n
\"integration\": {\n \"name\": \"endpoint\",\n \"version\":
\"8.14.2\",\n \"indices\": []\n
}\n}\n```","sha":"c8ee39d5f2f726c49573f9e3cf7464d756467b53"}},"sourceBranch":"main","suggestedTargetBranches":["8.19"],"targetPullRequestStates":[{"branch":"main","label":"v9.4.0","branchLabelMappingKey":"^v9.4.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/258418","number":258418,"mergeCommit":{"message":"[Security
Solution] [HDQ]: integration-based targeting and descriptor versioning
(#258418)\n\n## Summary\n\n### Changes\n- DHQ descriptor versioning:
each query descriptor now includes an\noptional version field
(defaulting to 1). Old v1 descriptors are fully\nunaffected.\n-
Integration-based targeting (v2): v2 descriptors replace the
index\nfield with an integrations field: a comma-separated list of
regex\npatterns matched against Fleet-installed packages. The executor
runs one\nquery per matched integration, using the datastream indices
resolved for\nthat integration.\n- Datastream type filtering (v2): an
optional datastreamTypes field\n(comma-separated regexes) narrows which
datastreams are included for the\nmatched integration.\n- Skip reasons:
when a v2+ query cannot execute, they are skipped and\nreported with one
of three reasons: integration_not_installed,\ndatastreams_not_matched,
or parse_failure.\n- EBT schema update: stats events gain
descriptorVersion, status, and\nintegration fields (matched patterns,
resolved indices, integration\nname/version) to allow downstream
consumers to distinguish v1 from v2\nexecutions.\n- Permission check
simplification: removed the indices.exists pre-check\nfrom
checkPermissions; privilege checking alone is sufficient, and\n`exists`
can lead to FP for indices patterns.\n\n### Query descriptor V2
example\n\n```yaml\nversion: 2\nid: \"endpoint-process-stats\"\nname:
\"Endpoint process event counts\"\nintegrations:
\"endpoint\"\ndatastreamTypes: \"logs\"\ntype: \"DSL\"\nquery: |\n {\n
\"aggs\": { \"by_action\": { \"terms\": { \"field\": \"event.action\",
\"size\": 20 } } },\n \"size\": 0\n }\nscheduleCron: \"1h\"\nenabled:
true\nfilterlist:\n \"by_action.key\": keep\n \"by_action.doc_count\":
keep\n```\n\n### Stats EBT document examples\n\n```json\n{\n \"name\":
\"endpoint-process-stats\",\n \"traceId\":
\"419cc487-3b17-48f0-bf7e-4e953ed9f050\",\n \"started\":
\"2026-03-18T10:00:00.000Z\",\n \"finished\":
\"2026-03-18T10:00:00.842Z\",\n \"descriptorVersion\": 2,\n \"status\":
\"success\",\n \"passed\": true,\n \"numDocs\": 1,\n \"fieldNames\": [\n
\"by_action.key\",\n \"by_action.doc_count\"\n ],\n \"integration\": {\n
\"name\": \"endpoint\",\n \"version\": \"8.14.2\",\n \"indices\": [\n
\"logs-endpoint.events.process-default\"\n ]\n },\n \"circuitBreakers\":
{ ...\n }\n}\n```\n\n```json\n{\n \"name\":
\"endpoint-process-stats\",\n \"traceId\": \"b2c3d4e5-...\",\n
\"started\": \"2026-03-18T10:00:00.000Z\",\n \"finished\":
\"2026-03-18T10:00:00.012Z\",\n \"descriptorVersion\": 2,\n \"status\":
\"skipped\",\n \"skipReason\": \"integration_not_installed\",\n
\"passed\": false,\n \"numDocs\": 0,\n \"fieldNames\":
[]\n}\n```\n\n```json\n{\n \"name\": \"endpoint-process-stats\",\n
\"traceId\": \"c3d4e5f6-...\",\n \"started\":
\"2026-03-18T10:00:00.000Z\",\n \"finished\":
\"2026-03-18T10:00:00.018Z\",\n \"descriptorVersion\": 2,\n \"status\":
\"skipped\",\n \"skipReason\": \"datastreams_not_matched\",\n
\"passed\": false,\n \"numDocs\": 0,\n \"fieldNames\": [],\n
\"integration\": {\n \"name\": \"endpoint\",\n \"version\":
\"8.14.2\",\n \"indices\": []\n
}\n}\n```","sha":"c8ee39d5f2f726c49573f9e3cf7464d756467b53"}},{"branch":"9.3","label":"v9.3.3","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"url":"https://github.com/elastic/kibana/pull/260877","number":260877,"state":"MERGED","mergeCommit":{"sha":"a33daa282e101a644cc7cad0b654c381111f3547","message":"[9.3]
[Security Solution] [HDQ]: integration-based targeting and descriptor
versioning (#258418) (#260877)\n\n# Backport\n\nThis will backport the
following commits from `main` to `9.3`:\n- [[Security Solution] [HDQ]:
integration-based targeting and descriptor\nversioning
(#258418)](https://github.com/elastic/kibana/pull/258418)\n\n\n\n###
Questions ?\nPlease refer to the [Backport
tool\ndocumentation](https://github.com/sorenlouv/backport)\n\n\n\nCo-authored-by:
Sebastián Zaffarano
<sebastian.zaffarano@elastic.co>"}},{"branch":"9.2","label":"v9.2.8","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"url":"https://github.com/elastic/kibana/pull/260876","number":260876,"state":"MERGED","mergeCommit":{"sha":"2484ea8af037aecc848b80cdf39f66b62eb7b5a0","message":"[9.2]
[Security Solution] [HDQ]: integration-based targeting and descriptor
versioning (#258418) (#260876)\n\n# Backport\n\nThis will backport the
following commits from `main` to `9.2`:\n- [[Security Solution] [HDQ]:
integration-based targeting and descriptor\nversioning
(#258418)](https://github.com/elastic/kibana/pull/258418)\n\n\n\n###
Questions ?\nPlease refer to the [Backport
tool\ndocumentation](https://github.com/sorenlouv/backport)\n\n\n\nCo-authored-by:
Sebastián Zaffarano
<sebastian.zaffarano@elastic.co>"}},{"branch":"8.19","label":"v8.19.14","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->
Copy link
Copy Markdown

@ShashankFC ShashankFC left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agentic Code Review

🚨 Needs Changes⚠️ 4 major


x-pack/solutions/security/plugins/security_solution/server/lib/telemetry/diagnostic/health_diagnostic_integration_resolver.test.ts

  • ⚠️ [MAJOR] Lines 362–363: The unknown object { version: 99, id: 'future', name: 'future', _raw: {} } is passed directly to resolver.resolve([unknown]) which expects HealthDiagnosticQuery[]. TypeScript excess-property checks on inline object literals will flag the extra version: 99 property since ParseFailureQuery has no version field, and neither HealthDiagnosticQueryV1 nor V2 accept version 99 — this is a type error that violates the guideline 'Never suppress type errors with @ts-ignore, @ts-expect-error; fix the root cause.' The same issue appears at line 376-377 in the mixed queries test.

    confidence=7 file_read of health_diagnostic_service.types.ts shows ParseFailureQuery = { id?: string; name?: string; _raw: unknown } with no version field. HealthDiagnosticQuery is a union of V1 (version:1), V2 (version:2), and ParseFailureQuery — none accommodates version:99. The test passes the literal inline to the typed array without a cast.

x-pack/solutions/security/plugins/security_solution/server/lib/telemetry/diagnostic/health_diagnostic_integration_resolver.ts

  • ⚠️ [MAJOR] Lines 83–85: When patterns is falsy in resolveV2, the method returns an empty array [], silently dropping the query with no EBT stats event, no skip record, and no log warning. Every other unexecutable code path (lines 58, 65, 101, 123, 137) returns a SkippedQuery for observability; this path is inconsistent and will cause the query to disappear from telemetry without any trace if a HealthDiagnosticQueryV2 with neither integrations nor index is ever constructed (the type allows both fields as optional). Should return [{ kind: 'skipped', query, reason: 'parse_failure' }] instead.

    confidence=7 file_read of the resolver confirmed all other non-executable paths return SkippedQuery (lines 58, 65, 101, 123, 137); file_read of health_diagnostic_service.types.ts confirmed HealthDiagnosticQueryV2 has integrations?: string[] and index?: string as optional fields, making both-absent legally constructible; grep confirmed line 84 is the only return [] in the file.

x-pack/solutions/security/plugins/security_solution/server/lib/telemetry/diagnostic/health_diagnostic_receiver.ts

  • ⚠️ [MAJOR] Lines 93–103: When indicesFor returns multiple indices (e.g. after ILM tier expansion for a v1 ESQL query that already contains a FROM clause), the regex.test(query.query) branch at line 96 re-uses the same verbatim FROM … query string for every index in the array — submitting the identical query N times instead of once. This causes duplicate results proportional to the number of tier-expanded indices.

    confidence=7 file_read lines 93-103 shows indices.map((index) => { const esqlQuery = regex.test(query.query) ? query.query : ... }) — when the regex matches, index is ignored and the same query string is sent once per element of indices, producing N identical executions.

x-pack/solutions/security/plugins/security_solution/server/lib/telemetry/event_based/events.ts

  • ⚠️ [MAJOR] Lines 1560–1562: The skipReason field description lists 'unknown_version' as a valid value, but this string does not exist anywhere in the codebase — it was likely a leftover from an earlier draft. The actual SkipReason type (health_diagnostic_service.types.ts lines 159–164) defines five values: 'datastreams_not_matched', 'integration_not_installed', 'parse_failure', 'fleet_unavailable', and 'unsupported_query'. The description omits the last three, all of which are actively emitted at runtime (e.g., integration_resolver.ts lines 58, 65, 137). This misleads downstream EBT consumers who rely on the schema description to understand what values can appear in the field.

    confidence=7 grep for 'unknown_version' across the telemetry directory returns only this description string (0 code references). grep for 'fleet_unavailable' and 'unsupported_query' confirms both are real values in health_diagnostic_service.types.ts and health_diagnostic_integration_resolver.ts. grep for 'parse_failure' shows it is used in integration_resolver.ts:137 and health_diagnostic_service.test.ts.


Generated by Agentic Reviewer · model: us.anthropic.claude-sonnet-4-6

Comment on lines +362 to +363
const unknown = { version: 99, id: 'future', name: 'future', _raw: {} };
const results = await resolver.resolve([unknown]);
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ MAJOR: The unknown object { version: 99, id: 'future', name: 'future', _raw: {} } is passed directly to resolver.resolve([unknown]) which expects HealthDiagnosticQuery[]. TypeScript excess-property checks on inline object literals will flag the extra version: 99 property since ParseFailureQuery has no version field, and neither HealthDiagnosticQueryV1 nor V2 accept version 99 — this is a type error that violates the guideline 'Never suppress type errors with @ts-ignore, @ts-expect-error; fix the root cause.' The same issue appears at line 376-377 in the mixed queries test.

Context: confidence=7 file_read of health_diagnostic_service.types.ts shows ParseFailureQuery = { id?: string; name?: string; _raw: unknown } with no version field. HealthDiagnosticQuery is a union of V1 (version:1), V2 (version:2), and ParseFailureQuery — none accommodates version:99. The test passes the literal inline to the typed array without a cast.

Comment on lines +83 to +85
if (!patterns) {
return [];
}
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ MAJOR: When patterns is falsy in resolveV2, the method returns an empty array [], silently dropping the query with no EBT stats event, no skip record, and no log warning. Every other unexecutable code path (lines 58, 65, 101, 123, 137) returns a SkippedQuery for observability; this path is inconsistent and will cause the query to disappear from telemetry without any trace if a HealthDiagnosticQueryV2 with neither integrations nor index is ever constructed (the type allows both fields as optional). Should return [{ kind: 'skipped', query, reason: 'parse_failure' }] instead.

Context: confidence=7 file_read of the resolver confirmed all other non-executable paths return SkippedQuery (lines 58, 65, 101, 123, 137); file_read of health_diagnostic_service.types.ts confirmed HealthDiagnosticQueryV2 has integrations?: string[] and index?: string as optional fields, making both-absent legally constructible; grep confirmed line 84 is the only return [] in the file.

Comment on lines +93 to +103
mergeMap((indices) =>
from(
indices.map((index) => {
const esqlQuery = regex.test(query.query)
? query.query
: `FROM ${index} | ${query.query}`;
return from(
this.client.helpers.esql({ query }, { signal: abortSignal }).toRecords()
).pipe(
mergeMap((resp) => {
return resp.records.map((r) => r as T);
})
);
this.client.helpers.esql({ query: esqlQuery }, { signal: abortSignal }).toRecords()
).pipe(mergeMap((resp) => resp.records.map((r) => r as T)));
})
)
).pipe(mergeMap((obs) => obs))
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ MAJOR: When indicesFor returns multiple indices (e.g. after ILM tier expansion for a v1 ESQL query that already contains a FROM clause), the regex.test(query.query) branch at line 96 re-uses the same verbatim FROM … query string for every index in the array — submitting the identical query N times instead of once. This causes duplicate results proportional to the number of tier-expanded indices.

Context: confidence=7 file_read lines 93-103 shows indices.map((index) => { const esqlQuery = regex.test(query.query) ? query.query : ... }) — when the regex matches, index is ignored and the same query string is sent once per element of indices, producing N identical executions.

Comment on lines +1560 to +1562
description:
"Reason for skipping: 'datastreams_not_matched', 'integration_not_installed' or 'unknown_version'.",
},
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ MAJOR: The skipReason field description lists 'unknown_version' as a valid value, but this string does not exist anywhere in the codebase — it was likely a leftover from an earlier draft. The actual SkipReason type (health_diagnostic_service.types.ts lines 159–164) defines five values: 'datastreams_not_matched', 'integration_not_installed', 'parse_failure', 'fleet_unavailable', and 'unsupported_query'. The description omits the last three, all of which are actively emitted at runtime (e.g., integration_resolver.ts lines 58, 65, 137). This misleads downstream EBT consumers who rely on the schema description to understand what values can appear in the field.

Context: confidence=7 grep for 'unknown_version' across the telemetry directory returns only this description string (0 code references). grep for 'fleet_unavailable' and 'unsupported_query' confirms both are real values in health_diagnostic_service.types.ts and health_diagnostic_integration_resolver.ts. grep for 'parse_failure' shows it is used in integration_resolver.ts:137 and health_diagnostic_service.test.ts.

Copy link
Copy Markdown

@ShashankFC ShashankFC left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agentic Code Review

🚨 Needs Changes⚠️ 5 major


x-pack/solutions/security/plugins/security_solution/server/lib/telemetry/diagnostic/health_diagnostic_integration_resolver.test.ts

  • ⚠️ [MAJOR] Lines 185–189: The fleetResult finder predicate !('resolution' in r) || resolution.name === 'fleet_server' is incorrect: it matches ANY result lacking a resolution property, not specifically the fleet_server result. If both integrations were skipped (e.g., order changed or both had no matching datastreams), the finder would return the first skipped result regardless of which integration it belongs to, making the assertion misleading. The predicate should be 'resolution' in r && resolution.name === 'fleet_server' to correctly find the fleet_server executable result, or explicitly !('resolution' in r) && r.query.name === 'fleet_server' to find the skipped fleet_server.

    confidence=7 file_read of lines 185-188 confirmed the predicate is !('resolution' in r) || (r as { resolution: IntegrationResolution }).resolution.name === 'fleet_server'. file_read of resolver (lines 104-128) confirms skipped results have no resolution field. The || logic means any skipped result (including a hypothetical skipped endpoint) would match this finder, not just the fleet_server one.

x-pack/solutions/security/plugins/security_solution/server/lib/telemetry/diagnostic/health_diagnostic_integration_resolver.ts

  • ⚠️ [MAJOR] Lines 83–85: When a v2 query has no integrations field (i.e. patterns is falsy), resolveV2 silently returns [], causing the query to be completely dropped from the resolved results with no SkippedQuery entry, no stats event, and no log message. Every other non-executable code path returns a SkippedQuery so the query appears in telemetry; this path is the only exception, making the query invisible to operators. It should return [{ kind: 'skipped', query, reason: 'parse_failure' }] (or a dedicated reason) instead.

    confidence=7 file_read of lines 83-85 confirms: if (!patterns) { return []; } — an empty array is returned with no SkippedQuery. The flatMap in the caller (line 50) produces zero entries for this query. All other skip paths (fleet_unavailable, integration_not_installed, datastreams_not_matched, unsupported_query, parse_failure) return a SkippedQuery. The type comment in health_diagnostic_service.types.ts says the parser enforces 'exactly one of integrations or index', but if that invariant is violated this drops the query silently.

x-pack/solutions/security/plugins/security_solution/server/lib/telemetry/diagnostic/health_diagnostic_query_parser.test.ts

  • ⚠️ [MAJOR] Lines 186–281: Missing test coverage for datastreamTypes that normalises to an empty array (e.g. datastreamTypes: ' , '). In the source, types would be computed as [] on the integrations path (parser.ts lines 88–94), and no guard equivalent to the integrations.length === 0 check at line 78 exists for types, so datastreamTypes: [] would be silently stored instead of triggering a ParseFailureQuery. A test case like integrations: endpoint + datastreamTypes: ' , ' is needed to assert this yields a ParseFailureQuery.

    confidence=7 file_read of parser.ts lines 78–94 confirmed: integrations has an explicit empty-array guard (lines 78–80), but types has no equivalent guard — an empty types array is spread into the result at line 101 as datastreamTypes: [] without error. file_read of test file lines 186–281 confirmed no test covers this path.

x-pack/solutions/security/plugins/security_solution/server/lib/telemetry/diagnostic/health_diagnostic_receiver.test.ts

  • ⚠️ [MAJOR] Lines 180–184: The error: resolve handler in the streamDSL uses resolved indices from v2 ExecutableQuery test silently swallows errors — if streamDSL throws or emits an error, the Promise resolves (passes) instead of failing. This means the test gives false confidence: it would pass even if the v2 index-resolution path is completely broken. The handler should be error: (e) => reject(e) (or equivalent) to surface failures.

    confidence=7 file_read lines 180-184 confirmed: error: resolve is used in the subscribe handler. The outer Promise only has a resolve callback and no reject, so any error path causes the test to pass.

x-pack/solutions/security/plugins/security_solution/server/lib/telemetry/event_based/events.ts

  • ⚠️ [MAJOR] Lines 1560–1562: The skipReason EBT schema description lists 'unknown_version' as a valid value, but this string does not exist in the SkipReason type. The actual type is 'datastreams_not_matched' | 'integration_not_installed' | 'parse_failure' | 'fleet_unavailable' | 'unsupported_query' — so 'parse_failure', 'fleet_unavailable', and 'unsupported_query' are missing from the description, and 'unknown_version' is fabricated. This causes downstream analytics consumers to have a false picture of the possible enum values emitted in production.

    confidence=7 grep for SkipReason in health_diagnostic_service.types.ts (lines 159-164) shows: 'datastreams_not_matched' | 'integration_not_installed' | 'parse_failure' | 'fleet_unavailable' | 'unsupported_query'. The description at line 1561-1562 of events.ts says 'datastreams_not_matched', 'integration_not_installed' or 'unknown_version' — three of the five real values are missing and one non-existent value is listed.


Generated by Agentic Reviewer · model: us.anthropic.claude-sonnet-4-6

Comment on lines +185 to +189
const fleetResult = results.find(
(r) =>
!('resolution' in r) ||
(r as { resolution: IntegrationResolution }).resolution.name === 'fleet_server'
);
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ MAJOR: The fleetResult finder predicate !('resolution' in r) || resolution.name === 'fleet_server' is incorrect: it matches ANY result lacking a resolution property, not specifically the fleet_server result. If both integrations were skipped (e.g., order changed or both had no matching datastreams), the finder would return the first skipped result regardless of which integration it belongs to, making the assertion misleading. The predicate should be 'resolution' in r && resolution.name === 'fleet_server' to correctly find the fleet_server executable result, or explicitly !('resolution' in r) && r.query.name === 'fleet_server' to find the skipped fleet_server.

Context: confidence=7 file_read of lines 185-188 confirmed the predicate is !('resolution' in r) || (r as { resolution: IntegrationResolution }).resolution.name === 'fleet_server'. file_read of resolver (lines 104-128) confirms skipped results have no resolution field. The || logic means any skipped result (including a hypothetical skipped endpoint) would match this finder, not just the fleet_server one.

Comment on lines +83 to +85
if (!patterns) {
return [];
}
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ MAJOR: When a v2 query has no integrations field (i.e. patterns is falsy), resolveV2 silently returns [], causing the query to be completely dropped from the resolved results with no SkippedQuery entry, no stats event, and no log message. Every other non-executable code path returns a SkippedQuery so the query appears in telemetry; this path is the only exception, making the query invisible to operators. It should return [{ kind: 'skipped', query, reason: 'parse_failure' }] (or a dedicated reason) instead.

Context: confidence=7 file_read of lines 83-85 confirms: if (!patterns) { return []; } — an empty array is returned with no SkippedQuery. The flatMap in the caller (line 50) produces zero entries for this query. All other skip paths (fleet_unavailable, integration_not_installed, datastreams_not_matched, unsupported_query, parse_failure) return a SkippedQuery. The type comment in health_diagnostic_service.types.ts says the parser enforces 'exactly one of integrations or index', but if that invariant is violated this drops the query silently.

Comment on lines +186 to +281
describe('types field', () => {
it('parses a comma-separated datastreamTypes string into an array', () => {
const yaml = `---
version: 2
id: q-types
name: q-types
integrations: 'endpoint.*'
datastreamTypes: 'logs,metrics.*'
type: DSL
query: '{"query": {"match_all": {}}}'
scheduleCron: 5m
filterlist:
user.name: keep
enabled: true`;
const [q] = parseHealthDiagnosticQueries(yaml);
const v2 = q as unknown as HealthDiagnosticQueryV2;
expect(v2.version).toBe(2);
expect(v2.datastreamTypes).toEqual(['logs', 'metrics.*']);
});

it('leaves types undefined when the field is absent', () => {
const [q] = parseHealthDiagnosticQueries(V2_YAML);
const v2 = q as unknown as HealthDiagnosticQueryV2;
expect(v2.datastreamTypes).toBeUndefined();
});

it('returns ParseFailureQuery when datastreamTypes is a number', () => {
const yaml = `
id: q1
name: q1
version: 2
type: DSL
query: '{"query":{"match_all":{}}}'
scheduleCron: 5m
filterlist:
user.name: keep
enabled: true
integrations: endpoint
datastreamTypes: 42`;
const [q] = parseHealthDiagnosticQueries(yaml);
expect((q as ParseFailureQuery)._raw).toBeDefined();
});

it('returns ParseFailureQuery when datastreamTypes is a YAML list', () => {
const yaml = `
id: q1
name: q1
version: 2
type: DSL
query: '{"query":{"match_all":{}}}'
scheduleCron: 5m
filterlist:
user.name: keep
enabled: true
integrations: endpoint
datastreamTypes:
- logs
- metrics`;
const [q] = parseHealthDiagnosticQueries(yaml);
expect((q as ParseFailureQuery)._raw).toBeDefined();
});

it('returns ParseFailureQuery when datastreamTypes is an empty string', () => {
const yaml = `
id: q1
name: q1
version: 2
type: DSL
query: '{"query":{"match_all":{}}}'
scheduleCron: 5m
filterlist:
user.name: keep
enabled: true
integrations: endpoint
datastreamTypes: ''`;
const [q] = parseHealthDiagnosticQueries(yaml);
expect((q as ParseFailureQuery)._raw).toBeDefined();
});

it('does not leak datastreamTypes on index-based path', () => {
const yaml = `
id: q1
name: q1
version: 2
type: DSL
query: '{"query":{"match_all":{}}}'
scheduleCron: 5m
filterlist:
user.name: keep
enabled: true
index: logs-test-*
datastreamTypes: logs`;
const [q] = parseHealthDiagnosticQueries(yaml) as HealthDiagnosticQueryV2[];
expect('datastreamTypes' in q).toBe(false);
});
});
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ MAJOR: Missing test coverage for datastreamTypes that normalises to an empty array (e.g. datastreamTypes: ' , '). In the source, types would be computed as [] on the integrations path (parser.ts lines 88–94), and no guard equivalent to the integrations.length === 0 check at line 78 exists for types, so datastreamTypes: [] would be silently stored instead of triggering a ParseFailureQuery. A test case like integrations: endpoint + datastreamTypes: ' , ' is needed to assert this yields a ParseFailureQuery.

Context: confidence=7 file_read of parser.ts lines 78–94 confirmed: integrations has an explicit empty-array guard (lines 78–80), but types has no equivalent guard — an empty types array is spread into the result at line 101 as datastreamTypes: [] without error. file_read of test file lines 186–281 confirmed no test covers this path.

Comment on lines +180 to +184
await new Promise<void>((resolve) => {
queryExecutor.streamDSL(execQuery, new AbortController().signal).subscribe({
complete: resolve,
error: resolve,
});
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ MAJOR: The error: resolve handler in the streamDSL uses resolved indices from v2 ExecutableQuery test silently swallows errors — if streamDSL throws or emits an error, the Promise resolves (passes) instead of failing. This means the test gives false confidence: it would pass even if the v2 index-resolution path is completely broken. The handler should be error: (e) => reject(e) (or equivalent) to surface failures.

Context: confidence=7 file_read lines 180-184 confirmed: error: resolve is used in the subscribe handler. The outer Promise only has a resolve callback and no reject, so any error path causes the test to pass.

Comment on lines +1560 to +1562
description:
"Reason for skipping: 'datastreams_not_matched', 'integration_not_installed' or 'unknown_version'.",
},
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ MAJOR: The skipReason EBT schema description lists 'unknown_version' as a valid value, but this string does not exist in the SkipReason type. The actual type is 'datastreams_not_matched' | 'integration_not_installed' | 'parse_failure' | 'fleet_unavailable' | 'unsupported_query' — so 'parse_failure', 'fleet_unavailable', and 'unsupported_query' are missing from the description, and 'unknown_version' is fabricated. This causes downstream analytics consumers to have a false picture of the possible enum values emitted in production.

Context: confidence=7 grep for SkipReason in health_diagnostic_service.types.ts (lines 159-164) shows: 'datastreams_not_matched' | 'integration_not_installed' | 'parse_failure' | 'fleet_unavailable' | 'unsupported_query'. The description at line 1561-1562 of events.ts says 'datastreams_not_matched', 'integration_not_installed' or 'unknown_version' — three of the five real values are missing and one non-existent value is listed.

Copy link
Copy Markdown

@ShashankFC ShashankFC left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agentic Code Review

🚨 Needs Changes⚠️ 6 major


x-pack/solutions/security/plugins/security_solution/server/lib/telemetry/diagnostic/health_diagnostic_integration_resolver.test.ts

  • ⚠️ [MAJOR] Lines 185–189: The fleetResult finder predicate !('resolution' in r) || r.resolution?.name === 'fleet_server' does not actually identify the fleet_server result — it finds "any result without a resolution property". It works coincidentally here because the skipped fleet_server result has no resolution key, but if the test data were extended with a second skipped result (e.g., for a different integration that also has no datastreams matching 'traces'), fleetResult would bind to whichever skipped result comes first, not necessarily fleet_server's. A correct predicate would be (r as any).resolution?.name === 'fleet_server' or finding by r.kind === 'skipped' combined with a known ordering assumption.

    confidence=7 file_read lines 185-189 confirmed the predicate; node simulation confirmed it matches by absence-of-resolution, not by integration name. The same node simulation showed a second skipped result without resolution would also satisfy the predicate, returning the wrong item.

  • ⚠️ [MAJOR] Lines 196–207: The test 'matches multiple types with a regex alternation pattern' passes 'logs|traces' as a single datastreamTypes entry. The source code wraps it as new RegExp('^' + pattern + '$'), producing /^logs|traces$/, which JavaScript parses as (^logs)|(traces$) — not ^(logs|traces)$. This means strings like 'logs_extra' or 'xtraces' would incorrectly match. The test only uses exact type strings ('logs', 'traces') so it passes without exposing this regex-anchor bug in the source. Adding a datastreamTypes: ['logs_extra'] or datastreamTypes: ['xtraces'] assertion would catch it.

    confidence=7 node simulation: new RegExp('^logs|traces$').test('logs_extra')true (wrong); new RegExp('^logs|traces$').test('xtraces')true (wrong). Source code at health_diagnostic_integration_resolver.ts line 109 confirmed: new RegExp('^' + pattern + '$').test(ds.type).

x-pack/solutions/security/plugins/security_solution/server/lib/telemetry/diagnostic/health_diagnostic_integration_resolver.ts

  • ⚠️ [MAJOR] Line 44: Accessing err.message in the catch block assumes err is an Error instance, but in JavaScript any value can be thrown (e.g., a string, number, or plain object). This will crash with TypeError: Cannot read properties of undefined if a non-Error is thrown — ironically inside the error handler that was meant to keep the service running.

    confidence=7 file_read of health_diagnostic_integration_resolver.ts line 44 shows error: err.message without verifying err instanceof Error. The guidelines also say to avoid any and handle errors explicitly.

  • ⚠️ [MAJOR] Lines 83–85: When resolveV2 is called with a v2 query that has neither integrations nor index (e.g., if the parser invariant is bypassed), it silently returns [] — the query is dropped from results with no SkippedQuery emitted and no reason reported. It should return a SkippedQuery with reason 'parse_failure' to ensure every input query produces at least one output entry.

    confidence=7 file_read of health_diagnostic_integration_resolver.ts lines 78–85 shows if (!patterns) { return []; } which produces zero results for that query. The HealthDiagnosticQueryV2 type in health_diagnostic_service.types.ts has integrations?: string[] (optional), so the guard can be reached.

x-pack/solutions/security/plugins/security_solution/server/lib/telemetry/diagnostic/health_diagnostic_receiver.test.ts

  • ⚠️ [MAJOR] Lines 175–190: The streamDSL uses resolved indices from v2 ExecutableQuery test uses error: resolve in the subscription, so if streamDSL errors (which it will if mockEsClient.security.hasPrivileges is not mocked to return { has_all_requested: true } — this beforeEach block only calls setupPointInTime), the promise resolves silently and the expect(mockEsClient.openPointInTime).toHaveBeenCalledWith(...) assertion at line 187 is never reached. The test passes vacuously without verifying the actual behaviour.

    confidence=7 file_read of receiver.ts lines 139-155 confirms streamDSL calls checkPermissionssecurity.hasPrivileges first. The DSL beforeEach (lines 58-60) only calls setupPointInTime, never mocking hasPrivileges. With an unmocked hasPrivileges, checkPermissions throws a PermissionError, the observable errors, error: resolve silently resolves the Promise, and the assertion on line 187 is skipped.

x-pack/solutions/security/plugins/security_solution/server/lib/telemetry/event_based/events.ts

  • ⚠️ [MAJOR] Lines 1558–1562: The skipReason schema description lists 'unknown_version' as a valid value, but this string does not exist anywhere in the codebase — the actual SkipReason type (in health_diagnostic_service.types.ts) defines five values: 'datastreams_not_matched', 'integration_not_installed', 'parse_failure', 'fleet_unavailable', and 'unsupported_query'. The description is also missing 'parse_failure' (used in tests and production code), 'fleet_unavailable' (emitted when Fleet is unavailable), and 'unsupported_query' (emitted for unsupported ES|QL queries), making it an inaccurate contract for downstream EBT consumers.

    confidence=7 grep for SkipReason in health_diagnostic_service.types.ts shows the type union is 'datastreams_not_matched' | 'integration_not_installed' | 'parse_failure' | 'fleet_unavailable' | 'unsupported_query'. grep for unknown_version across the diagnostic directory returns 0 matches — it only appears in this description string. grep confirmed fleet_unavailable and unsupported_query are actively emitted in health_diagnostic_integration_resolver.ts.


Generated by Agentic Reviewer · model: us.anthropic.claude-sonnet-4-6

Comment on lines +185 to +189
const fleetResult = results.find(
(r) =>
!('resolution' in r) ||
(r as { resolution: IntegrationResolution }).resolution.name === 'fleet_server'
);
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ MAJOR: The fleetResult finder predicate !('resolution' in r) || r.resolution?.name === 'fleet_server' does not actually identify the fleet_server result — it finds "any result without a resolution property". It works coincidentally here because the skipped fleet_server result has no resolution key, but if the test data were extended with a second skipped result (e.g., for a different integration that also has no datastreams matching 'traces'), fleetResult would bind to whichever skipped result comes first, not necessarily fleet_server's. A correct predicate would be (r as any).resolution?.name === 'fleet_server' or finding by r.kind === 'skipped' combined with a known ordering assumption.

Context: confidence=7 file_read lines 185-189 confirmed the predicate; node simulation confirmed it matches by absence-of-resolution, not by integration name. The same node simulation showed a second skipped result without resolution would also satisfy the predicate, returning the wrong item.

Comment on lines +196 to +207
it('matches multiple types with a regex alternation pattern', async () => {
const query = createMockQueryV2(QueryType.DSL, {
integrations: ['endpoint'],
datastreamTypes: ['logs|traces'],
});
const results = await resolver.resolve([query]);

expect(results).toHaveLength(1);
expect(results[0].kind).toBe('executable');
const resolution = (results[0] as { resolution: IntegrationResolution }).resolution;
expect(resolution.indices).toHaveLength(3);
});
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ MAJOR: The test 'matches multiple types with a regex alternation pattern' passes 'logs|traces' as a single datastreamTypes entry. The source code wraps it as new RegExp('^' + pattern + '$'), producing /^logs|traces$/, which JavaScript parses as (^logs)|(traces$) — not ^(logs|traces)$. This means strings like 'logs_extra' or 'xtraces' would incorrectly match. The test only uses exact type strings ('logs', 'traces') so it passes without exposing this regex-anchor bug in the source. Adding a datastreamTypes: ['logs_extra'] or datastreamTypes: ['xtraces'] assertion would catch it.

Context: confidence=7 node simulation: new RegExp('^logs|traces$').test('logs_extra')true (wrong); new RegExp('^logs|traces$').test('xtraces')true (wrong). Source code at health_diagnostic_integration_resolver.ts line 109 confirmed: new RegExp('^' + pattern + '$').test(ds.type).

this.logger.debug(
'Failed to fetch installed packages from Fleet; v2 queries will be skipped',
{
error: err.message,
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ MAJOR: Accessing err.message in the catch block assumes err is an Error instance, but in JavaScript any value can be thrown (e.g., a string, number, or plain object). This will crash with TypeError: Cannot read properties of undefined if a non-Error is thrown — ironically inside the error handler that was meant to keep the service running.

Context: confidence=7 file_read of health_diagnostic_integration_resolver.ts line 44 shows error: err.message without verifying err instanceof Error. The guidelines also say to avoid any and handle errors explicitly.

Comment on lines +83 to +85
if (!patterns) {
return [];
}
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ MAJOR: When resolveV2 is called with a v2 query that has neither integrations nor index (e.g., if the parser invariant is bypassed), it silently returns [] — the query is dropped from results with no SkippedQuery emitted and no reason reported. It should return a SkippedQuery with reason 'parse_failure' to ensure every input query produces at least one output entry.

Context: confidence=7 file_read of health_diagnostic_integration_resolver.ts lines 78–85 shows if (!patterns) { return []; } which produces zero results for that query. The HealthDiagnosticQueryV2 type in health_diagnostic_service.types.ts has integrations?: string[] (optional), so the guard can be reached.

Comment on lines +175 to +190
it('streamDSL uses resolved indices from v2 ExecutableQuery', async () => {
const execQuery = mkExecV2(QueryType.DSL, {}, ['logs-endpoint.events.process-default']);
setupPointInTime(mockEsClient, 'pit-id');
mockEsClient.search.mockResolvedValueOnce(createMockSearchResponse([], undefined, 'pit-id'));

await new Promise<void>((resolve) => {
queryExecutor.streamDSL(execQuery, new AbortController().signal).subscribe({
complete: resolve,
error: resolve,
});
});

expect(mockEsClient.openPointInTime).toHaveBeenCalledWith(
expect.objectContaining({ index: ['logs-endpoint.events.process-default'] })
);
});
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ MAJOR: The streamDSL uses resolved indices from v2 ExecutableQuery test uses error: resolve in the subscription, so if streamDSL errors (which it will if mockEsClient.security.hasPrivileges is not mocked to return { has_all_requested: true } — this beforeEach block only calls setupPointInTime), the promise resolves silently and the expect(mockEsClient.openPointInTime).toHaveBeenCalledWith(...) assertion at line 187 is never reached. The test passes vacuously without verifying the actual behaviour.

Context: confidence=7 file_read of receiver.ts lines 139-155 confirms streamDSL calls checkPermissionssecurity.hasPrivileges first. The DSL beforeEach (lines 58-60) only calls setupPointInTime, never mocking hasPrivileges. With an unmocked hasPrivileges, checkPermissions throws a PermissionError, the observable errors, error: resolve silently resolves the Promise, and the assertion on line 187 is skipped.

Comment on lines +1558 to +1562
_meta: {
optional: true,
description:
"Reason for skipping: 'datastreams_not_matched', 'integration_not_installed' or 'unknown_version'.",
},
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ MAJOR: The skipReason schema description lists 'unknown_version' as a valid value, but this string does not exist anywhere in the codebase — the actual SkipReason type (in health_diagnostic_service.types.ts) defines five values: 'datastreams_not_matched', 'integration_not_installed', 'parse_failure', 'fleet_unavailable', and 'unsupported_query'. The description is also missing 'parse_failure' (used in tests and production code), 'fleet_unavailable' (emitted when Fleet is unavailable), and 'unsupported_query' (emitted for unsupported ES|QL queries), making it an inaccurate contract for downstream EBT consumers.

Context: confidence=7 grep for SkipReason in health_diagnostic_service.types.ts shows the type union is 'datastreams_not_matched' | 'integration_not_installed' | 'parse_failure' | 'fleet_unavailable' | 'unsupported_query'. grep for unknown_version across the diagnostic directory returns 0 matches — it only appears in this description string. grep confirmed fleet_unavailable and unsupported_query are actively emitted in health_diagnostic_integration_resolver.ts.

Copy link
Copy Markdown

@ShashankFC ShashankFC left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agentic Code Review

🚨 Needs Changes⚠️ 4 major


x-pack/solutions/security/plugins/security_solution/server/lib/telemetry/diagnostic/__mocks__/index.ts

  • 📋 [GUIDELINE VIOLATION] Lines 18–19: [GUIDELINE] Rule: 'Prefer explicit import/exports over "*"' and standard import ordering — the export type { ... } statement on line 18 is interleaved between two import statements (lines 11–16 and line 19), which is unconventional and likely triggers an ESLint import-ordering rule (lint exited with code 2). All import declarations should be grouped together before any export statements.

    confidence=7 file_read confirmed the export type statement at line 18 sits between import blocks (lines 11-16 and line 19). Running lint on the file returned exit code 2 (error), consistent with an import-order violation.

x-pack/solutions/security/plugins/security_solution/server/lib/telemetry/diagnostic/health_diagnostic_integration_resolver.ts

  • ⚠️ [MAJOR] Lines 83–85: When a v2 query reaches resolveV2 with patterns undefined (i.e., neither integrations nor index is set), the method silently returns [], causing the query to vanish from the resolved results with no stats event emitted. Every other error path in this resolver returns a SkippedQuery (with a reason), so this inconsistency means a malformed-but-parseable v2 query is invisible to downstream consumers and telemetry. The guard should return [{ kind: 'skipped', query, reason: 'parse_failure' }] instead of [].

    confidence=7 file_read of resolver (line 83-85) confirms if (!patterns) return []. file_read of types (lines 154-157) confirms all other skipped paths return a SkippedQuery. The parser enforces the invariant (parser lines 64-65) but the type system allows both fields to be undefined, and any direct constructor of HealthDiagnosticQueryV2 can bypass the parser.

x-pack/solutions/security/plugins/security_solution/server/lib/telemetry/diagnostic/health_diagnostic_receiver.test.ts

  • ⚠️ [MAJOR] Lines 175–190: The streamDSL uses resolved indices from v2 ExecutableQuery test never sets up mockEsClient.security.hasPrivileges, so checkPermissions returns undefined (the mock's default), causing it to throw a PermissionError. The error: resolve handler silently swallows this, allowing the promise to resolve before the openPointInTime assertion is reached — making the test vacuously pass even when the v2 index-resolution feature is broken.

    confidence=7 file_read of mocks/index.ts line 80 shows security: { hasPrivileges: jest.fn() } with no default return value. The DSL beforeEach at line 58 only calls setupPointInTime, never configuring hasPrivileges. The EQL/ESQL beforeEach blocks DO configure it (line 200: mockEsClient.security.hasPrivileges.mockResolvedValue({ has_all_requested: true })), but the DSL describe block does not. The source at receiver.ts:181 calls checkPermissions before opening the PIT, so the assertion at line 187 would never run.

x-pack/solutions/security/plugins/security_solution/server/lib/telemetry/diagnostic/health_diagnostic_service.types.ts

  • ⚠️ [MAJOR] Lines 159–164: The SkipReason type defines 'parse_failure', 'fleet_unavailable', and 'unsupported_query' as valid values, but the EBT schema description for skipReason in events.ts (line 1561) only documents 'datastreams_not_matched', 'integration_not_installed', and 'unknown_version''unknown_version' does not exist in the type at all. This means the EBT schema description is stale and misrepresents the actual values that will appear in telemetry, breaking downstream consumers' ability to correctly interpret skipReason.

    Same pattern in 2 files: events.ts


Generated by Agentic Reviewer · model: us.anthropic.claude-sonnet-4-6

Comment on lines +18 to 19
export type { HealthDiagnosticQueryV1, HealthDiagnosticQueryV2 };
import type { TelemetryConfigProvider } from '../../../../../common/telemetry_config/telemetry_config_provider';
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📋 GUIDELINE VIOLATION: Rule: 'Prefer explicit import/exports over "*"' and standard import ordering — the export type { ... } statement on line 18 is interleaved between two import statements (lines 11–16 and line 19), which is unconventional and likely triggers an ESLint import-ordering rule (lint exited with code 2). All import declarations should be grouped together before any export statements.

Context: confidence=7 file_read confirmed the export type statement at line 18 sits between import blocks (lines 11-16 and line 19). Running lint on the file returned exit code 2 (error), consistent with an import-order violation.

Comment on lines +83 to +85
if (!patterns) {
return [];
}
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ MAJOR: When a v2 query reaches resolveV2 with patterns undefined (i.e., neither integrations nor index is set), the method silently returns [], causing the query to vanish from the resolved results with no stats event emitted. Every other error path in this resolver returns a SkippedQuery (with a reason), so this inconsistency means a malformed-but-parseable v2 query is invisible to downstream consumers and telemetry. The guard should return [{ kind: 'skipped', query, reason: 'parse_failure' }] instead of [].

Context: confidence=7 file_read of resolver (line 83-85) confirms if (!patterns) return []. file_read of types (lines 154-157) confirms all other skipped paths return a SkippedQuery. The parser enforces the invariant (parser lines 64-65) but the type system allows both fields to be undefined, and any direct constructor of HealthDiagnosticQueryV2 can bypass the parser.

Comment on lines +175 to +190
it('streamDSL uses resolved indices from v2 ExecutableQuery', async () => {
const execQuery = mkExecV2(QueryType.DSL, {}, ['logs-endpoint.events.process-default']);
setupPointInTime(mockEsClient, 'pit-id');
mockEsClient.search.mockResolvedValueOnce(createMockSearchResponse([], undefined, 'pit-id'));

await new Promise<void>((resolve) => {
queryExecutor.streamDSL(execQuery, new AbortController().signal).subscribe({
complete: resolve,
error: resolve,
});
});

expect(mockEsClient.openPointInTime).toHaveBeenCalledWith(
expect.objectContaining({ index: ['logs-endpoint.events.process-default'] })
);
});
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ MAJOR: The streamDSL uses resolved indices from v2 ExecutableQuery test never sets up mockEsClient.security.hasPrivileges, so checkPermissions returns undefined (the mock's default), causing it to throw a PermissionError. The error: resolve handler silently swallows this, allowing the promise to resolve before the openPointInTime assertion is reached — making the test vacuously pass even when the v2 index-resolution feature is broken.

Context: confidence=7 file_read of mocks/index.ts line 80 shows security: { hasPrivileges: jest.fn() } with no default return value. The DSL beforeEach at line 58 only calls setupPointInTime, never configuring hasPrivileges. The EQL/ESQL beforeEach blocks DO configure it (line 200: mockEsClient.security.hasPrivileges.mockResolvedValue({ has_all_requested: true })), but the DSL describe block does not. The source at receiver.ts:181 calls checkPermissions before opening the PIT, so the assertion at line 187 would never run.

Comment on lines +159 to +164
export type SkipReason =
| 'datastreams_not_matched'
| 'integration_not_installed'
| 'parse_failure'
| 'fleet_unavailable'
| 'unsupported_query';
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ MAJOR: The SkipReason type defines 'parse_failure', 'fleet_unavailable', and 'unsupported_query' as valid values, but the EBT schema description for skipReason in events.ts (line 1561) only documents 'datastreams_not_matched', 'integration_not_installed', and 'unknown_version''unknown_version' does not exist in the type at all. This means the EBT schema description is stale and misrepresents the actual values that will appear in telemetry, breaking downstream consumers' ability to correctly interpret skipReason.

Context: Same pattern in 2 files: events.ts

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants