Skip to content

[Fleet] ignore data streams starting with . in Fleet API#256625

Merged
juliaElastic merged 5 commits intoelastic:mainfrom
juliaElastic:data-streams-skip-hidden
Mar 11, 2026
Merged

[Fleet] ignore data streams starting with . in Fleet API#256625
juliaElastic merged 5 commits intoelastic:mainfrom
juliaElastic:data-streams-skip-hidden

Conversation

@juliaElastic
Copy link
Copy Markdown
Contributor

@juliaElastic juliaElastic commented Mar 9, 2026

Summary

Closes https://github.com/elastic/kibana-team/issues/3020

Tests started failing when a hidden .workflows-events data stream is being created by #254964 and the Fleet data streams API tries to query it and fails with missing access.

Tested by checking out the linked PR locally and applying the changes in this PR.

 └-: fleet
   └-> "before all" hook: beforeTestSuite.trigger in "fleet"
   └-> "before all" hook in "fleet"
     │ debg Creating new local SAML session for a user 'elastic_admin' with role 'admin'
     │ debg Created API key for role: [admin]
     │ debg Waiting up to 30000ms for get default fleet server...
     │ debg Waiting up to 30000ms for get default Elasticsearch output...
   └-: datastreams API
     └-> "before all" hook: beforeTestSuite.trigger for "it works"
     └-> "before all" hook for "it works"
     └-> it works
       └-> "before each" hook: global before each for "it works"
       └- ✓ pass  (64ms)
     └-> "after all" hook for "it works"
     └-> "after all" hook: afterTestSuite.trigger for "it works"
   └-> "after all" hook in "fleet"
   └-> "after all" hook: afterTestSuite.trigger in "fleet"

1 passing (7.2s)

✨  Done in 15.50s.

Printed out results of data stream queries from ES: https://github.com/elastic/kibana/pull/256625/changes#diff-3315d99712b4c4534754edf2249ca9b2a9553ccc84db4ab8e716dae526807a43R102-R108

 proc [kibana] dataStreamsInfoByName [
 proc [kibana]   '.workflows-events',
 proc [kibana]   '.alerts-transform.health.alerts-default',
 proc [kibana]   '.alerts-streams.alerts-default',
 proc [kibana]   '.alerts-default.alerts-default',
 proc [kibana]   '.alerts-ml.anomaly-detection.alerts-default',
 proc [kibana]   '.alerts-security.alerts-default',
 proc [kibana]   '.alerts-dataset.quality.alerts-default',
 proc [kibana]   'logs-nginx.access-default',
 proc [kibana]   '.alerts-ml.anomaly-detection-health.alerts-default',
 proc [kibana]   '.kibana-event-log-ds',
 proc [kibana]   '.alerts-stack.alerts-default',
 proc [kibana]   '.alerts-security.attack.discovery.alerts-default',
 proc [kibana]   '.edr-workflow-insights-default'
 proc [kibana] ]
 proc [kibana] dataStreamsStatsByName []
 proc [kibana] dataStreamsMeteringStatsByName [
 proc [kibana]   '.workflows-events',
 proc [kibana]   '.alerts-transform.health.alerts-default',
 proc [kibana]   '.alerts-streams.alerts-default',
 proc [kibana]   '.alerts-default.alerts-default',
 proc [kibana]   '.alerts-ml.anomaly-detection.alerts-default',
 proc [kibana]   '.alerts-security.alerts-default',
 proc [kibana]   '.alerts-dataset.quality.alerts-default',
 proc [kibana]   'logs-nginx.access-default',
 proc [kibana]   '.alerts-ml.anomaly-detection-health.alerts-default',
 proc [kibana]   '.kibana-event-log-ds',
 proc [kibana]   '.alerts-stack.alerts-default',
 proc [kibana]   '.alerts-security.attack.discovery.alerts-default',
 proc [kibana]   '.edr-workflow-insights-default'
 proc [kibana] ]
 proc [kibana] dataStreamNames [ 'logs-nginx.access-default' ]

Checklist

Check the PR satisfies following conditions.

Reviewers should verify this PR satisfies this list as well.

  • Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
  • Documentation was added for features that require explanation or tutorials
  • Unit or functional tests were updated or added to match the most common scenarios
  • If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the docker list
  • This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The release_note:breaking label should be applied in these situations.
  • Flaky Test Runner was used on any tests changed
  • The PR description includes the appropriate Release Notes section, and the correct release_note:* label is applied per the guidelines
  • Review the backport guidelines and apply applicable backport:* labels.

Identify risks

Does this PR introduce any risks? For example, consider risks like hard to test bugs, performance regression, potential of data loss.

Describe the risk, its severity, and mitigation for each identified risk. Invite stakeholders and evaluate how to proceed before merging.

@juliaElastic juliaElastic added release_note:skip Skip the PR/issue when compiling release notes backport:skip This PR does not require backporting labels Mar 9, 2026
@juliaElastic juliaElastic changed the title [Fleet] ignore hidden datastreams [Fleet] ignore data streams starting with . in Fleet API Mar 9, 2026
@juliaElastic juliaElastic marked this pull request as ready for review March 9, 2026 10:16
@juliaElastic juliaElastic requested a review from a team as a code owner March 9, 2026 10:16
@botelastic botelastic bot added the Team:Fleet Team label for Observability Data Collection Fleet team label Mar 9, 2026
@elasticmachine
Copy link
Copy Markdown
Contributor

Pinging @elastic/fleet (Team:Fleet)

Copy link
Copy Markdown
Member

@criamico criamico left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@elasticmachine
Copy link
Copy Markdown
Contributor

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

  • [job] [logs] FTR Configs #66 / Agent Builder agents Edit agent should edit agent name
  • [job] [logs] FTR Configs #127 / Endpoint plugin @ess @serverless @skipInServerlessMKI When attempting to call an endpoint api "before all" hook in "@ess @serverless @skipInServerlessMKI When attempting to call an endpoint api"

Metrics [docs]

✅ unchanged

History

@juliaElastic juliaElastic merged commit 7946c2c into elastic:main Mar 11, 2026
18 checks passed
mbondyra added a commit to mbondyra/kibana that referenced this pull request Mar 11, 2026
…e_fix

* commit '565f7545c422192218b803874fbdf93e8d8f08ee': (27 commits)
  [Lens API] ESQL schema for XY separately for Agent and some small token optimizations (elastic#256885)
  Fix "Accessing resource attributes before async attributes settled" telemetry error (elastic#256880)
  [Security Solution][Attacks/Alerts][Attacks page][Table section] Preserver "Sort by" state on Attacks page (elastic#256717) (elastic#256795)
  [APM] Improve redirect with default date range guard (elastic#256887)
  [Security Solution][Attacks/Alerts][Attacks page][Table section] Add assignees avatars to the group component (elastic#250126) (elastic#256901)
  [Docs] add xpack.alerting.rules.maxScheduledPerMinute setting description (elastic#257041)
  [SO] Fix non-deterministic ordering in nested find API integration tests (elastic#256447)
  [Write-restricted dashboards] Update user profile retrieval for getShouldAddAccessControl (elastic#255065)
  [One Workflow] Add Scout API test scaffold and execution tests (elastic#256300)
  [Fleet] add use_apm if dynamic_signal_types are enabled (elastic#256429)
  [Fleet] ignore data streams starting with `.` in Fleet API (elastic#256625)
  [ES|QL] METRICS_INFO support: columns_after & summary (elastic#256758)
  [Agent Builder] Agent plugins: initial installation support (elastic#256478)
  [Streams] Add field descriptions and documentation-only field overrides (elastic#255136)
  [api-docs] 2026-03-11 Daily api_docs build (elastic#257023)
  [Security Solution] fix alerts page infinite loading state due to data view error (elastic#256983)
  [Logging] Add `service.*` global fields (elastic#256878)
  [Canvas] Apply embeddable transforms to embeddable elements (elastic#252191)
  [table_list_view_table] stabilize jest test (elastic#254991)
  [Obs AI] get_index_info: add unit tests (elastic#256802)
  ...
sorenlouv pushed a commit that referenced this pull request Mar 17, 2026
## Summary

Closes https://github.com/elastic/kibana-team/issues/3020

Tests started failing when a hidden `.workflows-events` data stream is
being created by #254964 and the
Fleet data streams API tries to query it and fails with missing access.

Tested by checking out the linked PR locally and applying the changes in
this PR.

```
 └-: fleet
   └-> "before all" hook: beforeTestSuite.trigger in "fleet"
   └-> "before all" hook in "fleet"
     │ debg Creating new local SAML session for a user 'elastic_admin' with role 'admin'
     │ debg Created API key for role: [admin]
     │ debg Waiting up to 30000ms for get default fleet server...
     │ debg Waiting up to 30000ms for get default Elasticsearch output...
   └-: datastreams API
     └-> "before all" hook: beforeTestSuite.trigger for "it works"
     └-> "before all" hook for "it works"
     └-> it works
       └-> "before each" hook: global before each for "it works"
       └- ✓ pass  (64ms)
     └-> "after all" hook for "it works"
     └-> "after all" hook: afterTestSuite.trigger for "it works"
   └-> "after all" hook in "fleet"
   └-> "after all" hook: afterTestSuite.trigger in "fleet"

1 passing (7.2s)

✨  Done in 15.50s.
```

Printed out results of data stream queries from ES:
https://github.com/elastic/kibana/pull/256625/changes#diff-3315d99712b4c4534754edf2249ca9b2a9553ccc84db4ab8e716dae526807a43R102-R108

```
 proc [kibana] dataStreamsInfoByName [
 proc [kibana]   '.workflows-events',
 proc [kibana]   '.alerts-transform.health.alerts-default',
 proc [kibana]   '.alerts-streams.alerts-default',
 proc [kibana]   '.alerts-default.alerts-default',
 proc [kibana]   '.alerts-ml.anomaly-detection.alerts-default',
 proc [kibana]   '.alerts-security.alerts-default',
 proc [kibana]   '.alerts-dataset.quality.alerts-default',
 proc [kibana]   'logs-nginx.access-default',
 proc [kibana]   '.alerts-ml.anomaly-detection-health.alerts-default',
 proc [kibana]   '.kibana-event-log-ds',
 proc [kibana]   '.alerts-stack.alerts-default',
 proc [kibana]   '.alerts-security.attack.discovery.alerts-default',
 proc [kibana]   '.edr-workflow-insights-default'
 proc [kibana] ]
 proc [kibana] dataStreamsStatsByName []
 proc [kibana] dataStreamsMeteringStatsByName [
 proc [kibana]   '.workflows-events',
 proc [kibana]   '.alerts-transform.health.alerts-default',
 proc [kibana]   '.alerts-streams.alerts-default',
 proc [kibana]   '.alerts-default.alerts-default',
 proc [kibana]   '.alerts-ml.anomaly-detection.alerts-default',
 proc [kibana]   '.alerts-security.alerts-default',
 proc [kibana]   '.alerts-dataset.quality.alerts-default',
 proc [kibana]   'logs-nginx.access-default',
 proc [kibana]   '.alerts-ml.anomaly-detection-health.alerts-default',
 proc [kibana]   '.kibana-event-log-ds',
 proc [kibana]   '.alerts-stack.alerts-default',
 proc [kibana]   '.alerts-security.attack.discovery.alerts-default',
 proc [kibana]   '.edr-workflow-insights-default'
 proc [kibana] ]
 proc [kibana] dataStreamNames [ 'logs-nginx.access-default' ]
```

### Checklist

Check the PR satisfies following conditions. 

Reviewers should verify this PR satisfies this list as well.

- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This was checked for breaking HTTP API changes, and any breaking
changes have been approved by the breaking-change committee. The
`release_note:breaking` label should be applied in these situations.
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [ ] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
- [ ] Review the [backport
guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing)
and apply applicable `backport:*` labels.

### Identify risks

Does this PR introduce any risks? For example, consider risks like hard
to test bugs, performance regression, potential of data loss.

Describe the risk, its severity, and mitigation for each identified
risk. Invite stakeholders and evaluate how to proceed before merging.

- [ ] [See some risk
examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx)
- [ ] ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport:skip This PR does not require backporting release_note:skip Skip the PR/issue when compiling release notes Team:Fleet Team label for Observability Data Collection Fleet team v9.4.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants