Skip to content

[Fleet] Only show fleet managed data streams on data streams list page#143300

Merged
hop-dev merged 1 commit intoelastic:mainfrom
hop-dev:123569-only-show-fleet-managed-streams
Oct 13, 2022
Merged

[Fleet] Only show fleet managed data streams on data streams list page#143300
hop-dev merged 1 commit intoelastic:mainfrom
hop-dev:123569-only-show-fleet-managed-streams

Conversation

@hop-dev
Copy link
Copy Markdown
Contributor

@hop-dev hop-dev commented Oct 13, 2022

Summary

Closes #123569

Only return data stream info for fleet managed data streams.

I have updated the test data for the integration tests to cover this behaviour.

Manual test steps:

  • create a data stream doc which follows the data stream naming scheme and is not managed by fleet:
      POST /metrics-not_fleet.somedata-default/_doc
      {
          "@timestamp": "2022-01-01",
          "data_stream": {
            "dataset": "not_fleet.somedata",
            "namespace": "default",
            "type": "metrics"
          }
        }
  • view the data streams tab /app/fleet/data-streams, this stream should not be visible. Whereas previously we had something like:

Screenshot 2022-10-13 at 17 09 27

Checklist

Delete any items that are not applicable to this PR.

@hop-dev hop-dev force-pushed the 123569-only-show-fleet-managed-streams branch from 7cf5288 to d0571da Compare October 13, 2022 15:59
@hop-dev hop-dev force-pushed the 123569-only-show-fleet-managed-streams branch from d0571da to 4742a1f Compare October 13, 2022 15:59
@hop-dev hop-dev self-assigned this Oct 13, 2022
@hop-dev hop-dev added release_note:fix Team:Fleet Team label for Observability Data Collection Fleet team labels Oct 13, 2022
@hop-dev hop-dev changed the title [Fleet] Only return data streams managed by fleet [Fleet] Only show fleet managed data streams on data streams list page Oct 13, 2022
@hop-dev hop-dev marked this pull request as ready for review October 13, 2022 16:03
@hop-dev hop-dev requested a review from a team as a code owner October 13, 2022 16:03
@elasticmachine
Copy link
Copy Markdown
Contributor

Pinging @elastic/fleet (Team:Fleet)

@hop-dev hop-dev enabled auto-merge (squash) October 13, 2022 16:07
Copy link
Copy Markdown
Member

@kpollich kpollich left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Change looks simple enough - thanks for updating the test data as well. 🚀

@kibana-ci
Copy link
Copy Markdown

💚 Build Succeeded

Metrics [docs]

✅ unchanged

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @hop-dev

@hop-dev hop-dev merged commit 8ea512c into elastic:main Oct 13, 2022
@kibanamachine kibanamachine added v8.6.0 backport:skip This PR does not require backporting labels Oct 13, 2022
@hop-dev hop-dev deleted the 123569-only-show-fleet-managed-streams branch October 13, 2022 19:13
criamico added a commit that referenced this pull request Jan 26, 2023
…149504)

## Summary

Closes #147605

### Description
Clusters that exist since pre-8.0 might have data streams that have
`managed_by: ingest_manager` in their` _meta` properties, instead of the
current value `managed_by: fleet`.
However, with the merge of
#143300, the data streams view
filters out any data streams that don't have `managed_by: fleet`
metadata, so when updating from 7.x to 8.6.x no data streams are
returned.

### Solution
It was decided to simply add an additional check in the data stream
handler to allow for "legacy" metadata, and to avoid doing migrations
that can be dangerous for the users data.


### Testing
I'm looking for a way to reliably test this locally - I've only managed
to reproduce it on cloud
criamico added a commit to criamico/kibana that referenced this pull request Jan 26, 2023
…lastic#149504)

## Summary

Closes elastic#147605

### Description
Clusters that exist since pre-8.0 might have data streams that have
`managed_by: ingest_manager` in their` _meta` properties, instead of the
current value `managed_by: fleet`.
However, with the merge of
elastic#143300, the data streams view
filters out any data streams that don't have `managed_by: fleet`
metadata, so when updating from 7.x to 8.6.x no data streams are
returned.

### Solution
It was decided to simply add an additional check in the data stream
handler to allow for "legacy" metadata, and to avoid doing migrations
that can be dangerous for the users data.

### Testing
I'm looking for a way to reliably test this locally - I've only managed
to reproduce it on cloud

(cherry picked from commit 49ff27e)

# Conflicts:
#	x-pack/plugins/fleet/server/routes/data_streams/handlers.ts
criamico added a commit that referenced this pull request Jan 26, 2023
…pings (#149504) (#149583)

# Backport

This will backport the following commits from `main` to `8.6`:
- [[Fleet] Add check for legacy managed_by field in datastream mappings
(#149504)](#149504)

<!--- Backport version: 8.9.7 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Cristina
Amico","email":"criamico@users.noreply.github.com"},"sourceCommit":{"committedDate":"2023-01-26T08:15:14Z","message":"[Fleet]
Add check for legacy managed_by field in datastream mappings
(#149504)\n\n## Summary\r\n\r\nCloses
https://github.com/elastic/kibana/issues/147605\r\n\r\n###
Description\r\nClusters that exist since pre-8.0 might have data streams
that have\r\n`managed_by: ingest_manager` in their` _meta` properties,
instead of the\r\ncurrent value `managed_by: fleet`.\r\nHowever, with
the merge of\r\nhttps://github.com//pull/143300, the data
streams view\r\nfilters out any data streams that don't have
`managed_by: fleet`\r\nmetadata, so when updating from 7.x to 8.6.x no
data streams are\r\nreturned.\r\n\r\n### Solution\r\nIt was decided to
simply add an additional check in the data stream\r\nhandler to allow
for \"legacy\" metadata, and to avoid doing migrations\r\nthat can be
dangerous for the users data.\r\n\r\n\r\n### Testing\r\nI'm looking for
a way to reliably test this locally - I've only managed\r\nto reproduce
it on
cloud","sha":"49ff27e2ff109ef28a8640e056db1cda645986ca","branchLabelMapping":{"^v8.7.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Team:Fleet","v8.7.0","v8.6.1"],"number":149504,"url":"https://github.com/elastic/kibana/pull/149504","mergeCommit":{"message":"[Fleet]
Add check for legacy managed_by field in datastream mappings
(#149504)\n\n## Summary\r\n\r\nCloses
https://github.com/elastic/kibana/issues/147605\r\n\r\n###
Description\r\nClusters that exist since pre-8.0 might have data streams
that have\r\n`managed_by: ingest_manager` in their` _meta` properties,
instead of the\r\ncurrent value `managed_by: fleet`.\r\nHowever, with
the merge of\r\nhttps://github.com//pull/143300, the data
streams view\r\nfilters out any data streams that don't have
`managed_by: fleet`\r\nmetadata, so when updating from 7.x to 8.6.x no
data streams are\r\nreturned.\r\n\r\n### Solution\r\nIt was decided to
simply add an additional check in the data stream\r\nhandler to allow
for \"legacy\" metadata, and to avoid doing migrations\r\nthat can be
dangerous for the users data.\r\n\r\n\r\n### Testing\r\nI'm looking for
a way to reliably test this locally - I've only managed\r\nto reproduce
it on
cloud","sha":"49ff27e2ff109ef28a8640e056db1cda645986ca"}},"sourceBranch":"main","suggestedTargetBranches":["8.6"],"targetPullRequestStates":[{"branch":"main","label":"v8.7.0","labelRegex":"^v8.7.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/149504","number":149504,"mergeCommit":{"message":"[Fleet]
Add check for legacy managed_by field in datastream mappings
(#149504)\n\n## Summary\r\n\r\nCloses
https://github.com/elastic/kibana/issues/147605\r\n\r\n###
Description\r\nClusters that exist since pre-8.0 might have data streams
that have\r\n`managed_by: ingest_manager` in their` _meta` properties,
instead of the\r\ncurrent value `managed_by: fleet`.\r\nHowever, with
the merge of\r\nhttps://github.com//pull/143300, the data
streams view\r\nfilters out any data streams that don't have
`managed_by: fleet`\r\nmetadata, so when updating from 7.x to 8.6.x no
data streams are\r\nreturned.\r\n\r\n### Solution\r\nIt was decided to
simply add an additional check in the data stream\r\nhandler to allow
for \"legacy\" metadata, and to avoid doing migrations\r\nthat can be
dangerous for the users data.\r\n\r\n\r\n### Testing\r\nI'm looking for
a way to reliably test this locally - I've only managed\r\nto reproduce
it on
cloud","sha":"49ff27e2ff109ef28a8640e056db1cda645986ca"}},{"branch":"8.6","label":"v8.6.1","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->
kqualters-elastic pushed a commit to kqualters-elastic/kibana that referenced this pull request Feb 6, 2023
…lastic#149504)

## Summary

Closes elastic#147605

### Description
Clusters that exist since pre-8.0 might have data streams that have
`managed_by: ingest_manager` in their` _meta` properties, instead of the
current value `managed_by: fleet`.
However, with the merge of
elastic#143300, the data streams view
filters out any data streams that don't have `managed_by: fleet`
metadata, so when updating from 7.x to 8.6.x no data streams are
returned.

### Solution
It was decided to simply add an additional check in the data stream
handler to allow for "legacy" metadata, and to avoid doing migrations
that can be dangerous for the users data.


### Testing
I'm looking for a way to reliably test this locally - I've only managed
to reproduce it on cloud
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport:skip This PR does not require backporting release_note:fix Team:Fleet Team label for Observability Data Collection Fleet team v8.6.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Fleet]: Unknown dataset enterprise_search.api available under Data Streams tab.

5 participants