Add Managed label to data streams and a view switch for the table by yuliacech · Pull Request #83049 · elastic/kibana

yuliacech · 2020-11-10T13:14:09Z

Summary

This PR adds a 'Managed' label to data streams managed by Ingest Manager. There is also a new 'Include managed' switch, that is enabled by default. When disabled, managed data streams are hidden.

Screenshot

gif

How to test

start Kibana with yarn start --xpack.fleet.enabled=true --no-base-path
start ES with yarn es snapshot -E xpack.security.authc.api_key.enabled=true
Download an Elastic agent for your OS from here and unpack the archive to a folder
In Kibana go to Fleet -> Agents and click the button 'Create user and enable central management'
After that click 'Add agent' button and in the flyout look for the enrolment token in section 3. (but don't copy the command, it doesn't work)
In the folder where you unpacked downloaded Elastic agent, run ./elastic-agent enroll http://localhost:5601 $ENROLMENT_TOKEN --insecure
In the folder where you unpacked downloaded Elastic agent, run ./elastic-agent run and keep this terminal tab open.
You should see data streams in Fleet and in Index Management.

Checklist

Delete any items that are not applicable to this PR.

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Unit or functional tests were updated or added to match the most common scenarios

Release Note

In the Index Management app, data streams managed by Fleet can now be easily identified by a 'Managed' label.

elasticmachine · 2020-11-10T17:20:48Z

Pinging @elastic/es-ui (Team:Elasticsearch UI)

yuliacech · 2020-11-12T12:07:30Z

@elasticmachine merge upstream

alisonelizabeth

Great work @yuliacech! TIL how to set up Ingest Manager 😄.

Tested locally and everything worked as expected. I left a few comments, but nothing blocking.

alisonelizabeth · 2020-11-12T18:14:35Z

x-pack/plugins/index_management/public/application/lib/data_streams.tsx

+
+export const filterDataStreams = (dataStreams: DataStream[], managed: boolean): DataStream[] => {
+  return dataStreams.filter((dataStream: DataStream) =>
+    managed ? isManagedByIngestManager(dataStream) : !isManagedByIngestManager(dataStream)


This is a little hard for me to parse. If I'm understanding correctly, it looks like managed is only ever going to be false.

const filteredDataStreams = isIncludeManagedChecked ? dataStreams : filterDataStreams(dataStreams, isIncludeManagedChecked);

If that's correct, could the filter be simplified to this?

return dataStreams.filter((dataStream: DataStream) => isManagedByIngestManager(dataStream) === false)

alisonelizabeth · 2020-11-12T18:17:15Z

...nt/public/application/sections/home/data_stream_list/data_stream_table/data_stream_table.tsx

-        >
-          {name}
-        </EuiLink>
+        <Fragment>


(nit) you can use shorthand <></> for fragments.

alisonelizabeth · 2020-11-12T18:20:38Z

...gins/index_management/public/application/sections/home/data_stream_list/data_stream_list.tsx

+          <EuiFlexItem grow={false}>
+            <EuiSwitch
+              label={i18n.translate('xpack.idxMgmt.dataStreamListControls.viewManagedSwitchLabel', {
+                defaultMessage: 'View managed',


WDYT about changing this to "Include managed" to align with the other toggle?

alisonelizabeth · 2020-11-12T18:22:22Z

...nt/public/application/sections/home/data_stream_list/data_stream_table/data_stream_table.tsx

+          {isManagedByIngestManager(dataStream) ? (
+            <Fragment>
+              &nbsp;
+              <EuiBadge color="hollow">


What do you think about adding a tooltip or title to the badge to explain that "managed" means managed by Ingest Manager, specifically? Or maybe add this to the switch?

x-pack/plugins/index_management/__jest__/client_integration/home/data_streams_tab.test.ts

yuliacech · 2020-11-16T09:56:50Z

@elasticmachine merge upstream

…reams)

…me/data_streams_tab.test.ts Co-authored-by: Alison Goryachev <alisonmllr20@gmail.com>

yuliacech · 2020-11-16T14:13:49Z

Thank you for the review, @alisonelizabeth ! I renamed the switch and added some tooltips as you suggested :)

yuliacech · 2020-11-16T14:16:24Z

Hello @esdocs , could you please have a look at this PR's wording? I would appreciate some feedback on tooltips I added to the "Managed" label and to the "Include managed" switch (see screenshots in PR description). I'm not sure if I should use "Ingest Manager" or "Fleet" since the plugin name's change.

jrodewig

Thanks for reaching out about this PR @yuliacech. ~~At the least, I think we'll need to replace Ingest Manager with Fleet throughout. However, I also think the current copy places too much emphasis on "manage" and may confuse ILM users.~~

What we're really trying to communicate here is that these are internal system data streams that the user generally doesn't need to view or touch. We may not even need to point out that they're managed by Fleet.

EDIT: I was mistaken about the purpose of these managed streams. I'll submit another review.

I left some related suggestions for the copy. I'd also like to get @dedemorton's approval before merging. She's been writing the Fleet docs and is most familiar with the current terms.

...gins/index_management/public/application/sections/home/data_stream_list/data_stream_list.tsx

...nt/public/application/sections/home/data_stream_list/data_stream_table/data_stream_table.tsx

cjcenizal · 2020-11-17T19:25:52Z

@jrodewig I'm a little confused by this:

What we're really trying to communicate here is that these are internal system data streams that the user generally doesn't need to view or touch. We may not even need to point out that they're managed by Fleet.

My understanding of the term "system" is informed by my understanding of our original dot-prefixed system indices, like .kibana, .watches, and .security, which are being replaced by the formalized Elasticsearch system indices. These indices match your description of "internal" data that the user shouldn't be concerned with. Our Stack is responsible for managing them, not the user.

However, my understanding of Fleet data streams is that the user is concerned with them. These are created by the user when the user adds a new integration via Fleet, they store data ingested by the user's Elastic Agent, and the user will depend on visualizations and dashboards that consume this data. If the user wants to finesse storage costs, they'll fiddle with SLM and ILM to back up this data and move it to cheaper storage. Given this definition of Fleet's data streams, it seems useful to differentiate them from true system indices (and system data streams, once those exist).

So I'm sensing a gap between the way we're seeing things, and I'm wondering if I'm mistaken in my understanding of things. Can you help me understand your perspective so we can bridge the gap?

jrodewig · 2020-11-17T19:41:17Z

Hey @cjcenizal! Thanks for commenting. I'm likely wrong here. My main confusion arose from the -default streams vs streams like metric-system. But peaking at the data in those streams, you seem to be right. I'm a relative newbie to Elastic Agent, so my fault for not reaching out before submitting a review. Apologies.

I'll resolve my suggestions and submit a new review. However, I'll leave my original comments up for context. Thanks again!

jrodewig

LGTM. Apologies for my earlier misunderstanding. I left some terminology-related suggestions, but I'd still like approval from @dedemorton before merging. Thanks!

...gins/index_management/public/application/sections/home/data_stream_list/data_stream_list.tsx

...nt/public/application/sections/home/data_stream_list/data_stream_table/data_stream_table.tsx

yuliacech · 2020-11-18T11:00:13Z

@elasticmachine merge upstream

…me/data_stream_list/data_stream_list.tsx Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>

…me/data_stream_list/data_stream_table/data_stream_table.tsx Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>

dedemorton

Text strings LGTM.

…astic#83049) * Add Managed label to data streams and a view switch for the table * Fix i18n errors * Updated some wording and made filter function easier (managed data streams) * Update x-pack/plugins/index_management/__jest__/client_integration/home/data_streams_tab.test.ts Co-authored-by: Alison Goryachev <alisonmllr20@gmail.com> * Renamed view to include (managed data streams) * Update x-pack/plugins/index_management/public/application/sections/home/data_stream_list/data_stream_list.tsx Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com> * Update x-pack/plugins/index_management/public/application/sections/home/data_stream_list/data_stream_list.tsx Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com> * Update x-pack/plugins/index_management/public/application/sections/home/data_stream_list/data_stream_table/data_stream_table.tsx Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Alison Goryachev <alisonmllr20@gmail.com> Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>

…3049) (#83778) * Add Managed label to data streams and a view switch for the table * Fix i18n errors * Updated some wording and made filter function easier (managed data streams) * Update x-pack/plugins/index_management/__jest__/client_integration/home/data_streams_tab.test.ts Co-authored-by: Alison Goryachev <alisonmllr20@gmail.com> * Renamed view to include (managed data streams) * Update x-pack/plugins/index_management/public/application/sections/home/data_stream_list/data_stream_list.tsx Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com> * Update x-pack/plugins/index_management/public/application/sections/home/data_stream_list/data_stream_list.tsx Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com> * Update x-pack/plugins/index_management/public/application/sections/home/data_stream_list/data_stream_table/data_stream_table.tsx Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Alison Goryachev <alisonmllr20@gmail.com> Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Alison Goryachev <alisonmllr20@gmail.com> Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>

* master: (60 commits) Forward any registry cache-control header for files (elastic#83680) Revert "[Alerting] Add `alert.updatedAt` field to represent date of last user edit (elastic#83578)" [Security Solution][Detections] Fix adding an action to detection rules (elastic#83722) Make expectSnapshot available in all functional test runs (elastic#82932) Skip failing cypress test Increase bulk request timeout during esArchiver load (elastic#83657) [data.search] Server-side background session service (elastic#81099) [maps] convert VectorStyleEditor to TS (elastic#83582) Revert "[App Search] Engine overview layout stub (elastic#83504)" Adding documentation for global action configuration options (elastic#83557) [Metrics UI] Optimizations for Snapshot and Inventory Metadata (elastic#83596) chore(NA): update lmdb store to v0.8.15 (elastic#83726) [App Search] Engine overview layout stub (elastic#83504) [Workplace Search] Update SourceIcon to match latest changes in ent-search (elastic#83714) [Enterprise Search] Rename React Router helpers (elastic#83718) [Maps] Add 'crossed' & 'exited' events to tracking alert (elastic#82463) Updating code-owners to use new core/app-services team names (elastic#83731) Add Managed label to data streams and a view switch for the table (elastic#83049) [Maps] Add query bar inputs to geo threshold alerts tracked points & boundaries (elastic#80871) fix(NA): search examples kibana version declaration (elastic#83182) ...

… into add-logs-to-node-details * 'add-logs-to-node-details' of github.com:phillipb/kibana: (87 commits) [Maps] Add 'crossed' & 'exited' events to tracking alert (elastic#82463) Updating code-owners to use new core/app-services team names (elastic#83731) Add Managed label to data streams and a view switch for the table (elastic#83049) [Maps] Add query bar inputs to geo threshold alerts tracked points & boundaries (elastic#80871) fix(NA): search examples kibana version declaration (elastic#83182) Fixed console error, which appears when saving changes in Edit Alert flyout (elastic#83610) [Alerting] Add `alert.updatedAt` field to represent date of last user edit (elastic#83578) Not resetting server log level if level is defined (elastic#83651) disable incremenetal build for legacy tsconfig.json (elastic#82986) [Workplace Search] Migrate SourceLogic from ent-search (elastic#83593) [Workplace Search] Port Box changes from ent-search (elastic#83675) [APM] Improve router types (elastic#83620) Bump flat to v4.1.1 (elastic#83647) Bump y18n@5 to v5.0.5 (elastic#83644) Bump jsonpointer to v4.1.0 (elastic#83641) Bump is-my-json-valid to v2.20.5 (elastic#83642) [Telemetry] Move Monitoring collection strategy to a collector (elastic#82638) Update typescript eslint to v4.8 (elastic#83520) [ML] Persist URL state for Anomaly detection jobs using metric function (elastic#83507) [ML] Performance improvements to annotations editing in Single Metric Viewer & buttons placement (elastic#83216) ...

…astic#83049) * Add Managed label to data streams and a view switch for the table * Fix i18n errors * Updated some wording and made filter function easier (managed data streams) * Update x-pack/plugins/index_management/__jest__/client_integration/home/data_streams_tab.test.ts Co-authored-by: Alison Goryachev <alisonmllr20@gmail.com> * Renamed view to include (managed data streams) * Update x-pack/plugins/index_management/public/application/sections/home/data_stream_list/data_stream_list.tsx Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com> * Update x-pack/plugins/index_management/public/application/sections/home/data_stream_list/data_stream_list.tsx Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com> * Update x-pack/plugins/index_management/public/application/sections/home/data_stream_list/data_stream_table/data_stream_table.tsx Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Alison Goryachev <alisonmllr20@gmail.com> Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>

kibanamachine · 2020-11-19T18:37:06Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 69f4146

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`indexManagement`	511	512	+1

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`indexManagement`	1.5MB	1.5MB	+2.7KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`indexManagement`	114.1KB	114.2KB	+41.0B

History

💚 Build #88645 succeeded 230efd5
💚 Build #88646 succeeded 69f4146
💚 Build #88644 succeeded ea4c669
💚 Build #88522 succeeded 0c7efef
💚 Build #87988 succeeded 33469e4

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

Add Managed label to data streams and a view switch for the table

b669bca

yuliacech added Feature:Index Management Index and index templates UI Team:Kibana Management Dev Tools, Index Management, Upgrade Assistant, ILM, Ingest Node Pipelines, and more t// v7.11 v8.0.0 release_note:enhancement labels Nov 10, 2020

Fix i18n errors

7190c28

yuliacech marked this pull request as ready for review November 10, 2020 17:20

yuliacech requested a review from a team as a code owner November 10, 2020 17:20

yuliacech requested a review from alisonelizabeth November 10, 2020 17:20

Merge branch 'master' into managed_data_streams

2661b2e

alisonelizabeth approved these changes Nov 12, 2020

View reviewed changes

spalger added v7.11.0 and removed v7.11 labels Nov 12, 2020

kibanamachine and others added 4 commits November 16, 2020 04:56

Merge branch 'master' into managed_data_streams

77a02c4

Updated some wording and made filter function easier (managed data st…

2c781af

…reams)

Update x-pack/plugins/index_management/__jest__/client_integration/ho…

1048166

…me/data_streams_tab.test.ts Co-authored-by: Alison Goryachev <alisonmllr20@gmail.com>

Renamed view to include (managed data streams)

33469e4

yuliacech requested a review from a user November 16, 2020 14:14

jrodewig requested review from jrodewig and removed request for a user November 17, 2020 18:09

jrodewig suggested changes Nov 17, 2020

View reviewed changes

jrodewig requested a review from dedemorton November 17, 2020 19:08

jrodewig approved these changes Nov 17, 2020

View reviewed changes

kibanamachine and others added 4 commits November 18, 2020 06:00

Merge branch 'master' into managed_data_streams

0c7efef

Update x-pack/plugins/index_management/public/application/sections/ho…

ea4c669

…me/data_stream_list/data_stream_list.tsx Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>

Update x-pack/plugins/index_management/public/application/sections/ho…

230efd5

…me/data_stream_list/data_stream_list.tsx Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>

Update x-pack/plugins/index_management/public/application/sections/ho…

69f4146

…me/data_stream_list/data_stream_table/data_stream_table.tsx Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>

dedemorton approved these changes Nov 18, 2020

View reviewed changes

jrodewig mentioned this pull request Nov 18, 2020

[DOCS] Document get data stream API's _meta prop elastic/elasticsearch#65221

Merged

yuliacech merged commit 47d6612 into elastic:master Nov 18, 2020

yuliacech mentioned this pull request Nov 19, 2020

[7.x] Add Managed label to data streams and a view switch for the table (#83049) #83778

Merged

yuliacech deleted the managed_data_streams branch December 3, 2020 10:53

cjcenizal mentioned this pull request Dec 17, 2020

Some Fleet-managed data streams not marked as such #86403

Closed

yuliacech mentioned this pull request Jan 7, 2021

[DOCS] Update data streams list screenshots and description of "delete data stream" elastic/elasticsearch#67145

Merged

Conversation

yuliacech commented Nov 10, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Screenshot

Checklist

Release Note

Uh oh!

elasticmachine commented Nov 10, 2020

Uh oh!

yuliacech commented Nov 12, 2020

Uh oh!

alisonelizabeth left a comment

Choose a reason for hiding this comment

Uh oh!

alisonelizabeth Nov 12, 2020

Choose a reason for hiding this comment

Uh oh!

alisonelizabeth Nov 12, 2020

Choose a reason for hiding this comment

Uh oh!

alisonelizabeth Nov 12, 2020

Choose a reason for hiding this comment

Uh oh!

alisonelizabeth Nov 12, 2020

Choose a reason for hiding this comment

Uh oh!

Uh oh!

yuliacech commented Nov 16, 2020

Uh oh!

yuliacech commented Nov 16, 2020

Uh oh!

yuliacech commented Nov 16, 2020

Uh oh!

jrodewig left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cjcenizal commented Nov 17, 2020

Uh oh!

jrodewig commented Nov 17, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jrodewig left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yuliacech commented Nov 18, 2020

Uh oh!

dedemorton left a comment

Choose a reason for hiding this comment

Uh oh!

kibanamachine commented Nov 19, 2020

💚 Build Succeeded

Metrics [docs]

Module Count

Async chunks

Page load bundle

History

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

yuliacech commented Nov 10, 2020 •

edited

Loading

jrodewig left a comment •

edited

Loading

jrodewig commented Nov 17, 2020 •

edited

Loading

jrodewig left a comment •

edited

Loading