[Fleet] Add support for allow_routing data streams option by joshdover · Pull Request #131609 · elastic/kibana

joshdover · 2022-05-05T09:57:50Z

Summary

Explores adding support for elastic/package-spec#327, but it turns out we need to do #115032 first.

Doesn't work yet.

Checklist

Delete any items that are not applicable to this PR.

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios
Any UI touched in this PR is usable by keyboard only (learn more about keyboard accessibility)
Any UI touched in this PR does not create any new axe failures (run axe in browser: FF, Chrome)
If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the docker list
This renders correctly on smaller devices using a responsive layout. (You can test this in your browser)
This was checked for cross-browser compatibility

Risk Matrix

Delete this section if it is not applicable to this PR.

Before closing this PR, invite QA, stakeholders, and other developers to identify risks that should be tested prior to the change/feature release.

When forming the risk matrix, consider some of the following examples and how they may potentially impact the change:

Risk	Probability	Severity	Mitigation/Notes
Multiple Spaces—unexpected behavior in non-default Kibana Space.	Low	High	Integration tests will verify that all features are still supported in non-default Kibana Space and when user switches between spaces.
Multiple nodes—Elasticsearch polling might have race conditions when multiple Kibana nodes are polling for the same tasks.	High	Low	Tasks are idempotent, so executing them multiple times will not result in logical error, but will degrade performance. To test for this case we add plenty of unit tests around this logic and document manual testing procedure.
Code should gracefully handle cases when feature X or plugin Y are disabled.	Medium	High	Unit tests will verify that any feature flag or plugin combination still results in our service operational.
See more potential risk examples

For maintainers

This was checked for breaking API changes and was labeled appropriately

jsoriano · 2022-09-05T07:28:42Z

x-pack/plugins/fleet/server/services/package_policies_to_agent_permissions.ts

If this is going to give privileges on data streams outside of the ones of this package, shall we warn the user when this privilege is given? Otherwise any package could use this setting to inadvertently obtain too much privileges.

Sounds like a good idea. Do you have a suggestion on how and were to warn the user?

I guess that this would need to be done when installing a package with this flag enabled.

kibana-ci · 2022-10-11T13:06:07Z

💔 Build Failed

Failed CI Steps

Test Failures

[job] [logs] FTR Configs #19 / endpoint Endpoint permissions: "after all" hook in "Endpoint permissions:"
[job] [logs] FTR Configs #19 / endpoint Endpoint permissions: "after all" hook in "Endpoint permissions:"
[job] [logs] FTR Configs #19 / endpoint Endpoint permissions: "before all" hook in "Endpoint permissions:"
[job] [logs] FTR Configs #19 / endpoint Endpoint permissions: "before all" hook in "Endpoint permissions:"
[job] [logs] FTR Configs #19 / endpoint For each artifact list under management When on the Blocklist entries list "after all" hook for "should be able to delete the existing Blocklist entry"
[job] [logs] FTR Configs #19 / endpoint For each artifact list under management When on the Blocklist entries list "after all" hook for "should be able to delete the existing Blocklist entry"
[job] [logs] FTR Configs #19 / endpoint For each artifact list under management When on the Blocklist entries list "before all" hook for "should not show page title if there is no Blocklist entry"
[job] [logs] FTR Configs #19 / endpoint For each artifact list under management When on the Blocklist entries list "before all" hook for "should not show page title if there is no Blocklist entry"
[job] [logs] FTR Configs #19 / endpoint For each artifact list under management When on the Host isolation exceptions entries list "after all" hook for "should be able to delete the existing Host isolation exceptions entry"
[job] [logs] FTR Configs #19 / endpoint For each artifact list under management When on the Host isolation exceptions entries list "after all" hook for "should be able to delete the existing Host isolation exceptions entry"
[job] [logs] FTR Configs #19 / endpoint For each artifact list under management When on the Host isolation exceptions entries list "before all" hook for "should not show page title if there is no Host isolation exceptions entry"
[job] [logs] FTR Configs #19 / endpoint For each artifact list under management When on the Host isolation exceptions entries list "before all" hook for "should not show page title if there is no Host isolation exceptions entry"
[job] [logs] FTR Configs #19 / endpoint Response Actions Responder "before all" hook in "Response Actions Responder"
[job] [logs] FTR Configs #19 / endpoint Response Actions Responder "before all" hook in "Response Actions Responder"
[job] [logs] FTR Configs #19 / endpoint When on the Trusted Apps list "after all" hook for "should be able to add a new trusted app and remove it"
[job] [logs] FTR Configs #19 / endpoint When on the Trusted Apps list "after all" hook for "should be able to add a new trusted app and remove it"
[job] [logs] FTR Configs #19 / endpoint When on the Trusted Apps list "before all" hook for "should not show page title if there is no trusted app"
[job] [logs] FTR Configs #19 / endpoint When on the Trusted Apps list "before all" hook for "should not show page title if there is no trusted app"
[job] [logs] Jest Tests #9 / Fleet - packageToPackagePolicy packageToPackagePolicy returns package policy with multiple policy templates (aka has integrations
[job] [logs] FTR Configs #45 / Fleet Endpoints Package Policy - create Simplified package policy should work with valid values
[job] [logs] FTR Configs #45 / Fleet Endpoints Package Policy - create Simplified package policy should work with valid values
[job] [logs] FTR Configs #45 / Fleet Endpoints Package Policy - update "before all" hook for "should work with valid values on "regular" policies"
[job] [logs] FTR Configs #45 / Fleet Endpoints Package Policy - update "before all" hook for "should work with valid values on "regular" policies"
[job] [logs] FTR Configs #45 / Fleet Endpoints Package Policy - upgrade when upgrading from an integration package to an input package where a required variable has been added "before each" hook for "returns a diff with errors"
[job] [logs] FTR Configs #45 / Fleet Endpoints Package Policy - upgrade when upgrading from an integration package to an input package where a required variable has been added "before each" hook for "returns a diff with errors"
[job] [logs] Jest Tests #9 / toPackagePolicy With nginx package should work

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`fleet`	893	894	+1

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`fleet`	112.8KB	112.9KB	+62.0B

Unknown metric groups

API count

id	before	after	diff
`fleet`	996	997	+1

ESLint disabled in files

id	before	after	diff
`apm`	14	13	-1
`observability`	8	7	-1
total			-2

ESLint disabled line counts

id	before	after	diff
`apm`	81	79	-2
`observability`	45	44	-1
`securitySolution`	411	407	-4
`synthetics`	60	54	-6
`ux`	10	9	-1
total			-14

Total ESLint disabled count

id	before	after	diff
`apm`	95	92	-3
`observability`	53	51	-2
`securitySolution`	484	480	-4
`synthetics`	66	60	-6
`ux`	13	12	-1
total			-16

History

💔 Build #42757 failed 4ceb9551d43c9b80a8c83d78ad149ef4a404f563
💔 Build #42739 failed 6dfe4fbc1b0fc8fb69a988689cdf1c73e671ba96

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

joshdover · 2022-10-11T14:59:39Z

I updated this PR with the latest changes from #115032

I didn't quite get it to work yet with the correct permissions ending up in the resulting policy, but I think there's an issue with my test package. I will follow up on this by EOW.

felixbarny · 2022-10-17T07:26:58Z

Thanks for picking this up, Josh. What are the next steps here?

joshdover · 2022-10-17T11:12:08Z

There's some conflicts between this and other work for input packages that we're working through. It also seems there's additional code paths that are fetching details from the registry that we need to remove to unblock this and input packages. I'm working with @hop-dev in #143198 to find a solution.

joshdover · 2023-08-14T12:23:56Z

Already implemented

joshdover added the Team:Fleet Team label for Observability Data Collection Fleet team label May 5, 2022

joshdover mentioned this pull request May 5, 2022

Add flags to allow for routing to dynamic datasets and namespaces elastic/package-spec#327

Merged

2 tasks

joshdover force-pushed the fleet/allow-routing branch from 6dfe4fb to 4ceb955 Compare May 5, 2022 10:59

jsoriano reviewed Sep 5, 2022

View reviewed changes

Add support for allow_routing data stream option

d67bef4

joshdover force-pushed the fleet/allow-routing branch from 4ceb955 to d67bef4 Compare October 11, 2022 12:48

This was referenced Oct 12, 2022

[Fleet] Simplify skipArchive behaviour for input packages #141671

Merged

[Fleet] Return manifest info from get package info endpoints #143198

Closed

joshdover closed this Aug 14, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Fleet] Add support for allow_routing data streams option#131609

[Fleet] Add support for allow_routing data streams option#131609
joshdover wants to merge 1 commit intoelastic:mainfrom
joshdover:fleet/allow-routing

joshdover commented May 5, 2022 •

edited

Loading

Uh oh!

jsoriano Sep 5, 2022

Uh oh!

felixbarny Sep 5, 2022

Uh oh!

jsoriano Sep 5, 2022

Uh oh!

kibana-ci commented Oct 11, 2022 •

edited

Loading

API count

ESLint disabled in files

ESLint disabled line counts

Total ESLint disabled count

Uh oh!

joshdover commented Oct 11, 2022

Uh oh!

felixbarny commented Oct 17, 2022

Uh oh!

joshdover commented Oct 17, 2022

Uh oh!

joshdover commented Aug 14, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

joshdover commented May 5, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Checklist

Risk Matrix

For maintainers

Uh oh!

jsoriano Sep 5, 2022

Choose a reason for hiding this comment

Uh oh!

felixbarny Sep 5, 2022

Choose a reason for hiding this comment

Uh oh!

jsoriano Sep 5, 2022

Choose a reason for hiding this comment

Uh oh!

kibana-ci commented Oct 11, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💔 Build Failed

Failed CI Steps

Test Failures

Metrics [docs]

Public APIs missing comments

Page load bundle

API count

ESLint disabled in files

ESLint disabled line counts

Total ESLint disabled count

History

Uh oh!

joshdover commented Oct 11, 2022

Uh oh!

felixbarny commented Oct 17, 2022

Uh oh!

joshdover commented Oct 17, 2022

Uh oh!

joshdover commented Aug 14, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

joshdover commented May 5, 2022 •

edited

Loading

kibana-ci commented Oct 11, 2022 •

edited

Loading