Skip to content

[Fleet] Install final pipeline#100973

Merged
nchaulet merged 7 commits intoelastic:masterfrom
nchaulet:feature-fleet-final-pipeline
Jun 3, 2021
Merged

[Fleet] Install final pipeline#100973
nchaulet merged 7 commits intoelastic:masterfrom
nchaulet:feature-fleet-final-pipeline

Conversation

@nchaulet
Copy link
Copy Markdown
Member

@nchaulet nchaulet commented May 31, 2021 •
edited
Loading

Summary

Resolve #97181

To ensure the agent do not tamper agent id when sending event, we are going to use a final pipeline that check the agent id against the agent id contained in the api key.

For that:

  • we create a pipeline .fleet_final_pipeline during the fleet setup
  • we use that final pipeline on every index template installed by Fleet

Not done in that PR:

  • we do not update the existing index template installed by fleet without final_pipeline, should we migrate these index?

Decisions made in that PR:

  • If a template define a final_pipeline we will throw an error when installing it.

What to test?

every event ingested with Elastic agent should have the field event.agent_id_status.

@nchaulet nchaulet added v8.0.0 release_note:skip Skip the PR/issue when compiling release notes Team:Fleet Team label for Observability Data Collection Fleet team v7.14.0 labels May 31, 2021
@nchaulet nchaulet self-assigned this May 31, 2021
@nchaulet nchaulet marked this pull request as ready for review May 31, 2021 13:24
@nchaulet nchaulet requested a review from a team as a code owner May 31, 2021 13:24
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented May 31, 2021

Pinging @elastic/fleet (Team:Fleet)

@nchaulet nchaulet requested a review from andrewkroh June 1, 2021 12:39
@nchaulet nchaulet requested a review from kpollich June 1, 2021 14:40
@jen-huang jen-huang changed the title [Fleet] Fleet final pipeline [Fleet] Install final pipeline Jun 1, 2021
@P1llus
Copy link
Copy Markdown
Member

P1llus commented Jun 2, 2021

Just a small question/comment on this approach. Does this mean that final_pipeline will be set for all indices used by packages/integrations?
If so, this will then remove the possibility for end-users to apply a pipeline to overwrite module fields in any way.

@nchaulet
Copy link
Copy Markdown
Member Author

nchaulet commented Jun 2, 2021

Just a small question/comment on this approach. Does this mean that final_pipeline will be set for all indices used by packages/integrations?
If so, this will then remove the possibility for end-users to apply a pipeline to overwrite module fields in any way.

Yes with that PR the final_pipeline will be used for all indices, user will still be able to define a pipeline to overwrite module fields, but not a final_pipeline

@nchaulet
Copy link
Copy Markdown
Member Author

nchaulet commented Jun 2, 2021

@elasticmachine merge upstream

@nchaulet nchaulet added the auto-backport Deprecated - use backport:version if exact versions are needed label Jun 2, 2021
andrewkroh
andrewkroh approved these changes Jun 2, 2021
View reviewed changes
Copy link
Copy Markdown
Member

@andrewkroh andrewkroh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@nchaulet nchaulet enabled auto-merge (squash) June 3, 2021 12:31
@nchaulet
Copy link
Copy Markdown
Member Author

nchaulet commented Jun 3, 2021

@elasticmachine merge upstream

@kibanamachine
Copy link
Copy Markdown
Contributor

kibanamachine commented Jun 3, 2021

💚 Build Succeeded

Metrics [docs]

✅ unchanged

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @nchaulet

@nchaulet nchaulet merged commit 3b1e8b0 into elastic:master Jun 3, 2021
@kibanamachine kibanamachine mentioned this pull request Jun 3, 2021
Merged
kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Jun 3, 2021
@nchaulet @kibanamachine
[Fleet] Install final pipeline (elastic#100973)
94f9b0e
@kibanamachine
Copy link
Copy Markdown
Contributor

kibanamachine commented Jun 3, 2021

💚 Backport successful

Status Branch Result
✅ 7.x

This backport PR will be merged automatically after passing CI.

@nchaulet nchaulet mentioned this pull request Jun 3, 2021
Closed
kibanamachine added a commit that referenced this pull request Jun 3, 2021
@kibanamachine @nchaulet
[Fleet] Install final pipeline (#100973) (#101299)
1609e13 
Co-authored-by: Nicolas Chaulet <nicolas.chaulet@elastic.co>
@andrewkroh andrewkroh mentioned this pull request Jun 21, 2021
Merged
1 task
andrewkroh added a commit that referenced this pull request Jun 21, 2021
@andrewkroh
[Fleet] Update final pipeline based on ECS event.agent_id_status (#10…
2e3d527 
…2805)

This updates the Fleet final pipeline added in #100973 to match the specification of
`event.agent_id_status` field as defined in ECS. The field was added to ECS in
elastic/ecs#1454. Basically the values of the field were simplified
from what was originally proposed and implemented.
kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Jun 21, 2021
@andrewkroh @kibanamachine
[Fleet] Update final pipeline based on ECS event.agent_id_status (ela…
29979da 
…stic#102805)

This updates the Fleet final pipeline added in elastic#100973 to match the specification of
`event.agent_id_status` field as defined in ECS. The field was added to ECS in
elastic/ecs#1454. Basically the values of the field were simplified
from what was originally proposed and implemented.
kibanamachine added a commit that referenced this pull request Jun 22, 2021
@kibanamachine @andrewkroh
[Fleet] Update final pipeline based on ECS event.agent_id_status (#10…
7f7ba0b 
…2805) (#102832)

This updates the Fleet final pipeline added in #100973 to match the specification of
`event.agent_id_status` field as defined in ECS. The field was added to ECS in
elastic/ecs#1454. Basically the values of the field were simplified
from what was originally proposed and implemented.

Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>
@nchaulet nchaulet mentioned this pull request Aug 2, 2021
Closed
@ruflin ruflin mentioned this pull request Jun 17, 2022
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@andrewkroh andrewkroh andrewkroh approved these changes

@kpollich kpollich kpollich approved these changes

Assignees

@nchaulet nchaulet

Labels

auto-backport Deprecated - use backport:version if exact versions are needed release_note:skip Skip the PR/issue when compiling release notes Team:Fleet Team label for Observability Data Collection Fleet team v7.14.0 v8.0.0

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Fleet] Install a final_pipeline to help with host impersonation

6 participants

@nchaulet @elasticmachine @P1llus @kibanamachine @andrewkroh @kpollich