[Fleet] Refactor setup to load default packages/policies with preconfiguration by Zacqary · Pull Request #97328 · elastic/kibana

Zacqary · 2021-04-15T21:36:45Z

Summary

This removes the bespoke functions for installing default packages and configuring the default agent policies, and instead sets these as the default Kibana Config values for the preconfiguration API.

Testing

Default setup

Remove all references to xpack.fleet from your kibana.dev.yml
Start Kibana and navigate to the Fleet app
Ensure that the Default Agent Policy and Default Fleet Server Agent Policy have been created
Ensure that the System, Endpoint Security, Elastic Agent, and Fleet Server integrations are installed

Overriding the defaults

Restart your yarn es snapshot to clear your cluster, then set your kibana.dev.yml to:

xpack.fleet.packages:
  - name: system
    version: latest
  - name: endpoint
    version: latest
  - name: elastic_agent
    version: latest
  - name: fleet_server
    version: 0.2.0
  - name: apache
    version: latest

Ensure that all five of these integrations are installed, and that Fleet Server is installed with an outdated version.

Restart yarn es snapshot again, then set your kibana.dev.yml to:

xpack.fleet.agentPolicies:
  - name: Alternative Default Agent Policy
    id: 1
    namespace: test
    is_default: true
    monitoring_enabled: []
    package_policies:
      - package:
          name: system
        name: System Integration
  - name: Alternative Default Fleet Server Agent Policy
    id: 2
    namespace: test
    is_default_fleet_server: true
    package_policies:
      - package:
          name: fleet_server
        name: Fleet Server Integration

Ensure that the default policies are created with these alternative names and that agent monitoring is disabled for the default policy, but enabled for the default fleet server policy. Try to delete them, and ensure that they fail to be deleted.

Restart yarn es snapshot again, then change your kibana.dev.yml to:

xpack.fleet.agentPolicies:
  - name: Agent Policy With No Default
    id: 1
    namespace: test
    package_policies:
      - package:
          name: system
        name: System Integration

Ensure that Fleet starts with only this one policy, and that it can be deleted.

Checklist

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support

…iguration

elasticmachine · 2021-04-15T22:08:07Z

Pinging @elastic/fleet (Feature:Fleet)

x-pack/plugins/fleet/common/constants/agent_policy.ts

ruflin · 2021-04-16T12:09:51Z

Tried to run this locally and got the following error when accessing the Fleet page:

server    log   [14:08:07.108] [error][fleet][plugins] TypeError: Cannot read property 'search' of undefined
    at AgentPolicyService.getLatestFleetPolicy (/Users/ruflin/Dev/elastic/kibana/x-pack/plugins/fleet/server/services/agent_policy.ts:608:32)
    at map (/Users/ruflin/Dev/elastic/kibana/x-pack/plugins/fleet/server/services/agents/setup.ts:29:67)
    at Array.map (<anonymous>)
    at ensureAgentActionPolicyChangeExists (/Users/ruflin/Dev/elastic/kibana/x-pack/plugins/fleet/server/services/agents/setup.ts:28:19)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at createSetupSideEffects (/Users/ruflin/Dev/elastic/kibana/x-pack/plugins/fleet/server/services/setup.ts:74:3)
    at awaitIfPending (/Users/ruflin/Dev/elastic/kibana/x-pack/plugins/fleet/server/services/setup_utils.ts:30:20)
    at fleetSetupHandler (/Users/ruflin/Dev/elastic/kibana/x-pack/plugins/fleet/server/routes/setup/handlers.ts:50:5)
    at Router.handle (/Users/ruflin/Dev/elastic/kibana/src/core/server/http/router/router.ts:272:30)
    at handler (/Users/ruflin/Dev/elastic/kibana/src/core/server/http/router/router.ts:227:11)
    at exports.Manager.execute (/Users/ruflin/Dev/elastic/kibana/node_modules/@hapi/hapi/lib/toolkit.js:60:28)
    at Object.internals.handler (/Users/ruflin/Dev/elastic/kibana/node_modules/@hapi/hapi/lib/handler.js:46:20)
    at exports.execute (/Users/ruflin/Dev/elastic/kibana/node_modules/@hapi/hapi/lib/handler.js:31:20)
    at Request._lifecycle (/Users/ruflin/Dev/elastic/kibana/node_modules/@hapi/hapi/lib/request.js:370:32)
    at Request._execute (/Users/ruflin/Dev/elastic/kibana/node_modules/@hapi/hapi/lib/request.js:279:9)

The above is without any changes in the config file.

afgomez

I did a first pass downloading the code and clicking around. I noticed the following things

Default packages

I have this in my kibana.dev.yml

xpack.fleet.packages:
  - name: system
    version: latest
  - name: endpoint
    version: latest
  - name: elastic_agent
    version: latest
  - name: fleet_server
    version: 0.1.9
  - name: apache
    version: latest

Packages with latest version get installed, but fleet server is not.

Alternative default agent policies

I restarted everything with this on my kibana.dev.yml

xpack.fleet.agentPolicies:
  - name: Alternative Default Agent Policy
    id: 1
    namespace: test
    is_default: true
    package_policies:
      - package:
          name: system
        name: System Integration
  - name: Alternative Default Fleet Server Agent Policy
    id: 1
    namespace: test
    is_default_fleet_server: true
    package_policies:
      - package:
          name: fleet_server
        name: Fleet Server Integration

No agent policies were added

Agent policy with no default

Same happens as in point 2 :(

jen-huang · 2021-04-19T14:32:06Z

I am seeing the same as what @afgomez reported.

…fig-policy # Conflicts: # x-pack/plugins/fleet/server/services/epm/packages/install.ts # x-pack/plugins/fleet/server/services/setup.ts

skh · 2021-04-19T16:21:12Z

x-pack/plugins/fleet/server/services/setup.ts

 export interface SetupStatus {
  isInitialized: boolean;
  preconfigurationError: { name: string; message: string } | undefined;
-  nonFatalPackageUpgradeErrors: DefaultPackagesInstallationError[];


Is this undoing my work from #97404 ? Is this intentional?
Cc @ruflin

Not intentional, I just wasn't immediately sure how to deal with the merge conflict. I'll get this back in.

I think this actually mostly duplicates the functionality of preconfigurationError. I'll reconcile the differences and streamline.

As of the latest commit, running the test scenario from #97404 results in a preconfigurationError of:

I can tweak this functionality as needed.

I'm not sure about getting rid of nonFatalPackageUpgradeErrors and DefaultPackagesInstallationError. I like the explicitness of that interface wrt to package installation and various install types/ops: install/upgrade/rollback. package install handling is a sensitive area that we will want to keep iterating and improving on so I would like to keep the error granularity here.

in contrast, preconfigurationError can be related to package and agent policy errors.

my suggestion, what do you think?:

// export generic `nonFatalErrors` property export interface SetupStatus { isInitialized: boolean; nonFatalErrors?: Array<PreconfigurationError | DefaultPackagesInstallationError>; } // add back `DefaultPackagesInstallationError` and return it where appropriate export interface DefaultPackagesInstallationError { installType: InstallType; error: Error; } // define `PreconfigurationError` export interface PreconfigurationError { package?: { name: string; version: string; }; agentPolicy?: { name: string; } error: Error; }

Implemented this as best I could. Currently all properties besides error go unused on both error types, but the data is there at least.

Zacqary · 2021-04-19T18:43:11Z

Packages with latest version get installed, but fleet server is not.

@afgomez Try setting fleet_server to 0.2.0 instead of 0.1.9. I'll see if I can add some error handling for specifying invalid versions.

Zacqary · 2021-04-19T19:26:52Z

@afgomez As of latest commit, the original test scenario should result in:

Changing the version to 0.2.0 works as expected.

jfsiii · 2021-04-22T00:33:59Z

@spalger we want this to land in 7.13 so I restored the label. LMK if there's some reason we shouldn't use that label to backport to 7.13

spalger · 2021-04-22T00:34:51Z

@spalger we want this to land in 7.13 so I restored the label. LMK if there's some reason we shouldn't use that label to backport to 7.13

Nope, just wanted people to make that decision intentionally and ensure that we didn't have any more people auto-backport to 7.13 without backporting to 7.x.

afgomez · 2021-04-22T12:49:01Z

@elasticmachine merge upstream

afgomez · 2021-04-22T12:49:52Z

^^^ I think the CI failures are unrelated to this PR. Merging master to trigger another build before @Zacqary comes back online

jen-huang · 2021-04-22T14:58:06Z

x-pack/plugins/fleet/server/types/models/preconfiguration.ts

-  })
+  }),
+  {
+    defaultValue: [DEFAULT_AGENT_POLICY, DEFAULT_FLEET_SERVER_AGENT_POLICY],


what happens if the user has only specified either a default or a default fleet server policy in their preconfiguration? will we still ensure that whichever default one is missing, is added? what if their preconfiguration doesn't specify any default policies?

i think we might want to do the same sort of merging downstream like what you did for default packages

I was under the impression from discussion on the issue (#97016) that we explicitly did not want to do downstream merging for agent policies, and that the config file should be the source of truth for what the "default policies" are.

nchaulet

Tested the default and a few scenarios, it is working well 👍
Also tested the upgrade from 7.12 to this change and looks good.

Zacqary · 2021-04-22T16:23:22Z

@jen-huang @nchaulet I'm gonna merge this because it meets the acceptance criteria of the initial issue and I really don't want to wait for the entire CI to run again and then potentially fail.

Can we open a new issue to discuss whether default agentPolicies should also be downstream-merged like required packages, or if the current behavior is fine?

…iguration (elastic#97328) Co-authored-by: Nicolas Chaulet <nicolas.chaulet@elastic.co> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

jen-huang · 2021-04-22T16:27:59Z

@Zacqary 👍🏻 , I'm comfortable with that, thanks for the revisions on this PR over the past week. I'll continue poking and prodding on main branch and raise any issues that come up from there.

…iguration (#97328) (#98053) Co-authored-by: Nicolas Chaulet <nicolas.chaulet@elastic.co> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Nicolas Chaulet <nicolas.chaulet@elastic.co> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

…iguration (#97328) (#98052) Co-authored-by: Nicolas Chaulet <nicolas.chaulet@elastic.co> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Nicolas Chaulet <nicolas.chaulet@elastic.co> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

kibanamachine · 2021-04-28T17:57:03Z

💚 Build Succeeded

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`fleet`	991	993	+2

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`fleet`	722.4KB	722.1KB	-306.0B

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`fleet`	346.4KB	346.8KB	+399.0B

Unknown metric groups

API count

id	before	after	diff
`fleet`	1081	1083	+2

History

💚 Build #121710 succeeded 0658aaf
💔 Build #121595 failed 7142027
💔 Build #121505 failed 32630d8
💔 Build #121470 failed b25796e
💔 Build #121451 failed be0b09c

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @Zacqary

Zacqary added Feature:Fleet Fleet team's agent central management project Team:Fleet Team label for Observability Data Collection Fleet team v7.13.0 v8.0.0 release_note:skip Skip the PR/issue when compiling release notes labels Apr 15, 2021

Zacqary self-assigned this Apr 15, 2021

[Fleet] Refactor setup to load default packages/policies with preconf…

70d002e

…iguration

Zacqary force-pushed the 97106-default-config-policy branch from 1232e32 to 70d002e Compare April 15, 2021 22:03

Zacqary changed the title ~~[Fleet] Refactor setup to setup default packages/policies with preconfiguration~~ [Fleet] Refactor setup to load default packages/policies with preconfiguration Apr 15, 2021

Zacqary marked this pull request as ready for review April 15, 2021 22:08

Zacqary requested a review from a team as a code owner April 15, 2021 22:08

jfsiii reviewed Apr 16, 2021

View reviewed changes

x-pack/plugins/fleet/common/constants/agent_policy.ts Show resolved Hide resolved

jfsiii reviewed Apr 16, 2021

View reviewed changes

x-pack/plugins/fleet/common/constants/agent_policy.ts Show resolved Hide resolved

ruflin requested review from afgomez and jen-huang April 16, 2021 11:56

afgomez suggested changes Apr 19, 2021

View reviewed changes

jen-huang mentioned this pull request Apr 19, 2021

[Fleet] Preconfiguration allow agent.monitoring settings #97356

Closed

Zacqary added 2 commits April 19, 2021 11:03

Fix bad merge

ab14636

Merge remote-tracking branch 'upstream/master' into 97106-default-con…

1a2f891

…fig-policy # Conflicts: # x-pack/plugins/fleet/server/services/epm/packages/install.ts # x-pack/plugins/fleet/server/services/setup.ts

skh reviewed Apr 19, 2021

View reviewed changes

Zacqary mentioned this pull request Apr 19, 2021

[Fleet] Error during /setup now shows "Cannot read preconfigurationError of null" #97401

Closed

Fix missing preconfigurationError regression

54561b7

Fix nonfatal package errors

d0be8b8

Restore typedefs

324bcf5

Zacqary requested a review from afgomez April 19, 2021 19:39

spalger added v7.14.0 and removed v7.13.0 labels Apr 21, 2021

Fix integration test

7142027

jfsiii added the v7.13.0 label Apr 22, 2021

jen-huang self-requested a review April 22, 2021 03:56

Merge branch 'master' into 97106-default-config-policy

0658aaf

jen-huang reviewed Apr 22, 2021

View reviewed changes

nchaulet approved these changes Apr 22, 2021

View reviewed changes

afgomez approved these changes Apr 22, 2021

View reviewed changes

Zacqary merged commit 57f84f8 into elastic:master Apr 22, 2021

Zacqary deleted the 97106-default-config-policy branch April 22, 2021 16:25

Zacqary mentioned this pull request Apr 22, 2021

[7.x] [Fleet] Refactor setup to load default packages/policies with preconfiguration (#97328) #98052

Merged

Zacqary mentioned this pull request Apr 22, 2021

[7.13] [Fleet] Refactor setup to load default packages/policies with preconfiguration (#97328) #98053

Merged

jen-huang mentioned this pull request Apr 26, 2021

[Fleet][APM] Discuss: Setup of APM integration #90410

Closed

jfsiii mentioned this pull request Jun 22, 2021

[Fleet] Add UI and mappings for agent policy unenroll_timeout #102970

Merged

6 tasks

jen-huang mentioned this pull request Jun 28, 2021

[Fleet] Allow existing preconfigured policies to be edited #103187

Closed

an-tex mentioned this pull request Sep 20, 2021

Kibana YAML settings with empty array values get parsed as null elastic/cloud-on-k8s#4853

Closed

Conversation

Zacqary commented Apr 15, 2021 • edited by afgomez Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Testing

Default setup

Overriding the defaults

Checklist

Uh oh!

elasticmachine commented Apr 15, 2021

Uh oh!

Uh oh!

Uh oh!

ruflin commented Apr 16, 2021

Uh oh!

afgomez left a comment

Choose a reason for hiding this comment

Uh oh!

jen-huang commented Apr 19, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Zacqary commented Apr 19, 2021

Uh oh!

Zacqary commented Apr 19, 2021

Uh oh!

jfsiii commented Apr 22, 2021

Uh oh!

spalger commented Apr 22, 2021

Uh oh!

afgomez commented Apr 22, 2021

Uh oh!

afgomez commented Apr 22, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nchaulet left a comment

Choose a reason for hiding this comment

Uh oh!

Zacqary commented Apr 22, 2021

Uh oh!

jen-huang commented Apr 22, 2021

Uh oh!

kibanamachine commented Apr 28, 2021

💚 Build Succeeded

Metrics [docs]

Public APIs missing comments

Async chunks

Page load bundle

API count

History

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

11 participants

Zacqary commented Apr 15, 2021 •

edited by afgomez

Loading