[Fleet] Auto install content packages task by juliaElastic · Pull Request #226523 · elastic/kibana

juliaElastic · 2025-07-04T07:46:03Z

Summary

Adjusted to datasets field elastic/package-spec#922

Closes https://github.com/elastic/ingest-dev/issues/5685

Added kibana task to check discovery dataset fields of content packages in EPR, check recent data ingested for matching fields, and install content packages if not installed yet.

Added telemetry mappings: https://github.com/elastic/telemetry/pull/4940

To verify:

enable feature flag in kibana.dev.yml: xpack.fleet.enableExperimental: ['enableAutoInstallContentPackages']
TODO start local epr with test package that has a discovery.datasets with a value (e.g. system.cpu), depends on package-spec change to be available
enroll an agent and wait until some data is ingested to the dataset above
wait 10m for the task to run
verify that the test package is installed

Kibana logs:

[2025-07-10T14:07:13.177+02:00][INFO ][plugins.fleet.fleet:auto-install-content-packages-task:1.0.1] [runTask()] started
[2025-07-10T14:07:13.206+02:00][DEBUG][plugins.fleet.fleet:auto-install-content-packages-task:1.0.1] [AutoInstallContentPackagesTask] ESQL query took: 11ms
[2025-07-10T14:07:13.207+02:00][DEBUG][plugins.fleet.fleet:auto-install-content-packages-task:1.0.1] [AutoInstallContentPackagesTask] Found datasets with data: fleet_server.agent_versions, fleet_server.agent_status, elastic_agent.fleet_server
[2025-07-10T14:07:13.207+02:00][DEBUG][plugins.fleet.fleet:auto-install-content-packages-task:1.0.1] [AutoInstallContentPackagesTask] Content packages to install: kubernetes_otel@1.1.0
[2025-07-10T14:07:13.208+02:00][DEBUG][plugins.fleet] Kicking off install of kubernetes_otel-1.1.0 from registry
[2025-07-10T14:07:15.605+02:00][INFO ][plugins.fleet.fleet:auto-install-content-packages-task:1.0.1] [AutoInstallContentPackagesTask] runTask ended: success

Checklist

Unit or functional tests were updated or added to match the most common scenarios

elasticmachine · 2025-07-07T12:09:24Z

Pinging @elastic/fleet (Team:Fleet)

juliaElastic · 2025-07-08T10:52:41Z

x-pack/platform/plugins/shared/fleet/server/services/epm/packages/install.ts

    telemetryEvent.currentVersion = installedPkg?.attributes.version || 'not_installed';
+    telemetryEvent.packageType = packageInfo.type;
+    telemetryEvent.discoveryFields = packageInfo.discovery?.fields;
+    telemetryEvent.automaticInstall = automaticInstall;


sending telemetry event like this:

{ packageName: 'kubernetes_otel', currentVersion: 'not_installed', newVersion: '1.1.0', status: 'success', dryRun: false, eventType: 'package-install', installType: 'install', packageType: 'content', discoveryDatasets: [ { name: 'system.cpu' } ], automaticInstall: true }

juliaElastic · 2025-07-09T10:50:49Z

x-pack/platform/plugins/shared/fleet/server/tasks/auto_install_content_packages_task.ts

+        continue;
+      }
+
+      const hasData = await this.hasDiscoveryFieldData(esClient, dataset);


I'm not sure how much overlap there will be in content packs dataset fields.
Currently doing an ES search for each dataset value (as they can map to multiple packages).

Alternatively we could go by package and do a single terms query with all dataset values at once.
Another option would be to do a single terms aggregation on the dataset field, and compare results with dataset fields in content packages.

Is it worth exploring ESQL for these queries at all to simplify anything here, or is the DSL our best bet? We definitely want to minimize the amount of queries we run here and keep them lightweight as the index patterns we query are going to be under heavy write pressure from agents, collectors, etc in large environments.

Changed the implementation to use a single ESQL query to get the unique datasets that appear in the recently ingested data, filtering out the dataset fields from the already installed content packs.
We would have to test the performance of this query on large clusters.

MichelLosier

Code looks good! Working on the local setup to verify

x-pack/platform/plugins/shared/fleet/server/tasks/auto_install_content_packages_task.ts

…_content_packages_task.ts Co-authored-by: Michel Losier <mikelosier@gmail.com>

elasticmachine · 2025-07-14T09:51:56Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: ed57b89

Failed CI Steps

Fleet Cypress Tests #2

Test Failures

[job] [logs] Fleet Cypress Tests #2 / View agents list Agent status filter should filter on healthy and unhealthy
[job] [logs] Fleet Cypress Tests #2 / View agents list Bulk actions should allow to bulk upgrade agents and cancel that upgrade

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`fleet`	1433	1435	+2

Public APIs missing exports

Total count of every type that is part of your API that should be exported but is not. This will cause broken links in the API documentation system. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats exports for more detailed information.

id	before	after	diff
`fleet`	97	98	+1

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`fleet`	166.9KB	166.9KB	+36.0B

Unknown metric groups

API count

id	before	after	diff
`fleet`	1564	1566	+2

History

💚 Build #318020 succeeded 6224872
💛 Build #317673 was flaky 024b5e1
💛 Build #317588 was flaky ed9aac7
💚 Build #317233 succeeded bdd5c27
💔 Build #317165 failed 4d3b6f9
💔 Build #316614 failed 1466a1a

cc @juliaElastic

ersin-erdal

LGTM

MichelLosier

Runs as expected!

Ran it with this package-registry PR: elastic/package-registry#1338
For the good_content package discovery fields to get picked up I had to:
- In auto_install_content_packages_task:322 I set prerelease to true
- In Registry.fetchList comment out the setConstraints
- Also added elastic_agent.filebeat as a discovery field to good_content

## Summary Adjusted to datasets field elastic/package-spec#922 Closes elastic/ingest-dev#5685 Added kibana task to check discovery dataset fields of content packages in EPR, check recent data ingested for matching fields, and install content packages if not installed yet. Added telemetry mappings: elastic/telemetry#4940 To verify: - enable feature flag in `kibana.dev.yml`: `xpack.fleet.enableExperimental: ['enableAutoInstallContentPackages']` - TODO start local epr with test package that has a `discovery.datasets` with a value (e.g. system.cpu), depends on [package-spec](elastic/package-spec#922) change to be available - enroll an agent and wait until some data is ingested to the dataset above - wait 10m for the task to run - verify that the test package is installed Kibana logs: ``` [2025-07-10T14:07:13.177+02:00][INFO ][plugins.fleet.fleet:auto-install-content-packages-task:1.0.1] [runTask()] started [2025-07-10T14:07:13.206+02:00][DEBUG][plugins.fleet.fleet:auto-install-content-packages-task:1.0.1] [AutoInstallContentPackagesTask] ESQL query took: 11ms [2025-07-10T14:07:13.207+02:00][DEBUG][plugins.fleet.fleet:auto-install-content-packages-task:1.0.1] [AutoInstallContentPackagesTask] Found datasets with data: fleet_server.agent_versions, fleet_server.agent_status, elastic_agent.fleet_server [2025-07-10T14:07:13.207+02:00][DEBUG][plugins.fleet.fleet:auto-install-content-packages-task:1.0.1] [AutoInstallContentPackagesTask] Content packages to install: kubernetes_otel@1.1.0 [2025-07-10T14:07:13.208+02:00][DEBUG][plugins.fleet] Kicking off install of kubernetes_otel-1.1.0 from registry [2025-07-10T14:07:15.605+02:00][INFO ][plugins.fleet.fleet:auto-install-content-packages-task:1.0.1] [AutoInstallContentPackagesTask] runTask ended: success ``` ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios --------- Co-authored-by: Michel Losier <mikelosier@gmail.com>

wip auto install content packages task

f0c66ad

juliaElastic self-assigned this Jul 4, 2025

juliaElastic added backport:skip This PR does not require backporting release_note:feature Makes this part of the condensed release notes labels Jul 4, 2025

juliaElastic and others added 6 commits July 4, 2025 09:46

Merge branch 'main' into auto-install-content-packages

a148d0d

fix checks

16f6e09

cache discovery map

c1378a4

add tests

def5ec2

cleanup

3a7fb31

Merge branch 'main' into auto-install-content-packages

ffe5e88

juliaElastic marked this pull request as ready for review July 7, 2025 11:36

juliaElastic requested review from a team as code owners July 7, 2025 11:36

update test

1b5e6c1

botelastic bot added the Team:Fleet Team label for Observability Data Collection Fleet team label Jul 7, 2025

add telemetry and use streaming

1466a1a

juliaElastic marked this pull request as draft July 8, 2025 08:25

juliaElastic commented Jul 8, 2025

View reviewed changes

juliaElastic and others added 3 commits July 8, 2025 13:04

fix tests

054780e

Merge branch 'main' into auto-install-content-packages

1c2fe45

update to use discovery.datasets

bd0379c

juliaElastic marked this pull request as ready for review July 9, 2025 07:46

fix index patterns

4d3b6f9

juliaElastic commented Jul 9, 2025

View reviewed changes

juliaElastic and others added 5 commits July 9, 2025 13:40

remove installed packages before doing search on dataset

5002f00

Merge branch 'main' into auto-install-content-packages

bdd5c27

use esql grouping to fetch datasets

ed9aac7

filter out installed packages datasets

15aba02

early return if no datasets that are not installed

024b5e1

juliaElastic and others added 2 commits July 11, 2025 09:38

move timestamp filter to esql

14091fe

Merge branch 'main' into auto-install-content-packages

6224872

MichelLosier reviewed Jul 11, 2025

View reviewed changes

x-pack/platform/plugins/shared/fleet/server/tasks/auto_install_content_packages_task.ts Outdated Show resolved Hide resolved

juliaElastic and others added 2 commits July 14, 2025 09:00

Update x-pack/platform/plugins/shared/fleet/server/tasks/auto_install…

16df27a

…_content_packages_task.ts Co-authored-by: Michel Losier <mikelosier@gmail.com>

Merge branch 'main' into auto-install-content-packages

ed57b89

juliaElastic requested a review from MichelLosier July 14, 2025 07:00

ersin-erdal approved these changes Jul 14, 2025

View reviewed changes

MichelLosier approved these changes Jul 14, 2025

View reviewed changes

juliaElastic merged commit 4df9932 into elastic:main Jul 15, 2025
11 of 12 checks passed

kibanamachine added the v9.2.0 label Jul 15, 2025

juliaElastic added release_note:skip Skip the PR/issue when compiling release notes and removed release_note:feature Makes this part of the condensed release notes labels Aug 22, 2025

kpollich mentioned this pull request Sep 24, 2025

[REQUEST]: Document auto-install content packs feature for integration authors elastic/ingest-docs#1857

Closed

mrodm mentioned this pull request Feb 13, 2026

Support Kibana 9.3+ Fleet experimental features format elastic/elastic-package#3276

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Fleet] Auto install content packages task#226523

[Fleet] Auto install content packages task#226523
juliaElastic merged 22 commits intoelastic:mainfrom
juliaElastic:auto-install-content-packages

juliaElastic commented Jul 4, 2025 •

edited

Loading

Uh oh!

elasticmachine commented Jul 7, 2025

Uh oh!

juliaElastic Jul 8, 2025 •

edited

Loading

Uh oh!

juliaElastic Jul 9, 2025

Uh oh!

kpollich Jul 9, 2025

Uh oh!

juliaElastic Jul 10, 2025 •

edited

Loading

Uh oh!

MichelLosier left a comment

Uh oh!

Uh oh!

elasticmachine commented Jul 14, 2025

API count

Uh oh!

ersin-erdal left a comment

Uh oh!

MichelLosier left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

juliaElastic commented Jul 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Checklist

Uh oh!

elasticmachine commented Jul 7, 2025

Uh oh!

juliaElastic Jul 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

juliaElastic Jul 9, 2025

Choose a reason for hiding this comment

Uh oh!

kpollich Jul 9, 2025

Choose a reason for hiding this comment

Uh oh!

juliaElastic Jul 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MichelLosier left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

elasticmachine commented Jul 14, 2025

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

Metrics [docs]

Public APIs missing comments

Public APIs missing exports

Page load bundle

API count

History

Uh oh!

ersin-erdal left a comment

Choose a reason for hiding this comment

Uh oh!

MichelLosier left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

juliaElastic commented Jul 4, 2025 •

edited

Loading

juliaElastic Jul 8, 2025 •

edited

Loading

juliaElastic Jul 10, 2025 •

edited

Loading