Skip to content

[EPM] Implement ingest pipeline installation from packages #49752

@skh

Description

@skh

This task is to implement rewriting and installation of ingest pipelines from packages.

For each input/dataset defined in a package:

  • The main ("entry") pipeline is defined in the manifest.yml of the input/dataset.
  • This main pipeline may contain references to other pipelines. The reference in the pipeline looks like this: {{IngestPipeline 'my-pipeline'}} and refers to a pipeline called my-pipeline.json in the same directory as the main pipeline in the package.
  • The main pipeline will be called like the index alias it refers to (e.g. logs-nginx-prod-access-aabbcc)
  • (40752 rewrite ingest pipeline #50627) Rewrite the content of the main pipeline, replacing every instance of {{IngestPipeline 'some-pipeline'}} with logs-nginx-prod-access-aabbcc-some-pipeline (the name of the main pipeline and the name of the referenced pipeline joined by - to one long string)
  • Install all referenced pipelines with their new names, e.g. logs-nginx-prod-access-aabbcc-some-pipeline
  • Finally install the rewritten main pipeline with the name logs-nginx-prod-access-aabbcc
  • Pipelines may also be yml-files. The same rules apply.
  • It is possible that an input/dataset doesn't have any ingest pipelines defined.
  • If the format ({{IngestPipeline 'my-pipeline'}}) turns out to be hard to parse we can still change it.

Metadata

Metadata

Assignees

Labels

Feature:EPMFleet team's Elastic Package Manager (aka Integrations) projectTeam:FleetTeam label for Observability Data Collection Fleet team

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions