Set pipeline name automatically in transform definitions by mrodm · Pull Request #2973 · elastic/elastic-package

mrodm · 2025-10-02T11:08:19Z

This PR adds support to set automatically the pipeline name used in transforms adding the current package version as a prefix.

In order to achieve that elasticsearch/transform/*/transform.yml is managed by elastic-package as a template file (Golang text template). And it is allowed to use a function in the template named ingestPipelineName that adds the current version of the package to the pipeline.

Example:

dest:
  index: "logs-ti_anomali_latest.threatstream-2"
  pipeline: '{{ ingestPipelineName "transform" }}'
  aliases:
    - alias: "logs-ti_anomali_latest.threatstream"
      move_on_creation: true

This rendering of the template happens at different stages to ensure always is performed:

Building package: elastic-package build
Installing package: elastic-package install
System tests: elastic-package test system

Author's checklist

Test rendering while building
Test installing package
Test running system tests
Add note in Revert stack version pinning for packages failing with 9.x #2773 for the new test package.
Create PR to fix packages using wrong transform definitions if any.

How to test this PR locally

elastic-package stack up -v -d
package_path="test/packages/parallel/ti_anomali_template"

# Build and check that pipeline has been replaced
elastic-package -C "${package_path}" build -v
grep -A 5 "dest:"  build/packages/ti_anomali_template/1.22.1/elasticsearch/transform/latest_ioc/transform.yml 
# Expected output:
# dest:
#   index: "logs-ti_anomali_latest.threatstream-2"
#   pipeline: '1.22.1-transform'
#   aliases:
#     - alias: "logs-ti_anomali_latest.threatstream"
#       move_on_creation: true


# Install package
elastic-package -C "${package_path}" install -v

# check in Kibana that the transform has been installed with the right pipeline
curl -k -s -u elastic:changeme 'https://localhost:9200/_transform/logs-ti_anomali_template.latest_ioc-default-0.3.0' | jq -r '.transforms[] | .dest'
# Expected output:
# {
#   "index": "logs-ti_anomali_latest.threatstream-2",
#   "aliases": [
#     {
#       "alias": "logs-ti_anomali_latest.threatstream",
#       "move_on_creation": true
#     }
#   ],
#   "pipeline": "1.22.1-transform"
# }

# Running system tests for this package should work successfully
# Transform should be detected and validated
elastic-package -C "${package_path}" test system -v

elastic-package stack down -v

Allow to set an IngestPipelineName function in these templates in order to set the right ingest template filename. This filename must contain as a prefix the package version (from the manifest).

mrodm · 2025-10-02T14:44:53Z

/test

mrodm · 2025-10-02T15:55:43Z

/test

mrodm · 2025-10-02T17:03:04Z

/test

mrodm · 2025-10-02T17:42:45Z

/test

mrodm · 2025-10-02T18:17:45Z

/test

mrodm · 2025-10-03T08:14:18Z

/test

mrodm · 2025-10-03T11:09:20Z

/test

jsoriano

LGTM, some nitpicking and wondering if we should use text/template instead of html/template.

internal/packages/packages.go

jsoriano · 2025-10-07T14:34:24Z

internal/packages/packages.go

+	_, err = os.Stat(filepath.Join(packageRootPath, "elasticsearch", "ingest_pipeline", pipelineFileName))
+	if err != nil {
+		return nil, TransformDefinition{}, fmt.Errorf("destination ingest pipeline file %s not found: %w", pipelineFileName, err)
+	}


Did you test with integrations? Maybe some package fails with this 😬 Though if it does we should fix the package.

Testing to build locally all packages in the integrations repository, there is one package that sets as dest.pipeline the ingest pipeline from the data stream:

dest: index: "aws_billing.cur-v1" pipeline: "metrics-aws_billing.cur-0.1.0-pipeline_extract_metadata" aliases: - alias: "aws_billing.cur_latest" move_on_creation: true

That fails with the current validation 🤔

That package (aws_billing) is failing this validation because of:

It's using a pipeline from a data stream

It set an old version in the destination pipeline, it should be 0.2.0.

That pipeline from cur data stream looks like it is not used:
https://github.com/elastic/integrations/blob/34df8c08914a678dc3310bd000e7908a1af42ef6/packages/aws_billing/data_stream/cur/elasticsearch/ingest_pipeline/pipeline_extract_metadata.yml

At least from the default pipeline there is no pipeline processor to trigger that other pipeline:
https://github.com/elastic/integrations/blob/34df8c08914a678dc3310bd000e7908a1af42ef6/packages/aws_billing/data_stream/cur/elasticsearch/ingest_pipeline/default.yml

Here I've added the support to detect if the ingest pipeline exists on any data stream:
4f2756e

Probably, it could be moved the pipeline in that package to be on the elasticsearch folder in the root, but in any case it could be interesting to keep the support to check if pipelines are located in data streams.

WDYT ?

I thought that for those cases using ingest pipelines for data streams, the transform definition could use the function like this:

dest: index: "aws_billing.cur-v1" pipeline: "metrics-aws_billing.cur{{ ingestPipelineName "pipeline_extract_metadata"}}" aliases: - alias: "aws_billing.cur_latest" move_on_creation: true

Currently, this would be the minimum change required for aws_billing package
elastic/integrations#15593 since the latest version of the package is 0.2.0.

Another option could be move the Ingest Pipeline to the elasticsearch folder from the root of the package, but I'm not totally sure if this could lead to some issue.

Currently, this would be the minimum change required for aws_billing package
elastic/integrations#15593 since the latest version of the package is 0.2.0.

Yep, we can apply this change by now, but using a pipeline from a data stream in a transform sounds wrong.

We can also use the placeholder you proposed in the previous comment, right?

pipeline: "metrics-aws_billing.cur{{ ingestPipelineName "pipeline_extract_metadata"}}"

We can also use the placeholder you proposed in the previous comment, right?

pipeline: "metrics-aws_billing.cur{{ ingestPipelineName "pipeline_extract_metadata"}}"

Exactly, but that placeholder could be used there once the elastic-package version containing this feature is available in the integrations repository.

Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co>

mrodm · 2025-10-07T19:18:32Z

test integrations

elastic-vault-github-plugin-prod · 2025-10-07T19:25:16Z

Created or updated PR in integrations repository to test this version. Check elastic/integrations#15594

elasticmachine · 2025-10-07T19:39:46Z

💚 Build Succeeded

Buildkite Build
Commit: 4f2756e

History

💔 Build #6300 failed a44d6cc
💚 Build #6288 succeeded 351171c
💚 Build #6287 succeeded 351171c
💚 Build #6284 succeeded 351171c
💔 Build #6283 failed 351171c

cc @mrodm

…cally in transforms (#16175) ti_google_threat_intelligence: Set destination pipeline name automatically in transforms Due to an earlier limitation of elastic-package to unable to extract integration version into the transform, the destination pipeline needed to be set/updated manually. Currently the integration requires users to manually add destination pipelines into the transform during the package installation and upgrade. This presents significant inconvenience to users and is often error-prone. elastic-package#2973 [1] removes the dest pipeline limitation which now allows the pipeline names to be templated. This PR updates the transform definition to set the pipeline names using a template which renders integration version. Change involves updating all transforms with: dest: pipeline: '{{ ingestPipelineName "<pipeline-name>"}}' Also: - With this the transform's does not require any manual intervention, hence they are now auto-enabled. - README is updated accordingly. [1] elastic/elastic-package#2973

…s missing and set destination pipeline name automatically in transforms The foreach processor in the ingest pipeline now uses ignore_missing: true to handle cases where threat_connect.indicator.associated_groups.data but lacks the attributes field, preventing pipeline failures during data processing. elastic-package#2973 [1] removes the dest pipeline limitation which now allows the pipeline names to be templated. This PR updates the transform definition to set the pipeline names using a template which renders integration version. [1] elastic/elastic-package#2973

mrodm added 2 commits October 2, 2025 12:26

Set transform manifest as template

07f3e15

Allow to set an IngestPipelineName function in these templates in order to set the right ingest template filename. This filename must contain as a prefix the package version (from the manifest).

Add a copy of ti_anomaly with a dummy ingest pipeline for transform

5961b7e

mrodm self-assigned this Oct 2, 2025

mrodm added 6 commits October 2, 2025 13:29

Skip check if there is no destination pipeline

951defe

Check for empty pipelines

6cf6d08

Add test

9093315

Pin stack version for ti_anomali_template

ff9aa3f

Update stack version used for ti_anomali and ti_anomali_logsdb

bf90e2b

Update index and alias names in tranform

351171c

mrodm marked this pull request as ready for review October 3, 2025 11:46

mrodm requested a review from a team October 6, 2025 08:42

jsoriano reviewed Oct 7, 2025

View reviewed changes

mrodm and others added 5 commits October 7, 2025 19:24

Apply suggestions from code review

789693b

Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co>

Set to lowercase first letter in template function

d8f0447

Use text/template

ff0daa5

Re-phrase error message

a44d6cc

Validate if ingest pipeline exists on data stream

4f2756e

elastic-vault-github-plugin-prod bot mentioned this pull request Oct 7, 2025

Test elastic-package#2973 - DO NOT MERGE elastic/integrations#15594

Closed

mrodm mentioned this pull request Oct 8, 2025

Revert stack version pinning for packages failing with 9.x #2773

Open

mrodm requested a review from jsoriano October 8, 2025 15:11

jsoriano approved these changes Oct 9, 2025

View reviewed changes

mrodm merged commit d2907e2 into elastic:main Oct 14, 2025
3 checks passed

mrodm deleted the render-transform-manifest branch October 14, 2025 09:51

Conversation

mrodm commented Oct 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Author's checklist

How to test this PR locally

Uh oh!

mrodm commented Oct 2, 2025

Uh oh!

mrodm commented Oct 2, 2025

Uh oh!

mrodm commented Oct 2, 2025

Uh oh!

mrodm commented Oct 2, 2025

Uh oh!

mrodm commented Oct 2, 2025

Uh oh!

mrodm commented Oct 3, 2025

Uh oh!

mrodm commented Oct 3, 2025

Uh oh!

jsoriano left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jsoriano Oct 7, 2025

Choose a reason for hiding this comment

Uh oh!

mrodm Oct 7, 2025

Choose a reason for hiding this comment

Uh oh!

mrodm Oct 7, 2025

Choose a reason for hiding this comment

Uh oh!

mrodm Oct 7, 2025

Choose a reason for hiding this comment

Uh oh!

mrodm Oct 8, 2025

Choose a reason for hiding this comment

Uh oh!

mrodm Oct 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jsoriano Oct 9, 2025

Choose a reason for hiding this comment

Uh oh!

mrodm Oct 14, 2025

Choose a reason for hiding this comment

Uh oh!

mrodm commented Oct 7, 2025

Uh oh!

elastic-vault-github-plugin-prod bot commented Oct 7, 2025

Uh oh!

elasticmachine commented Oct 7, 2025

💚 Build Succeeded

History

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mrodm commented Oct 2, 2025 •

edited

Loading

mrodm Oct 8, 2025 •

edited

Loading