Package policy dataset mismatch for otelcol input packages

## Summary

When elastic-package creates Fleet package policies for **input packages** whose policy template uses `input: otelcol`, the dataset value it sends does not match the value Fleet uses when the same integration is added via the Fleet UI. As a result, policy tests that create a policy via elastic-package and compare it to the downloaded policy fail, because the downloaded policy reflects Fleet’s compilation (e.g. `statsdreceiver`) while the test was written to expect that value, and elastic-package is currently sending a different value (e.g. `statsd_input_otel.statsdreceiver`).

## Background

- **OTel input packages** are packages with `type: input` and policy templates that declare a single `otelcol` `input`.
- For such packages, the **dataset** is sent to Fleet as part of the package policy (in the data stream and in vars such as `data_stream.dataset`). Fleet compiles and stores the policy; the **download agent policy API** returns the compiled policy.
- **Policy tests** in elastic-package create a Fleet policy (via the same APIs), download it, and compare the result to an expected file. The expected file is often generated or updated to match what the Fleet download API returns when the integration is added through the Fleet UI.

## Current behavior (wrong for otelcol)

- For every input package, elastic-package builds the dataset as **package name + policy template name** (e.g. `statsd_input_otel.statsdreceiver`), sends it in the package policy request, and Fleet stores/uses that value.
- For **otelcol** input packages, when the integration is added via the **Fleet UI**, Fleet compiles the dataset as **policy template name only** (e.g. `statsdreceiver`). The download API then returns that compiled value.

So for otelcol:

- **Fleet UI / download API:** dataset = `statsdreceiver` (policy template name only).
- **elastic-package–created policy:** dataset = `statsd_input_otel.statsdreceiver` (package name + policy template name).

The policy created by the policy test therefore differs from the policy created in the UI: same package, different dataset in the compiled agent policy. Once the expected file is updated to match Fleet (e.g. `statsdreceiver`), the test fails until elastic-package stops sending `statsd_input_otel.statsdreceiver` for that package.

Relevant code: https://github.com/elastic/elastic-package/blob/b7fb0463c604fa46e1741bc4c5d6e93731930597/internal/resources/fleetpolicy.go#L245-L310

StatsD Input Package sample event: https://github.com/elastic/integrations/blob/592c26bfe301286231d416606ca20d3f11de44c9/packages/statsd_input_otel/sample_event.json#L5-L9

Mismatched dataset value in expected policy: https://github.com/elastic/integrations/blob/592c26bfe301286231d416606ca20d3f11de44c9/packages/statsd_input_otel/_dev/test/policy/test-default.expected#L27-L32

## Expected behavior

- For input packages whose policy template uses **`input: otelcol`**: the dataset sent by elastic-package (and thus the compiled policy) should match Fleet’s behavior when the integration is added via the UI—i.e. dataset should be **policy template name only** (e.g. `statsdreceiver`), so that the download API and the expected file align and policy tests pass.
- For **other** input packages (non-otelcol): existing behavior may still be correct (dataset = package name + policy template name); that should be confirmed or left as-is depending on product expectations.

## Impact

- **Policy tests** for otelcol input packages (e.g. `statsd_input_otel`) fail when the expected policy file is set to Fleet’s compiled value (policy template name only), because elastic-package sends a different dataset.
- Consistency between “add via UI” and “add via elastic-package / policy test” is broken for otelcol packages.

## Additional context

Here's an example of Fleet's compiled policy for the statsd input package:

<img width="767" height="229" alt="Image" src="https://github.com/user-attachments/assets/201158a9-6934-45ad-9b4a-6e498df686c7" />


	func createInputPackagePolicy(policy FleetAgentPolicy, manifest packages.PackageManifest, packagePolicy FleetPackagePolicy) (*kibana.PackageDataStream, error) {
	if dsName := packagePolicy.DataStreamName; dsName != "" {
	return nil, fmt.Errorf("no data stream expected for input package policy %q, found %q", packagePolicy.Name, dsName)
	}

	policyTemplateName := packagePolicy.TemplateName
	if policyTemplateName == "" {
	name, err := findPolicyTemplateForInputPackage(manifest, packagePolicy.InputName)
	if err != nil {
	return nil, fmt.Errorf("failed to determine the associated policy_template: %w", err)
	}
	policyTemplateName = name
	}

	policyTemplate, err := selectPolicyTemplateByName(manifest.PolicyTemplates, policyTemplateName)
	if err != nil {
	return nil, fmt.Errorf("failed to find the selected policy_template: %w", err)
	}

	ds := kibana.PackageDataStream{
	Name: packagePolicy.Name,
	Namespace: policy.Namespace,
	PolicyID: policy.ID,
	Enabled: !packagePolicy.Disabled,
	Inputs: []kibana.Input{
	{
	PolicyTemplate: policyTemplate.Name,
	Enabled: !packagePolicy.Disabled,
	},
	},
	}
	ds.Package.Name = manifest.Name
	ds.Package.Title = manifest.Title
	ds.Package.Version = manifest.Version

	streamInput := policyTemplate.Input
	ds.Inputs[0].Type = streamInput

	dataset := fmt.Sprintf("%s.%s", manifest.Name, policyTemplate.Name)
	streams := []kibana.Stream{
	{
	ID: fmt.Sprintf("%s-%s.%s", streamInput, manifest.Name, policyTemplate.Name),
	Enabled: !packagePolicy.Disabled,
	DataStream: kibana.DataStream{
	Type: policyTemplate.Type,
	Dataset: dataset,
	},
	},
	}

	// Add policyTemplate-level vars.
	vars := setKibanaVariables(policyTemplate.Vars, common.MapStr(packagePolicy.Vars))
	if _, found := vars["data_stream.dataset"]; !found {
	var value packages.VarValue
	value.Unpack(dataset)
	vars["data_stream.dataset"] = kibana.Var{
	Value: value,
	Type: "text",
	}
	}

	streams[0].Vars = vars
	ds.Inputs[0].Streams = streams

	return &ds, nil
	}

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Package policy dataset mismatch for otelcol input packages #3275

Summary

Background

Current behavior (wrong for otelcol)

Expected behavior

Impact

Additional context

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Package policy dataset mismatch for otelcol input packages #3275

Description

Summary

Background

Current behavior (wrong for otelcol)

Expected behavior

Impact

Additional context

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions