Support policy tests in input packages with input type otelcol by mrodm · Pull Request #2873 · elastic/elastic-package

mrodm · 2025-09-04T15:22:42Z

Supersedes #2799
Relates #2835
Relates elastic/kibana#227673

Add support to run policy tests in input packages defining a policy template with the input type otelcol. Continuing the work done in #2799

In order to support policy tests in those packages, it is needed to add filters to remove the ids that may appear in policies with OTel configuration. Some examples:

receivers:
    httpcheck/b0f518d6-4e2d-4c5d-bda7-f9808df537b7:
        collection_interval: 1m
        targets:
            - endpoints:
                - https://epr.elastic.co
              method: GET
secret_references: []
service:
    pipelines:
        logs:
            receivers:
                - httpcheck/b0f518d6-4e2d-4c5d-bda7-f9808df537b7

This PR adds a new test package httpcheck that includes this new input type (copied from elastic/integrations#14315).

In order to test this new input type, this PR also enables the feature flag enableOtelIntegration in Kibana configuration file. That feature flag was added in elastic/kibana#227673.

Examples:

Stack versions between 8.7.0 and 8.10.0-SNAPSHOT

xpack.fleet.enableExperimental: [
  # Enable experimental toggles in Fleet UI
  "experimentalDataStreamSettings"]

Stack versions starting in 9.2.0-SNAPSHOT:

xpack.fleet.enableExperimental: [
  # Support for otelcol input
  "enableOtelIntegrations"]

Stack versions between 8.3.0 and 8.7.0

xpack.fleet.enableExperimental: []

Older stack versions (<8.3.0) do not define this configuration key.

How to test this PR locally

elastic-package stack up -v -d --version 9.2.0-SNAPSHOT

elastic-package -C test/packages/parallel/httpcheck test policy -v

elastic-package stack down -v

This reverts commit b649672.

internal/stack/_static/kibana.yml.tmpl

mrodm · 2025-09-04T15:36:32Z

test/packages/parallel/httpcheck.stack_version

@@ -0,0 +1 @@
+9.2.0-SNAPSHOT


Forced to run this package with 9.2.0-SNAPSHOT.

This should be removed once the default version is 9.2.0 or higher.

test/packages/parallel/httpcheck/manifest.yml

mrodm · 2025-09-04T15:56:00Z

internal/testrunner/runners/policy/policy_test.go

+receivers:
+    httpcheck/b0f518d6-4e2d-4c5d-bda7-f9808df537b7:
+        collection_interval: 1m
+        targets:
+            - endpoints:
+                - https://epr.elastic.co
+              method: GET


@jsoriano in these policies, could it be more than one receiver/exporter/extensions/processors?
Focusing on receiver as example, if there is another receiver defined, would it start also with httpcheck ?

receivers: httpcheck/b0f518d6-4e2d-4c5d-bda7-f9808df537b7: collection_interval: 1m targets: - endpoints: - https://epr.elastic.co method: GET httpcheck/b0f518d6-4e2d-4c5d-bda7-11111111111: collection_interval: 2m targets: - endpoints: - https://example.com method: GET

If that could happen, the resulting yaml would contain just one receiver httpcheck/componentid

In practice I don't think this has a real use case for receivers, but in theory this will be allowed, yes. And it will be definitely possible with processors.

So you are right that we may need some replacement that is not always the same to avoid collisions, but is still predictable.

internal/testrunner/runners/policy/policy.go

test/packages/parallel/httpcheck/manifest.yml

mrodm · 2025-09-05T15:45:48Z

/test

mrodm · 2025-09-05T15:47:59Z

internal/testrunner/runners/policy/policy_test.go

+processors:
+    batch/11c35ad0-4351-49d4-9c78-fa679ce9d950:
+        send_batch_size: 10
+        timeout: 1s
+    batch/e6e379c5-6446-4090-af10-a9e5f8fc4640:
+        send_batch_size: 10000
+        timeout: 10s


With the current implementation, this could fail in some test executions:

--- FAIL: TestComparePolicies (0.00s) --- FAIL: TestComparePolicies/otel_ids (0.00s) policy_test.go:472: Error Trace: /home/mariorodriguez/Coding/work/elastic-package-otel/internal/testrunner/runners/policy/policy_test.go:472 Error: Should be empty, but was --- want +++ got @@ -17,4 +17,4 @@ batch/componentid: - send_batch_size: 10 - timeout: 1s + send_batch_size: 10000 + timeout: 10s receivers: Test: TestComparePolicies/otel_ids FAIL FAIL github.com/elastic/elastic-package/internal/testrunner/runners/policy 0.061s FAIL

Working to replace the OTEL ids (directly on the []byte variable) before running the cleanup process.

Hopefully, this is solved by replacing the OTEL IDs before cleaning up the policy with the current filters. This was added in 50de2aa

mrodm · 2025-09-08T15:22:43Z

/test

…s-otel-input

internal/testrunner/runners/policy/policy.go

jsoriano

Looks good overall, added some suggestions. In any case we will need to make changes here as the E2E implementation consolidates, so we will have opportunities to revisit this.

jsoriano · 2025-09-09T09:42:21Z

internal/testrunner/runners/policy/policy_test.go

+                - httpcheck/4bae34b3-8f66-49c1-b04f-d58af1b5f743
+            processors:
+                - batch/11c35ad0-4351-49d4-9c78-fa679ce9d950
+                - batch/e6e379c5-6446-4090-af10-a9e5f8fc4640


This is likely going to change as more plumbing is needed, see elastic/kibana#233090.

But we can go on with this change and regenerate files later.

internal/testrunner/runners/policy/policy.go

jsoriano · 2025-09-09T11:07:19Z

internal/testrunner/runners/policy/policy.go

+	policy = otelComponentIDsRegexp.ReplaceAllFunc(policy, func(match []byte) []byte {
+		count := 0
+		lines := bytes.Split(match, []byte("\n"))
+		for i, line := range lines {
+			line = bytes.TrimRight(line, "\r\n")
+			stringLine := string(line)
+			if strings.Contains(stringLine, ":") {
+				if uniqueOtelComponentIDReplace.regexp.MatchString(stringLine) {
+					replacement := fmt.Sprintf(uniqueOtelComponentIDReplace.replace, strconv.Itoa(count))
+					count++
+					lines[i] = []byte(uniqueOtelComponentIDReplace.regexp.ReplaceAllString(stringLine, replacement))
+
+					// store the otel ID replaced without the space indentation and the colon to be replaced later
+					// (e.g. http_check/4391d954-1ffe-4014-a256-5eda78a71828 replaced by http_check/componentid-0)
+					otelID := strings.SplitN(strings.TrimSpace(stringLine), ":", 2)[0]
+					replacementsDone[otelID] = strings.SplitN(strings.TrimSpace(string(lines[i])), ":", 2)[0]
+				}
+			}
+		}
+		return bytes.Join(lines, []byte("\n"))
+	})


Is it neccesary to take into account the root level? Couldn't we directly replace the matching ids?

Suggested change

policy = otelComponentIDsRegexp.ReplaceAllFunc(policy, func(match []byte) []byte {

count := 0

lines := bytes.Split(match, []byte("\n"))

for i, line := range lines {

line = bytes.TrimRight(line, "\r\n")

stringLine := string(line)

if strings.Contains(stringLine, ":") {

if uniqueOtelComponentIDReplace.regexp.MatchString(stringLine) {

replacement := fmt.Sprintf(uniqueOtelComponentIDReplace.replace, strconv.Itoa(count))

count++

lines[i] = []byte(uniqueOtelComponentIDReplace.regexp.ReplaceAllString(stringLine, replacement))

// store the otel ID replaced without the space indentation and the colon to be replaced later

// (e.g. http_check/4391d954-1ffe-4014-a256-5eda78a71828 replaced by http_check/componentid-0)

otelID := strings.SplitN(strings.TrimSpace(stringLine), ":", 2)[0]

replacementsDone[otelID] = strings.SplitN(strings.TrimSpace(string(lines[i])), ":", 2)[0]

}

}

}

return bytes.Join(lines, []byte("\n"))

})

policy = uniqueOtelComponentIDReplace.regexp.ReplaceAllStringFunc(policy, func(match string) string {

replacement := fmt.Sprintf(uniqueOtelComponentIDReplace.replace, strconv.Itoa(count))

replaced := uniqueOtelComponentIDReplace.regexp.ReplaceAllString(match, replacement)

otelID, _, _ := strings.Cut(strings.TrimSpace(match), ":")

replacementsDone[otelID], _, _ = strings.Cut(strings.TrimSpace(string(replaced)), ":")

})

Maybe it could be done like that, but it could not be ensured that all the replacements performed would be related to OTEL.

The current implementation, at least, looks for OTEL IDs just in the block definitions detailed by the regexp otelComponentIDsRegexp:

var otelComponentIDsRegexp = regexp.MustCompile(`(?m)^(?:extensions|receivers|processors|connectors|exporters):(?:\s\{\}\n|\n(?:\s{2,}.+\n)+)`)

I'd prefer to keep it for now like this to avoid replacing unexpected strings in other settings/fields while validating these policies.

internal/testrunner/runners/policy/policy.go

Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co>

mrodm · 2025-09-09T12:36:05Z

test integrations

elastic-vault-github-plugin-prod · 2025-09-09T12:45:17Z

Created or updated PR in integrations repository to test this version. Check elastic/integrations#15254

jsoriano

👍

test/packages/parallel/httpcheck/_dev/test/system/test-local-config.yml

Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co>

elasticmachine · 2025-09-09T14:54:59Z

💚 Build Succeeded

Buildkite Build
Commit: 7a7dc37

History

💚 Build #6058 succeeded 0c5483c
💚 Build #6052 succeeded ea59a6b
💚 Build #6051 succeeded 6c0ccf4
💚 Build #6048 succeeded 075be2a
💔 Build #6047 failed 2e85b42
💔 Build #6040 failed a5e135a

cc @mrodm

jsoriano and others added 10 commits September 4, 2025 17:20

Support OTel config in policy tests

acedfd0

Add test package

2c5c553

Add kibana feature flag

6f8de38

Remove debug level for fleet plugin

b19a6d8

Fix error in kibana config server.ssl for stack versions < 8.15.0

304ef01

Move comments

6b25a60

Add namespaces field to test package

b649672

Set specific version

0ad6db6

Do not replace values in memberReplace case

37b53c9

Revert "Add namespaces field to test package"

5fe45aa

This reverts commit b649672.

mrodm self-assigned this Sep 4, 2025

mrodm commented Sep 4, 2025

View reviewed changes

internal/stack/_static/kibana.yml.tmpl Outdated Show resolved Hide resolved

mrodm commented Sep 4, 2025

View reviewed changes

internal/stack/_static/kibana.yml.tmpl Show resolved Hide resolved

mrodm mentioned this pull request Sep 4, 2025

Ensure input packages with otelcol input type (e.g. "OTel integrations") can be built, tested, and published with elastic-package #2835

Closed

jsoriano mentioned this pull request Sep 4, 2025

Support OTel config in policy tests #2799

Closed

mrodm commented Sep 4, 2025

View reviewed changes

test/packages/parallel/httpcheck/manifest.yml Show resolved Hide resolved

mrodm commented Sep 4, 2025

View reviewed changes

internal/testrunner/runners/policy/policy.go Outdated Show resolved Hide resolved

Skip system test for httpcheck test package

94ed714

jsoriano reviewed Sep 4, 2025

View reviewed changes

test/packages/parallel/httpcheck/manifest.yml Outdated Show resolved Hide resolved

mrodm added 2 commits September 4, 2025 18:48

Update spec version for httpcheck

ad580ae

Do not replace strings in memberReplace

b5ac0b5

mrodm mentioned this pull request Sep 5, 2025

Update kibana template used in the stack #2875

Merged

Update tests including more than one OTEL processor

a5e135a

mrodm commented Sep 5, 2025

View reviewed changes

mrodm added 3 commits September 8, 2025 15:50

Update otel IDs in test

2e85b42

Replace otel IDs definitions and usages using regexes

50de2aa

Allow to be otel IDs with empty maps

075be2a

mrodm added 2 commits September 8, 2025 18:01

Add comments and remove unused code

68f0e09

Merge remote-tracking branch 'upstream/main' into support-policy-test…

6c0ccf4

…s-otel-input

mrodm commented Sep 8, 2025

View reviewed changes

internal/testrunner/runners/policy/policy.go Show resolved Hide resolved

Update function documentation

ea59a6b

mrodm marked this pull request as ready for review September 8, 2025 17:48

mrodm requested a review from jsoriano September 8, 2025 17:48

jsoriano reviewed Sep 9, 2025

View reviewed changes

mrodm and others added 4 commits September 9, 2025 13:37

Add connectors to OTEL components regex

3750339

Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co>

Remove unnecessary if

c68ae5c

Use strings.Cut instead of strings.SplitN

640a700

Use scanner to read matching lines of the given policy

0c5483c

elastic-vault-github-plugin-prod bot mentioned this pull request Sep 9, 2025

Test elastic-package#2873 - DO NOT MERGE elastic/integrations#15254

Closed

jsoriano approved these changes Sep 9, 2025

View reviewed changes

test/packages/parallel/httpcheck/_dev/test/system/test-local-config.yml Show resolved Hide resolved

Add variable to system test - httpcheck package

7a7dc37

Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co>

mrodm merged commit ffa80e9 into elastic:main Sep 9, 2025
3 checks passed

mrodm deleted the support-policy-tests-otel-input branch September 9, 2025 14:57

mrodm mentioned this pull request Sep 10, 2025

[policy tests] Replace also OTEL ids under service.pipelines #2886

Merged

mrodm mentioned this pull request Feb 26, 2026

Remove pinning stack version for httpcheck #3313

Merged

Conversation

mrodm commented Sep 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

How to test this PR locally

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

mrodm commented Sep 5, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mrodm commented Sep 8, 2025

Uh oh!

Uh oh!

jsoriano left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

mrodm commented Sep 9, 2025

Uh oh!

elastic-vault-github-plugin-prod bot commented Sep 9, 2025

Uh oh!

jsoriano left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

elasticmachine commented Sep 9, 2025

💚 Build Succeeded

History

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mrodm commented Sep 4, 2025 •

edited

Loading