internal/testrunner/script: add script testing package by efd6 · Pull Request #3012 · elastic/elastic-package

efd6 · 2025-10-21T02:14:46Z

This adds script testing for data streams. This is the MVP, future versions can be generalised to operate on input packages.

The current implementation supports:

pipeline testing
system testing
package upgrade testing (only to latest, not to arbitrary versions)
shared stack
independent stack
docker services

Not supported:

test coverage
report output configuration
k8s services
tf services (ish, these can be shimmed via docker)

Depends on a v3.5.1 of package-spec (currently shimmed in via a go.mod replace directive, to be removed)

Please take a look.

For #2944

efd6 · 2025-10-21T05:26:28Z

/test

.go-version

Makefile

efd6 · 2025-10-21T21:12:29Z

cmd/testrunner.go

+	return cmd
+}
+
+func testRunnerScriptCommandAction(cmd *cobra.Command, args []string) error {


The logic in this function is rudimentary and only intended to allow an MVP to be demonstrated. In order to be able to render reports and redirect to file output this needs enhancement.

go.mod

jsoriano

Thanks! This is a pretty interesting feature.

I haven't checked the implementation in detail, added by now some comments and questions about the overall behavior.

docs/howto/script_testing.md

jsoriano · 2025-11-04T19:36:55Z

internal/testrunner/script/script.go

+	if err != nil {
+		return err
+	}
+	workRoot := filepath.Join(home, filepath.FromSlash(".elastic-package/tmp/script_tests"))


Config directory should be obtained using the location manager. It can be customized with ELASTIC_PACKAGE_DATA_HOME.

The tmp directory can be obtained with loc.TempDir().

Suggested change

workRoot := filepath.Join(home, filepath.FromSlash(".elastic-package/tmp/script_tests"))

workRoot := filepath.Join(home, loc.TempDir(), "script_tests")

jsoriano · 2025-11-04T19:40:10Z

internal/testrunner/script/script.go

+		// root is non-zero, so just let testscript put it where it wants in the
+		// case that we have not requested work to be retained. This will be in
+		// os.MkdirTemp(os.Getenv("GOTMPDIR"), "go-test-script") which on most
+		// systems will be /tmp/go-test-script. However, due to… decisions, we


What decisions are you referring to? 🙂

The use of file system jailing prevents using other paths.

Maybe this should be part of the coment 🙂

test/packages/other/with_script/data_stream/first/_dev/test/scripts/agent_up_down.txt

docs/howto/script_testing.md

jsoriano · 2025-11-04T19:51:59Z

docs/howto/script_testing.md

+# Only run the test if --external-stack=true.
+[!external_stack] skip 'Skipping external stack test.'
+# Only run the test if the jq executable is in $PATH. This is needed for a test below.
+[!exec:jq] skip 'Skipping test requiring absent jq command'


It looks like several sample tests here depend on jq to compare fields in JSON files. If this is going to be frequent, maybe we can provide a function to do these comparisons? And maybe the same with basic http requests to avoid depending on curl.

This would help to avoid depending on external tools that may not be installed, specially on Windows.

If it becomes necessary, those can be added. Both curl and jq are available in CI and I'd prefer to avoid having to reimplement and maintain tools that already exist.

Agree this can be added later.

I am not so worried about the current CI, but about future deployments where we might forget to install jq (or curl in Windows) if not required for other things, and the test would be silently skipped. Same thing for local execution.

jq and curl are powerful tools, in no case I would think on re-implement them, but the limited logic that we need, also reusing existing tools, in the form of Go code.

mrodm

Added a comment about CI scripts.

The error raised in the latest Buildkite build should be fixed now, since it has been merged the new spec version #3049

mrodm · 2025-11-05T13:24:30Z

Makefile

 	PACKAGE_TEST_TYPE=other ./scripts/test-check-packages.sh

+test-check-packages-independent-script:
+	elastic-package test script -C test/packages/other/with_script --external-stack=false --defer-cleanup 1s


I would create a new folder for this test package like test/packages/independent-script/ and just keep (move) that new test package into this new folder.

The reasoning of this is that currently, this with_script test package is being tested twice in Buildkite builds. In these two steps:

Integration test: other

Integration test: independent-script

So, I would create a new folder and update this target:

Suggested change

elastic-package test script -C test/packages/other/with_script --external-stack=false --defer-cleanup 1s

elastic-package test script -C test/packages/independent-script/with_script --external-stack=false --defer-cleanup 1s

I would create a new folder for this test package like test/packages/independent-script/ and just keep (move) that new test package into this new folder.

The twice use of this is intentional.

The twice use of this is intentional.

Why is it needed? Within the other step, what is it different from the other CI step (independent-script)?
Maybe it is tested with the stack triggered by the CI itself ? I mean not using this external-stack=false ?

When external-stack=false, the script brings up a stack. This would fail if a stack is already running due to port collision. The external and non-external stack tests need to be in the same package to ensure that the conditional is correctly implemented.

docs/howto/script_testing.md

efd6 · 2025-11-06T21:37:12Z

/test

andrewkroh

LGTM. I left a few minor comments and have two questions:

When we run dump_logs, it captures logs like $WORK/logs/elastic-agent-internal/elastic-agent-YYYYMMDD.ndjson and $WORK/logs/elastic-agent-internal/cel/cel-UUID-URL.ndjson. Will these less predictable file names make it difficult to write assertions on their file contents from testscript?
Can we add a mechanism to capture the registry files from the Elastic Agent container? I would like to be able to run assertions on their contents, like checking that the httpjson cursor contains a specific timestamp or pagination token after test completion.

cmd/testrunner.go

internal/testrunner/script/script.go

efd6 · 2025-11-17T20:32:19Z

When we run dump_logs, it captures logs like $WORK/logs/elastic-agent-internal/elastic-agent-YYYYMMDD.ndjson and $WORK/logs/elastic-agent-internal/cel/cel-UUID-URL.ndjson. Will these less predictable file names make it difficult to write assertions on their file contents from testscript?

It will to a degree; from what I can see elastic-package's testing was not written with a concern for bit-reproducible testing. The filenames there are a consequence of the way that logs are written by the underlying routines in elastic-agent though AFAICS. At the moment, the way to handle these would be to have a helper shell script or maybe jq over the glob. If this is seen as too complex, we can maybe harmonise all the files in those directories into single file collections that have been temporally ordered and written to a file with a known name.

The current situation does not make it impossible, but depending on the threshold for "difficult", yes, no or maybe.

Can we add a mechanism to capture the registry files from the Elastic Agent container? I would like to be able to run assertions on their contents, like checking that the httpjson cursor contains a specific timestamp or pagination token after test completion.

We can; in fact that is one of my unchecked TODO boxes. Would you like that added in this change, or can it be added in a follow-up?

efd6 · 2025-11-17T21:16:01Z

If this is seen as too complex, we can maybe harmonise all the files in those directories into single file collections that have been temporally ordered and written to a file with a known name.

I've done this.

andrewkroh · 2025-11-17T21:19:42Z

In a follow-up would be best.

mrodm

LGTM
Just added a few comments.

It would be interesting to wait also for @jsoriano's review before merging, since he had some comments here too.

mrodm · 2025-11-19T11:06:59Z

docs/howto/script_testing.md

+While the testscript package allows reference to paths outside the configuration
+root and the package's root, the backing `elastic-package` infrastructure does
+not, so it is advised that tests only refer to paths within the `$WORK` and
+`$PKG_ROOT` directories.


Looking into the environment variables listed above, should this be PACKAGE_ROOT?

Suggested change

`$PKG_ROOT` directories.

`$PACKAGE_ROOT` directories.

mrodm · 2025-11-19T11:07:49Z

docs/howto/script_testing.md

+
+While the testscript package allows reference to paths outside the configuration
+root and the package's root, the backing `elastic-package` infrastructure does
+not, so it is advised that tests only refer to paths within the `$WORK` and


Should WORK be added as part of the environment listed here?

https://github.com/elastic/elastic-package/pull/3012/files#diff-827ffc58440ddcc24a65a13e49b1ec463282c31886d69cfa50f775e549e183cdR75-R87

jsoriano

LGTM, thanks!

elasticmachine · 2025-11-19T21:19:22Z

💚 Build Succeeded

Buildkite Build
Commit: 78bb1cd

History

💔 Build #6640 failed cfe3ac1
💚 Build #6625 succeeded fcd4486
💚 Build #6538 succeeded 054ae48
💔 Build #6537 failed 054ae48
💚 Build #6527 succeeded d5eced9

cc @efd6

efd6 self-assigned this Oct 21, 2025

efd6 added the enhancement New feature or request label Oct 21, 2025

efd6 force-pushed the script_tests branch 5 times, most recently from 411a39e to bc55805 Compare October 21, 2025 04:31

efd6 force-pushed the script_tests branch 2 times, most recently from d9e0c1a to 8433c67 Compare October 21, 2025 20:16

efd6 marked this pull request as ready for review October 21, 2025 21:09

efd6 requested a review from a team as a code owner October 21, 2025 21:09

efd6 commented Oct 21, 2025

View reviewed changes

efd6 requested a review from andrewkroh October 21, 2025 21:14

jsoriano self-requested a review October 23, 2025 08:34

efd6 force-pushed the script_tests branch 5 times, most recently from 7f4815f to fdc0951 Compare October 28, 2025 06:46

efd6 force-pushed the script_tests branch from 0e84c58 to 0f9015a Compare November 4, 2025 01:01

jsoriano reviewed Nov 4, 2025

View reviewed changes

efd6 force-pushed the script_tests branch from c8ce65a to c2e7dad Compare November 5, 2025 03:57

mrodm reviewed Nov 5, 2025

View reviewed changes

efd6 force-pushed the script_tests branch 5 times, most recently from d5eced9 to 054ae48 Compare November 6, 2025 21:05

belimawr mentioned this pull request Nov 11, 2025

[Filebeat] Filestream running as Log input under Elastic Agent or feature flag elastic/beats#46587

Merged

15 tasks

andrewkroh reviewed Nov 17, 2025

View reviewed changes

cmd/testrunner.go Outdated Show resolved Hide resolved

internal/testrunner/script/script.go Outdated Show resolved Hide resolved

andrewkroh approved these changes Nov 17, 2025

View reviewed changes

mrodm approved these changes Nov 19, 2025

View reviewed changes

jsoriano mentioned this pull request Nov 19, 2025

Use named loggers and add a selector flag #3058

Open

jsoriano approved these changes Nov 19, 2025

View reviewed changes

efd6 added 15 commits November 20, 2025 07:01

internal/testrunner/script: add script testing package

0ff1cae

repair

e031044

escape

f0d966f

more repair

4495394

run independent stack script test in ci

fca8b2e

bump github.com/elastic/package-spec to v3.5.1

02a0ad1

address pr comments: documentation

1815b36

address pr comment: use loc.TempDir

6197b9d

address pr comment: expose PROFILE env var

98fe2e2

names

ac60606

more names

76807cf

address pr comments

9c1e18b

collect internal logs into canonical locations

c09e65c

address pr comments

0a203e2

repair

78bb1cd

efd6 force-pushed the script_tests branch from cfe3ac1 to 78bb1cd Compare November 19, 2025 20:44

efd6 merged commit bd294f5 into elastic:main Nov 19, 2025
3 checks passed

mrodm mentioned this pull request Nov 21, 2025

Test upgrade from previous version to latest version in CI #1831

Open

	workRoot := filepath.Join(home, filepath.FromSlash(".elastic-package/tmp/script_tests"))
	workRoot := filepath.Join(home, loc.TempDir(), "script_tests")

	elastic-package test script -C test/packages/other/with_script --external-stack=false --defer-cleanup 1s
	elastic-package test script -C test/packages/independent-script/with_script --external-stack=false --defer-cleanup 1s

Conversation

efd6 commented Oct 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

efd6 commented Oct 21, 2025

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jsoriano left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mrodm left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

efd6 commented Nov 6, 2025

Uh oh!

andrewkroh left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

efd6 commented Nov 17, 2025

Uh oh!

efd6 commented Nov 17, 2025

Uh oh!

andrewkroh commented Nov 17, 2025

Uh oh!

mrodm left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jsoriano left a comment

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Nov 19, 2025

💚 Build Succeeded

History

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

efd6 commented Oct 21, 2025 •

edited

Loading