Testing to run system tests using Elastic Agent docker images based on Wolfi images #10933, raised the issue that system tests are failing for these two packages (iptables and journald).
The errors that comes from the buildkite build are that elastic-package could not find hits in the data stream:
test case failed: could not find hits in logs-iptables.log-68254 data stream
test case failed: could not find hits in logs-journald.logs-31807 data stream
Reviewing Elastic Agent logs, it looks like the agent uses journalctl:
{"log.level":"info","@timestamp":"2024-08-30T09:27:50.576Z","message":"Journalctl command: journalctl --utc --output=json --follow --file /run/service_logs/test.journal --no-tail","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"journald-default","type":"journald"},"log":{"source":"journald-default"},"log.logger":"input.journald","service.name":"filebeat","id":"journald-journald.logs-e11b66f5-06a3-421b-b4cb-2562c40b18ba","input_source":"/run/service_logs/test.journal","ecs.version":"1.6.0","path":"/run/service_logs/test.journal","log.origin":{"file.line":158,"file.name":"journalctl/reader.go","function":"github.com/elastic/beats/v7/filebeat/input/journald/pkg/journalctl.New"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-08-30T09:27:50.576Z","message":"cannot read from journalctl stderr: read |0: file already closed","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"journald-default","type":"journald"},"log":{"source":"journald-default"},"path":"/run/service_logs/test.journal","ecs.version":"1.6.0","log.logger":"input.journald","log.origin":{"file.line":193,"file.name":"journalctl/reader.go","function":"github.com/elastic/beats/v7/filebeat/input/journald/pkg/journalctl.New.func1"},"input_source":"/run/service_logs/test.journal","service.name":"filebeat","id":"journald-journald.logs-e11b66f5-06a3-421b-b4cb-2562c40b18ba","ecs.version":"1.6.0"}
but this command does not exist in docker images based on Wolfi. Example:
$ docker exec -it elastic-package-agent-journald-26598-elastic-agent-1 /bin/bash
bash-5.2# journalctl --utc --output=json --follow --file /run/service_logs/test.journal --no-tail
bash: journalctl: command not found
bash-5.2#
Should these packages be using for these tests the Elastic Agent Ubuntu based image (e.g. docker.elastic.co/elastic-agent/elastic-agent)?
cc @elastic/elastic-agent-control-plane
Testing to run system tests using Elastic Agent docker images based on Wolfi images #10933, raised the issue that system tests are failing for these two packages (iptables and journald).
The errors that comes from the buildkite build are that
elastic-packagecould not find hits in the data stream:Reviewing Elastic Agent logs, it looks like the agent uses
journalctl:{"log.level":"info","@timestamp":"2024-08-30T09:27:50.576Z","message":"Journalctl command: journalctl --utc --output=json --follow --file /run/service_logs/test.journal --no-tail","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"journald-default","type":"journald"},"log":{"source":"journald-default"},"log.logger":"input.journald","service.name":"filebeat","id":"journald-journald.logs-e11b66f5-06a3-421b-b4cb-2562c40b18ba","input_source":"/run/service_logs/test.journal","ecs.version":"1.6.0","path":"/run/service_logs/test.journal","log.origin":{"file.line":158,"file.name":"journalctl/reader.go","function":"github.com/elastic/beats/v7/filebeat/input/journald/pkg/journalctl.New"},"ecs.version":"1.6.0"} {"log.level":"error","@timestamp":"2024-08-30T09:27:50.576Z","message":"cannot read from journalctl stderr: read |0: file already closed","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"journald-default","type":"journald"},"log":{"source":"journald-default"},"path":"/run/service_logs/test.journal","ecs.version":"1.6.0","log.logger":"input.journald","log.origin":{"file.line":193,"file.name":"journalctl/reader.go","function":"github.com/elastic/beats/v7/filebeat/input/journald/pkg/journalctl.New.func1"},"input_source":"/run/service_logs/test.journal","service.name":"filebeat","id":"journald-journald.logs-e11b66f5-06a3-421b-b4cb-2562c40b18ba","ecs.version":"1.6.0"}but this command does not exist in docker images based on Wolfi. Example:
Should these packages be using for these tests the Elastic Agent Ubuntu based image (e.g. docker.elastic.co/elastic-agent/elastic-agent)?
cc @elastic/elastic-agent-control-plane