Testing to run system tests using Elastic Agent docker images based on Wolfi images #10933, raised the issue that system tests are failing for these packages (apache_tomcat, microsoft_sqlserver and stan).
The errors that comes from the buildkite build are that elastic-package could not find hits in the data stream:
test case failed: could not find hits in logs-apache_tomcat.catalina-84034 data stream
test case failed: could not find hits in logs-apache_tomcat.access-60721 data stream
test case failed: could not find hits in logs-microsoft_sqlserver.log-20239 data stream
test case failed: could not find hits in logs-stan.log-82655 data stream
For these packages, documents were not ingested due to permission errors while reading those files by Elastic Agent.
{"log.level":"error","@timestamp":"2024-08-29T16:48:47.669Z","message":"File could not be opened for reading: failed opening /tmp/service_logs/catalina.2024-08-29.log: open /tmp/service_logs/catalina.2024-08-29.log: permission denied","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"source_file":"filestream::filestream-apache_tomcat.catalina-e5df2680-3558-4118-a15a-804e746d2ae9::native::9454105-64515","path":"/tmp/service_logs/catalina.2024-08-29.log","ecs.version":"1.6.0","log.logger":"input.filestream","log.origin":{"file.line":146,"file.name":"filestream/input.go","function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).Run"},"service.name":"filebeat","id":"filestream-apache_tomcat.catalina-e5df2680-3558-4118-a15a-804e746d2ae9","state-id":"native::9454105-64515","ecs.version":"1.6.0"}{"log.level":"error","@timestamp":"2024-09-04T10:59:01.500Z","message":"Harvester could not be started on new file: /tmp/service_logs/errorlog, Err: error setting up harvester: Harvester setup failed. Unexpected file opening error: Failed opening /tmp/service_logs/errorlog: open /tmp/service_logs/errorlog: permission denied","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"log-default","type":"log"},"log":{"source":"log-default"},"finished":false,"log.logger":"input","source_file":"/tmp/service_logs/errorlog","state_id":"native::9844199-64515","ecs.version":"1.6.0","log.origin":{"file.line":570,"file.name":"log/input.go","function":"github.com/elastic/beats/v7/filebeat/input/log.(*Input).scan"},"service.name":"filebeat","input_id":"940fda73-b249-4ae7-b07d-5757cb30ecdb","os_id":"9844199-64515","ecs.version":"1.6.0"}{"log.level":"warn","@timestamp":"2024-08-29T17:01:36.831Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":663},"message":"Unit state changed log-default-logfile-stan-2dd61177-25e0-47e0-9d06-9aa86ef6b41e (HEALTHY->DEGRADED): Harvester could not be started on new file: /tmp/service_logs/stan.log, Err: error setting up harvester: Harvester setup failed. Unexpected file opening error: Failed opening /tmp/service_logs/stan.log: open /tmp/service_logs/stan.log: permission denied","log":{"source":"elastic-agent"},"component":{"id":"log-default","state":"HEALTHY"},"unit":{"id":"log-default-logfile-stan-2dd61177-25e0-47e0-9d06-9aa86ef6b41e","type":"input","state":"DEGRADED","old_state":"HEALTHY"},"ecs.version":"1.6.0"}Permission of the files for easch package:
bash-5.2$ ls -l /tmp/service_logs/
total 16
-rw-r----- 1 root root 5855 Aug 29 16:48 catalina.2024-08-29.log
-rw-r--r-- 1 root root 319 Aug 29 16:48 localhost.log
-rw-r----- 1 root root 980 Aug 29 16:55 localhost_access_log.2024-08-29.txt
$ docker exec -it elastic-package-agent-microsoft_sqlserver-log-37113-elastic-agent-1 /bin/bash
bash-5.2$ ls -l /tmp/service_logs/ |grep errorlog
-rw-r----- 1 root root 10754 Sep 4 10:54 errorlog
-rw-r----- 1 root root 0 Sep 4 10:54 errorlog.1
$ docker exec -it elastic-package-agent-stan-log-64803-elastic-agent-1 /bin/bash
bash-5.2$ ls -l /tmp/service_logs/
total 1940
-rw-r----- 1 root root 1985018 Aug 29 17:02 stan.log
It is needed to check if the permissions of these files could be changed somehow in tests so Elastic Agent user could read those.
As a workarounds, there could be some different approaches:
- Force to use Elastic Agent Ubuntu image by updating the configuration file of the given system tests (
agent.base_image: systemd).
- Run system tests with user root by updating the configuration file of the given system tests (
agent.user: root).
This elastic-package issue elastic/elastic-package#1852 could be related to avoid permission errors related to the parent folder, but probably the packages would have to change the user that writes the logs.
Testing to run system tests using Elastic Agent docker images based on Wolfi images #10933, raised the issue that system tests are failing for these packages (apache_tomcat, microsoft_sqlserver and stan).
The errors that comes from the buildkite build are that
elastic-packagecould not find hits in the data stream:For these packages, documents were not ingested due to permission errors while reading those files by Elastic Agent.
{"log.level":"error","@timestamp":"2024-08-29T16:48:47.669Z","message":"File could not be opened for reading: failed opening /tmp/service_logs/catalina.2024-08-29.log: open /tmp/service_logs/catalina.2024-08-29.log: permission denied","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"source_file":"filestream::filestream-apache_tomcat.catalina-e5df2680-3558-4118-a15a-804e746d2ae9::native::9454105-64515","path":"/tmp/service_logs/catalina.2024-08-29.log","ecs.version":"1.6.0","log.logger":"input.filestream","log.origin":{"file.line":146,"file.name":"filestream/input.go","function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).Run"},"service.name":"filebeat","id":"filestream-apache_tomcat.catalina-e5df2680-3558-4118-a15a-804e746d2ae9","state-id":"native::9454105-64515","ecs.version":"1.6.0"}{"log.level":"error","@timestamp":"2024-09-04T10:59:01.500Z","message":"Harvester could not be started on new file: /tmp/service_logs/errorlog, Err: error setting up harvester: Harvester setup failed. Unexpected file opening error: Failed opening /tmp/service_logs/errorlog: open /tmp/service_logs/errorlog: permission denied","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"log-default","type":"log"},"log":{"source":"log-default"},"finished":false,"log.logger":"input","source_file":"/tmp/service_logs/errorlog","state_id":"native::9844199-64515","ecs.version":"1.6.0","log.origin":{"file.line":570,"file.name":"log/input.go","function":"github.com/elastic/beats/v7/filebeat/input/log.(*Input).scan"},"service.name":"filebeat","input_id":"940fda73-b249-4ae7-b07d-5757cb30ecdb","os_id":"9844199-64515","ecs.version":"1.6.0"}{"log.level":"warn","@timestamp":"2024-08-29T17:01:36.831Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":663},"message":"Unit state changed log-default-logfile-stan-2dd61177-25e0-47e0-9d06-9aa86ef6b41e (HEALTHY->DEGRADED): Harvester could not be started on new file: /tmp/service_logs/stan.log, Err: error setting up harvester: Harvester setup failed. Unexpected file opening error: Failed opening /tmp/service_logs/stan.log: open /tmp/service_logs/stan.log: permission denied","log":{"source":"elastic-agent"},"component":{"id":"log-default","state":"HEALTHY"},"unit":{"id":"log-default-logfile-stan-2dd61177-25e0-47e0-9d06-9aa86ef6b41e","type":"input","state":"DEGRADED","old_state":"HEALTHY"},"ecs.version":"1.6.0"}Permission of the files for easch package:
It is needed to check if the permissions of these files could be changed somehow in tests so Elastic Agent user could read those.
As a workarounds, there could be some different approaches:
agent.base_image: systemd).agent.user: root).This elastic-package issue elastic/elastic-package#1852 could be related to avoid permission errors related to the parent folder, but probably the packages would have to change the user that writes the logs.