We ran into an issue on Kubernetes with ECK and Elastic Agent. The released version of ECK has a bug in that it does not use a hostPath volume we configure for the Elastic Agent data directory. The intention behind using a hostPath volume was to create a location for the runtime state of Agent to be persistent across container restarts. Fixing this issue has surfaced a problem with the planned approach in that we cannot just configure --path.home to be the hostPath volume on Kubernetes because Elastic Agent installs the programs it wants to run into that filesystem and tries to execute them from there as well but hostPath volumes are mounted noexec on Kubernetes. Running Elastic Agent as DaemonSet, as probably many users will want to do, means there is no good way to use persistent volumes as an alternative, which might not have the noexec restriction.

The current filesystem structure for the Elastic Agent Docker container seems to be roughly:

/usr/share/elastic-agent => home --path.home
/usr/share/elastic-agent/data => data directory derived from home
/usr/share/elastic-agent/data/elastic-agent-${hash}/run => runtime state derived from home
/usr/share/elastic-agent/data/elastic-agent-${hash}/install => install location for the programs to execute agent.download.install_path
/usr/share/elastic-agent/data/elastic-agent-${hash}/download => download location for packages to install agent.download.target_directory

If we had a way to separately configure the directory for the runtime state (presumably everything below run) that would allow us to sidestep this issue. The fact that the hash of the Elastic Agent build is part of the directory structure makes it very hard for an orchestrator to mount something on top of the directory without knowing that hash upfront or having such an explicit config option.

But even with the ability to configure a separate directory for runtime state, it would still be problematic that Elastic Agent would download and execute the packages in the filesystem of the Docker container. But I don't see a good way of addressing this other than effectively reimplementing a DaemonSet in the ECK operator and using persistent volumes that are not hostPath based.