Install systemd on Filebeat Docker images by belimawr · Pull Request #44056 · elastic/beats

belimawr · 2025-04-24T16:41:38Z

Proposed commit message

The journald input from Filebeat requires the journalctl binary to ingest journal logs, this commit adds it by installing systemd in all Filebeat Docker container images

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
~~I have made corresponding change to the default configuration files~~
~~I have added tests that prove my fix is effective or that my feature works~~
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

~~## Disruptive User Impact~~
~~## Author's Checklist~~

How to test this PR locally

Package Filebeat

DEV=true SNAPSHOT=true PACKAGES="docker" PLATFORMS=linux/amd64 mage -v package

Ensure journalctl is present in the containers (adjust the image name/tag according to the version you built, oss or not)

docker run --rm -it --user root --entrypoint "journalctl" docker.elastic.co/beats/filebeat-wolfi:9.1.0-SNAPSHOT --version
docker run --rm -it --user root --entrypoint "journalctl" docker.elastic.co/beats/filebeat-ubi:9.1.0-SNAPSHOT --version
# Only for x-pack
docker run --rm -it --user root --entrypoint "journalctl" docker.elastic.co/beats/filebeat:9.1.0-SNAPSHOT --version

Start a stack. E.g: using elastic-package: elastic-package stack up --version=9.1.0-SNAPSHOT -v -d

Create the following filebeat.yml (adjust the IP address/ES host as needed)

filebeat.yml

filebeat.inputs:
  - type: journald
    id: foo-bar
    paths:
      - /var/log/journal/*/*.journal

output.elasticsearch:
  hosts:
    - https://192.168.42.42:9200
  username: elastic
  password: changeme
  ssl.verification_mode: none

Then run Filebeat from the docker image (adjust the journald folder if needed)

docker run --rm \
  --name=filebeat \
  --user=root \
  --volume="/var/log/journal:/var/log/journal:ro" \
  --volume="$(pwd)/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro" \
  --volume="registry:/usr/share/filebeat/data:rw" \
  docker.elastic.co/beats/filebeat-wolfi:9.1.0-SNAPSHOT filebeat -e --strict.perms=false setup

Ensure there are no errors and data is ingested

Related issues

~~## Use cases~~
~~## Screenshots~~
~~## Logs~~

The journald input from Filebeat requires the `journalctl` binary to ingest journal logs, this commit adds it by installing systemd in all Filebeat Docker container images

mergify · 2025-04-24T16:42:21Z

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @belimawr? 🙏.
For such, you'll need to label your PR with:

The upcoming major version of the Elastic Stack
The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit
backport-active-all is the label that automatically backports to all active branches.
backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

elasticmachine · 2025-04-24T18:49:37Z

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

leehinman

FYI opened #44062 for us to improve updating of packages.

docs/reference/filebeat/filebeat-input-journald.md

kilfoyle

LGTM for the docs! 🏎️
(I added just a small suggestion.)

rdner · 2025-04-25T06:02:26Z

docs/reference/filebeat/filebeat-input-journald.md

+```sh
+docker run --rm -it --entrypoint "journalctl" docker.elastic.co/beats/filebeat-wolfi:<VERSION> --version
+```


This is very helpful, thanks!

docs/release-notes/index.md

rdner · 2025-04-25T06:05:38Z

@belimawr I see this PR is marked to close #44040 I don't think it's correct.

This PR only addresses the standalone Filebeat container, not running integrations under Elastic Agent, right?

belimawr · 2025-04-25T10:15:00Z

@belimawr I see this PR is marked to close #44040 I don't think it's correct.

This PR only addresses the standalone Filebeat container, not running integrations under Elastic Agent, right?

Yes, you're correct. Thanks for catching this. I updated the PR description.

belimawr · 2025-04-25T10:17:10Z

FYI opened #44062 for us to improve updating of packages.

Thanks Lee!

…lctl-to-containers

github-actions · 2025-04-25T15:12:42Z

@Mergifyio backport 8.17 8.18 8.19 9.0

mergify · 2025-04-25T15:12:50Z

backport 8.17 8.18 8.19 9.0

✅ Backports have been created

Details

#44085 [8.17](backport #44056) Install systemd on Filebeat Docker images has been created for branch 8.17 but encountered conflicts
#44086 [8.18](backport #44056) Install systemd on Filebeat Docker images has been created for branch 8.18 but encountered conflicts
#44087 [8.19](backport #44056) Install systemd on Filebeat Docker images has been created for branch 8.19 but encountered conflicts
#44088 [9.0](backport #44056) Install systemd on Filebeat Docker images has been created for branch 9.0

The journald input from Filebeat requires the `journalctl` binary to ingest journal logs, this commit adds it by installing systemd in all Filebeat Docker container images (cherry picked from commit 32ecb7c) # Conflicts: # docs/reference/filebeat/filebeat-input-journald.md # docs/release-notes/index.md

The journald input from Filebeat requires the `journalctl` binary to ingest journal logs, this commit adds it by installing systemd in all Filebeat Docker container images (cherry picked from commit 32ecb7c)

…4085) The journald input from Filebeat requires the `journalctl` binary to ingest journal logs, this commit adds it by installing systemd in all Filebeat Docker container images (cherry picked from commit 32ecb7c) # Conflicts: # docs/reference/filebeat/filebeat-input-journald.md # docs/release-notes/index.md * Convert the docs and changelog to markdown --------- Co-authored-by: Tiago Queiroz <tiago.queiroz@elastic.co>

…4086) The journald input from Filebeat requires the `journalctl` binary to ingest journal logs, this commit adds it by installing systemd in all Filebeat Docker container images (cherry picked from commit 32ecb7c) # Conflicts: # docs/reference/filebeat/filebeat-input-journald.md # docs/release-notes/index.md * Convert the docs and changelog to markdown --------- Co-authored-by: Tiago Queiroz <tiago.queiroz@elastic.co>

…4087) The journald input from Filebeat requires the `journalctl` binary to ingest journal logs, this commit adds it by installing systemd in all Filebeat Docker container images (cherry picked from commit 32ecb7c) # Conflicts: # docs/reference/filebeat/filebeat-input-journald.md # docs/release-notes/index.md * Convert the docs and changelog to markdown --------- Co-authored-by: Tiago Queiroz <tiago.queiroz@elastic.co>

The journald input from Filebeat requires the `journalctl` binary to ingest journal logs, this commit adds it by installing systemd in all Filebeat Docker container images (cherry picked from commit 32ecb7c) Co-authored-by: Tiago Queiroz <tiago.queiroz@elastic.co>

rdner · 2025-04-28T07:52:48Z

@belimawr could you document the image size increase in the description of this PR?

cmacknz · 2025-04-28T20:45:35Z

dev-tools/packaging/templates/docker/Dockerfile.tmpl

+    {{- if (contains .from "ubi") }}
+    RUN for iter in {1..10}; do \
+            microdnf -y update && \
+            microdnf -y install systemd && \


What systemd version does this install and how compatible is it with debian?

I ask because I expect users will frequently use this to read journald logs from debian or ubuntu based k8s nodes so if it isn't compatible we need to clearly document that.

UBI at least already exposes us to CVEs in the rest of the OS packages so it doesn't have the same concerns as Wolfi for that problem.

What systemd version does this install and how compatible is it with debian?

I ask because I expect users will frequently use this to read journald logs from debian or ubuntu based k8s nodes so if it isn't compatible we need to clearly document that.

Is not it sufficiently documented here? https://github.com/elastic/beats/pull/44056/files#diff-7a8b715909f505c1dc033e00d9fb48d50576d9c128e1c50538fe697218b97ff7R11-R21

I think almost, that doesn't explain why this is important or what the compatibility rules are. It just says users should check.

What is journactl's version compatibility policy? Backwards compatibility (new versions can read old versions) I assume, but does it also provide forward compatibility (old versions can read new versions of journal files)?

If the Filebeat version is 255 but the host system is 257 what does that mean for users?

cmacknz · 2025-04-28T20:46:51Z

dev-tools/packaging/templates/docker/Dockerfile.tmpl

+    {{- if (contains .from "wolfi") }}
+    RUN for iter in {1..10}; do \
+            apk update && \
+            apk add --no-interactive --no-progress --no-cache systemd && \


This is going to have the side effect of exposing us to CVEs in the entirety of systemd which might turn out to be a significant problem. We probably want to keep this out of the wolfi image whose point is to minimize exposure to OS based CVEs.

In general it would actually be better to build only what we need or find a way to include just journalctl and nothing else but I'm not sure how possible that is.

rdner · 2025-04-29T07:39:03Z

PR to remove systemd from Wolfi #44108

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Apr 24, 2025

Install systemd on Filebeat Docker images

1317cd2

The journald input from Filebeat requires the `journalctl` binary to ingest journal logs, this commit adds it by installing systemd in all Filebeat Docker container images

belimawr force-pushed the 44040-add-journalctl-to-containers branch from ff2a227 to 1317cd2 Compare April 24, 2025 16:42

mergify bot assigned belimawr Apr 24, 2025

belimawr added Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team bugfix backport-active-all Automated backport with mergify to all the active branches labels Apr 24, 2025

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Apr 24, 2025

This was referenced Apr 24, 2025

[Filebeat] Journald input doesn't work in container #41278

Closed

GA support for reading from journald #37086

Open

belimawr marked this pull request as ready for review April 24, 2025 18:49

belimawr requested review from a team as code owners April 24, 2025 18:49

belimawr requested review from faec and leehinman April 24, 2025 18:49

leehinman approved these changes Apr 24, 2025

View reviewed changes

kilfoyle reviewed Apr 25, 2025

View reviewed changes

docs/reference/filebeat/filebeat-input-journald.md Outdated Show resolved Hide resolved

kilfoyle approved these changes Apr 25, 2025

View reviewed changes

rdner reviewed Apr 25, 2025

View reviewed changes

belimawr added 2 commits April 25, 2025 06:20

implement review suggestions

02f0929

Merge branch 'main' of github.com:elastic/beats into 44040-add-journa…

9943ffc

…lctl-to-containers

belimawr merged commit 32ecb7c into elastic:main Apr 25, 2025
196 checks passed

mergify bot mentioned this pull request Apr 25, 2025

[8.17](backport #44056) Install systemd on Filebeat Docker images #44085

Merged

6 tasks

mergify bot mentioned this pull request Apr 25, 2025

[8.18](backport #44056) Install systemd on Filebeat Docker images #44086

Merged

6 tasks

mergify bot mentioned this pull request Apr 25, 2025

[8.19](backport #44056) Install systemd on Filebeat Docker images #44087

Merged

6 tasks

mergify bot mentioned this pull request Apr 25, 2025

[9.0](backport #44056) Install systemd on Filebeat Docker images #44088

Merged

6 tasks

cmacknz reviewed Apr 28, 2025

View reviewed changes

rdner mentioned this pull request Apr 29, 2025

Remove systemd from the Wolfi image #44108

Merged

Conversation

belimawr commented Apr 24, 2025 • edited by rdner Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Checklist

How to test this PR locally

Related issues

Uh oh!

mergify bot commented Apr 24, 2025

Uh oh!

elasticmachine commented Apr 24, 2025

Uh oh!

leehinman left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kilfoyle left a comment

Choose a reason for hiding this comment

Uh oh!

rdner Apr 25, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

rdner commented Apr 25, 2025

Uh oh!

belimawr commented Apr 25, 2025

Uh oh!

belimawr commented Apr 25, 2025

Uh oh!

Uh oh!

github-actions bot commented Apr 25, 2025

Uh oh!

mergify bot commented Apr 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Backports have been created

Uh oh!

rdner commented Apr 28, 2025

Uh oh!

cmacknz Apr 28, 2025

Choose a reason for hiding this comment

Uh oh!

cmacknz Apr 28, 2025

Choose a reason for hiding this comment

Uh oh!

rdner Apr 29, 2025

Choose a reason for hiding this comment

Uh oh!

cmacknz Apr 30, 2025

Choose a reason for hiding this comment

Uh oh!

cmacknz Apr 28, 2025

Choose a reason for hiding this comment

Uh oh!

rdner Apr 29, 2025

Choose a reason for hiding this comment

Uh oh!

rdner commented Apr 29, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

belimawr commented Apr 24, 2025 •

edited by rdner

Loading

mergify bot commented Apr 25, 2025 •

edited

Loading