Journald support for System module by belimawr · Pull Request #41555 · elastic/beats

belimawr · 2024-11-07T18:05:53Z

Proposed commit message

This commit adds journald support for the System module, both filesets now have a use_journald variable
that can be set to force using Journald to ingest syslog and auth logs.

The ingest pipelines are updated, now there is an entrypoint pipeline that selects the correct one according to the field
input.type.

System tests are also added.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

~~## Disruptive User Impact~~

Author's Checklist

Ensure the journald input is ingesting data correctly

How to test this PR locally

Package Filebeat from this PR
Configure the ES output and Kibana credentials
Enable the system module (./filebeat modules enable system), set var.use_journald: true for both filesets (edit modules.d/system.yml)
Setup the assets: ./filebeat setup --modules system (this requires Kibana credentials correctly set)
Run Filebeat as root
Look at the logs in the filebeat-* data view, filter by event.dataset: system.syslog or event.dataset: system.auth, ensure the logs are correctly ingested
Look at the system module dashboards, ensure they're working/show data.

Related issues

Closes Investigate the best way to decide when to read system logs from files or journald #40526

~~## Use cases~~

Screenshots

Dashboards, journald and logs side by side

Events, journald and logs side by side

~~## Logs~~

mergify · 2024-11-07T18:06:32Z

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @belimawr? 🙏.
For such, you'll need to label your PR with:

The upcoming major version of the Elastic Stack
The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit

mergify · 2024-11-07T18:06:32Z

backport-8.x has been added to help with the transition to the new branch 8.x.
If you don't need it please use backport-skip label and remove the backport-8.x label.

belimawr · 2024-11-08T16:01:47Z

I've been doing a comparison test between the system module with journald and log input, and they report a different amount of messages, there are more entries in the log files than in the journal. I did this test on a fresh Debian 11 VM that uses both: jouranld and traditional log files (via rsyslog):

root@bullseye:/var/log# wc -l syslog auth.log messages 
   679 syslog
   147 auth.log
   423 messages
  1249 total
root@bullseye:/var/log# journalctl --no-tail | wc -l
919
root@bullseye:/var/log#

The dashboards seem correctly populated (screenshots in the PR description).

I'm still investigating this difference.

…ule-debian-12-V2

elasticmachine · 2024-11-13T15:56:18Z

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

AndersonQ · 2024-11-14T12:59:16Z

filebeat/module/system/auth/test/debian-12.journal-expected.json

any reason for not placing it inside a testdata folder?

Yes, I placed it where the other test files are. Those are used by the python system tests.

AndersonQ · 2024-11-14T13:00:02Z

filebeat/module/system/syslog/test/debian-12.journal-expected.json

any reason for not placing it inside a testdata folder?

Yes, I placed it where the other test files are. Those are used by the python system tests.

AndersonQ · 2024-11-14T14:22:18Z

I've been doing a comparison test between the system module with journald and log input, and they report a different amount of messages, there are more entries in the log files than in the journal. I did this test on a fresh Debian 11 VM that uses both: jouranld and traditional log files (via rsyslog):
root@bullseye:/var/log# wc -l syslog auth.log messages 
   679 syslog
   147 auth.log
   423 messages
  1249 total
root@bullseye:/var/log# journalctl --no-tail | wc -l
919
root@bullseye:/var/log# 
The dashboards seem correctly populated (screenshots in the PR description).

I'm still investigating this difference.

so, that means the PR isn't yet ready to be merged, right? We don't wanna it to go in unless we're sure we're not loosing logs.

Perhaps it should go back to draft? What do you think?

AndersonQ · 2024-11-14T14:22:29Z

Anyway I'll test it

AndersonQ · 2024-11-14T15:18:19Z

Anyway I'll test it

it works. I don't see the map SSH failed login attempts source locations [Filebeat System] ECS populated. @belimawr do you know where the date to populate this map comes from?

AndersonQ · 2024-11-14T15:26:04Z

CHANGELOG.next.asciidoc

 - Improved Azure Blob Storage input documentation. {pull}41252[41252]
 - Make ETW input GA. {pull}41389[41389]
 - Add support for Okta entity analytics provider to collect role and factor data for users. {pull}41460[41460]
+- Add support for Journald in the System module. {pull}41555[41555]


I'm not sure where it needs to be placed, but if there is a difference between he number of logs collected between journald and the log input it might be good to have it documented somewhere.

We're already collecting all the journald logs, to me the difference in the number of logs from journald and the files is a OS detail, also some OSes only use jouranld, which makes comparing impossible unless something like rsyslog is installed and setup to generate the log files.

AndersonQ · 2024-11-14T15:48:36Z

I've been doing a comparison test between the system module with journald and log input, and they report a different amount of messages, there are more entries in the log files than in the journal. I did this test on a fresh Debian 11 VM that uses both: jouranld and traditional log files (via rsyslog):
root@bullseye:/var/log# wc -l syslog auth.log messages 
   679 syslog
   147 auth.log
   423 messages
  1249 total
root@bullseye:/var/log# journalctl --no-tail | wc -l
919
root@bullseye:/var/log# 
The dashboards seem correctly populated (screenshots in the PR description).
I'm still investigating this difference.
so, that means the PR isn't yet ready to be merged, right? We don't wanna it to go in unless we're sure we're not loosing logs.

Perhaps it should go back to draft? What do you think?

we had a call, the journald input consumes all the logs on journald, so I don't see it as a blocker here

belimawr · 2024-11-14T15:49:53Z

I've been doing a comparison test between the system module with journald and log input, and they report a different amount of messages, there are more entries in the log files than in the journal. I did this test on a fresh Debian 11 VM that uses both: jouranld and traditional log files (via rsyslog):
root@bullseye:/var/log# wc -l syslog auth.log messages 
   679 syslog
   147 auth.log
   423 messages
  1249 total
root@bullseye:/var/log# journalctl --no-tail | wc -l
919
root@bullseye:/var/log# 
The dashboards seem correctly populated (screenshots in the PR description).
I'm still investigating this difference.
so, that means the PR isn't yet ready to be merged, right? We don't wanna it to go in unless we're sure we're not loosing logs.

Perhaps it should go back to draft? What do you think?

It's ready for review. I've investigated more and the system module can ingest without errors all logs in the journal, so any difference in the amount of logs available is at the OS level, outside of the scope of this PR.

If needed, I could dig deeper to understand the difference. Let me know if you, or any other reviewer, believes it's necessary.

AndersonQ

I don't see https://github.com/elastic/beats/pull/41555/files#r1842419453 as a blocker. We just need to check if we need to add any doc explaining the difference between the log input and journald. Also the whole point is to be able to have the system module on systems where there is only journald, so, again not a blocker

belimawr · 2024-11-14T18:52:13Z

Anyway I'll test it

it works. I don't see the map SSH failed login attempts source locations [Filebeat System] ECS populated. @belimawr do you know where the date to populate this map comes from?

No, I don't.

mergify · 2024-11-18T14:55:41Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b 40526-system-module-debian-12-V2 upstream/40526-system-module-debian-12-V2
git merge upstream/main
git push upstream 40526-system-module-debian-12-V2

…ule-debian-12-V2

This commit adds journald support for the System module, both filesets now have a `use_journald` variable that can be set to force using Journald to ingest syslog and auth logs. The ingest pipelines are updated, now there is an entrypoint pipeline that selects the correct one according to the field `input.type`. System tests are also added. (cherry picked from commit f4b80fd)

This commit adds journald support for the System module, both filesets now have a `use_journald` variable that can be set to force using Journald to ingest syslog and auth logs. The ingest pipelines are updated, now there is an entrypoint pipeline that selects the correct one according to the field `input.type`. System tests are also added. (cherry picked from commit f4b80fd) Co-authored-by: Tiago Queiroz <tiago.queiroz@elastic.co>

belimawr added 5 commits November 7, 2024 11:37

[WIP] support journald for system module

55ac589

Update generated files (input.type)

f3f9912

Add journald to the config and update journal test files

9deda04

Update generated files

0741b9b

Remove debug

24cba1b

belimawr added the Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team label Nov 7, 2024

belimawr self-assigned this Nov 7, 2024

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Nov 7, 2024

mergify bot added the backport-8.x Automated backport to the 8.x branch with mergify label Nov 7, 2024

fix syslog pipeline

b9d9f00

belimawr added the enhancement label Nov 8, 2024

belimawr added 4 commits November 8, 2024 11:13

Merge branch 'main' of github.com:elastic/beats into 40526-system-mod…

2a53030

…ule-debian-12-V2

Merge branch 'main' of github.com:elastic/beats into 40526-system-mod…

c16c886

…ule-debian-12-V2

Select ingest pipeline by input.type and small fixes

592b8ee

Add changelog entry

bec3bf2

belimawr marked this pull request as ready for review November 13, 2024 15:56

belimawr requested a review from a team as a code owner November 13, 2024 15:56

belimawr requested review from AndersonQ and rdner November 13, 2024 15:56

belimawr mentioned this pull request Nov 13, 2024

Investigate the best way to decide when to read system logs from files or journald #40526

Closed

AndersonQ reviewed Nov 14, 2024

View reviewed changes

AndersonQ approved these changes Nov 14, 2024

View reviewed changes

Merge branch 'main' of github.com:elastic/beats into 40526-system-mod…

c9bd65e

…ule-debian-12-V2

belimawr merged commit f4b80fd into elastic:main Nov 18, 2024

mergify bot mentioned this pull request Nov 18, 2024

[8.x](backport #41555) Journald support for System module #41670

Merged

7 tasks

belimawr deleted the 40526-system-module-debian-12-V2 branch December 19, 2024 13:56

khushijain21 mentioned this pull request Jun 23, 2025

test4 #44958

Closed

Conversation

belimawr commented Nov 7, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

Uh oh!

mergify bot commented Nov 7, 2024

Uh oh!

mergify bot commented Nov 7, 2024

Uh oh!

belimawr commented Nov 8, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticmachine commented Nov 13, 2024

Uh oh!

AndersonQ Nov 14, 2024

Choose a reason for hiding this comment

Uh oh!

belimawr Nov 14, 2024

Choose a reason for hiding this comment

Uh oh!

AndersonQ Nov 14, 2024

Choose a reason for hiding this comment

Uh oh!

belimawr Nov 14, 2024

Choose a reason for hiding this comment

Uh oh!

AndersonQ commented Nov 14, 2024

Uh oh!

AndersonQ commented Nov 14, 2024

Uh oh!

AndersonQ commented Nov 14, 2024

Uh oh!

AndersonQ Nov 14, 2024

Choose a reason for hiding this comment

Uh oh!

belimawr Nov 18, 2024

Choose a reason for hiding this comment

Uh oh!

AndersonQ commented Nov 14, 2024

Uh oh!

belimawr commented Nov 14, 2024

Uh oh!

AndersonQ left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

belimawr commented Nov 14, 2024

Uh oh!

mergify bot commented Nov 18, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

belimawr commented Nov 7, 2024 •

edited

Loading

belimawr commented Nov 8, 2024 •

edited

Loading

AndersonQ left a comment •

edited

Loading