[System] Add journald support for auth and syslog data streams#11618
[System] Add journald support for auth and syslog data streams#11618belimawr merged 0 commit intoelastic:mainfrom
Conversation
d8ffe53 to
77e4c83
Compare
🚀 Benchmarks reportTo see the full report comment with |
|
Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane) |
|
Pinging @elastic/sec-linux-platform (Team:Security-Linux Platform) |
|
Pinging @elastic/sec-windows-platform (Team:Security-Windows Platform) |
packages/system/data_stream/auth_journald/agent/stream/journald.yml.hbs
Outdated
Show resolved
Hide resolved
packages/system/data_stream/auth_journald/agent/stream/journald.yml.hbs
Outdated
Show resolved
Hide resolved
packages/system/data_stream/auth_journald/agent/stream/journald.yml.hbs
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
The fields removed on this file were duplicated before my PR.
There was a problem hiding this comment.
Adding input.type is the only real change from this PR on this file, the rest is indentation from elastic-package format
|
We should:
|
|
possible crazy thought. What if we had an autodisover module that set a variable that for "system module input" to something like "log" or "journald". Could the condition then use variable provided autodiscover? |
The biggest problem with that is that the Elastic-Agent needs to know the input type (that's defined in the integration manifest and does not need to map to a Beat input) before sending it to Filebeat. That's used to choose which OS process will run the input (e.g: one process for all journald inputs, another for all log inputs, etc), it is also used to aggregate and report status in multiple places, like the overview page for the Elastic-Agent. This also has implications on the state folder used. Given all that, at the moment, we cannot postpone the decision about the input type, the input type needs to be known before the Elastic-Agent sends the configuration to Filebeat. Another interesting challenge is that many things in Beats run "outside" of the input code, e.g: |
I like this experience @cmacknz, however as far as I know it's not possible to do that with the current features we have on Fleet-UI. Maybe @kpollich can chime in and help understand what is supported by Fleet UI. Clarifying more on this "use journald" toggle, does that mean having only one section for system logs, adding the "use journald" toggle that will show/hide a second level of "advanced options" Instead of having the two configuration blocks (currently aggregated by input type) that's currently implemented on this PR: |
Could that be addressed by having a "new" input type called "autodiscover"? |
|
@lalit-satapathy can we get someone from your team review it too please? |
|
@belimawr shouldn't this version only be available from 8.17.0 as only this one will include the latest changes on Beats for the journald input? |
fff7c2f to
4427c93
Compare
andrewkroh
left a comment
There was a problem hiding this comment.
Looks good, just a few minor comments. I did not review in detail the contents of the new pipelines under the assumption that they are close derivatives of the existing ones.
I do think some documentation should be added to the README to explain a bit of how it works (or how it is supposed to work) w.r.t. using journald on newer OSes and that this is based on the condition.
packages/system/changelog.yml
Outdated
There was a problem hiding this comment.
The changelog message no longer matches the current approach.
packages/system/data_stream/auth/elasticsearch/ingest_pipeline/default.yml
Outdated
Show resolved
Hide resolved
packages/system/data_stream/auth/elasticsearch/ingest_pipeline/journald.yml
Outdated
Show resolved
Hide resolved
I added something on a9731b8, I tried to keep it generic as the current implementation will likely change a bit. |
packages/system/manifest.yml
Outdated
There was a problem hiding this comment.
What happens if you remove this? What do older agent versions do if they don't support the new host.* parameters for the journald condition?
Doing this means users won't get bug fixes for the integration unless they upgrade to 8.16.0. I want to double check that this is actually solving a problem that can't instead be solved by documentation (e.g. that journald conditions don't work until this version).
There was a problem hiding this comment.
What happens if you remove this?
The Jouranld filters won't work, auth and syslog datastreams will both ingest the whole journal.
I'll test with an older version to see the overall behaviour.
There was a problem hiding this comment.
Any chance we can put in a condition on the agent version to get around this?
There was a problem hiding this comment.
So, I did some testing and the results are interesting:
- The conditions silently break, so the log input runs on Debian 12 and the journald does not.
- Because the filters do not work, both datasets (auth and syslog) ingest all the data
- The dashboards seem to work well, because they use specific fields set by the ingest pipelines, I believe their data is accurate.
So, overall, it does not seem to bad, we duplicate data, but that's about it.
Any chance we can put in a condition on the agent version to get around this?
That's a good idea, I'll check. However it's gonna be a bit unclear for the user why the journald input shows as not available.
There was a problem hiding this comment.
I've updated the PR with the latest changes and the description with the new screenshot
new screenshot
There was a problem hiding this comment.
So, overall, it does not seem to bad, we duplicate data, but that's about it.
We duplicate data on all non-8.16.0 agents? This is going to be a very large number of agents, and this integration is installed by default, so if there's any consequence or effect to removing the version constraint like this let's put the constraint back. I'm confident we did our diligence that it has to be there.
The system integration is the first integration most users interact with, it can't have any quirks or weird behavior or duplicate anything for currently supported 8.x versions (which is more than 8.16.x).
There was a problem hiding this comment.
We duplicate data on all non-8.16.0 agents?
No, that's only for Elastic-Agents where the Jouranld input ends up running, the user would have to change the conditions to get the Journald input running on agents < 8.16.0.
So, by default, there will be no issue, however this does introduce a weird behaviour, which is undocumented and un-intuitive.
I'll revert to require at least 8.16.0
|
Folks, that's the final version. We've reverted to using the conditions default value and documenting it. |
cmacknz
left a comment
There was a problem hiding this comment.
The only two changes I want are to put the stack version constraint back, and to show the user that the condition exists by default. Let's not hide the fact that the inputs are conditional.
I think the documentation for the condition is great, and the simplification to have the condition apply to the whole input instead of every stream makes things easier to see.
|
@cmacknz I added all changes you requested on f3bd0cfd66. |
cmacknz
left a comment
There was a problem hiding this comment.
Thanks!
Remember that once you merge this it will be available to all 8.16+ stacks rapidly afterwards
I'm aware of this. Is there any extra testing you'd like to do? |
|
I've just realised that while our tests do a good job at ensuring the inputs and ingest pipelines work as expected, they do not test the The main challenge is that the system tests run the Elastic-Agent in its own docker container, which limits how much OS-level information we can actually test. I'll look if there is a way to circumvent this. |
💚 Build Succeeded
History
cc @belimawr |
|
|
Package system - 1.63.0 containing this change is available at https://epr.elastic.co/package/system/1.63.0/ |
…ic#11618) This commit introduces support for Journald in the system integration, adding a "journald" input for both auth and syslog data. Conditions are used to select when to run the log input or the journal input, it defaults to running the Journald input on Debian 12 and Amazon Linux 2023. The minimum stack version is updated to 8.17.0 so the integration can use the facilities filtering option from the journald input added in 8.16.0 and some extra fields from the host provider added in 8.17.0.
…ic#11618) This commit introduces support for Journald in the system integration, adding a "journald" input for both auth and syslog data. Conditions are used to select when to run the log input or the journal input, it defaults to running the Journald input on Debian 12 and Amazon Linux 2023. The minimum stack version is updated to 8.17.0 so the integration can use the facilities filtering option from the journald input added in 8.16.0 and some extra fields from the host provider added in 8.17.0.
Note for reviewers
I did my best to get events that would test all the grok expressions, I mostly tried to get similar messages to the
syslogandauthdataset. Theauthdataset might be missing some from PAM because I wasn't sure how to generate them.Proposed commit message
This commit introduces support for Journald in the system integration,
adding a "journald" input for both auth and syslog data.
Conditions are used to select when to run the log input or the journal input, it
defaults to running the Journald input on Debian 12 and Amazon Linux 2023.
The minimum stack version is updated to
8.17.0so the integration can use thefacilities filtering option from the journald input added in
8.16.0and some extra fieldsfrom the host provider added in
8.17.0.Checklist
changelog.ymlfile.Author's Checklist
data_stream.datasetandevent.datasetmatch their log input counterpartos_family,os_platformandos_versionto host provider elastic-agent#5941 before merging/releasing this integration version.How to test this PR locally
main, you need the changes from (Addos_family,os_platformandos_versionto host provider elastic-agent#5941)Related issues
Screenshots
Integration configuration
Dashboards with data from Journald (Debain 12)
Journald x Log input
I enabled both, the journald and the log input for the auth and syslog datasets on a fresh Debian 11 VM, both inputs collected the same amount of events. On the left is the journald input and on the right the log input.