Skip to content

Commit f4b80fd

Browse files
belimawrbelimawr
authored
Journald support for System module (#41555)
This commit adds journald support for the System module, both filesets now have a `use_journald` variable that can be set to force using Journald to ingest syslog and auth logs. The ingest pipelines are updated, now there is an entrypoint pipeline that selects the correct one according to the field `input.type`. System tests are also added.
1 parent 18e256f commit f4b80fd

28 files changed

Lines changed: 817 additions & 15 deletions

‎CHANGELOG.next.asciidoc‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -323,6 +323,7 @@ https://github.com/elastic/beats/compare/v8.8.1\...main[Check the HEAD diff]
323323
- Make ETW input GA. {pull}41389[41389]
324324
- Added input metrics to GCS input. {issue}36640[36640] {pull}41505[41505]
325325
- Add support for Okta entity analytics provider to collect role and factor data for users. {pull}41460[41460]
326+
- Add support for Journald in the System module. {pull}41555[41555]
326327
- Add ability to remove request trace logs from http_endpoint input. {pull}40005[40005]
327328
- Add ability to remove request trace logs from entityanalytics input. {pull}40004[40004]
328329

‎filebeat/docs/include/use-journald.asciidoc‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
*`var.use_journald`*::
2+
3+
A boolean that when set to `true` will read logs from Journald. When
4+
Journald is used all events contain the tag `journald`.

‎filebeat/docs/modules/system.asciidoc‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -64,11 +64,13 @@ include::../include/config-option-intro.asciidoc[]
6464
==== `syslog` fileset settings
6565

6666
include::../include/var-paths.asciidoc[]
67+
include::../include/use-journald.asciidoc[]
6768

6869
[float]
6970
==== `auth` fileset settings
7071

7172
include::../include/var-paths.asciidoc[]
73+
include::../include/use-journald.asciidoc[]
7274

7375
*`var.tags`*::
7476

‎filebeat/filebeat.reference.yml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,9 @@ filebeat.modules:
2121
# Filebeat will choose the paths depending on your OS.
2222
#var.paths:
2323

24+
# Use journald to collect system logs
25+
#var.use_journald: false
26+
2427
# Input configuration (advanced). Any input configuration option
2528
# can be added under this section.
2629
#input:
@@ -33,6 +36,9 @@ filebeat.modules:
3336
# Filebeat will choose the paths depending on your OS.
3437
#var.paths:
3538

39+
# Use journald to collect auth logs
40+
#var.use_journald: false
41+
3642
# Input configuration (advanced). Any input configuration option
3743
# can be added under this section.
3844
#input:

‎filebeat/module/system/README.md‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
# Journald tests (Debian 12)
2+
The tests for the journald input (currently only used for Debian 12
3+
testing) require journal files (test files ending in `.journal`), those
4+
files are generated using `systemd-journal-remote` (see the [Journald
5+
input README.md](../../input/journald/README.md) for more details).
6+
7+
The source for those journal files are the `.export` files in the test
8+
folder. Those files are the raw output of `journalctl -o export`. They
9+
are added here because journal files format change with different
10+
versions of journald, which can cause `journalclt` to fail reading
11+
them, which leads to test failures. So if tests start failing because
12+
`journalctl` cannot read the journal files as expected, new ones can
13+
easily be generated with the same version of journalctl used on CI
14+
and the original dataset.

‎filebeat/module/system/_meta/config.reference.yml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,9 @@
77
# Filebeat will choose the paths depending on your OS.
88
#var.paths:
99

10+
# Use journald to collect system logs
11+
#var.use_journald: false
12+
1013
# Input configuration (advanced). Any input configuration option
1114
# can be added under this section.
1215
#input:
@@ -19,6 +22,9 @@
1922
# Filebeat will choose the paths depending on your OS.
2023
#var.paths:
2124

25+
# Use journald to collect auth logs
26+
#var.use_journald: false
27+
2228
# Input configuration (advanced). Any input configuration option
2329
# can be added under this section.
2430
#input:

‎filebeat/module/system/_meta/config.yml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,10 +7,16 @@
77
# Filebeat will choose the paths depending on your OS.
88
#var.paths:
99

10+
# Use journald to collect system logs
11+
#var.use_journald: false
12+
1013
# Authorization logs
1114
auth:
1215
enabled: false
1316

1417
# Set custom paths for the log files. If left empty,
1518
# Filebeat will choose the paths depending on your OS.
1619
#var.paths:
20+
21+
# Use journald to collect auth logs
22+
#var.use_journald: false

‎filebeat/module/system/_meta/docs.asciidoc‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -57,11 +57,13 @@ include::../include/config-option-intro.asciidoc[]
5757
==== `syslog` fileset settings
5858

5959
include::../include/var-paths.asciidoc[]
60+
include::../include/use-journald.asciidoc[]
6061

6162
[float]
6263
==== `auth` fileset settings
6364

6465
include::../include/var-paths.asciidoc[]
66+
include::../include/use-journald.asciidoc[]
6567

6668
*`var.tags`*::
6769

‎filebeat/module/system/auth/config/auth.yml‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,22 @@
1+
{{ if .use_journald }}
2+
type: journald
3+
id: system-auth
4+
facilities:
5+
- 4
6+
- 10
7+
{{ else }}
18
type: log
29
paths:
310
{{ range $i, $path := .paths }}
411
- {{$path}}
512
{{ end }}
613
exclude_files: [".gz$"]
7-
814
multiline:
915
pattern: "^\\s"
1016
match: after
17+
{{ end }}
1118

19+
# Common configuration
1220
processors:
1321
- add_locale: ~
1422

‎filebeat/module/system/auth/ingest/entrypoint.yml‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
description: Entrypoint Pipeline for system/auth Filebeat module
2+
processors:
3+
- pipeline:
4+
if: ctx?.input?.type == "journald"
5+
name: '{< IngestPipeline "journald" >}'
6+
7+
- pipeline:
8+
if: ctx?.input?.type == "log"
9+
name: '{< IngestPipeline "files" >}'

0 commit comments

Comments
 (0)