Filebeat system pipeline fails to grok pam_unix(sudo:session) messages in /var/log/secure.
- Version: 8.5.3
- OS: Amazon Linux 2
- Steps to Reproduce:
Enable system module with filebeat modules enable, and enable the auth fileset.
# modules.d/system.yml
# Module: system
# Docs: https://www.elastic.co/guide/en/beats/filebeat/8.5/filebeat-module-system.html
- module: system
# Syslog
syslog:
enabled: true
# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
#var.paths:
# Authorization logs
auth:
enabled: true
# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
#var.paths:Then become root from ec2-user with sudo -i.
Redacted output:
{
"agent": {
"type": "filebeat",
"version": "8.5.3"
},
"@timestamp": "2023-01-12T18:17:05.112Z",
"ecs": {
"version": "8.0.0"
},
"log": {
"file": {
"path": "/var/log/secure"
},
"offset": 27533
},
"fileset": {
"name": "auth"
},
"message": [
"Jan 12 18:10:48 REDACTED sudo: pam_unix(sudo:session): session opened for user root by ec2-user(uid=0)"
],
"error.message": [
"Provided Grok expressions do not match field value: [{\"@timestamp\":\"2023-01-12T18:17:05.112Z\",\"@metadata\":{\"beat\":\"filebeat\",\"type\":\"_doc\",\"version\":\"8.5.3\",\"pipeline\":\"filebeat-8.5.3-system-auth-pipeline\"},\"fileset\":{\"name\":\"auth\"},\"ecs\":{\"version\":\"8.0.0\"},\"log\":{\"file\":{\"path\":\"/var/log/secure\"},\"offset\":27533},\"service\":{\"type\":\"system\"},\"input\":{\"type\":\"log\"},\"agent\":{\"id\":\"REDACTED\",\"name\":\"REDACTED.compute.internal\",\"type\":\"filebeat\",\"version\":\"8.5.3\",\"ephemeral_id\":\"REDACTED\"},\"message\":\"Jan 12 18:10:48 REDACTED sudo: pam_unix(sudo:session): session opened for user root by ec2-user(uid=0)\",\"event\":{\"module\":\"system\",\"dataset\":\"system.auth\",\"timezone\":\"+00:00\"},\"host\":{\"name\":\"REDACTED.compute.internal\"}}]"
],
}
Filebeat
systempipeline fails to grokpam_unix(sudo:session)messages in/var/log/secure.beats/filebeat/module/system/auth/ingest/pipeline.yml
Line 40 in bb64650
Enable
systemmodule withfilebeat modules enable, and enable theauthfileset.Then become root from ec2-user with
sudo -i.Redacted output:
{ "agent": { "type": "filebeat", "version": "8.5.3" }, "@timestamp": "2023-01-12T18:17:05.112Z", "ecs": { "version": "8.0.0" }, "log": { "file": { "path": "/var/log/secure" }, "offset": 27533 }, "fileset": { "name": "auth" }, "message": [ "Jan 12 18:10:48 REDACTED sudo: pam_unix(sudo:session): session opened for user root by ec2-user(uid=0)" ], "error.message": [ "Provided Grok expressions do not match field value: [{\"@timestamp\":\"2023-01-12T18:17:05.112Z\",\"@metadata\":{\"beat\":\"filebeat\",\"type\":\"_doc\",\"version\":\"8.5.3\",\"pipeline\":\"filebeat-8.5.3-system-auth-pipeline\"},\"fileset\":{\"name\":\"auth\"},\"ecs\":{\"version\":\"8.0.0\"},\"log\":{\"file\":{\"path\":\"/var/log/secure\"},\"offset\":27533},\"service\":{\"type\":\"system\"},\"input\":{\"type\":\"log\"},\"agent\":{\"id\":\"REDACTED\",\"name\":\"REDACTED.compute.internal\",\"type\":\"filebeat\",\"version\":\"8.5.3\",\"ephemeral_id\":\"REDACTED\"},\"message\":\"Jan 12 18:10:48 REDACTED sudo: pam_unix(sudo:session): session opened for user root by ec2-user(uid=0)\",\"event\":{\"module\":\"system\",\"dataset\":\"system.auth\",\"timezone\":\"+00:00\"},\"host\":{\"name\":\"REDACTED.compute.internal\"}}]" ], }