[system][syslog] add a pattern in filebeat.system module to capture greedy multiline logs with ISO timestamps by stefans-elastic · Pull Request #13427 · elastic/integrations

stefans-elastic · 2025-04-04T12:28:48Z

Proposed commit message

add a pattern in filebeat.system module to capture greedy multiline logs with ISO timestamps

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

[ ]

How to test this PR locally

Related issues

Closes https://github.com/elastic/obs-integration-team/issues/198

Screenshots

…reedy multiline logs with ISO timestamps

elastic-vault-github-plugin-prod · 2025-04-04T13:40:34Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

packages/system/changelog.yml

...a_stream/syslog/_dev/test/pipeline/test-iso8601-timestamp-multiline-syslog.log-expected.json

ishleenk17 · 2025-04-05T10:36:41Z

Also, can you state details about the second ask as per this issue.

…ystem-syslog-iso-grok-pattern

stefans-elastic · 2025-04-07T07:51:38Z

Also, can you state details about the second ask as per this issue.

Sure.

The gh issue contains two points:

Include the greedy grok pattern not only for SYSLOGTIMESTAMP, but for ISO8601 as well.
A process might send logs to /var/log/messages with white space characters and not the white spaces themselves, something like this
So this PR covets point 1. As for point 2: I'm not sure it is possible to have log entries with escaped whitespace characters. The example given in the issue seems to be coming from result event and for those the whitespace characters are escaped (as can be seen here: https://github.com/elastic/beats/blob/main/filebeat/module/system/syslog/test/darwin-syslog.log-expected.json#L31)

ishleenk17 · 2025-04-08T05:15:14Z

...a_stream/syslog/_dev/test/pipeline/test-iso8601-timestamp-multiline-syslog.log-expected.json

+      "input": {
+        "type": "log"
+      },
+      "message": "2022-04-21T14:30:00Z\n\t\tGoogleSoftwareUpdateAgent[21412/0x700007399000] [lvl=2] -[KSUpdateEngine updateAllExceptProduct:] KSUpdateEngine updating all\n\t\tinstalled products, except:'com.google.Keystone'.",


This is still matching the 3rd pattern.
If you see we are getting fields as host.hostname, process.name which matches the 3rd pattern .
To match the 4th pattern , one of these fields should be sipped. So that other than timestamp the complete message becomes part of the greedymessage.

hmm, what is the "tell" of it (I think I'm missing it)?

ishleenk17 · 2025-04-08T05:32:48Z

Also, can you state details about the second ask as per this issue.

Sure.

The gh issue contains two points:

Include the greedy grok pattern not only for SYSLOGTIMESTAMP, but for ISO8601 as well. A process might send logs to /var/log/messages with white space characters and not the white spaces themselves, something like this So this PR covets point 1. As for point 2: I'm not sure it is possible to have log entries with escaped whitespace characters. The example given in the issue seems to be coming from result event and for those the whitespace characters are escaped (as can be seen here: https://github.com/elastic/beats/blob/main/filebeat/module/system/syslog/test/darwin-syslog.log-expected.json#L31)

Since, we don't observe these logs and this could be a corner case.lets focus on just adding the ISO timestamp for this PR.

ishleenk17 · 2025-04-08T12:54:13Z

packages/system/manifest.yml

 name: system
 title: System
-version: "1.67.3"
+version: "1.67.4"


Suggested change

version: "1.67.4"

version: "1.68.0"

ishleenk17

Please make the version change. Otherwise, LGTM.

ishleenk17 · 2025-04-08T12:55:34Z

packages/system/changelog.yml

@@ -1,4 +1,9 @@
 # newer versions go on top
+- version: "1.67.4"


Suggested change

- version: "1.67.4"

- version: "1.68.0"

stefans-elastic · 2025-04-08T12:58:59Z

Please make the version change. Otherwise, LGTM.

updated the version

elastic-sonarqube · 2025-04-08T13:24:30Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

elasticmachine · 2025-04-08T13:24:58Z

💚 Build Succeeded

Buildkite Build
Commit: 0ed1fa7

History

💚 Build #24404 succeeded 8e62018
💚 Build #24352 succeeded ecfc8d7
💔 Build #24351 failed 05cf08f
💔 Build #24349 failed 09e786f

cc @stefans-elastic

elastic-vault-github-plugin-prod · 2025-04-09T10:59:20Z

Package system - 1.68.0 containing this change is available at https://epr.elastic.co/package/system/1.68.0/

[system][syslog] add a pattern in filebeat.system module to capture g…

09e786f

…reedy multiline logs with ISO timestamps

stefans-elastic added the bug Something isn't working, use only for issues label Apr 4, 2025

stefans-elastic self-assigned this Apr 4, 2025

stefans-elastic requested review from a team as code owners April 4, 2025 12:28

stefans-elastic mentioned this pull request Apr 4, 2025

add a pattern in filebeat.system module to capture greedy multiline logs with ISO timestamps elastic/beats#43047

Closed

6 tasks

stefans-elastic added 2 commits April 4, 2025 15:44

update manifest and changelog

05cf08f

fix test file names to make linter pass

ecfc8d7

andrewkroh added Integration:system System Team:Obs-InfraObs Observability Infrastructure Monitoring team [elastic/obs-infraobs-integrations] labels Apr 4, 2025

ishleenk17 reviewed Apr 5, 2025

View reviewed changes

packages/system/changelog.yml Outdated Show resolved Hide resolved

...a_stream/syslog/_dev/test/pipeline/test-iso8601-timestamp-multiline-syslog.log-expected.json Outdated Show resolved Hide resolved

stefans-elastic added 3 commits April 7, 2025 10:29

Merge branch 'main' of github.com:stefans-elastic/integrations into s…

a9a0bd5

…ystem-syslog-iso-grok-pattern

fix change type in changelog

d4461dc

change timestamp to ISO 8601 format in test-case

8e62018

ishleenk17 reviewed Apr 8, 2025

View reviewed changes

fix iso test

c4f281e

ishleenk17 reviewed Apr 8, 2025

View reviewed changes

ishleenk17 approved these changes Apr 8, 2025

View reviewed changes

ishleenk17 reviewed Apr 8, 2025

View reviewed changes

package version fix

0ed1fa7

stefans-elastic merged commit 3d765a1 into elastic:main Apr 9, 2025
7 checks passed

stefans-elastic deleted the system-syslog-iso-grok-pattern branch April 9, 2025 10:39

github-actions bot mentioned this pull request Mar 19, 2026

[system]: syslog parsed with future timestamps #17911

Open

		@@ -1,4 +1,9 @@
		# newer versions go on top
		- version: "1.67.4"

Conversation

stefans-elastic commented Apr 4, 2025

Proposed commit message

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

Uh oh!

elastic-vault-github-plugin-prod bot commented Apr 4, 2025

🚀 Benchmarks report

Uh oh!

Uh oh!

Uh oh!

ishleenk17 commented Apr 5, 2025

Uh oh!

stefans-elastic commented Apr 7, 2025

Uh oh!

ishleenk17 Apr 8, 2025

Choose a reason for hiding this comment

Uh oh!

stefans-elastic Apr 8, 2025

Choose a reason for hiding this comment

Uh oh!

ishleenk17 commented Apr 8, 2025

Uh oh!

ishleenk17 Apr 8, 2025

Choose a reason for hiding this comment

Uh oh!

ishleenk17 left a comment

Choose a reason for hiding this comment

Uh oh!

ishleenk17 Apr 8, 2025

Choose a reason for hiding this comment

Uh oh!

stefans-elastic commented Apr 8, 2025

Uh oh!

elastic-sonarqube bot commented Apr 8, 2025

Quality Gate passed

Uh oh!

elasticmachine commented Apr 8, 2025

💚 Build Succeeded

History

Uh oh!

Uh oh!

elastic-vault-github-plugin-prod bot commented Apr 9, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants