Overview

Syslog parsing in Beats is currently implemented as a dedicated input. It supports accepting data over UDP, TCP, and Unix sockets. Because it is implemented as an input, the parsing cannot be used by other inputs such as filestream, httpjson, or kafka.

The goal is to decouple the input from parsing by moving the parsing to a processor. This also provides the opportunity for to improve other areas of processing at the same time.

Summary of Intended Changes

• Implement a parser for syslog
  • Strongly follow RFC 3164, allow for flexible timestamps. RFC 3146 is also rather vague in areas, and a best effort parser will be necessary for some elements (tag and content fields, for instance)
  • Strictly follow RFC 5424
  • Automatically detect format by default
  • Parsers will allow multiline to work with syslog
• Always retain event.original
• Populate ECS fields
• Deprecate syslog input

Flexible Timestamp Parsing for RFC 3146

In addition to the RFC-defined timestamp format ("Mmm dd hh:mm:ss", e.g. Feb 5 17:32:18), the processor should also allow for parsing RFC 3339 timestamps. There tends to be low adherence to this part of the specfication due to lack of subsecond precision, year, and timezone. These deviations will be documented with examples.

Other Notes

• Cisco does not follow RFCs. Cisco modules will use plain udp/tcp/tls inputs (without parser) and use grok or dissect to do best effort syslog header parser. Even if header parsing fails they should pick up parsing from the Cisco message ID (e.g. %SEC-6-IPACCESSLOGP)

See Also

• https://datatracker.ietf.org/doc/html/rfc3164
• https://datatracker.ietf.org/doc/html/rfc5424