Describe the bug
The syslog parser is including the hostname and ident as part of the time field
To Reproduce
We are sending syslog messages in rfc3164 format but with timestamps in "%Y-%m-%dT%H:%M:%SZ" format eg:
<14>2020-02-25T10:37:41Z podname-v1-5cc4869f45-sd7fr servicename[1089]: {\"level\":\"info\",\"msg\":\"http: TLS handshake error from 127.0.0.1:59100: tls: failed to verify client's certificate: x509: certificate signed by unknown authority\\n\",\"Version\":\"0.0.1\",\"Caller File\":\"/usr/local/go/src/log/log.go\",\"Caller Line\":172}
The logs as they appear in stdout include the hostname and ident as part of the date field. The syslog parser does not appear to be parsing the date correctly. eg:
logs we are seeing:
2020-03-03 10:14:29.000000000 +0000 syslog.user.info: {"time":"2020-03-03T10:14:29Z podname-v1-5cc4869f45-kqf2g servicename[260]:","host":"{\"level\":\"info\",\"msg\":\"http:","ident":"TLS","message":"41282: tls: failed to verify client's certificate: x509: certificate signed by unknown authority\\n\",\"Version\":\"0.0.1\",\"Caller File\":\"/usr/local/go/src/log/log.go\",\"Caller Line\":172}"}
--
Expected behavior
The delimiter for the end of the timestamp field should be recognised and the timestamp/host and ident fields should be parsed seperately.
Your Environment
Runing in kubernetes pods
fluentd-1.3.3 from gcr.io/google-containers/fluentd-elasticsearch:v2.4.0 image which is the latest version available
Your Configuration
<match fluent.**>
@type null
</match>
<source>
@type syslog
port 10514
bind "0.0.0.0"
tag "syslog"
<parse>
@type "syslog"
keep_time_key true
with_priority true
message_format rfc3164
time_format "%Y-%m-%dT%H:%M:%SZ"
</parse>
</source>
<source>
@type syslog
port 24224
bind "0.0.0.0"
protocol_type tcp
tag "syslog"
<parse>
@type "syslog"
keep_time_key true
with_priority true
message_format rfc3164
time_format "%Y-%m-%dT%H:%M:%SZ"
</parse>
</source>
<match **>
@type stdout
<buffer>
@type "memory"
flush_interval 1s
</buffer>
</match>
<match syslog.**>
@type elasticsearch
log_level info
include_tag_key true
host "elasticsearch"
port 9200
logstash_format true
buffer_chunk_limit 2M
buffer_queue_limit 8
flush_interval 5s
max_retry_wait 30
disable_retry_limit
num_threads 2
<buffer>
flush_thread_count 2
flush_interval 5s
retry_forever
retry_max_interval 30
chunk_limit_size 2M
queue_limit_length 8
</buffer>
</match>
Your Error Log
$ kubectl logs -n logging fluentd-es-6f9d5c58dc-twxwz
2020-03-03 10:14:28 +0000 [info]: starting fluentd-1.3.3 without supervision pid=1 ruby="2.3.3"
2020-03-03 10:14:28 +0000 [info]: gem 'fluent-plugin-concat' version '2.3.0'
2020-03-03 10:14:28 +0000 [info]: gem 'fluent-plugin-detect-exceptions' version '0.0.11'
2020-03-03 10:14:28 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '3.0.2'
2020-03-03 10:14:28 +0000 [info]: gem 'fluent-plugin-kubernetes_metadata_filter' version '2.1.6'
2020-03-03 10:14:28 +0000 [info]: gem 'fluent-plugin-multi-format-parser' version '1.0.0'
2020-03-03 10:14:28 +0000 [info]: gem 'fluent-plugin-prometheus' version '1.3.0'
2020-03-03 10:14:28 +0000 [info]: gem 'fluent-plugin-systemd' version '1.0.1'
2020-03-03 10:14:28 +0000 [info]: gem 'fluentd' version '1.3.3'
2020-03-03 10:14:28 +0000 [info]: adding match pattern="fluent.**" type="null"
2020-03-03 10:14:28 +0000 [info]: adding match pattern="**" type="stdout"
2020-03-03 10:14:28 +0000 [info]: adding match pattern="syslog.**" type="elasticsearch"
2020-03-03 10:14:28 +0000 [warn]: 'log_level' is deprecated parameter name. use '@log_level' instead.
2020-03-03 10:14:28 +0000 [warn]: 'log_level' is deprecated parameter name. use '@log_level' instead.
2020-03-03 10:14:28 +0000 [info]: 'flush_interval' is configured at out side of <buffer>. 'flush_mode' is set to 'interval' to keep existing behaviour
2020-03-03 10:14:28 +0000 [warn]: Could not connect Elasticsearch or obtain version. Assuming Elasticsearch 5.
2020-03-03 10:14:28 +0000 [info]: adding source type="syslog"
2020-03-03 10:14:28 +0000 [info]: adding source type="syslog"
2020-03-03 10:14:28 +0000 [info]: using configuration file: <ROOT>
<match fluent.**>
@type null
</match>
<source>
@type syslog
port 10514
bind "0.0.0.0"
tag "syslog"
<parse>
@type "syslog"
keep_time_key true
with_priority true
message_format rfc3164
time_format "%Y-%m-%dT%H:%M:%SZ"
</parse>
</source>
<source>
@type syslog
port 24224
bind "0.0.0.0"
protocol_type tcp
tag "syslog"
<parse>
@type "syslog"
keep_time_key true
with_priority true
message_format rfc3164
time_format "%Y-%m-%dT%H:%M:%SZ"
</parse>
</source>
<match **>
@type stdout
<buffer>
@type "memory"
flush_interval 1s
</buffer>
</match>
<match syslog.**>
@type elasticsearch
log_level info
include_tag_key true
host "elasticsearch"
port 9200
logstash_format true
buffer_chunk_limit 2M
buffer_queue_limit 8
flush_interval 5s
max_retry_wait 30
disable_retry_limit
num_threads 2
<buffer>
flush_thread_count 2
flush_interval 5s
retry_forever
retry_max_interval 30
chunk_limit_size 2M
queue_limit_length 8
</buffer>
</match>
</ROOT>
2020-03-03 10:14:28 +0000 [info]: starting fluentd worker pid=1 ppid=0 worker=0
2020-03-03 10:14:28 +0000 [info]: listening syslog socket on 0.0.0.0:24224 with tcp
2020-03-03 10:14:28 +0000 [info]: listening syslog socket on 0.0.0.0:10514 with udp
2020-03-03 10:14:28 +0000 [info]: fluentd worker is now running worker=0
2020-03-03 10:14:29.000000000 +0000 syslog.user.info: {"time":"2020-03-03T10:14:29Z podname-v1-5cc4869f45-kqf2g servicename[260]:","host":"{\"level\":\"info\",\"msg\":\"http:","ident":"TLS","message":"41282: tls: failed to verify client's certificate: x509: certificate signed by unknown authority\\n\",\"Version\":\"0.0.1\",\"Caller File\":\"/usr/local/go/src/log/log.go\",\"Caller Line\":172}"}
Additional context
Describe the bug
The syslog parser is including the hostname and ident as part of the time field
To Reproduce
We are sending syslog messages in rfc3164 format but with timestamps in "%Y-%m-%dT%H:%M:%SZ" format eg:
The logs as they appear in stdout include the hostname and ident as part of the date field. The syslog parser does not appear to be parsing the date correctly. eg:
logs we are seeing:
--
Expected behavior
The delimiter for the end of the timestamp field should be recognised and the timestamp/host and ident fields should be parsed seperately.
Your Environment
Runing in kubernetes pods
fluentd-1.3.3 from gcr.io/google-containers/fluentd-elasticsearch:v2.4.0 image which is the latest version available
Your Configuration
Your Error Log
Additional context