Describe the bug
The parser_syslog plugin accepts a parameter parser_type (equal to string or regexp):
|
config_param :parser_type, :enum, list: [:regexp, :string], default: :regexp |
This configuration key has a naming conflict with a method that's also called parser_type defined here:
When used in conjunction with an input plugin that changes how it calls parse() depending on the value of parser_type(), it receives a value that's outside PARSER_TYPES, namely string or regexp rather than say :text_per_line
I reproduced this with input_exec input plugin, which falls back to calling parse() with the entire input buffer if the parser plugin doesn't return :text_per_line from parser_type(). This causes the parser to get confused (all lines end up in the message) if the exec exits, or seems to completely not parse anything if the exec'd process is long-running (presumably because the fallback buffer relies on the exec'd command exiting before it passes the buffer over).
To Reproduce
This config standalone is enough to reproduce the issue:
<source>
@type exec
command printf '%bMar 16 21:28:01 hostname CRON[6477]: blah\nMar 16 21:28:11 hostname puppet-agent[1388]: foo'
tag test
<parse>
@type syslog
with_priority false
parser_type string
message_format rfc3164
</parse>
</source>
<match *>
@type stdout
</match>
Expected behavior
The above config should output:
2021-03-16 21:28:01.000000000 +0000 test: {"host":"hostname","ident":"CRON","pid":"6477","message":"blah"}
2021-03-16 21:28:11.000000000 +0000 test: {"host":"hostname","ident":"puppet-agent","pid":"1388","message":"foo"}
Instead it outputs:
1970-01-01 00:00:00.000000000 +0000 test: {"host":"hostname","ident":"CRON","pid":"6477","message":"blah\nMar 16 21:28:11 hostname puppet-agent[1388]: foo"}
Your Environment
- Fluentd or td-agent version: td-agent 4.1.0 fluentd 1.12.1 (e3effa3)
- Operating system: Debian Buster
- Kernel version: 4.19.118
Your Configuration
Similar to the above, though with a different command that's long-running and streams syslog.
Your Error Log
2021-03-16 09:16:37 +0000 [info]: fluent/log.rb:329:info: parsing config file is succeeded path="/etc/td-agent/td-agent.conf"
2021-03-16 09:16:37 +0000 [info]: fluent/log.rb:329:info: gem 'fluent-plugin-elasticsearch' version '4.3.3'
2021-03-16 09:16:37 +0000 [info]: fluent/log.rb:329:info: gem 'fluent-plugin-flowcounter-simple' version '0.1.0'
2021-03-16 09:16:37 +0000 [info]: fluent/log.rb:329:info: gem 'fluent-plugin-kafka' version '0.16.0'
2021-03-16 09:16:37 +0000 [info]: fluent/log.rb:329:info: gem 'fluent-plugin-prometheus' version '1.8.5'
2021-03-16 09:16:37 +0000 [info]: fluent/log.rb:329:info: gem 'fluent-plugin-prometheus_pushgateway' version '0.0.2'
2021-03-16 09:16:37 +0000 [info]: fluent/log.rb:329:info: gem 'fluent-plugin-record-modifier' version '2.1.0'
2021-03-16 09:16:37 +0000 [info]: fluent/log.rb:329:info: gem 'fluent-plugin-rewrite-tag-filter' version '2.3.0'
2021-03-16 09:16:37 +0000 [info]: fluent/log.rb:329:info: gem 'fluent-plugin-s3' version '1.5.1'
2021-03-16 09:16:37 +0000 [info]: fluent/log.rb:329:info: gem 'fluent-plugin-sd-dns' version '0.1.0'
2021-03-16 09:16:37 +0000 [info]: fluent/log.rb:329:info: gem 'fluent-plugin-stdin' version '0.1.2'
2021-03-16 09:16:37 +0000 [info]: fluent/log.rb:329:info: gem 'fluent-plugin-systemd' version '1.0.2'
2021-03-16 09:16:37 +0000 [info]: fluent/log.rb:329:info: gem 'fluent-plugin-td' version '1.1.0'
2021-03-16 09:16:37 +0000 [info]: fluent/log.rb:329:info: gem 'fluent-plugin-webhdfs' version '1.4.0'
2021-03-16 09:16:37 +0000 [info]: fluent/log.rb:329:info: gem 'fluentd' version '1.12.1'
2021-03-16 09:16:37 +0000 [trace]: fluent/log.rb:286:trace: registered output plugin 'stdout'
2021-03-16 09:16:38 +0000 [trace]: fluent/log.rb:286:trace: registered buffer plugin 'memory'
2021-03-16 09:16:38 +0000 [trace]: fluent/log.rb:286:trace: registered formatter plugin 'stdout'
2021-03-16 09:16:38 +0000 [trace]: fluent/log.rb:286:trace: registered formatter plugin 'json'
2021-03-16 09:16:39 +0000 [trace]: fluent/log.rb:286:trace: registered input plugin 'exec'
2021-03-16 09:16:39 +0000 [trace]: fluent/log.rb:286:trace: registered parser plugin 'syslog'
2021-03-16 09:16:39 +0000 [debug]: fluent/log.rb:308:debug: No fluent logger for internal event
2021-03-16 09:16:39 +0000 [info]: fluent/log.rb:329:info: using configuration file: <ROOT>
<source>
@type exec
command "printf \'%bMar 16 21:28:01 hostname CRON[6477]: blah\\nMar 16 21:28:11 hostname puppet-agent[1388]: foo\'"
tag "test"
<parse>
@type "syslog"
with_priority false
parser_type string
message_format rfc3164
</parse>
</source>
<match *>
@type stdout
</match>
</ROOT>
2021-03-16 09:16:39 +0000 [info]: fluent/log.rb:329:info: starting fluentd-1.12.1 pid=3683 ruby="2.7.2"
2021-03-16 09:16:39 +0000 [info]: fluent/log.rb:329:info: spawn command to main: cmdline=["/opt/td-agent/bin/ruby", "-Eascii-8bit:ascii-8bit", "/usr/sbin/td-agent", "-vv", "--under-supervisor"]
2021-03-16 09:16:50 +0000 [info]: fluent/log.rb:329:info: adding match pattern="*" type="stdout"
2021-03-16 09:16:51 +0000 [trace]: #0 fluent/log.rb:286:trace: registered output plugin 'stdout'
2021-03-16 09:16:51 +0000 [trace]: #0 fluent/log.rb:286:trace: registered buffer plugin 'memory'
2021-03-16 09:16:51 +0000 [trace]: #0 fluent/log.rb:286:trace: registered formatter plugin 'stdout'
2021-03-16 09:16:51 +0000 [trace]: #0 fluent/log.rb:286:trace: registered formatter plugin 'json'
2021-03-16 09:16:51 +0000 [info]: fluent/log.rb:329:info: adding source type="exec"
2021-03-16 09:16:51 +0000 [trace]: #0 fluent/log.rb:286:trace: registered input plugin 'exec'
2021-03-16 09:16:51 +0000 [trace]: #0 fluent/log.rb:286:trace: registered parser plugin 'syslog'
2021-03-16 09:16:51 +0000 [debug]: #0 fluent/log.rb:308:debug: No fluent logger for internal event
2021-03-16 09:16:51 +0000 [info]: #0 fluent/log.rb:329:info: starting fluentd worker pid=3688 ppid=3683 worker=0
2021-03-16 09:16:51 +0000 [debug]: #0 fluent/log.rb:308:debug: Executing command title=:exec_input spawn=[{}, "printf '%bMar 16 21:28:01 hostname CRON[6477]: blah\\nMar 16 21:28:11 hostname puppet-agent[1388]: foo'"] mode=[:read] stderr=:discard
2021-03-16 09:16:51 +0000 [info]: #0 fluent/log.rb:329:info: fluentd worker is now running worker=0
1970-01-01 00:00:00.000000000 +0000 test: {"host":"hostname","ident":"CRON","pid":"6477","message":"blah\nMar 16 21:28:11 hostname puppet-agent[1388]: foo"}
Describe the bug
The parser_syslog plugin accepts a parameter
parser_type(equal tostringorregexp):fluentd/lib/fluent/plugin/parser_syslog.rb
Line 59 in 436c7da
This configuration key has a naming conflict with a method that's also called
parser_typedefined here:fluentd/lib/fluent/plugin/parser.rb
Line 104 in 436c7da
When used in conjunction with an input plugin that changes how it calls
parse()depending on the value ofparser_type(), it receives a value that's outsidePARSER_TYPES, namelystringorregexprather than say:text_per_lineI reproduced this with input_exec input plugin, which falls back to calling
parse()with the entire input buffer if the parser plugin doesn't return:text_per_linefromparser_type(). This causes the parser to get confused (all lines end up in the message) if the exec exits, or seems to completely not parse anything if the exec'd process is long-running (presumably because the fallback buffer relies on the exec'd command exiting before it passes the buffer over).To Reproduce
This config standalone is enough to reproduce the issue:
Expected behavior
The above config should output:
Instead it outputs:
Your Environment
Your Configuration
Similar to the above, though with a different command that's long-running and streams syslog.
Your Error Log