Describe the bug
We are using EFK stack for logging with our AWS EKS cluster. We have recently upgrade the fluentd version to 1.13.3 and found this error in fluentd pod logs.
[error]: #0 [in_tail_container_logs] Unexpected error raised. Stopping the timer. title=:in_tail_close_watcher error_class=NoMethodError error="undefined method `eof?' for nil:NilClass"
To Reproduce
The logs are flowing but given a period of time, the mentioned issue is creeping up badly.
Expected behavior
NO error
Your Environment
- Fluentd version: v1.13.3
- Operating system: Debian GNU/Linux
- Kernel version:5.4.117-58.216.amzn2.x86_64
Your Configuration
<source>
@type tail
@id in_tail_container_logs
path /var/log/containers/*.log
pos_file /var/log/fluentd-containers.log.pos
tag kubernetes.*
read_from_head true
follow_inodes true
pos_file_compaction_interval 24h
read_bytes_limit_per_second 8192
<parse>
@type "#{ENV['FLUENT_CONTAINER_TAIL_PARSER_TYPE'] || 'json'}"
time_format %Y-%m-%dT%H:%M:%S.%NZ
</parse>
</source>
<match **>
@type aws-elasticsearch-service
@id out_aws_es
@log_level "#{ENV['FLUENT_LOG_LEVEL'] || 'info'}"
include_tag_key true
<endpoint>
url "#{ENV['FLUENT_AWS_ELASTICSEARCH_ENDPOINT']}"
region "#{ENV['FLUENT_AWS_ELASTICSEARCH_REGION']}"
</endpoint>
reload_connections "#{ENV['FLUENT_ELASTICSEARCH_RELOAD_CONNECTIONS'] || 'false'}"
reconnect_on_error "#{ENV['FLUENT_ELASTICSEARCH_RECONNECT_ON_ERROR'] || 'true'}"
reload_on_failure "#{ENV['FLUENT_ELASTICSEARCH_RELOAD_ON_FAILURE'] || 'true'}"
logstash_prefix "#{ENV['FLUENT_ELASTICSEARCH_LOGSTASH_PREFIX'] || 'logstash-apiplatform'}"
logstash_format "#{ENV['FLUENT_ELASTICSEARCH_LOGSTASH_FORMAT'] || 'true'}"
index_name "#{ENV['FLUENT_ELASTICSEARCH_LOGSTASH_INDEX_NAME'] || 'logstash-apiplatform'}"
type_name "#{ENV['FLUENT_ELASTICSEARCH_LOGSTASH_TYPE_NAME'] || 'fluentd'}"
<buffer>
@type file
path /fluentd/log/elastic-buffer
flush_mode interval
flush_thread_count "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_FLUSH_THREAD_COUNT'] || '8'}"
flush_interval "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_FLUSH_INTERVAL'] || '5s'}"
chunk_limit_size "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_CHUNK_LIMIT_SIZE'] || '16M'}"
retry_max_interval "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_RETRY_MAX_INTERVAL'] || '30'}"
retry_timeout 1h
</buffer>
</match>
Your Error Log
2021-09-08 11:57:05 +0000 [error]: #0 [in_tail_container_logs] Unexpected error raised. Stopping the timer. title=:in_tail_close_watcher error_class=NoMethodError error="undefined method `eof?' for nil:NilClass"
2021-09-08 11:57:05 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluentd-1.13.3/lib/fluent/plugin/in_tail.rb:765:in `eof?'
2021-09-08 11:57:05 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluentd-1.13.3/lib/fluent/plugin/in_tail.rb:547:in `block in detach_watcher_after_rotate_wait'
2021-09-08 11:57:05 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluentd-1.13.3/lib/fluent/plugin_helper/timer.rb:80:in `on_timer'
2021-09-08 11:57:05 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in `run_once'
2021-09-08 11:57:05 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in `run'
2021-09-08 11:57:05 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluentd-1.13.3/lib/fluent/plugin_helper/event_loop.rb:93:in `block in start'
2021-09-08 11:57:05 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluentd-1.13.3/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2021-09-08 11:57:05 +0000 [error]: #0 [in_tail_container_logs] Timer detached. title=:in_tail_close_watcher
Additional context
No response
Describe the bug
We are using EFK stack for logging with our AWS EKS cluster. We have recently upgrade the fluentd version to 1.13.3 and found this error in fluentd pod logs.
[error]: #0 [in_tail_container_logs] Unexpected error raised. Stopping the timer. title=:in_tail_close_watcher error_class=NoMethodError error="undefined method `eof?' for nil:NilClass"
To Reproduce
The logs are flowing but given a period of time, the mentioned issue is creeping up badly.
Expected behavior
NO error
Your Environment
Your Configuration
Your Error Log
Additional context
No response