Skip to content

Unexpected error Broken pipe when trying to shutdown fluentd with multiple workers #3066

@philipsabri

Description

@philipsabri

Describe the bug
When trying to shutdown with ctrl+c in the td-agent command promt it crashes and wont shutdown properly.

To Reproduce
Run config and press ctrl+c or shut in the command promt

Expected behavior
Should shutdown properly withouth getting stuck with errors

Your Environment
Windows Server 2019
fluentd 1.10.2
Tried with both td-agent 4.0.0 and 3.7.1

Your Configuration

<system>
    workers                             2
</system>

<worker 0>
    <source>
        @type                           windows_eventlog2
        @id                             windows_eventlog2_0
        read_interval                   2
        tag                             serverwindows.general
        rate_limit                      200
        channels                        x
        parse_description               true
        render_as_xml                   true
        <storage>   
            @type                       local
            persistent                  true
            path                        x
        </storage>
    </source>

    <filter serverwindows.**>
        @type                           record_transformer
        enable_ruby                     true
        <record>
            time                        ${if record["TimeCreated"] then record["TimeCreated"] end}
            fluentd_host                ${hostname}
        </record>
    </filter>

    <match serverwindows.general>
        @type                           kafka2
        brokers                         x
        username                        x
        password                        x
        scram_mechanism                 x
        ssl_ca_cert                     x
        default_topic                   x

        <format>
            @type                       json
        </format>

        <buffer>
            @type                       file
            path                        x
            flush_mode                  interval
            flush_interval              2s
            flush_thread_count          8
            retry_wait                  5s
            chunk_limit_size            5m
            flush_at_shutdown           true
        </buffer>
    </match>

</worker>

<worker 1>
    <source>
        @type                           windows_eventlog2
        @id                             windows_eventlog2_1
        read_interval                   2
        tag                             serverwindows.security
        rate_limit                      200
        channels                        x
        parse_description               true
        render_as_xml                   true
        <storage>
            @type                       local
            persistent                  true
            path                        x
        </storage>
    </source>
	
    <filter serverwindows.**>
        @type                           record_transformer
        enable_ruby                     true
        <record>
            time                        ${if record["TimeCreated"] then record["TimeCreated"] end}
            fluentd_host                ${hostname}

        </record>
    </filter>

    <match serverwindows.security>
        @type                           kafka2
        brokers                         x
        username                        x
        password                        x
        scram_mechanism                 x
        ssl_ca_cert                     x
        default_topic                   x

        <format>
        	@type                       json
        </format>

        <buffer>
            @type                       file
            path                        x
            flush_mode                  interval
            flush_interval              2s
            flush_thread_count          8
            retry_wait                  5s
            chunk_limit_size            5m
            flush_at_shutdown           true
        </buffer>
    </match>
</worker>

Your Error Log

2020-07-07 13:08:04 +0200 [info]: #1 fluentd worker is now running worker=1
2020-07-07 13:08:04 +0200 [info]: #0 fluentd worker is now running worker=0
2020-07-07 13:08:30 +0200 [info]: Received graceful stop
2020-07-07 13:08:30 +0200 [info]: #1 fluentd worker is now stopping worker=1
2020-07-07 13:08:30 +0200 [info]: #1 shutting down fluentd worker worker=1
2020-07-07 13:08:30 +0200 [info]: #1 shutting down input plugin type=:windows_eventlog2 plugin_id="windows_eventlog2_1"
2020-07-07 13:08:30 +0200 [info]: #1 shutting down output plugin type=:kafka2 plugin_id="object:2b15510"
2020-07-07 13:08:30 +0200 [info]: #1 shutting down filter plugin type=:record_transformer plugin_id="object:21a7410"
2020-07-07 13:08:30 +0200 [info]: Worker 1 finished with status 0
Unexpected error Broken pipe
  C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/serverengine-2.2.1/lib/serverengine/multi_spawn_server.rb:51:in `write'
  C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/serverengine-2.2.1/lib/serverengine/multi_spawn_server.rb:51:in `stop'
  C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/serverengine-2.2.1/lib/serverengine/server.rb:104:in `block (2 levels) in install_signal_handlers'
  C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/serverengine-2.2.1/lib/serverengine/signal_thread.rb:96:in `main'
Unexpected error Broken pipe
  C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/serverengine-2.2.1/lib/serverengine/multi_spawn_server.rb:51:in `write'
  C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/serverengine-2.2.1/lib/serverengine/multi_spawn_server.rb:51:in `stop'
  C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/serverengine-2.2.1/lib/serverengine/server.rb:104:in `block (2 levels) in install_signal_handlers'
  C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/serverengine-2.2.1/lib/serverengine/signal_thread.rb:96:in `main'
Unexpected error Broken pipe
  C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/serverengine-2.2.1/lib/serverengine/multi_spawn_server.rb:51:in `write'
  C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/serverengine-2.2.1/lib/serverengine/multi_spawn_server.rb:51:in `stop'
  C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/serverengine-2.2.1/lib/serverengine/server.rb:104:in `block (2 levels) in install_signal_handlers'
  C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/serverengine-2.2.1/lib/serverengine/signal_thread.rb:96:in `main'
Unexpected error Broken pipe
  C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/serverengine-2.2.1/lib/serverengine/multi_spawn_server.rb:51:in `write'
  C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/serverengine-2.2.1/lib/serverengine/multi_spawn_server.rb:51:in `stop'
  C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/serverengine-2.2.1/lib/serverengine/server.rb:104:in `block (2 levels) in install_signal_handlers'
  C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/serverengine-2.2.1/lib/serverengine/signal_thread.rb:96:in `main'

After the first ctrl+c it stops at 2020-07-07 13:08:30 +0200 [info]: Worker 1 finished with status 0 and then for every ctrl+c it gives the Unexpected error Broken pipe. If I wait after the first ctrl+c it nothing will happenes.

It works fine when restarting and shuting it down via services.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions