Skip to content

Verbose close connection logging during cluster formation #82360

@ywelsch

Description

@ywelsch

Cluster formation / node joining looks to unexpectedly close connections.

I've observed this while running tests on master branch (e.g. SimpleSearchIT):

[2022-01-10T09:06:14,442][INFO ][o.e.n.Node               ] [[test_SUITE-TEST_WORKER_VM=[--not-gradle--]-CLUSTER_SEED=[7454109354569661897]-HASH=[7B0771AAE426]-cluster[T#1]]] starting ...
[2022-01-10T09:06:14,443][INFO ][o.e.n.Node               ] [[test_SUITE-TEST_WORKER_VM=[--not-gradle--]-CLUSTER_SEED=[7454109354569661897]-HASH=[7B0771AAE426]-cluster[T#2]]] starting ...
[2022-01-10T09:06:14,445][INFO ][o.e.n.Node               ] [[test_SUITE-TEST_WORKER_VM=[--not-gradle--]-CLUSTER_SEED=[7454109354569661897]-HASH=[7B0771AAE426]-cluster[T#3]]] starting ...
[2022-01-10T09:06:14,455][INFO ][o.e.t.TransportService   ] [[test_SUITE-TEST_WORKER_VM=[--not-gradle--]-CLUSTER_SEED=[7454109354569661897]-HASH=[7B0771AAE426]-cluster[T#3]]] publish_address {127.0.0.1:53925}, bound_addresses {[::1]:53922}, {127.0.0.1:53925}
[2022-01-10T09:06:14,455][INFO ][o.e.t.TransportService   ] [[test_SUITE-TEST_WORKER_VM=[--not-gradle--]-CLUSTER_SEED=[7454109354569661897]-HASH=[7B0771AAE426]-cluster[T#1]]] publish_address {127.0.0.1:53924}, bound_addresses {[::1]:53920}, {127.0.0.1:53924}
[2022-01-10T09:06:14,455][INFO ][o.e.t.TransportService   ] [[test_SUITE-TEST_WORKER_VM=[--not-gradle--]-CLUSTER_SEED=[7454109354569661897]-HASH=[7B0771AAE426]-cluster[T#2]]] publish_address {127.0.0.1:53923}, bound_addresses {[::1]:53921}, {127.0.0.1:53923}
[2022-01-10T09:06:14,506][INFO ][o.e.t.TcpTransport       ] [node_s0] close connection exception caught on transport layer [NioSocketChannel{localAddress=/127.0.0.1:53924, remoteAddress=/127.0.0.1:53928}], disconnecting from relevant node: Connection reset
[2022-01-10T09:06:14,510][INFO ][o.e.t.TcpTransport       ] [node_s0] close connection exception caught on transport layer [NioSocketChannel{localAddress=/127.0.0.1:53924, remoteAddress=/127.0.0.1:53927}], disconnecting from relevant node: Connection reset
[2022-01-10T09:06:14,510][INFO ][o.e.t.TcpTransport       ] [node_s1] close connection exception caught on transport layer [NioSocketChannel{localAddress=/127.0.0.1:53923, remoteAddress=/127.0.0.1:53926}], disconnecting from relevant node: Connection reset
[2022-01-10T09:06:14,511][INFO ][o.e.t.TcpTransport       ] [node_s2] close connection exception caught on transport layer [NioSocketChannel{localAddress=/127.0.0.1:53925, remoteAddress=/127.0.0.1:53930}], disconnecting from relevant node: Connection reset
[2022-01-10T09:06:14,512][INFO ][o.e.t.TcpTransport       ] [node_s1] close connection exception caught on transport layer [NioSocketChannel{localAddress=/127.0.0.1:53923, remoteAddress=/127.0.0.1:53929}], disconnecting from relevant node: Connection reset
[2022-01-10T09:06:14,513][INFO ][o.e.t.TcpTransport       ] [node_s2] close connection exception caught on transport layer [NioSocketChannel{localAddress=/127.0.0.1:53925, remoteAddress=/127.0.0.1:53931}], disconnecting from relevant node: Connection reset
[2022-01-10T09:06:14,516][INFO ][o.e.c.c.Coordinator      ] [node_s1] setting initial configuration to VotingConfiguration{mThCwBWdQ9uaMF2-CYDl6w,{bootstrap-placeholder}-node_s2,QCKJYiJwRB-GZAtWV7tEBg}
[2022-01-10T09:06:14,628][INFO ][o.e.c.s.MasterService    ] [node_s1] elected-as-master ([3] nodes joined)[{node_s1}{mThCwBWdQ9uaMF2-CYDl6w}{ZYIwe8tHTFqLSu-mXqH-Zg}{127.0.0.1}{127.0.0.1:53923}{cdfhilmrstw} completing election, {node_s2}{3TGv6GqTQVSxZ9w0Dho0NQ}{EF3qKJ1oSaOxkM41r-E3pg}{127.0.0.1}{127.0.0.1:53925}{cdfhilmrstw} completing election, {node_s0}{QCKJYiJwRB-GZAtWV7tEBg}{vKsINkoSRzihDZ19mrKiFA}{127.0.0.1}{127.0.0.1:53924}{cdfhilmrstw} completing election, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 1, version: 1, delta: master node changed {previous [], current [{node_s1}{mThCwBWdQ9uaMF2-CYDl6w}{ZYIwe8tHTFqLSu-mXqH-Zg}{127.0.0.1}{127.0.0.1:53923}{cdfhilmrstw}]}, added {{node_s2}{3TGv6GqTQVSxZ9w0Dho0NQ}{EF3qKJ1oSaOxkM41r-E3pg}{127.0.0.1}{127.0.0.1:53925}{cdfhilmrstw}, {node_s0}{QCKJYiJwRB-GZAtWV7tEBg}{vKsINkoSRzihDZ19mrKiFA}{127.0.0.1}{127.0.0.1:53924}{cdfhilmrstw}}
[2022-01-10T09:06:14,629][INFO ][o.e.t.TcpTransport       ] [node_s2] close connection exception caught on transport layer [NioSocketChannel{localAddress=/127.0.0.1:53925, remoteAddress=/127.0.0.1:53945}], disconnecting from relevant node: Connection reset
[2022-01-10T09:06:14,629][INFO ][o.e.t.TcpTransport       ] [node_s0] close connection exception caught on transport layer [NioSocketChannel{localAddress=/127.0.0.1:53924, remoteAddress=/127.0.0.1:53942}], disconnecting from relevant node: Connection reset
[2022-01-10T09:06:14,630][INFO ][o.e.t.TcpTransport       ] [node_s0] close connection exception caught on transport layer [NioSocketChannel{localAddress=/127.0.0.1:53924, remoteAddress=/127.0.0.1:53941}], disconnecting from relevant node: Connection reset
[2022-01-10T09:06:14,630][INFO ][o.e.t.TcpTransport       ] [node_s2] close connection exception caught on transport layer [NioSocketChannel{localAddress=/127.0.0.1:53925, remoteAddress=/127.0.0.1:53947}], disconnecting from relevant node: Connection reset
[2022-01-10T09:06:14,630][INFO ][o.e.t.TcpTransport       ] [node_s0] close connection exception caught on transport layer [NioSocketChannel{localAddress=/127.0.0.1:53924, remoteAddress=/127.0.0.1:53943}], disconnecting from relevant node: Connection reset
[2022-01-10T09:06:14,630][INFO ][o.e.t.TcpTransport       ] [node_s2] close connection exception caught on transport layer [NioSocketChannel{localAddress=/127.0.0.1:53925, remoteAddress=/127.0.0.1:53949}], disconnecting from relevant node: Connection reset
[2022-01-10T09:06:14,652][INFO ][o.e.c.c.CoordinationState] [node_s1] cluster UUID set to [tL_byD7ATLKHXJwjeptqow]
[2022-01-10T09:06:14,652][INFO ][o.e.c.c.CoordinationState] [node_s0] cluster UUID set to [tL_byD7ATLKHXJwjeptqow]

Relates #81768

Metadata

Metadata

Assignees

No one assigned

    Labels

    :Distributed/Cluster CoordinationCluster formation and cluster state publication, including cluster membership and fault detection.Team:DistributedMeta label for distributed team.

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions