Possible deadlock in StopPeerForError -> RemovePeer #4591
Description
Tendermint version (use tendermint version or git rev-parse --verify HEAD if installed from source): v0.33.2
ABCI app (name for built-in, URL for self-written if it's publicly available): https://github.com/MinterTeam/minter-go-node
Environment:
- OS (e.g. from /etc/os-release): Debian GNU/Linux 10 (buster)
What happened: Node cannot be terminated by SIGTERM after some time of operating. Many chan send-ing goroutines found in kill -6 output.
Examples (there are 100+ of them in logs):
goroutine 122508 [chan send, 2320 minutes]:
github.com/tendermint/tendermint/blockchain/v1.(*BlockchainReactor).RemovePeer(0xc03ad5a600, 0x14098c0, 0xc06b9bb5c0, 0xf6dca0, 0xc00007e0f0)
/go/pkg/mod/github.com/tendermint/tendermint@v0.33.2/blockchain/v1/reactor.go:211 +0xef
goroutine 3759947 [chan send, 2330 minutes]:
github.com/tendermint/tendermint/blockchain/v1.(*BlockchainReactor).RemovePeer(0xc03ad5a600, 0x14098c0, 0xc026134a80, 0xf6dca0, 0xc00007e0f0)
/go/pkg/mod/github.com/tendermint/tendermint@v0.33.2/blockchain/v1/reactor.go:211 +0xef
github.com/tendermint/tendermint/p2p.(*Switch).stopAndRemovePeer(0xc0002d57a0, 0x14098c0, 0xc026134a80, 0xf6dca0, 0xc00007e0f0)
/go/pkg/mod/github.com/tendermint/tendermint@v0.33.2/p2p/switch.go:356 +0x11c
github.com/tendermint/tendermint/p2p.(*Switch).StopPeerForError(0xc0002d57a0, 0x14098c0, 0xc026134a80, 0xf6dca0, 0xc00007e0f0)
/go/pkg/mod/github.com/tendermint/tendermint@v0.33.2/p2p/switch.go:325 +0x127
github.com/tendermint/tendermint/p2p.createMConnection.func2(0xf6dca0, 0xc00007e0f0)
/go/pkg/mod/github.com/tendermint/tendermint@v0.33.2/p2p/peer.go:390 +0x4e
github.com/tendermint/tendermint/p2p/conn.(*MConnection).stopForError(0xc05aad5600, 0xf6dca0, 0xc00007e0f0)
/go/pkg/mod/github.com/tendermint/tendermint@v0.33.2/p2p/conn/connection.go:341 +0x76
github.com/tendermint/tendermint/p2p/conn.(*MConnection).recvRoutine(0xc05aad5600)
/go/pkg/mod/github.com/tendermint/tendermint@v0.33.2/p2p/conn/connection.go:596 +0xd2d
created by github.com/tendermint/tendermint/p2p/conn.(*MConnection).OnStart
/go/pkg/mod/github.com/tendermint/tendermint@v0.33.2/p2p/conn/connection.go:230 +0x1d9
TrapSignal routine is also asleep:
goroutine 743 [chan send]:
github.com/tendermint/tendermint/blockchain/v1.(*BlockchainReactor).RemovePeer(0xc03ad5a600, 0x14098c0, 0xc05aac0f00, 0x0, 0x0)
/go/pkg/mod/github.com/tendermint/tendermint@v0.33.2/blockchain/v1/reactor.go:211 +0xef
github.com/tendermint/tendermint/p2p.(*Switch).stopAndRemovePeer(0xc0002d57a0, 0x14098c0, 0xc05aac0f00, 0x0, 0x0)
/go/pkg/mod/github.com/tendermint/tendermint@v0.33.2/p2p/switch.go:356 +0x11c
github.com/tendermint/tendermint/p2p.(*Switch).OnStop(0xc0002d57a0)
/go/pkg/mod/github.com/tendermint/tendermint@v0.33.2/p2p/switch.go:243 +0x9f
github.com/tendermint/tendermint/libs/service.(*BaseService).Stop(0xc0002d57a0, 0x0, 0x0)
/go/pkg/mod/github.com/tendermint/tendermint@v0.33.2/libs/service/service.go:167 +0x2f4
github.com/tendermint/tendermint/node.(*Node).OnStop(0xc03aea6000)
/go/pkg/mod/github.com/tendermint/tendermint@v0.33.2/node/node.go:816 +0xb9
github.com/tendermint/tendermint/libs/service.(*BaseService).Stop(0xc03aea6000, 0x4000, 0xc089b763c0)
/go/pkg/mod/github.com/tendermint/tendermint@v0.33.2/libs/service/service.go:167 +0x2f4
github.com/MinterTeam/minter-go-node/cmd/minter/cmd.runNode.func4()
/go/src/github.com/MinterTeam/minter-go-node/cmd/minter/cmd/node.go:109 +0x4c
github.com/tendermint/tendermint/libs/os.TrapSignal.func1(0xc03a5e7080, 0x7f9aa292df60, 0xc039cb94e0, 0xc039cb9500)
/go/pkg/mod/github.com/tendermint/tendermint@v0.33.2/libs/os/os.go:24 +0x109
created by github.com/tendermint/tendermint/libs/os.TrapSignal
/go/pkg/mod/github.com/tendermint/tendermint@v0.33.2/libs/os/os.go:20 +0xe6
What you expected to happen: Node should terminate correctly.
Have you tried the latest version: yes