I am still experiencing this issue running gaiad v.2.0.3 with tendermint/kms v0.7.0.

Experiencing the same experience as described in kms issue-116.

After restarting my remote signer I see the following in my logs.

E[2019-12-29|19:14:16.928] SignerListener: Ping timeout                 module=privval
I[2019-12-29|19:14:17.473] Timed out                                    module=consensus dur=4.96053053s height=133946 round=0 step=RoundStepNewHeight
E[2019-12-29|19:14:19.929] SignerListener: Ping timeout                 module=privval
E[2019-12-29|19:14:22.929] SignerClient::GetPubKey                      module=privval err="endpoint connection timed out"
E[2019-12-29|19:14:22.929] CONSENSUS FAILURE!!!                         module=consensus err="runtime error: invalid memory address or nil pointer dereference" stack="goroutine 976 [running]:\nruntime/debug.Stack(0xc000eb9358, 0x102fa00, 0x1d768e0)\n\t/usr/local/go/src/runtime/debug/stack.go:24 +0x9d\ngithub.com/tendermint/tendermint/consensus.(*ConsensusState).receiveRoutine.func2(0xc000074700, 0x138f910)\n\t/go/pkg/mod/github.com/tendermint/tendermint@v0.32.7/consensus/state.go:602 +0x57\npanic(0x102fa00, 0x1d768e0)\n\t/usr/local/go/src/runtime/panic.go:679 +0x1b2\ngithub.com/tendermint/tendermint/consensus.(*ConsensusState).signAddVote(0xc000074700, 0x20b02, 0xc003f97e00, 0x20, 0x20, 0x1, 0xc003f97e20, 0x20, 0x20, 0xa)\n\t/go/pkg/mod/github.com/tendermint/tendermint@v0.32.7/consensus/state.go:1745 +0x585\ngithub.com/tendermint/tendermint/consensus.(*ConsensusState).enterPrecommit(0xc000074700, 0x20b3a, 0x0)\n\t/go/pkg/mod/github.com/tendermint/tendermint@v0.32.7/consensus/state.go:1155 +0x12ec\ngithub.com/tendermint/tendermint/consensus.(*ConsensusState).addVote(0xc000074700, 0xc000d8b400, 0xc000a574d0, 0x28, 0xc000eb9a38, 0xbff162, 0x0)\n\t/go/pkg/mod/github.com/tendermint/tendermint@v0.32.7/consensus/state.go:1663 +0xdfa\ngithub.com/tendermint/tendermint/consensus.(*ConsensusState).tryAddVote(0xc000074700, 0xc000d8b400, 0xc000a574d0, 0x28, 0x4922ea91d59af208, 0x109, 0x101)\n\t/go/pkg/mod/github.com/tendermint/tendermint@v0.32.7/consensus/state.go:1527 +0x59\ngithub.com/tendermint/tendermint/consensus.(*ConsensusState).handleMsg(0xc000074700, 0x1545be0, 0xc004be4160, 0xc000a574d0, 0x28)\n\t/go/pkg/mod/github.com/tendermint/tendermint@v0.32.7/consensus/state.go:691 +0x252\ngithub.com/tendermint/tendermint/consensus.(*ConsensusState).receiveRoutine(0xc000074700, 0x0)\n\t/go/pkg/mod/github.com/tendermint/tendermint@v0.32.7/consensus/state.go:633 +0x6eb\ncreated by github.com/tendermint/tendermint/consensus.(*ConsensusState).OnStart\n\t/go/pkg/mod/github.com/tendermint/tendermint@v0.32.7/consensus/state.go:334 +0x13a\n"
I[2019-12-29|19:14:22.929] Stopping baseWAL                             module=consensus wal=/root/.gaiad/data/cs.wal/wal impl=baseWAL
I[2019-12-29|19:14:22.931] Stopping Group                               module=consensus wal=/root/.gaiad/data/cs.wal/wal impl=Group
E[2019-12-29|19:14:25.929] SignerListener: Ping timeout                 module=privval
E[2019-12-29|19:14:28.929] SignerListener: Ping timeout                 module=privval
E[2019-12-29|19:14:31.929] SignerListener: Ping timeout                 module=privval
E[2019-12-29|19:14:34.929] SignerListener: Ping timeout                 module=privval

The node has never recovered from this unless I restart gaiad process.

privval: close and retry RemoteSigner on error #3114

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions