fatal error: concurrent map read and map write #3509
Description
Tendermint version (use tendermint version or git rev-parse --verify HEAD if installed from source):
0.31.1
ABCI app (name for built-in, URL for self-written if it's publicly available):
Environment:
- OS (e.g. from /etc/os-release): ec2
- Install tools:
- Others:
What happened:
tendermint crashed when there's about 9k unconfirmed txs in mempool
What you expected to happen:
4 validator nodes with some traffic, tendermint will crash with concurrent map read/write error
Have you tried the latest version: yes
How to reproduce it (as minimally and precisely as possible):
Logs (paste a small part showing an error (< 10 lines) or link a pastebin, gist, etc. containing more of the log file):
I[2019-03-29|23:51:40.361] Timed out module=consensus dur=2s height=1614 round=0 step=RoundStepPropose
E[2019-03-29|23:51:42.006] Error on broadcastTxCommit module=rpc err="Timed out waiting for tx to be included in a block"
fatal error: concurrent map read and map write
goroutine 205042 [running]:
runtime.throw(0xe83c9f, 0x21)
/home/jslave/goroot/src/runtime/panic.go:608 +0x72 fp=0xc00157bbb0 sp=0xc00157bb80 pc=0x42c432
runtime.mapaccess2(0xd3b340, 0xc002163890, 0xc00157bc58, 0xfca081abf0ce081f, 0x39d3c0655bb83629)
/home/jslave/goroot/src/runtime/map.go:453 +0x223 fp=0xc00157bbf8 sp=0xc00157bbb0 pc=0x40da33
github.com/tendermint/tendermint/mempool.(*Mempool).CheckTxWithInfo(0xc0000d0d10, 0xc0057e07e0, 0x10e, 0x10e, 0x0, 0x3, 0x0, 0x0)
/home/jslave/workspace/02.Fetch.Binary/go/src/github.com/tendermint/tendermint/mempool/mempool.go:384 +0x6f6 fp=0xc00157bcd8 sp=0xc00157bbf8 pc=0xb73076
github.com/tendermint/tendermint/mempool.(*MempoolReactor).Receive(0xc0008ac4e0, 0x4071500000000030, 0x10a4bc0, 0xc001b2e600, 0xc00158c000, 0x115, 0x1000)
/home/jslave/workspace/02.Fetch.Binary/go/src/github.com/tendermint/tendermint/mempool/reactor.go:170 +0x710 fp=0xc00157bdc8 sp=0xc00157bcd8 pc=0xb78080
github.com/tendermint/tendermint/p2p.createMConnection.func1(0xc001560b30, 0xc00158c000, 0x115, 0x1000)
/home/jslave/workspace/02.Fetch.Binary/go/src/github.com/tendermint/tendermint/p2p/peer.go:374 +0x1ba fp=0xc00157be68 sp=0xc00157bdc8 pc=0x97ba9a
github.com/tendermint/tendermint/p2p/conn.(*MConnection).recvRoutine(0xc00155ef20)
/home/jslave/workspace/02.Fetch.Binary/go/src/github.com/tendermint/tendermint/p2p/conn/connection.go:599 +0x8ad fp=0xc00157bfd8 sp=0xc00157be68 pc=0x92852d
runtime.goexit()
/home/jslave/goroot/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc00157bfe0 sp=0xc00157bfd8 pc=0x45be01
created by github.com/tendermint/tendermint/p2p/conn.(*MConnection).OnStart
/home/jslave/workspace/02.Fetch.Binary/go/src/github.com/tendermint/tendermint/p2p/conn/connection.go:210 +0x1a2
goroutine 1 [select (no cases)]:
github.com/tendermint/tendermint/cmd/tendermint/commands.NewRunNodeCmd.func1(0xc0000c1200, 0xc00014b3c0, 0x0, 0x2, 0x0, 0x0)
/home/jslave/workspace/02.Fetch.Binary/go/src/github.com/tendermint/tendermint/cmd/tendermint/commands/run_node.go:71 +0x2fd
github.com/tendermint/tendermint/vendor/github.com/spf13/cobra.(*Command).execute(0xc0000c1200, 0xc00014b380, 0x2, 0x2, 0xc0000c1200, 0xc00014b380)
Config (you can paste only the changes you've made):
node command runtime flags:
/dump_consensus_state output for consensus bugs
Anything else we need to know:
Seems start to seeing this issue with newer version.
Thanks for taking a look!