Tendermint node restart after app failure sometimes doesn't work #350
Description
I have 4 nodes running. Each of them is mapped to tmsp app. In one of the tmsp app tree got corrupted and merkle root hash was different. All the other nodes started giving following warning
NOTE[01-03|09:31:42] enterNewRound(64429/20). Current: 64429/19/RoundStepPrecommit module=consensus
WARN[01-03|09:31:43] enterPrevote: ProposalBlock is invalid module=consensus error="Wrong Block.Header.AppHash. Expected 7820B976E6E7FC55A9124B7D94B6F31E8FFE0179, got 87A01D3C30FAA6F239AC114F6D24840447106690"
NOTE[01-03|09:31:54] enterNewRound(64429/21). Current: 64429/20/RoundStepPrecommit module=consensus
After this, we are not able to produce any new block. The system is never able to recover. We need to reset everything and start it from scratch.