It would be nice to recover from an app restart without having to restart tendermint - p2p reactors not dependent on the app could thus keep running. Implementing this requires changes in tmsp, consensus, and mempool:

• tmsp client:
  • when a request fails, flush the queue, close the connection, and drop back into the connect loop to attempt reconnect
  • expose a channel to alert consumer when we have reconnected
• consensus:
  • when execBlock fails due to TMSP request failure, stop the consensus state
  • wait for the reconnection and restart consensus state
• mempool:
  • stop caching txs while the app is disconnected or anti-replay will cause problems
  • eventually, buffer txs received while the app is off and replay when it comes on (buffer mempool txs for replay if app restarts #256)

If we can only stop the consensus state and mempool (and not the reactors) then we don't have to worry about re-adding peers (eg. looping over Switch.Peers and running AddPeer). However, some peer messages wont be able to be processed and may queue up.

Related

• Ability to replay txs from LevelDB on startup #227
• TMSP handshake message #211
• Better err/warn output upon app connection failure #199