The fact that the PrivValidator object updates non-atomically with the vote being received by the state machine means there is a halt condition.

The halt condition was originally dealt with using replay/WAL (https://github.com/tendermint/tendermint/blob/master/consensus/replay_test.go#L59) - if we crash after signing a precommit but before we process it, we cannot recover because the PrivValidator won't let us sign again. Even with the current cswal, writing to PrivValidator is non-atomic with writing to the wal, so the halt can still happen if we crash after signing and before writing.

To fix this the PrivValidator should track the last thing signed and return that signature if the height/round/step match.

Also, I think there's also a bug in the PrivValidator.SignVote, namely,

        if privVal.LastRound == vote.Round && privVal.LastStep > voteToStep(vote) {
            return errors.New("Step regression in SignVote")
        }

The privVal.LastStep > voteToStep(vote) means we could sign a precommit even if we already precommit. It should be >=. Instead, let's make it:

    if privVal.LastRound == vote.Round {
        if privVal.LastStep > voteToStep(vote) {
            return errors.New("Step regression in SignVote")
        } else if privVal.LastStep == voteToStep(vote) {
            vote.Signature = privVal.LastSignature
        }
    }

EDIT: this issue used to consider eliminating the need for the WAL and moving catchup/replay to the DB. Instead, we'll just use a standard process to truncate the WAL.