e2e: testnet fails to start #6208
Description
Tendermint version (use tendermint version or git rev-parse --verify HEAD if installed from source):
master - e2e testnets
What happened:
Chain failed to start. It seems the three starting validators can't produce the first block within the timeout.
I[2021-03-04|09:21:25.858] Waiting for initial height 1...
E[2021-03-04|09:21:45.922] chain stalled at unknown height
All three validators receive a proposal
validator01 |�[0m I[2021-03-04|09:21:26.792] received proposal module=consensus proposal="Proposal{1/0 (FF1FF7F6580E60319526683D93BE198DAB21BF5F5503B3E158885B8A629414FD:2:164E1D0660E0, -1) 2FAC5666B49E @ 2021-03-04T09:21:26.76167382Z}"
validator02 |�[0m I[2021-03-04|09:21:26.863] received proposal module=consensus proposal="Proposal{1/0 (FF1FF7F6580E60319526683D93BE198DAB21BF5F5503B3E158885B8A629414FD:2:164E1D0660E0, -1) 2FAC5666B49E @ 2021-03-04T09:21:26.76167382Z}"
validator03 |�[0m I[2021-03-04|09:21:26.763] received proposal module=consensus proposal="Proposal{1/0 (FF1FF7F6580E60319526683D93BE198DAB21BF5F5503B3E158885B8A629414FD:2:164E1D0660E0, -1) 2FAC5666B49E @ 2021-03-04T09:21:26.76167382Z}"
They all send prevotes
validator01 |�[0m I[2021-03-04|09:21:21.545] signed and pushed vote module=consensus height=1 round=0 vote="Vote{2:D3B5BFD2AEAC 1/00/SIGNED_MSG_TYPE_PREVOTE(Prevote) 000000000000 7326E9023629 @ 2021-03-04T09:21:21.543659674Z}"
validator02 |�[0m I[2021-03-04|09:21:25.682] signed and pushed vote module=consensus height=1 round=0 vote="Vote{1:99F30C473E41 1/00/SIGNED_MSG_TYPE_PREVOTE(Prevote) 000000000000 2B2332FF684C @ 2021-03-04T09:21:25.680681969Z}"
validator03 |�[0m I[2021-03-04|09:21:26.769] signed and pushed vote module=consensus height=1 round=0 vote="Vote{0:A54399B792EC 1/00/SIGNED_MSG_TYPE_PREVOTE(Prevote) FF1FF7F6580E 2AED7F1A78B2 @ 2021-03-04T09:21:26.768116926Z}"
validator 02 even sends a precommit message
But somehow they fail to deliver votes to the proposal and all receive timeouts
validator01 |�[0m I[2021-03-04|09:21:21.542] Timed out module=consensus dur=3s height=1 round=0 step=RoundStepPropose
validator02 |�[0m I[2021-03-04|09:21:25.676] Timed out module=consensus dur=3s height=1 round=0 step=RoundStepPropose
validator03 |�[0m I[2021-03-04|09:21:29.760] Timed out module=consensus dur=3s height=1 round=0 step=RoundStepPropose
There is at least 20 seconds, between all validators starting up and the test net finishing where a block isn't able to be created