@adizere, @andynog I tried rolling updates to comet1.0 build, here are my first set of observations:

Repo: trufnetwork/kwil-db#760

Consensus:

No consensus breakage with a rolling upgrades.

P2P panics:

nodes on older versions panic as they receive messages from the nodes on newer versions, potentially due to the changes in this PR which introduces this new message (HasProposalBlockPartMessage), that the old nodes are not aware of. The panic closes the connection to the peer and reestablish it with every block. So there is significant p2p connection activity happening with every block.

• Is panic necessary here? can the protocol be changed to ignore these messages and not cause the old nodes to panic?

• What are the consequences of not consuming these HasProposalBlockPartMessage messages? Are there any liveness issues? I have not noticed any, but want to confirm. Also, Is coordinated upgrades the recommended approach for this?

• Upgrade guidelines doc doesn't mention about this breaking nature in P2P layer, you might wanna add it there.

Node with older versions: Panicking

Peer disconnects with every block:

Logging issues:

There are lots of noise from the mempool, which I don't think is necessary. Below are the screenshots

This is one other log message I used to observe a lot while doing stress test and even on smaller loads. But I haven't observed this anymore on comet1.0, looks like u guys fixed it. But I want to bring it up.