spec: overview of p2p in v0.34 by jmalicevic · Pull Request #9120 · tendermint/tendermint

jmalicevic · 2022-07-28T12:48:25Z

Initial overview of the p2p layer in Tenderming v0.34.

cason

I don't have a full comprehension of this implementation yet, so my comments focus more on the switch implementation.

General recommendation: try to use line breaks more often, in particular, split long lines into several smaller lines (single line breaks don't have formatting effect in the MD syntax). This facilitate adding comments and suggesting changes, as Git does versioning per line.

cason · 2022-07-29T05:58:54Z

spec/p2p/v0.34/README.md

+
+Every node connects to a set of peers to whom it sends requests via the Switch. 
+
+The PEX reactor ensures a node is connected to peers by running a routine to keep dialing peers in the background until a threshold of connected peers is reached (`MaxNumOutboundPeers`). This way, if a peers' address has been added via an incoming request, it will eventually be dialed. Peers to dial are chosen with some configurable bias for unvetted peers. The bias should be lower when we have fewer peers and can increase as we obtain more, ensuring that our first peers are more trustworthy, but always giving us the chance to discover new good peers. 


if a peers' address has been added via an incoming request, it will eventually be dialed.

I didn't fully get this. You mean that if a peer was discovered, the node eventually dials it?

The peers address is added into the address book only when a peer dials the node or if the address already exists in the config file. What I wanted to say here is that the peer will eventually be dialed once we add its address when we accept an incoming connection (this dialing will not happen immediately after we accept the connection).

cason · 2022-07-29T06:01:53Z

spec/p2p/v0.34/README.md

+
+As the number of outgoing peers is limited, the reactor will choose the number of addresses to dial taking into account the current number of outgoing connections and ongoing dialings. The addresses to dial will be picked with a bias towards new and vetted peers (TODO define ).
+
+Except for persistent peers, all other peers can be dialed at most `maxAttemptsToDial`. If a node is not connected by that time it is marked as bad. Otherwise, if the dial fails, the node will wait before the next dial. (exponential backoff mechanism). 


There is a linear backoff, followed by an exponential backoff. See reconnectToPeer method in the switch.

cason · 2022-07-29T06:03:23Z

spec/p2p/v0.34/README.md

+
+Once the addresses to dial are known, they are forwarded to the `DialPeersAsync` routine of the switch. Each address is then dialed in parallel and the corresponding peer is added to the `dialing` list. 
+
+**Note** When dialing a peer, the peers' address is not added into the address book as it had to be there in order for the node to dial it in the first place. 


You mean, it is not added again?

Yes, it was already there so it is not added upon successfully dialing it. As in, we do not first dial and then ad the address but we rather read the addresses and dail them. Now that I write it like this, it seems obvious.

cason · 2022-07-29T06:04:26Z

spec/p2p/v0.34/README.md

+
+When a peer is successfully dialed, it is removed from the `dialing` list and added to the `peers` list. The switch then calls the `InitPeer` and `AddPeer` routines of all the reactors that have registered to it. 
+
+Once a peer is successfully dialed, we ask it to provide us with more peers.


The PEX reactor does that, correct?

Yes, it is done from the PEX reactor.

cason · 2022-07-29T06:12:40Z

spec/p2p/v0.34/README.md

+
+### *Removing peers*
+
+Each reactor can call the `StopPeerForError` method of the switch with the ID of the peer that needs to be removed. Then the switch handles stopping the peer (closing the connection to it), and calls the `RemovePeer` method of all reactors registered to the switch.


I have to confirm, but the peer instance itself my also trigger this method in case of errors in the connection.

In fact, the onPeerError field in the peerConfig struct is configured as a pointer to this method. This same pointer is passed to the connection, becoming the onError field of the MConnection struct. This method is then called in several error situations in the connection managament.

cason · 2022-07-29T06:16:57Z

spec/p2p/v0.34/README.md

+
+## Switch
+
+Every Tendermint reactor regiters itself with the Switch by prividing a Channel ID it is listening to to it. The Switch then forwards messages destined to a particular channel. It is the reactors responisibility to process the messages. 


A reactor can use multiple channels, the consensus reactor reactor for sure does that.

Yes, this is badly phrased here, it provides Channel IDs not just one channel ID.

cason · 2022-07-29T06:17:59Z

spec/p2p/v0.34/README.md

+
+The switch is implemented as a service and, on start, listens for incoming connections in the background (calling the`acceptRoutine` function). 
+
+Dialing peers, either via the PEX reactor or dialing persistent peers on startup, is done by calling the `DialPeersAsync` routine of the Switch.


Is this method invoked by anyone that is not the PEX reactor?

It is called from the node on startup to dial persistent peers and from the rpc/core package to dial seeds.

cason · 2022-07-29T06:18:39Z

spec/p2p/v0.34/README.md

+
+When the connection to a peer is established (either by dialing it or accepting an incoming connection from it), the peer address is added to the address book. (The address book is managed by the PEX reactor). 
+
+When a peer is added to the address book, it is marked as connected to and no further new connections are established between the node and this peer. 


Who guarantees that, I mean, in the implementation?

It is checked within filterPeers and IsDialingOrExistingAddress.

But we should add this to the text.

cason · 2022-07-29T06:21:55Z

spec/p2p/v0.34/README.md

+
+When a peer is added to the address book, it is marked as connected to and no further new connections are established between the node and this peer. 
+
+The number of peers a node can connect to is set by `MaxNumInboundPeers` and `MaxNumOutboundPeers` respectively. 


The switch enforces the MaxNumInboundPeers limit for peers, except for those set as unconditional peers.

The MaxNumOutboundPeers does not appear to be a configuration considered by the switch.

MaxNumOutboundPeers is considered within the ensurePeers routing of the PEX reactor when choosing the number of peers to dial.

cason · 2022-07-29T06:22:59Z

spec/p2p/v0.34/README.md

+
+## Pex reactor
+
+The PEX reactor is responsible for peer discovery and providing other peers with information about peers a node is already connected to. The PEX reactor receive routine listens to two types of messages: `PexRequest` and `PexAddress`. 


providing other peers with information about peers a node is already connected to

Known peers, or connected peers? In the second case, the provided information is pretty restricted, isn't it?

It selects addresses from the address book by calling GetSelectionWithBias.

* Iniital comments on v0.34 p2p * Added conf, updated text * Moved everything to spec * Update README.md

* spec: overview of p2p in v0.34 (#9120) * Iniital comments on v0.34 p2p * Added conf, updated text * Moved everything to spec * Update README.md * spec: overview of the p2p implementation in v0.34 (#9126) * Spec: p2p v0.34 doc, switch initial documentation * Spec: p2p v0.34 doc, list of source files * Spec: p2p v0.34 doc, transport documentation * Spec: p2p v0.34 doc, transport error handling * Spec: p2p v0.34 doc, PEX initial documentation * PEX protocol documentation is a separated file * PEX reactor documentation with a general documentation, including the address book and its role as (outbound) peer manager. * Spec: p2p v0.34 doc, PEX protocol documentation * Spec: p2p v0.34 doc, PEX protocol on seed nodes * Spec: p2p v0.34 doc, address book * Spec: p2p v0.34 doc, address book, more details * Spec: p2p v0.34 doc, address book persistence * Spec: p2p v0.34 doc, address book random samples * Spec: p2p v0.34 doc, status of this documentation * Spec: p2p v0.34 doc, pex reactor documentation * Spec: p2p v0.34 doc, addressing PR #9126 comments Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com> * Spec: p2p v0.34 doc, peer manager, outbound peers Co-authored-by: Daniel Cason <cason@gandria> Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com> * spec:p2p v0.34 introduction (#9319) * restructure README.md initial * Fix typos * Reorganization * spec: overview of p2p in v0.34 (#9120) * Iniital comments on v0.34 p2p * Added conf, updated text * Moved everything to spec * Update README.md * spec: overview of the p2p implementation in v0.34 (#9126) * Spec: p2p v0.34 doc, switch initial documentation * Spec: p2p v0.34 doc, list of source files * Spec: p2p v0.34 doc, transport documentation * Spec: p2p v0.34 doc, transport error handling * Spec: p2p v0.34 doc, PEX initial documentation * PEX protocol documentation is a separated file * PEX reactor documentation with a general documentation, including the address book and its role as (outbound) peer manager. * Spec: p2p v0.34 doc, PEX protocol documentation * Spec: p2p v0.34 doc, PEX protocol on seed nodes * Spec: p2p v0.34 doc, address book * Spec: p2p v0.34 doc, address book, more details * Spec: p2p v0.34 doc, address book persistence * Spec: p2p v0.34 doc, address book random samples * Spec: p2p v0.34 doc, status of this documentation * Spec: p2p v0.34 doc, pex reactor documentation * Spec: p2p v0.34 doc, addressing PR #9126 comments Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com> * Spec: p2p v0.34 doc, peer manager, outbound peers Co-authored-by: Daniel Cason <cason@gandria> Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com> * spec:p2p v0.34 introduction (#9319) * restructure README.md initial * Fix typos * Reorganization * spec: p2p v0.34, addressbook review * spec: p2p v0.34, peer manager review * spec: p2p v0.34, peer manager review * spec: p2p v0.34, peer manager review * spec: p2p v0.34, peer manager review * spec: p2p v0.34, peer manager review * spec: p2p v0.34, peer manager review * spec: p2p v0.34, peer manager review * Filled config description * spec: p2p v0.34, transport review * spec: p2p v0.34, switch review * spec: p2p v0.34, overview, first version * spec: p2p v0.34, peer manager review * spec: p2p v0.34, shorter readme * Configuration update * Configuration update * Shortened README * spec: p2p v0.34, readme intro * spec: p2p v0.34, readme contents * spec: p2p v0.34, readme references * spec: p2p readme points to v0.34 * spec: p2p, v0.34, fixing brokend markdown links * Makrdown fix * Apply suggestions from code review Co-authored-by: Adi Seredinschi <adizere@gmail.com> Co-authored-by: Zarko Milosevic <zarko@informal.systems> * spec: p2p v0.34, address book new intro * spec: p2p v0.34, address book buckets summary * spec: p2p v0.34, peer manager, issue link * spec: p2p v0.34, fixing links * spec: p2p v0.34, addressing comments from reviews * spec: p2p v0.34, addressing comments from reviews * Apply suggestions from Jasmina's code review Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com> * spec: p2p v0.34, addressing comments from reviews * Apply suggestions from code review Co-authored-by: Sergio Mena <sergio@informal.systems> * Apply suggestions from code review Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com> Co-authored-by: Sergio Mena <sergio@informal.systems> * spec: p2p, v0.34, address book section reorganized * spec: p2p, v0.34, addressing review comments * Typos * Typo * spec: p2p, v0.34, address book markbad Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com> Co-authored-by: Daniel Cason <cason@gandria> Co-authored-by: Adi Seredinschi <adizere@gmail.com> Co-authored-by: Zarko Milosevic <zarko@informal.systems> Co-authored-by: Sergio Mena <sergio@informal.systems>

jmalicevic added 4 commits July 20, 2022 07:41

Iniital comments on v0.34 p2p

9f5407f

Added conf, updated text

40d0aca

Merge remote-tracking branch 'origin/v0.34.x' into jasmina/v0.34-doc-p2p

ef0c289

Moved everything to spec

60b9bd4

cason reviewed Jul 29, 2022

View reviewed changes

jmalicevic changed the title ~~spec: p2p overview~~ spec: overview of p2p in v0.34 Jul 29, 2022

Update README.md

840b8e3

cason marked this pull request as ready for review August 10, 2022 13:45

cason requested review from cmwaters, ebuchman, tychoish and williambanfield as code owners August 10, 2022 13:45

cason merged commit 3797a3a into spec/p2p-v0.34 Aug 10, 2022

cason deleted the jasmina/spec-v0.34-p2p branch August 10, 2022 13:45

cason pushed a commit that referenced this pull request Aug 29, 2022

spec: overview of p2p in v0.34 (#9120)

4a7b2f3

* Iniital comments on v0.34 p2p * Added conf, updated text * Moved everything to spec * Update README.md


		Every node connects to a set of peers to whom it sends requests via the Switch.

		The PEX reactor ensures a node is connected to peers by running a routine to keep dialing peers in the background until a threshold of connected peers is reached (`MaxNumOutboundPeers`). This way, if a peers' address has been added via an incoming request, it will eventually be dialed. Peers to dial are chosen with some configurable bias for unvetted peers. The bias should be lower when we have fewer peers and can increase as we obtain more, ensuring that our first peers are more trustworthy, but always giving us the chance to discover new good peers.


		As the number of outgoing peers is limited, the reactor will choose the number of addresses to dial taking into account the current number of outgoing connections and ongoing dialings. The addresses to dial will be picked with a bias towards new and vetted peers (TODO define ).

		Except for persistent peers, all other peers can be dialed at most `maxAttemptsToDial`. If a node is not connected by that time it is marked as bad. Otherwise, if the dial fails, the node will wait before the next dial. (exponential backoff mechanism).


		Once the addresses to dial are known, they are forwarded to the `DialPeersAsync` routine of the switch. Each address is then dialed in parallel and the corresponding peer is added to the `dialing` list.

		Note When dialing a peer, the peers' address is not added into the address book as it had to be there in order for the node to dial it in the first place.


		When a peer is successfully dialed, it is removed from the `dialing` list and added to the `peers` list. The switch then calls the `InitPeer` and `AddPeer` routines of all the reactors that have registered to it.

		Once a peer is successfully dialed, we ask it to provide us with more peers.


		### Removing peers

		Each reactor can call the `StopPeerForError` method of the switch with the ID of the peer that needs to be removed. Then the switch handles stopping the peer (closing the connection to it), and calls the `RemovePeer` method of all reactors registered to the switch.


		## Switch

		Every Tendermint reactor regiters itself with the Switch by prividing a Channel ID it is listening to to it. The Switch then forwards messages destined to a particular channel. It is the reactors responisibility to process the messages.


		The switch is implemented as a service and, on start, listens for incoming connections in the background (calling the`acceptRoutine` function).

		Dialing peers, either via the PEX reactor or dialing persistent peers on startup, is done by calling the `DialPeersAsync` routine of the Switch.


		When the connection to a peer is established (either by dialing it or accepting an incoming connection from it), the peer address is added to the address book. (The address book is managed by the PEX reactor).

		When a peer is added to the address book, it is marked as connected to and no further new connections are established between the node and this peer.


		When a peer is added to the address book, it is marked as connected to and no further new connections are established between the node and this peer.

		The number of peers a node can connect to is set by `MaxNumInboundPeers` and `MaxNumOutboundPeers` respectively.


		## Pex reactor

		The PEX reactor is responsible for peer discovery and providing other peers with information about peers a node is already connected to. The PEX reactor receive routine listens to two types of messages: `PexRequest` and `PexAddress`.

Conversation

jmalicevic commented Jul 28, 2022

Uh oh!

cason left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants