Just dumping some thoughts on the P2P PeerManager API model. I think I prefer one of these approaches, which are likely to involve equivalent internal implementations with different external interfaces.

• Make DialNext() and EvictNext() trivially cheap when there is nothing to do, such that they can be called in a tight loop with e.g. a 1ms interval.
• Make DialNext() and EvictNext() return channels with peer IDs to dial/evict.
• Make DialNext() and EvictNext() block until a peer is available, and take a context for cancellation.

I'm leaning towards channels or blocking.

Below are the options I've been considering.

Polling

Current model, Router calls e.g. DialNext() in loop to check if there's another peer to dial, then Dialed() or DailFailed when dialed, before Ready() when set up, and Disconnected() when disconnected.

Positive:

• Simple to reason about
• No concurrency (all operations are serialized by a mutex)

Negative:

• Polling timeout can cause e.g. retries to be coarser than necessary and harder to calculate (i.e. an intended 100 ms retry delay will take 1 second to retry if the polling interval is 1 second).
• Polling can be expensive (e.g. Ranked calculations) and there may not be any changes.
• Rest of router uses channels and is event-based.
• Logic is spread across peer manager and router.

Director

The Router simply has methods for e.g. Dial(), Accept(), and Disconnect(). The PeerManager takes the Router as a parameter and controls it, doing peer scheduling itself, but the router manages the actual connections.

Positive:

• Simplifies router somewhat, since it does not have to deal with peer scheduling.

Negative:

• Complicates manager, since it now needs to be a concurrent, event-driven system with internal goroutines and scheduling. Logic is already complex.
• Router needs to "talk back" to the peer manager, e.g. for remote peer disconnections and peer errors (from channels), so it will be a hybrid solution anyway with info flowing in both directions.

Event-driven

The Manager and Router communicate with each other using channels.

Positive:

• Fits better with Router model, which is channel-based.

Negative:

• All communication between router and manager becomes asynchronous, making it harder to reason about.
• Manager logic is already somewhat difficult to reason about, even without being async.

Pulling with channels

Keep the same model as polling, but instead of polling DialNext() and EvictNext() these return channels that dispense peers. Other events (e.g. Dialed or DialFailed) are called synchronously.

Positive:

• Fits better with Router model.
• Remains fairly simple to reason about.
• Internally the channels can be implemented in any number of ways.
• May improve performance, and has more accurate retry delays.

Negative:

• May need internal goroutines and/or start/stop methods, especially to avoid blocking during shutdown and/or contention.

Pulling with context

Instead of DialNext() and EvictNext() returning channels, they instead block until there is a peer to return. A context is passed for cancellation (e.g. for Router shutdown).

Positive:

• Same benefits and downsides as Pulling.
• Since all other PeerManager APIs are method calls, this fits better with the overall API.

Negative:

• Doesn't fit as well with Router, since it now has to manage the context based on its own channels (e.g. stopCh) and cancel the context when appropriate.