As discussed in #2067, the peer-to-peer architecture needs to be redesigned and refactored. The background and scope for this is outlined in ADR 061 (as "phase 1"), and the proposed new architecture design is described in ADR 062.

Phase I

Items that are deemed necessary to complete or at least revisit before the next major release (v0.35.0)

• Implement Channel and PeerUpdates APIs as shims on top of current Switch and Peer APIs. p2p refactor: state sync reactor #5671
• Update reactors to use channel shims.
  • state-sync
  • evidence
  • blockchain v0
  • mempool
  • consensus
  • pex
• New transport API
  • Implement new Transport interface, port MConnection, and migrate legacy P2P stack. p2p: implement new Transport interface #5791
  • Duplicate SecretConnection in the privval package until it's replaced by gRPC. privval: duplicate SecretConnection from p2p package #5672
  • Add in-memory transport for testing. p2p: add MemoryTransport, an in-memory transport for testing #5827
  • Write unit- and integration-tests. p2p: tighten up and test Transport API #6020
• New P2P core
  • Implement Router prototype (for message/connection routing). p2p: add Router prototype #5831
  • Implement PeerManager prototype (for peer lifecycle management). p2p: add prototype peer lifecycle manager #5882
  • Implement peerStore prototype with persistence. p2p: improve peerStore prototype #5954
  • Specify API semantics, write unit- and integration-tests. p2p: tighten up and test PeerManager #6034 p2p: tighten up Router and add tests #6044
  • Write ADR documenting design. ADR-062: update with new P2P core implementation #6051
  • Implement seed mode, either in Tendermint or as separate daemon. Also needs special handling of seed peers in PeerManager (i.e. don't keep connecting to them). p2p: seed mode support #6271
  • Add connection limits (num and rate) for inbound connections from same IP. (p2p: rate-limit incoming connections by IP #6286)
  • Legacy ABCI IP filtering must either be removed or integrated with router/manager.
  • Add ABCI-based peer ID filtering in PeerManager.
  • Audit DoS vectors and security vulnerabilities.
  • Redesign exposed metrics, including RPC peer info APIs. (p2p: metrics #6278)
  • Implement a queue scheduler that prioritizes and drops messages as appropriate. p2p: revised router message scheduling #6126
  • Improve peer scoring policy for PeerManager.
  • Add API for reporting peer quality events from reactors. (p2p: simple peer scoring #6277)
  • Evaluate handling of all PeerUpdate.Status usage in reactors.
• Migrate to new P2P API
  • Temporarily support using both old and new P2P cores (Router vs Switch) Feature Flag for Legacy P2P Support #6056
  • Improve E2E tests and pass with new P2P core (new network topologies, sentry nodes, NAT/UPnP handling, etc)
  • Benchmark both P2P cores, and make sure we don't get deadlocks/stalls under high load/contention.
  • Remove old abstractions: Switch, Peer, NetAddress, PeerSet, AddressBook, and so on.
  • Remove shims.
  • Review reactor concurrency models (old stack called Receive() concurrently, now it is up to the reactor to decide its concurrency model -- if single-treaded per channel, this can hamper performance, but may or may not matter).
  • Clean up P2P configuration settings.
    • Merge MaxNumInboundPeers and MaxNumOutboundPeers into MaxConnections. (p2p: minor cleanup + update router options #6353)
    • Remove UnconditionalPeerIDs and AllowDuplicateIP
    • Consider how to update peer address changes in PeerManager (replace all other addresses for peer?).

Phase II

Items that should be completed for a future release but are not required for a fully functional p2p layer that maintains feature and performance parity with the legacy stack.

• New transport API
  • Consider stream-based transport API. rfc: add P2P stream proposal spec#227
  • Refactor and clean up MConnection and SecretConnection code (possibly after stream-based API).
  • Remove traffic scheduling (priorities and bandwidth limits) in MConnection, move into Router (probably requires stream API)
• New P2P core
  • Design improved address detection and advertisement system.
    • Differentiate between listen address and advertised addresses, by detecting advertised addresses.
    • UPnP address detection.
    • Differentiate advertisement based on remote peer IP network (e.g. peers on 192.168.0.0 can get internal addresses, but not peers on public networks).
  • Improve peerStore.Ranked() performance with ordered data structure.
  • Consider CLI to manage peer database (e.g. manually evicting invalid addresses).
• Migrate to new P2P API
  • Review P2P and reactor code to make better use of new abstractions.
• Future work
  • Remove PeerManager.SetHeight() and GetHeight(), as reactors should not share state (requires protocol changes such that reactors retrieve this information themselves).
  • Remove V1 PEX Reactor (PEX: eventually remove v1 reactor  #6371)
  • Make PEX and MConn handshakes exchange address URLs, not IP/port endpoints.