1. Network Overview

2. Physical Topology

3. Device Inventory & IP Addressing

Network Switches

GPU Servers

Other Devices

4. Fabric Link Addressing

Server-Facing Links

5. eBGP Underlay Configuration

5.1 NX_AI_Spine1 — BGP Configuration

! NX_AI_Spine1 (192.168.51.232) - AS 65000

router bgp 65000
  router-id 10.2.0.1
  address-family ipv4 unicast
    network 10.2.0.1/32

  ! Peer to Leaf1 via Eth1/14
  neighbor 10.4.0.5
    remote-as 65001
    description to-NX-AI-Leaf1
    bfd
    address-family ipv4 unicast

  ! Peer to Leaf2 via Eth1/18
  neighbor 10.4.0.14
    remote-as 65002
    description to-NX-AI-Leaf2
    bfd
    address-family ipv4 unicast

5.2 NX_AI_Spine2 — BGP Configuration

! NX_AI_Spine2 (192.168.51.231) - AS 65000

router bgp 65000
  router-id 10.2.0.2
  address-family ipv4 unicast
    network 10.2.0.2/32

  ! Peer to Leaf1 via Eth1/13
  neighbor 10.4.0.1
    remote-as 65001
    description to-NX-AI-Leaf1
    bfd
    address-family ipv4 unicast

  ! Peer to Leaf2 via Eth1/17
  neighbor 10.4.0.9
    remote-as 65002
    description to-NX-AI-Leaf2
    bfd
    address-family ipv4 unicast

5.3 NX_AI_Leaf1 — BGP Configuration

! NX_AI_Leaf1 (192.168.50.229) - AS 65001

router bgp 65001
  router-id 10.2.0.3
  address-family ipv4 unicast
    network 10.2.0.3/32
    network 10.3.0.1/32
    network 10.0.0.0/24           ! Rail 0 SVI subnet
    maximum-paths 2              ! ECMP across both spines

  ! Peer to Spine1 via Eth1/14
  neighbor 10.4.0.6
    remote-as 65000
    description to-NX-AI-Spine1
    bfd
    address-family ipv4 unicast

  ! Peer to Spine2 via Eth1/13
  neighbor 10.4.0.2
    remote-as 65000
    description to-NX-AI-Spine2
    bfd
    address-family ipv4 unicast

5.4 NX_AI_Leaf2 — BGP Configuration

! NX_AI_Leaf2 (192.168.51.230) - AS 65002

router bgp 65002
  router-id 10.2.0.4
  address-family ipv4 unicast
    network 10.2.0.4/32
    network 10.3.0.2/32
    network 10.0.1.0/24           ! Rail 1 SVI subnet
    maximum-paths 2              ! ECMP across both spines

  ! Peer to Spine1 via Eth1/18
  neighbor 10.4.0.13
    remote-as 65000
    description to-NX-AI-Spine1
    bfd
    address-family ipv4 unicast

  ! Peer to Spine2 via Eth1/17
  neighbor 10.4.0.10
    remote-as 65000
    description to-NX-AI-Spine2
    bfd
    address-family ipv4 unicast

6. QoS / PFC / ECN Configuration (Lossless RDMA)

6.1 Classification & Marking

ACL — RoCE UDP Port Matching

ip access-list rdma
  10 permit udp any any eq 4741
  20 permit udp any eq 4741 any
  30 permit udp any eq 4791 any
  40 permit udp any any eq 4791
! UDP 4741 = RoCE v1, UDP 4791 = RoCE v2

Class Maps — RDMA Traffic Identification

! Match DSCP 26 (CS3/AF31 — standard RoCE marking)
class-map type qos match-all RDMA
  match dscp 26

! Match DSCP 26 OR RoCE UDP ports (broader catch-all)
class-map type qos match-any RDMA_UDP
  match dscp 26
  match access-group name rdma

Input Marking Policy

! Classify RDMA traffic into qos-group 3 for downstream processing
policy-map type qos QOS_MARKING
  class RDMA
    set qos-group 3
  class RDMA_UDP
    set qos-group 3

6.2 Network QoS (PFC + MTU)

! Network QoS: controls MTU per queue and PFC behavior
policy-map type network-qos QOS_NETWORK
  class type network-qos c-nq3
    mtu 9216              ← jumbo frames for RDMA queue
    pause pfc-cos 3       ← IEEE 802.1Qbb PFC on CoS 3
  class type network-qos c-nq-default
    mtu 9216              ← jumbo for all other traffic too

Per-Interface PFC Settings

! Applied to ALL server-facing and fabric-facing interfaces:
priority-flow-control mode on

! Global features enabled:
feature lldp     ← Link Layer Discovery Protocol
feature dcbx     ← Data Center Bridging Capability Exchange

6.3 Egress Queuing (ECN + Priority)

! Egress queuing: scheduling + ECN for congestion signaling
policy-map type queuing RDMA_ECN_OUT
  class type queuing c-out-q3
    priority level 1                              ← strict priority (lowest latency)
    random-detect threshold burst-optimized ecn   ← DCQCN congestion signaling
  class type queuing c-out-q2
    bandwidth remaining percent 0
  class type queuing c-out-q1
    bandwidth remaining percent 0
  class type queuing c-out-q-default
    bandwidth remaining percent 100               ← all remaining BW for non-RDMA

System QoS Application

! Apply policies globally to the switching ASIC
system qos
  service-policy type network-qos QOS_NETWORK      ← PFC + MTU
  service-policy type queuing output RDMA_ECN_OUT  ← ECN + scheduling

6.4 Design Summary

Cleanup Scripts

7. RDMA Rail Design

7.1 Rail 0 — Leaf1 / VLAN 100

! Leaf1 - Rail 0 Switch Configuration

system jumbomtu 9216

interface Eth1/27
  switchport access vlan 100
  mtu 9216
  no shutdown

interface Eth1/28
  switchport access vlan 100
  mtu 9216
  no shutdown

interface Vlan100
  no shutdown
  mtu 9216
  ip address 10.0.0.254/24

7.2 Rail 1 — Leaf2 / VLAN 101

! Leaf2 - Rail 1 Switch Configuration

system jumbomtu 9216

interface Eth1/27
  switchport access vlan 101
  mtu 9216
  no shutdown

interface Eth1/28
  switchport access vlan 101
  mtu 9216
  no shutdown

interface Vlan101
  no shutdown
  mtu 9216
  ip address 10.0.1.254/24

8. GPU Server Configuration

8.1 NIC IP Addressing & MTU

# ens6d1 (Rail 0 - Leaf1)
ip addr add 10.0.0.1/24 dev ens6d1
ip link set ens6d1 mtu 9000
ip link set ens6d1 up

# ens6 (Rail 1 - Leaf2)
ip addr add 10.0.1.1/24 dev ens6
ip link set ens6 mtu 9000
ip link set ens6 up

# ens6d1 (Rail 0 - Leaf1)
ip addr add 10.0.0.2/24 dev ens6d1
ip link set ens6d1 mtu 9000
ip link set ens6d1 up

# ens6 (Rail 1 - Leaf2)
ip addr add 10.0.1.2/24 dev ens6
ip link set ens6 mtu 9000
ip link set ens6 up

8.2 Per-NIC Policy Routing

# Step 1: Register routing table names in /etc/iproute2/rt_tables
echo '100 rail0' >> /etc/iproute2/rt_tables
echo '101 rail1' >> /etc/iproute2/rt_tables

# Step 2: Rail 0 routing (ens6d1 → Leaf1 SVI 10.0.0.254)
ip route add 10.0.0.0/24 dev ens6d1 scope link table 100
ip route add default via 10.0.0.254 dev ens6d1 table 100
ip rule add from <ens6d1_ip> table 100

# Step 3: Rail 1 routing (ens6 → Leaf2 SVI 10.0.1.254)
ip route add 10.0.1.0/24 dev ens6 scope link table 101
ip route add default via 10.0.1.254 dev ens6 table 101
ip rule add from <ens6_ip> table 101

Policy Routing per Server

8.3 Netplan Persistence

# /etc/netplan/60-rdma-rails.yaml (gpuserver1 example)
network:
  version: 2
  ethernets:
    ens6d1:
      addresses:
        - 10.0.0.1/24
      mtu: 9000
      routing-policy:
        - from: 10.0.0.1
          table: 100
      routes:
        - to: 0.0.0.0/0
          via: 10.0.0.254
          table: 100
        - to: 10.0.0.0/24
          scope: link
          table: 100
    ens6:
      addresses:
        - 10.0.1.1/24
      mtu: 9000
      routing-policy:
        - from: 10.0.1.1
          table: 101
      routes:
        - to: 0.0.0.0/0
          via: 10.0.1.254
          table: 101
        - to: 10.0.1.0/24
          scope: link
          table: 101

9. MTU Configuration

10. Change Log & Issues Resolved

11. Automation Scripts

12. Verification Results

eBGP Sessions — All Established

Cross-Rail Routing — ECMP Working

! Leaf1: route to Rail 1 subnet (10.0.1.0/24) - 2 paths
10.0.1.0/24, ubest/mbest: 2/0
    *via 10.4.0.6, [20/0], BGP-65000   ← via Spine1
    *via 10.4.0.2, [20/0], BGP-65000   ← via Spine2

! Leaf2: route to Rail 0 subnet (10.0.0.0/24) - 2 paths
10.0.0.0/24, ubest/mbest: 2/0
    *via 10.4.0.13, [20/0], BGP-65000  ← via Spine1
    *via 10.4.0.10, [20/0], BGP-65000  ← via Spine2

Cross-Server Reachability — All Passed

13. RDMA Performance Results

13.1 Bandwidth Tests (ib_write_bw)

Results — IB MTU 4096

MTU Comparison — 2048 vs 4096

13.2 Latency Tests (ib_write_lat)

Results — IB MTU 4096

13.3 How Our Lab Compares

Test Commands Reference

# Bandwidth test -- Server side (gpuserver2):
ib_write_bw -d rocep130s0 -i <ib_port> --source_ip <server_ip> --port=<tcp_port> -m 4096 -F --report_gbits -a

# Bandwidth test -- Client side (gpuserver1):
ib_write_bw -d rocep130s0 -i <ib_port> --source_ip <client_ip> --port=<tcp_port> -m 4096 -F --report_gbits -a <server_ip>

# Latency test -- same flags, replace ib_write_bw with ib_write_lat (no --report_gbits)
ib_write_lat -d rocep130s0 -i <ib_port> --source_ip <ip> --port=<tcp_port> -m 4096 -F -a [server_ip]

# IB Port mapping:
#   -i 1 = ens6   (Rail 1, 10.0.1.x via Leaf2)
#   -i 2 = ens6d1 (Rail 0, 10.0.0.x via Leaf1)
# IMPORTANT: Use -i for IB port, NOT -p (which sets TCP port)
# Use -m 4096 for jumbo IB MTU (default is only 2048)
# Use unique --port values (19001, 19002...) to avoid TCP conflicts