Summary

• Rework GKE NCCL all-reduce bandwidth validation from raw Pods + kubectl exec to Kubeflow TrainJob + MPI pattern, matching the EKS approach
• Delete nccl_gke.go (raw Pod flow), nccl_gke_test.go, and nccl-test-tcpxo.yaml template — replaced by per-platform TrainingRuntime + shared TrainJob
• Both platforms now share the same code path: applyNCCLResources() → apply runtime → apply trainjob → waitForLauncherCompletion() → parse bandwidth
• Remove GKE-only helper methods (WaitForPodReady, ExecInContainer) and GKE-specific timeouts
• Add documentation for testing with custom validator images and private registry authentication

Architecture

Shared TrainJob with per-platform TrainingRuntimes:

testdata/
├── trainjob.yaml              ← shared (runtimeRef + numNodes only)
└── h100/
    ├── eks/
    │   └── runtime.yaml       ← EFA image, EFA mpirun args, p5 nodeSelector
    └── gke/
        └── runtime.yaml       ← TCPXO sidecar, FastRak args, hostNetwork, a3-megagpu nodeSelector

The TrainJob is platform-agnostic — it only sets runtimeRef and numNodes. All platform-specific configuration lives in the per-platform TrainingRuntime. The per-platform split is at the runtime level (not TrainJob overrides) because EKS and GKE have fundamentally different GPU networking stacks. The TrainJob API's spec.podSpecOverrides cannot inject native sidecars (initContainers with restartPolicy: Always), set hostNetwork: true, or set pod-level dnsPolicy — all required for GKE TCPXO.

EKS vs GKE Runtime Comparison

Both use Kubeflow Trainer MPI plugin with the same test parameters (-b 1K -e 16G -f 2 -g 1).

Config alignment effort

GKE TCPXO required extensive debugging to match EKS quality:

• SSH port 2222: hostNetwork: true binds host's sshd on port 22, forcing alternate port
• MCA/UCX NIC restriction: With hostNetwork, UCX discovers 169.254.x.x GPU NIC link-local addresses that aren't routable cross-node — restricted to control NIC
• NIC name shift: hostNetwork on a3-megagpu-8g shifts all interfaces by 1 (control: eth1 not eth0, GPU NICs: eth2-eth9 not eth1-eth8) — discovered via nccl-env-profile.sh
• Guest Config Checker compliance: Removed 4 stale env vars (NCCL_ALGO, NCCL_DYNAMIC_CHUNK_SIZE, NCCL_FASTRAK_LLCM_DEVICE_DIRECTORY, NCCL_NVLS_ENABLE) flagged as "expected unset"
• PCI BAR access: FastRak plugin needs /sys/bus/pci/devices/.../resource0_wc — requires privileged: true (not just IPC_LOCK)
• Test binary: EKS uses all_reduce_perf, GKE uses all_reduce_perf_mpi — both MPI-linked, different image conventions. Both now use ${TEST_TYPE} template variable
• Matched test sweep: Both use identical -b ${MIN_MESSAGE_SIZE} -e ${MAX_MESSAGE_SIZE} -f 2 -g 1

Bandwidth results

• EKS (p5.48xlarge, 2 nodes, 3.2 Tbps EFA): 485 GB/s busBW
• GKE (a3-megagpu-8g, 2 nodes, 1.6 Tbps TCPXO): 335 GB/s busBW

GPU hardware is the same on both (8x H100 80GB HBM3). The ~30% gap is explained by 2x network fabric difference (3.2 vs 1.6 Tbps). On 2 nodes the gap is narrower since most all-reduce traffic stays intra-node over NVLink (identical on both) — the fabric difference matters more as node count scales.

Changes

Test plan

• go test -race ./validators/performance/... — all pass
• go test -race ./validators/helper/... — all pass
• golangci-lint run — 0 issues
• make tidy — vendor/ synced (removed unused SPDY/websocket/remotecommand deps)
• E2E on EKS H100 cluster (p5.48xlarge, 2 nodes): 485 GB/s busBW — PASS
• E2E on GKE H100 cluster (a3-megagpu-8g, 2 nodes): 335 GB/s busBW — PASS