High finding: The no-EFA fallback is currently incomplete.

In validators/performance/nccl_all_reduce_bw_constraint.go, efaCount == 0 is treated as a TCP fallback path, but the EKS runtime template still hardcodes FI_PROVIDER=efa (and related EFA-specific env behavior) in validators/performance/testdata/h100/eks/runtime.yaml.

With no EFA devices/plugin, forcing FI_PROVIDER=efa can prevent libfabric/NCCL from using TCP, so this path may fail instead of degrading gracefully.

Suggested fix:

1. Template FI_PROVIDER (and EFA-only env vars) based on discovered efaCount.
2. For efaCount == 0, render TCP-compatible settings.
3. Add tests that verify rendered runtime content for both EFA and non-EFA branches.

High finding: The no-EFA fallback is currently incomplete.

In validators/performance/nccl_all_reduce_bw_constraint.go, efaCount == 0 is treated as a TCP fallback path, but the EKS runtime template still hardcodes FI_PROVIDER=efa (and related EFA-specific env behavior) in validators/performance/testdata/h100/eks/runtime.yaml.

With no EFA devices/plugin, forcing FI_PROVIDER=efa can prevent libfabric/NCCL from using TCP, so this path may fail instead of degrading gracefully.

Suggested fix:

1. Template FI_PROVIDER (and EFA-only env vars) based on discovered efaCount.
2. For efaCount == 0, render TCP-compatible settings.
3. Add tests that verify rendered runtime content for both EFA and non-EFA branches.

In practice our EKS test did complete successfully with FI_PROVIDER=efa and no EFA devices — NCCL fell back to TCP automatically and got ~3 GB/s. The NCCL/libfabric stack handles the missing provider gracefully.

LGTM — good improvement. Dynamic discovery eliminates the p5.48xlarge/EFA hardcoding and the TCP fallback is well-handled.

A few observations (non-blocking):

Medium:

1. Single-node assumption for EKS config discovery. discoverEKSNodeConfig(config.Nodes[0]) reads instance type and EFA count from the first GPU node only. In heterogeneous clusters (e.g., mixed p5.48xlarge and p4d.24xlarge), this could select the wrong instance type for the nodeSelector, causing pods to schedule on nodes with a different EFA count than expected. Consider validating that all GPU nodes have the same instance type, or at minimum logging a warning if they differ.
2. Empty EFA resource lines produce blank lines in YAML. When buildEFAResourceLine returns "", the template substitution for ${EFA_RESOURCE_LIMITS} and ${EFA_RESOURCE_REQUESTS} will leave empty lines in the rendered YAML. While most YAML parsers tolerate this, it could cause yamllint warnings or confuse users inspecting the rendered output. Consider having the template handle the conditional (e.g., an {{if}} block) or trimming blank lines after substitution.

Low:

3. Nodes field added to gpuConfiguration but only used by EKS. GKE doesn't use config.Nodes — it calls discoverGKEGPUNICNetworks via the dynamic client instead. The field is fine for now, but if more platforms need node-level discovery, consider a dedicated method on gpuConfiguration rather than carrying the full node list.
4. Test coverage gap. No test for the TCP fallback path end-to-end (EFA count 0 → maxMessageSizeTCP in templateData). The unit tests cover discoverEKSNodeConfig and buildEFAResourceLine individually, but the integration in applyNCCLResources is only validated manually.

For 1:

The current implementation uses node.kubernetes.io/instance-type as the nodeSelector, which implicitly guarantees GPU homogeneity on cloud (e.g., p5.48xlarge = always H100 SXM5 + 32 EFA). A heterogeneous node warning was added to catch misconfigurations early.

For a future PR, we plan to add a GPU product compatibility layer using nvidia.com/gpu.product labels (set by GPU Feature Discovery). This would:

1. Define NCCL-compatible families — group GPU products by NVLink topology:

2. Validate node compatibility — ensure all GPU nodes belong to the same family before scheduling NCCL workers. Products within the same family (e.g., HBM3 + HBM3e) can participate in the same collective, though performance is limited by the slowest variant.