Summary

Upgrade KAI Scheduler from v0.12.17 to v0.13.0 and fix DRA compatibility for gang scheduling validation.

Motivation / Context

KAI v0.12.x injected runtimeClassName: nvidia into GPU pods via admission webhook, which conflicted with GPU Operator v25.10.0+ CDI defaults (causing "unresolvable CDI devices" errors). We worked around this with a post-install Job that patched the Config CR. KAI v0.13.0 fixes this upstream (KAI-Scheduler#1035).

Additionally, KAI v0.13.0 requires kai.scheduler/queue labels on DRA ResourceClaims — without this label, the scheduler rejects device-plugin-style GPU requests on DRA-enabled nodes with "device-plugin GPU requests cannot be scheduled on DRA-only nodes".

Fixes: N/A
Related: kai-scheduler/KAI-Scheduler#1035

Type of Change

• Bug fix (non-breaking change that fixes an issue)
• New feature (non-breaking change that adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Documentation update
• Refactoring (no functional changes)
• Build/CI/tooling

Component(s) Affected

• CLI (cmd/aicr, pkg/cli)
• API server (cmd/aicrd, pkg/api, pkg/server)
• Recipe engine / data (pkg/recipe)
• Bundlers (pkg/bundler, pkg/component/*)
• Collectors / snapshotter (pkg/collector, pkg/snapshotter)
• Validator (pkg/validator)
• Core libraries (pkg/errors, pkg/k8s)
• Docs/examples (docs/, examples/)
• Other: recipes/overlays, recipes/registry, evidence scripts

Implementation Notes

KAI v0.13.0 upgrade:

• recipes/registry.yaml: version bump v0.12.17 → v0.13.0
• recipes/overlays/base.yaml: version bump + remove manifestFiles workaround
• recipes/components/kai-scheduler/values.yaml: add admission.gpuPodRuntimeClassName: ""
• Delete recipes/components/kai-scheduler/manifests/disable-runtime-class-injection.yaml

DRA gang scheduling fix:

• validators/conformance/gang_scheduling_check.go: add kai.scheduler/queue: default-queue label to buildGangResourceClaim
• pkg/evidence/scripts/manifests/gang-scheduling-test.yaml: replace device-plugin nvidia.com/gpu: 1 with DRA ResourceClaims + queue labels

Testing

# Conformance validation (all 11 checks)
aicr validate --phase conformance  # 11/11 PASS

# Gang scheduling evidence collection
aicr validate --phase conformance --cncf-submission \
  --feature gang-scheduling --evidence-dir ./evidence  # PASS

Tested on EKS with H100 (p5.48xlarge), KAI v0.13.0, DRA driver, GPU Operator v25.12.0.

Risk Assessment

• Low — Isolated change, well-tested, easy to revert

Rollout notes: Existing clusters using KAI v0.12.x with the post-install workaround will upgrade cleanly — the workaround Job is idempotent and its removal is safe. New deployments get the fix natively via Helm values.

Checklist

• Tests pass locally (make test with -race)
• Linter passes (make lint)
• I did not skip/disable tests to make CI green
• I added/updated tests for new functionality
• I updated docs if user-facing behavior changed
• Changes follow existing patterns in the codebase
• Commits are cryptographically signed (git commit -S)