Cross-DGD complement to test_actuation_knobs_live.py (the single-DGD live
test that already ships with this PR). Validates invariants that
single-DGD tests structurally cannot reach:

  - A2  Three planners patch DISJOINT pod sets in one namespace
  - A3  AggPlanner-C uses only the DECODE branch (no PREFILL calls)
  - B2  Power Agent applies 8 distinct caps on one tick
  - B4  Happy-path zero-counter invariant
  - B5  Multi-pod conflict + safe-default behaviour (skipped pending
        Finding #7 — CDI-safe bystander rewrite tracked separately)
  - 8.5.6 / 8.6.2  NVML / DCGM cap-application proven by reading the
                   live driver state (not via mocks)

Layout introduced by this commit:

  components/src/dynamo/planner/tests/integration/test_multi_dgd_live.py
      ~1000 LoC pytest module. Opt-in via RUN_MULTI_DGD_LIVE=1 +
      TEST_NODE=<8-GPU node> env vars; auto-detects Path A (production
      shape: 3 real DGDs + power-aware planner) vs Path B (5 stub-worker
      pods with hardcoded annotations) and skips Path-A-only tests
      cleanly when running Path B. DCGM tests gated by
      DCGM_HOSTENGINE_AVAILABLE=1.

  examples/multi-dgd-live-test/
      Manifests + helm values overrides + teardown script + README
      walk-through that an operator applies before running the test:
        00-namespace.yaml                 — namespace + pull secret
        10-power-agent-values-nvml.yaml   — Power Agent NVML helm values
        11-power-agent-values-dcgm.yaml   — Power Agent DCGM helm values
        20-nvidia-dcgm-standalone-ds.yaml — DCGM 4.5.1 hostengine DS
        30/31/32-vllm-dgd-{A,B,C}.yaml    — three DGDs (Path A)
        40-conflict-bystander-pod.yaml    — multi-pod conflict fixture
        50-stub-workers.yaml              — Path-B stub workers
        99-teardown.sh                    — idempotent cleanup w/ cap-
                                            restore verification
        README.md                         — full reproduction playbook
                                            including a 2026-05-21 live
                                            run record + 11 findings

  docs/design-docs/multi-tenant-test.md
      Design doc (§8 — multi-DGD test plan) that establishes the
      ground-truth topology, the planner / agent contract under test,
      and the rationale for the artifact split above.

  pyproject.toml
      Registers two new test markers consumed by the live test's
      `pytestmark`:
        multi_dgd_live  — scenario selector for this specific file
        power_agent     — shared with components/power_agent/tests/

Module-level gating fails fast on misconfiguration (missing TEST_NODE,
missing kubeconfig) with actionable error messages, and skips silently
when RUN_MULTI_DGD_LIVE is unset — so this file is a no-op under the
default unit-test invocation. Verified: pytest --collect-only is a
clean no-op without the env vars, and full-import works under
RUN_MULTI_DGD_LIVE=1 (collection fails cleanly on absent k8s creds, not
ImportError).

Refs: docs/design-docs/multi-tenant-test.md §8, PR #9683 review.
Signed-off-by: Kai Ma <kaim@nvidia.com>

…nd DCGM 4.5

Two stacked defects surfaced on the 2026-05-21T20:11Z Path-B re-run
caused `test_dcgm_per_gpu_cap_matches_topology` and
`test_dcgm_target_config_registered` to skip silently when they should
have PASSED:

1. **`kubernetes-client>=35.0.0` mangles JSON pod-exec stdout.**
   `kubernetes.stream.stream(..., _preload_content=True)` sniffs
   JSON-shaped stdout, parses it client-side, and returns `str(dict)`.
   That means `printf '%s' '{"a":1}'` comes back as the literal 8-char
   string `{'a': 1}` (single quotes, Python-dict repr), which then
   trips `json.loads` at every read-back site — including the four
   inside `test_dcgm_*`. The intended-as-defensive
   "did not return JSON" skip branch fired instead.

2. **DCGM 4.5+ rotated the `dcgmi config --get -g <id> -j` JSON shape.**
   Pre-4.5: `body` is a list of per-GPU entries keyed by `"Power Limit (W)"`
   with flat `{"Current": "...", "Target": "..."}` children.
   4.5+:    `body` is a dict of fields keyed by `"Power Limit"`
            (no `(W)` suffix) with a `"children"` wrapper whose leaves
            are `{"value": "..."}` objects.
   Three breaking changes in one upgrade — `body` flipped list→dict,
   the field name lost its suffix, the leaves gained a `value` wrapper.

The previous implementation handled neither, so even if defect (1)
were absent, the strict per-field reader would still skip.

This commit lands a single combined fix:

- **`_exec_in_pod`**: switch from `_preload_content=True` (default) to
  `_preload_content=False` + raw stdout/stderr accumulation off the
  `WSClient` via `peek_stdout`/`read_stdout`. Bypasses the mangling
  regardless of which `kubernetes-client` version is installed, and is
  the documented "I want bytes back" pattern in the kubernetes-client
  examples.

- **New `_extract_power_limit_from_dcgmi_body(body)` helper**: walks
  both shapes uniformly and returns `(current_w, target_w)` ints or
  `(None, None)` if neither shape matches. Both DCGM read-back tests
  (`test_dcgm_per_gpu_cap_matches_topology` and
  `test_dcgm_target_config_registered`) now consume this helper
  instead of inlining duplicated nested-`dict.get` chains.

Verified locally:
  - synthetic 4.0-shape payload `[{"Power Limit (W)": {"Current": "350",
    "Target": "350"}}]`  → `(350, 350)` ✓
  - synthetic 4.5-shape payload `{"Power Limit": {"children":
    {"Current": {"value": "375"}, "Target": {"value": "375"}}}}`
      → `(375, 375)` ✓
  - missing field, wrong types, partial children: graceful `(None, None)`
    / `(int, None)` returns ✓
  - test module still collects cleanly (`pytest --collect-only`)

`examples/multi-dgd-live-test/README.md` Finding #11 updated to
"Status: fixed" with a cross-reference to this commit and the expected
SKIPPED→PASSED flip on the next live re-run.

Refs: PR #9683 review (Finding #11 in
examples/multi-dgd-live-test/README.md).

Signed-off-by: Kai Ma <kaim@nvidia.com>

`TestMultiPodConflict` previously created a bystander pod with
`NVIDIA_VISIBLE_DEVICES=<gpu-uuid>` to pin it to a specific GPU. AKS
clusters running the NVIDIA Container Toolkit in CDI device mode (the
dpp-dev-env default, and the direction NVIDIA is pushing for newer
operator releases) reject that with
`unresolvable CDI devices management.nvidia.com/gpu=*` and refuse to
admit the pod — so the test class was carrying a
`@pytest.mark.skip(_CDI_CONFLICT_SKIP_REASON)` decorator that disabled
the whole §8.5.7 contract on every run.

This commit lands the CDI-mode-safe rewrite:

  - Bystander now claims `nvidia.com/gpu: 1` from the device plugin
    (both `requests` and `limits` per K8s extended-resource rules).
    CDI-mode-agnostic — the device plugin owns the actual GPU wiring
    via its own machinery.

  - `examples/multi-dgd-live-test/40-conflict-bystander-pod.yaml`
    mirrors the inline Python manifest exactly so YAML-debugging and
    test-debugging agree (the README explicitly calls this out).

  - New helper `_per_gpu_compute_pids()` snapshots
    `nvmlDeviceGetComputeRunningProcesses` across all 8 GPUs from the
    agent pod (which has hostPID + NVML access) and returns
    `{gpu_idx: {pid, ...}}`. The test polls this until it sees a new
    PID on some index after the bystander reaches Running — that
    index IS the GPU the device plugin assigned, no NVIDIA_VISIBLE_DEVICES
    pin needed.

  - Test renamed `test_conflict_triggers_safe_default_on_gpu0_only` →
    `test_conflict_triggers_safe_default_on_assigned_gpu` to reflect
    the index-agnostic contract. Blast-radius / recovery asserts key
    off the discovered `assigned_gpu`; the loop over the other GPUs
    iterates `range(8)` and skips that index instead of hard-coding
    `range(1, 8)`.

  - Class-level `@pytest.mark.skip` decorator removed. The supporting
    `_CDI_CONFLICT_SKIP_REASON` constant is dropped too — kept neither
    in code nor as a runtime branch, since neither path is needed
    once the CDI-safe flow works.

  - `bystander_uuid` class fixture removed (no longer needed — the
    UUID/index discovery is post-creation, not pre).

`examples/multi-dgd-live-test/README.md` updated:
  - Finding #7 marked **Status: fixed** with cross-reference to this
    commit and an explicit description of the new (a)/(b)/(c) flow.
  - Expected-outcome counts updated: Path B now expects 5 PASSED /
    2 SKIPPED (was 4 PASSED / 3 SKIPPED — multi-pod conflict moves
    from SKIP to PASS); DCGM pass expects 7 PASSED / 2 SKIPPED on the
    next live run.
  - Live-run record row updated to note the rewrite and expected
    PASS outcome.

Verified locally: test module compiles + collects cleanly. The
end-to-end CDI-vs-non-CDI assignment behaviour is by construction the
device plugin's responsibility, so the failure mode of this rewrite
is bounded to "device plugin returns no GPU within 90 s" — handled
with a clear pytest.fail message that includes a before/after PID
snapshot for debugging.

Refs: PR #9683 review (Finding #7 in
examples/multi-dgd-live-test/README.md).

Signed-off-by: Kai Ma <kaim@nvidia.com>

…dd SPDX header

Two follow-up fixes to commit 552be92 caught in PR #9683 review.

1. Bystander cannot schedule. The previous CDI-safe rewrite swapped
   `NVIDIA_VISIBLE_DEVICES=<uuid>` for a `nvidia.com/gpu: 1` device-
   plugin claim, expecting the plugin to allocate a free GPU. But
   the 5 stub workers in `50-stub-workers.yaml` already claim every
   `nvidia.com/gpu` slot on the test node (1+1+2+2+2 = 8 of 8), so
   the bystander's claim sits Pending forever and the conflict never
   materialises.

   Switch to the same pattern the Power Agent itself uses
   (`deploy/helm/charts/power-agent/values.yaml`):
   `NVIDIA_VISIBLE_DEVICES: "all"` exposes every GPU through the
   toolkit without consuming a device-plugin claim. The bystander
   pins itself to `cuda:0` (deterministically the first visible
   device — same NVML index 0 as the agent, since both have
   `NVIDIA_VISIBLE_DEVICES=all`). The existing PID-based discovery
   loop in the test still verifies which physical GPU the context
   landed on rather than trusting the mapping, so the test stays
   robust under future toolkit / CDI changes.

   Touches the inline `bystander_manifest` in
   `test_multi_dgd_live.py::TestMultiPodConflict` and the YAML
   companion `examples/multi-dgd-live-test/40-conflict-bystander-pod.yaml`,
   plus the README's Finding #7 status to document both
   non-options (the original `NVIDIA_VISIBLE_DEVICES=<uuid>` CDI
   rejection AND the `nvidia.com/gpu: 1` over-subscription) and
   the surviving fix-shape.

2. Copyright check failed. `40-conflict-bystander-pod.yaml` was
   imported from `.aic_sandbox/` in commit 004df25 without the
   standard two-line SPDX header that every other YAML in
   `examples/multi-dgd-live-test/` carries (00-namespace through
   50-stub-workers — verified via head-of-file scan). Add the
   header.

Signed-off-by: Kai Ma <kaim@nvidia.com>

… lychee

Lychee on PR #9683 fails on 5 URLs that aren't really URLs (caught
in review). All five are inherited from f7f3226 (the original
"feat(planner): power infrastructure …" commit on this branch) but
block #9683 from going green either way.

The 5 cited locations and fix shape:

- `components/src/dynamo/planner/config/defaults.py:48`
- `components/src/dynamo/planner/config/planner_config.py:147`
  In-cluster Prometheus URL
  `http://prometheus-kube-prometheus-prometheus.monitoring.svc.cluster.local:9090`.
  `*.svc.cluster.local` is the canonical Kubernetes-internal DNS
  suffix and is not externally resolvable. Add an ignore entry for
  the suffix so any cluster-internal default URL passes lychee.

- `components/src/dynamo/planner/config/planner_config.py:317`
  Docstring placeholder `http://host:port/`.
- `deploy/helm/charts/platform/README.md:118-119`
  Helm-rendered Format hints `"nats://hostname:port"` /
  `"http://hostname:port"` / `"https://hostname:port"`.
  Add three anchored ignore entries for these exact placeholder
  forms (kept narrow so a real `*.com` hostname can't accidentally
  slip past as `hostname:port`).

- `deploy/helm/charts/platform/Chart.yaml:23`
  `home: https://nvidia.com`. The same file already uses
  `https://www.nvidia.com` on line 19 for `url:` and that one passes
  lychee, so the bare-apex form on line 23 must be hitting a 4xx
  (Cloudflare bot challenge, redirect-loop, etc.). Canonicalise to
  the `www.` form so the two `nvidia.com` URLs in the same file
  agree, and no ignore entry is needed.

Verified all 5 URLs match the new ignore patterns via local regex
spot-check; `https://www.nvidia.com` (existing line 19 + line 23 fix)
deliberately does NOT match any ignore pattern.

Signed-off-by: Kai Ma <kaim@nvidia.com>

Cross-DGD complement to test_actuation_knobs_live.py (the single-DGD live
test that already ships with this PR). Validates invariants that
single-DGD tests structurally cannot reach:

  - A2  Three planners patch DISJOINT pod sets in one namespace
  - A3  AggPlanner-C uses only the DECODE branch (no PREFILL calls)
  - B2  Power Agent applies 8 distinct caps on one tick
  - B4  Happy-path zero-counter invariant
  - B5  Multi-pod conflict + safe-default behaviour (skipped pending
        Finding #7 — CDI-safe bystander rewrite tracked separately)
  - 8.5.6 / 8.6.2  NVML / DCGM cap-application proven by reading the
                   live driver state (not via mocks)

Layout introduced by this commit:

  components/src/dynamo/planner/tests/integration/test_multi_dgd_live.py
      ~1000 LoC pytest module. Opt-in via RUN_MULTI_DGD_LIVE=1 +
      TEST_NODE=<8-GPU node> env vars; auto-detects Path A (production
      shape: 3 real DGDs + power-aware planner) vs Path B (5 stub-worker
      pods with hardcoded annotations) and skips Path-A-only tests
      cleanly when running Path B. DCGM tests gated by
      DCGM_HOSTENGINE_AVAILABLE=1.

  examples/multi-dgd-live-test/
      Manifests + helm values overrides + teardown script + README
      walk-through that an operator applies before running the test:
        00-namespace.yaml                 — namespace + pull secret
        10-power-agent-values-nvml.yaml   — Power Agent NVML helm values
        11-power-agent-values-dcgm.yaml   — Power Agent DCGM helm values
        20-nvidia-dcgm-standalone-ds.yaml — DCGM 4.5.1 hostengine DS
        30/31/32-vllm-dgd-{A,B,C}.yaml    — three DGDs (Path A)
        40-conflict-bystander-pod.yaml    — multi-pod conflict fixture
        50-stub-workers.yaml              — Path-B stub workers
        99-teardown.sh                    — idempotent cleanup w/ cap-
                                            restore verification
        README.md                         — full reproduction playbook
                                            including a 2026-05-21 live
                                            run record + 11 findings

  docs/design-docs/multi-tenant-test.md
      Design doc (§8 — multi-DGD test plan) that establishes the
      ground-truth topology, the planner / agent contract under test,
      and the rationale for the artifact split above.

  pyproject.toml
      Registers two new test markers consumed by the live test's
      `pytestmark`:
        multi_dgd_live  — scenario selector for this specific file
        power_agent     — shared with components/power_agent/tests/

Module-level gating fails fast on misconfiguration (missing TEST_NODE,
missing kubeconfig) with actionable error messages, and skips silently
when RUN_MULTI_DGD_LIVE is unset — so this file is a no-op under the
default unit-test invocation. Verified: pytest --collect-only is a
clean no-op without the env vars, and full-import works under
RUN_MULTI_DGD_LIVE=1 (collection fails cleanly on absent k8s creds, not
ImportError).

Refs: docs/design-docs/multi-tenant-test.md §8, PR #9683 review.
Signed-off-by: Kai Ma <kaim@nvidia.com>

…nd DCGM 4.5

Two stacked defects surfaced on the 2026-05-21T20:11Z Path-B re-run
caused `test_dcgm_per_gpu_cap_matches_topology` and
`test_dcgm_target_config_registered` to skip silently when they should
have PASSED:

1. **`kubernetes-client>=35.0.0` mangles JSON pod-exec stdout.**
   `kubernetes.stream.stream(..., _preload_content=True)` sniffs
   JSON-shaped stdout, parses it client-side, and returns `str(dict)`.
   That means `printf '%s' '{"a":1}'` comes back as the literal 8-char
   string `{'a': 1}` (single quotes, Python-dict repr), which then
   trips `json.loads` at every read-back site — including the four
   inside `test_dcgm_*`. The intended-as-defensive
   "did not return JSON" skip branch fired instead.

2. **DCGM 4.5+ rotated the `dcgmi config --get -g <id> -j` JSON shape.**
   Pre-4.5: `body` is a list of per-GPU entries keyed by `"Power Limit (W)"`
   with flat `{"Current": "...", "Target": "..."}` children.
   4.5+:    `body` is a dict of fields keyed by `"Power Limit"`
            (no `(W)` suffix) with a `"children"` wrapper whose leaves
            are `{"value": "..."}` objects.
   Three breaking changes in one upgrade — `body` flipped list→dict,
   the field name lost its suffix, the leaves gained a `value` wrapper.

The previous implementation handled neither, so even if defect (1)
were absent, the strict per-field reader would still skip.

This commit lands a single combined fix:

- **`_exec_in_pod`**: switch from `_preload_content=True` (default) to
  `_preload_content=False` + raw stdout/stderr accumulation off the
  `WSClient` via `peek_stdout`/`read_stdout`. Bypasses the mangling
  regardless of which `kubernetes-client` version is installed, and is
  the documented "I want bytes back" pattern in the kubernetes-client
  examples.

- **New `_extract_power_limit_from_dcgmi_body(body)` helper**: walks
  both shapes uniformly and returns `(current_w, target_w)` ints or
  `(None, None)` if neither shape matches. Both DCGM read-back tests
  (`test_dcgm_per_gpu_cap_matches_topology` and
  `test_dcgm_target_config_registered`) now consume this helper
  instead of inlining duplicated nested-`dict.get` chains.

Verified locally:
  - synthetic 4.0-shape payload `[{"Power Limit (W)": {"Current": "350",
    "Target": "350"}}]`  → `(350, 350)` ✓
  - synthetic 4.5-shape payload `{"Power Limit": {"children":
    {"Current": {"value": "375"}, "Target": {"value": "375"}}}}`
      → `(375, 375)` ✓
  - missing field, wrong types, partial children: graceful `(None, None)`
    / `(int, None)` returns ✓
  - test module still collects cleanly (`pytest --collect-only`)

`examples/multi-dgd-live-test/README.md` Finding #11 updated to
"Status: fixed" with a cross-reference to this commit and the expected
SKIPPED→PASSED flip on the next live re-run.

Refs: PR #9683 review (Finding #11 in
examples/multi-dgd-live-test/README.md).

Signed-off-by: Kai Ma <kaim@nvidia.com>