Motivation

Fix two compounded issues in framework_final of docker/Dockerfile that have been silently breaking aarch64 nightly Docker builds since #22160 (2026-04-09):

1. Silent-failure bug. The cubin-download retry block ends with ... || true (intended only to guard the trailing find cleanup), but bash's left-associative &&/|| precedence makes that swallow the entire chain — including the [ "$success" = "1" ] fail-fast check that [Docker] Fix Trivy CVEs, cubin download 403s, and kernels command order #22322 added the same day. When kernels download python fails permanently, the RUN reports DONE 0, the subsequent mkdir -p /root/.cache/sglang && mv python/kernels.lock steps are skipped, and the runtime stage's COPY --from=framework_final /root/.cache/sglang /root/.cache/sglang then fails with a misleading "not found" two stages later.

2. No aarch64 cubin variants. kernels-community/sgl-flash-attn3 (pinned via [tool.kernels.dependencies] in python/pyproject.toml) only publishes x86_64-linux build variants. On aarch64 hosts (e.g., Grace Blackwell builds), all 3 retries always fail; the silent-failure bug above hides this inside a DONE 0 layer.

This affects every aarch64 build of this Dockerfile — observed in NVIDIA-internal nightly pipelines for several weeks. x86_64 builds are unaffected (they find a matching prebuilt cubin variant).

Modifications

In docker/Dockerfile, the framework_final editable-install RUN block:

• Scope || true to the cleanup find only. Wrap the trailing find ... -exec rm -rf {} + in a subshell so the || true no longer swallows earlier failures in the && chain. This is the actual silent-failure fix; it makes future regressions in the retry/install chain fail loudly at the right step, not three stages later in COPY.
• Skip the cubin download on aarch64 with an explicit log message naming kernels-community/sgl-flash-attn3. JIT compilation handles the kernels at runtime (which the silent-failure path was implicitly already doing). A short reference comment marks the branch for removal once arm cubins are published upstream.
• Defensive mkdir -p /root/.cache/huggingface /root/.cache/sglang. /root/.cache/huggingface was previously created as a side effect of kernels download python (HF Hub cache). With that call skipped on aarch64, the adjacent COPY --from=framework_final /root/.cache/huggingface ... in the runtime stage would fail the same way /root/.cache/sglang did. Eliminating this class of silent-skip-then-COPY-fails bug in one place.
• Replace [ -f kernels.lock ] && mv ... || true with if [ -f ... ]; then mv ...; fi. Loud-fails on real mv errors (filesystem, permissions) rather than masking them — consistent with the silent-failure lesson of this very fix.

python/pyproject.toml is intentionally untouched: removing kernels-community/sgl-flash-attn3 from [tool.kernels.dependencies] would also break x86, which can use the prebuilt cubins. Selective skip in the Dockerfile is the right scope.

Diff size: 1 file changed, 18 insertions(+), 9 deletions(-).

Accuracy Tests

N/A — Dockerfile-only build fix. No model output, no kernel/forward changes.

Speed Tests and Profiling

N/A — Dockerfile-only build fix. The functional behavior on aarch64 is unchanged from the prior silent-failure path: cubins were not actually being installed (the retry was failing every time), so kernels were already being JIT-compiled at runtime. This PR just makes that explicit and removes the misleading COPY: not found failure mode.

Verification

Before (representative aarch64 nightly run, sglang bcb34da9f)

#71 [framework_final 4/8] RUN ... kernels download python && success=1 && break ...
#71 21.45 Cannot find a build variant for this system in
          kernels-community/sgl-flash-attn3 (revision: b73eb6a16dea3a785f3c491f4d174d339684c4a3).
          Available variants: torch211-cxx11-cu130-x86_64-linux,
                              torch210-cxx11-cu128-x86_64-linux,
                              torch29-cxx11-cu129-x86_64-linux,
                              torch29-cxx11-cu128-x86_64-linux,
                              torch211-cxx11-cu128-x86_64-linux,
                              torch210-cxx11-cu130-x86_64-linux,
                              torch29-cxx11-cu130-x86_64-linux
... (3 attempts, all fail with same message) ...
#71 DONE 115.4s         <-- RUN reports success despite all failures
...
#81 [runtime 10/16] COPY --from=framework_final /root/.cache/sglang /root/.cache/sglang
#81 ERROR: failed to calculate checksum ... "/root/.cache/sglang": not found
ERROR: failed to build: ...

After (expected)

On aarch64:

#71 [framework_final 4/8] RUN ...
Skipping kernels-community/sgl-flash-attn3 cubin download on aarch64
(no variants published upstream); kernels will be JIT-compiled at runtime
#71 DONE ~5s
...
[runtime 10/16] COPY --from=framework_final /root/.cache/sglang /root/.cache/sglang
... build completes

On x86_64: behavior unchanged — retry loop runs as before, fail-fast on permanent download error preserved.

I will attach links to a green NVIDIA-internal aarch64 nightly Docker build run once the patched build cycle completes.

Checklist

• Code style matches existing patterns in this Dockerfile ($(uname -m) is used elsewhere at lines 146, 195, 201).
• [N/A] Unit tests — Dockerfile build fix; covered by repo's existing CI build of docker/Dockerfile.
• [N/A] Documentation — no user-facing API change.
• [N/A] Accuracy/speed benchmarks (see above).
• Pre-commit: change is whitespace-sensitive Dockerfile shell continuation; no formatter-relevant changes.

Review and Merge Process

1. Ping Merge Oncalls to start the process. See the PR Merge Process.
2. Get approvals from CODEOWNERS and other reviewers.
3. Trigger CI tests with comments or contact authorized users to do so.
   • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.