Port PR sgl-project#23136 (Yuhao Yang): cudaMemcpyBatchAsync lost its failIdx
parameter in CUDA 13, so the dlsym-based call was passing the stream
handle at the wrong slot and segfaulting inside cuMemcpyBatchAsync_v2.
Use driver_version at runtime to dispatch to either the CUDA 12 or
CUDA 13 signature.

With the segfault fixed, move the 7 hicache tests that were parked
under test/manual in PR sgl-project#23119 and subsequent cu13 flake sweeps back
into test/registered so they run in CI again:

- hicache/test_hicache_storage.py
- hicache/test_hicache_storage_3fs_backend.py
- hicache/test_hicache_storage_file_backend.py
- hicache/test_hicache_storage_mooncake_backend.py
- hicache/test_hicache_storage_runtime_attach_detach.py
- hicache/test_hicache_variants.py
- 4-gpu-models/test_qwen35_hicache.py

TODO "move back after fixed" docstrings are stripped and the
register_cuda_ci call that was dropped from the mooncake backend test
on its way to manual is restored.

Co-Authored-By: Yuhao Yang <yhyang201@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

When a USE_VENV=false runner had flashinfer-cubin installed
("already installed, keeping it"), `uv pip uninstall flashinfer-python`
left the flashinfer/data/ subdirectory behind (cubin files still owned
entries below it). The next `uv pip install -e python[dev,runai,tracing]`
then failed with:

  error: Failed to install: flashinfer_python-0.6.7.post3-py3-none-any.whl
    Caused by: failed to create directory
    `/usr/local/lib/python3.10/dist-packages/flashinfer/data`: File exists

Seen on stage-a-test-1-gpu-small in
https://github.com/sgl-project/sglang/actions/runs/24634237642/job/72027123887

Two-layer fix:

1. ci_install_dependency.sh (in-flight safeguard): right after the
   flashinfer uninstall step, if <site-packages>/flashinfer/ still
   exists, rm -rf it and force flashinfer-cubin to reinstall.
   `uv pip install -e python[...]` then resolves both flashinfer-python
   and flashinfer-cubin (both declared in pyproject.toml) and repopulates
   flashinfer/data/ cleanly. This makes the PR self-healing on its
   first run without depending on a prior job's post-cleanup.

2. ci_cleanup_venv.sh (post-job hygiene): the USE_VENV=false arm used
   to `exit 0` immediately. It now uninstalls the flashinfer trio and
   purges residual flashinfer/, flashinfer_cubin/, flashinfer_jit_cache/
   trees from system site-packages so the next job's runner starts
   clean even if the in-flight safeguard ever regresses. Cached wheels
   under ~/.cache/flashinfer-wheels/ keep the reinstall fast.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Addresses the review on sgl-project#23172:
sgl-project#23172 (comment)

cudaMemcpyBatchAsync is a libcudart (runtime) symbol; the ABI of the
function dlsym'd into this process is owned by the libcudart that's
actually loaded, not by the host's kernel driver. Dispatching on
cudaDriverGetVersion() breaks in the common container case where a
cu12 runtime is paired with a cu13-capable host driver: driver=13000
steers us to the 8-param v13 call, but the symbol resolves to v12
(9 params with failIdx), so the stream argument lands in a wrong slot
and we segfault — the exact crash this fix was supposed to prevent.

Reproduced on ion-user-9 with lmsysorg/sglang:dev (cu12.9 runtime):

    cudaDriverGetVersion()  = 13000
    cudaRuntimeGetVersion() = 12090
    v12 dispatch of dlsym'd symbol: cudaSuccess, exit 0
    v13 dispatch of dlsym'd symbol: Segmentation fault (core dumped)

Switching the signature-selection to cudaRuntimeGetVersion makes the
choice follow the loaded libcudart, which is what actually determines
the ABI. The existing cudaDriverGetVersion guard above is kept — it
remains the right knob for the capability check since cudaMemcpyBatch
requires a 12.8+ driver regardless of the runtime version.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

After the main `uv pip install -e python[...]` step, runners that carried
state from the pre-sgl-project#23119 (cu129) era keep `nvidia-cuda-runtime-cu12`
installed as an orphan (Required-by: empty) alongside the cu13 runtime.
Its libcudart.so.12 sits under `nvidia/cuda_runtime/lib/` while cu13's
lives under `nvidia/cu13/lib/`. Both dirs end up on LD_LIBRARY_PATH, so
cudnn_frontend_shim.h's probe

    for lib in ["libcudart.so.12", "libcudart.so.13"]:
        dlopen(lib)

loads both and throws:

    RuntimeError: Multiple libcudart libraries found:
    libcudart.so.12 and libcudart.so.13

Tests hit this during server setUpClass → CUDA graph capture (e.g.
test_nvfp4_gemm_sm120.py on stage-b-test-1-gpu-small). The same failure
reproduces on main, so this is not PR-specific — it's a leftover cleanup
step the cu13 migration missed.

Fix: uninstall nvidia-cuda-runtime-cu12 right after the main install.
Its install dir is disjoint from cu13's so the uninstall doesn't touch
any files shared with cu13 packages (a blunter sweep of all
`nvidia-*-cu12` breaks torch because several pairs share dirs under
`nvidia/<name>/lib/` and uninstalling one deletes files that the cu13
variant still references through its RECORD).

Reproduced and verified on 5090-novita-ci-runner-d (runner-1 container):

    before: libcudart.so.12 + libcudart.so.13 both loadable
    after : only libcudart.so.13 loadable, torch.cuda.randn works

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>