…state

Build LMCache#2599 with the `vllm --help` probe in place proved the env is
already broken immediately after `uv pip install -U vllm ...`, before
LMCache install and before any post-install eviction: the auto-heal
loop trips the "non-ModuleNotFoundError" branch with the exact
ImportError traceback from vllm/transformers_utils/config.py:18.

The same install recipe replayed in a fresh local venv (including a
full requirements/cuda.txt-based base-image emulation) always
succeeds. The divergence is therefore filesystem state on the K3s
pod coming out of the cached base image, not something we can fix
by regenerating bytecode after the fact.

Apply the minimum-blast-radius fix: tell uv to uninstall-and-
reinstall the full vllm serve import chain (transformers, tokenizers,
huggingface-hub, safetensors, vllm) even when it thinks the existing
install is already up to date. `--reinstall-package` implies
`--refresh-package`, so the wheels come down fresh and are extracted
over freshly cleared paths. Combined with a pre-install
`uv cache clean` + `__pycache__` wipe and the existing post-install
eviction, this puts the import chain on guaranteed-clean ground
regardless of what the base image had.

Cost is a few extra seconds of re-download; the base image stays
the same. If a future job hits the same failure, the setup still
fails fast with the full traceback (via the pre-install auto-heal
loop), pointing at whatever upstream break is actually at fault.

Signed-off-by: Samuel Shen <slshen@tensormesh.ai>

…gging

Build LMCache#2652 with --reinstall-package on the whole import chain still
fails with the same ImportError: freshly extracted transformers 5.5.4
wheel, GenerationConfig still missing from the top-level namespace
according to Python, while an identical recipe in any fresh local
venv produces a working transformers import.

I'm out of remote-debuggable hypotheses for why this is CI-specific.
Add a diagnostic block that the auto-heal loop runs when the probe
hits the "non-ModuleNotFoundError" branch. It dumps:

- `uv pip list` for the transformers chain
- ls+stat of transformers/__init__.py and its .pyc
- the dist-info METADATA Version
- the __version__ and _import_structure["generation"] block from the
  actual __init__.py on disk
- what Python itself sees: sys.executable, sys.path,
  transformers.__file__, whether GenerationConfig is in dir() and in
  _class_to_module / _import_structure, and the traceback of an
  isolated `from transformers import GenerationConfig` attempt

Three outcomes, each unblocks the next step:

1. The file-on-disk _import_structure does *not* contain
   GenerationConfig -> the wheel or its extraction is corrupt; pin
   transformers or change the index.
2. Python loads a different transformers.__file__ than we expect, or
   _import_structure is absent -> shadowing/.pth/PYTHONPATH issue;
   inspect sys.path.
3. Isolated `from transformers import GenerationConfig` WORKS in
   the diagnostic block -> the failure depends on vllm's prior
   imports; we can then bisect the vllm import chain.

This commit just adds the dump. Once a build runs with this script
the real fix will be obvious from the diagnostic output.

Signed-off-by: Samuel Shen <slshen@tensormesh.ai>

Build LMCache#2653's diagnostic dump proved the transformers install is
correct and that isolated `from transformers import GenerationConfig`
works fine inside the failing pod. The failure only manifests through
vllm's CLI entry point.

Root cause is in vllm/entrypoints/cli/main.py itself: the module spawns
a daemon thread (`_bg_preload_torch`) that calls `import torch` and
then `import transformers` at module-scope, racing the main thread
which proceeds into main() -> vllm.entrypoints.cli.benchmark.main ->
... -> vllm.transformers_utils.config:18 ->
`from transformers import GenerationConfig, PretrainedConfig`.

On the K3s pods the race lands deterministically in a state where
transformers' _LazyModule._class_to_module cannot resolve
'GenerationConfig' (even though, as the diagnostic confirms, the
fully-initialized module contains it). A fresh local venv with
identical versions cannot reproduce it, consistent with a
timing-sensitive race. The diagnostic ran `import transformers` on
the main thread as its first action, which is exactly why it didn't
trip the race.

Fix: after `uv pip install -U vllm ...`, patch
vllm/entrypoints/cli/main.py to add `import transformers` at module
top, before the BG thread is spawned. Once transformers is already
in sys.modules with _LazyModule fully initialized, the BG thread's
`import transformers` becomes a no-op and the later
`from transformers import ...` on the main thread is just an
attribute lookup against a fully-ready module.

The patch is idempotent (marker comment prevents double-application)
and fails loudly if vllm restructures the file. Once upstream vllm
fixes this on their side, this patch block can be removed.

Signed-off-by: Samuel Shen <slshen@tensormesh.ai>