Overview

The mm_agg_router_* post-merge tests were passing for the wrong reasons. The base assertion cached_tokens >= 1 is satisfied any time vLLM's per-worker engine prefix cache fires — even when MM-aware KV routing has silently degraded to text-prefix routing and load-balancing just happens to put both repeats on the same worker. We could not distinguish "MM routing engaged" from "lucky round-robin landing." Real regressions would have slipped through.

This PR tightens those tests so they fail closed when MM-aware routing degrades, and addresses the structural mismatches that the tighter gate then exposed for Phi-3, Qwen2-VL, Qwen2.5-VL, and the fastokens tokenizer path. All four affected model profiles now route deterministically with non-zero [ROUTING] N/M blocks overlap on the repeat (load-balance luck no longer required).

How the test was hardened

CachedTokensChatPayload grows three opt-in kwargs (each off by default, no behavior change for callers that don't set them):

• require_lightseek_init=True — asserts the MM-aware KV routing enabled (lightseek) init log fired (proves the routing path is wired up at deployment time).
• min_avg_kv_hit_rate=0.9 — scrapes the router_kv_hit_rate Prometheus histogram from the frontend /metrics endpoint, snapshots it after R1, and asserts the R2+ delta mean is at or above the threshold. In MM-aware mode every routed model converges to (M-1)/M (≥0.944 observed across the four profiles); in the broken-hash regression class fixed in this PR the rate is ~0. Pure-text-fallback is caught separately by require_lightseek_init.
• require_vllm_mm_processor_init=True — chat-processor path equivalent for [mm-routing] Transfer mode:.

make_image_payload_cached_tokens also bumps repeat_count from 2 to 3 so the metric mean covers both R2 and R3 — a single bad routing decision on either one drops the mean below 0.9 and fails the test.

The four routed MM profiles (mm_agg_router_qwen3-vl-2b, mm_agg_router_qwen2.5-vl-3b, mm_agg_router_qwen2-vl-2b, mm_agg_router_phi3-vision) opt into the strong gate.

What the strong gate exposed (and how each was fixed)

1. Phi-3-vision — chat template emits BPE-shatterable text + vLLM does a decode-roundtrip

Phi-3's chat template renders <|image_{n}|> as plain text. The Rust tokenizers crate BPE-shatters that into ~7 sub-tokens, and the leading < can merge with the preceding . (word.< → token id 19423). vLLM's _apply_token_matches therefore can't find the placeholder in the prompt tokens and falls back to _apply_text_matches, which decodes the prompt tokens back to text, substitutes, and re-encodes — and the decode inserts a space after each special token (e.g. <|user|>\n → <|user|> \n), bumping the SentencePiece prefix from ▁ (id 29871) to ▁▁ (id 259). The worker therefore hashes its KV blocks on a token sequence that differs from the router's routing_token_ids by a few ids — and every block hash from byte 0 onward diverges.

The fix mirrors vLLM's behavior exactly: the routing path does an encode → decode → re-encode roundtrip on the prompt, then splits at <|image_N|> and tokenizes each segment with the segment-BOS that Phi-3's HF processor produces. Locally captured all_token_ids from vLLM's HF processor and the router's routing_token_ids now match byte-for-byte.

Phi-3's tokenizer_config.json also declares add_bos_token: true, so the routing-side sequence prepends <s> (id 1) at the very front to match the backend's HF-processor output. No-op for add_bos_token: false models.

2. Qwen2-VL / Qwen2.5-VL — chat-template token id ≠ per-patch expansion token id

These families' config.json declares image_token_id: 151655 (<|image_pad|>) — what the chat template emits per image and what the backend's HF processor fills the expanded sequence with. Lightseek's placeholder_token_id() returns a different id (151654 = <|vision_pad|>); that id never appears in the worker's stored sequence, so using it as the routing-side fill token leaves every image-region block hashing differently from the backend.

Resolve chat_placeholder_token_id from config.json's image_token_id field and use it as both the find-target and the expansion fill token. Falls back to lightseek's value when config.json doesn't expose the field. Families where the two coincide (Qwen3-VL, LLaVA, Phi-3 after substitution) take the same code path as before — no behavior change.

3. Qwen2-VL-2B — Rust tokenizer loader misses tokenizer_config.json

Qwen2-VL-2B's release declares <|image_pad|> as a special token only in tokenizer_config.json's added_tokens_decoder field — not in tokenizer.json's added_tokens list. HF Python's AutoTokenizer.from_pretrained() reads both and merges. The Rust tokenizers crate reads only tokenizer.json and BPE-shatters <|image_pad|> into 6 sub-tokens. This gap predates MM routing; text-prefix routing didn't care because it didn't require token-identity alignment with vLLM's view.

HuggingFaceTokenizer::from_file (and a new from_tokenizer_with_model_dir variant called from model_card.rs) now reads the sibling tokenizer_config.json at load time and merges any added_tokens_decoder entries with special: true via the tokenizers crate's add_special_tokens() API. Aligns Rust with HF Python reference behavior. Idempotent for models that already have those tokens in tokenizer.json.

Cost: load-time only (~10-30 ms one-time, one extra file read + JSON parse). Encode hot path: zero overhead — tokenizers uses a trie for special-token recognition, lookup is O(input length) independent of trie size.

4. fastokens — no tokenizer_config.json merge available

The opt-in DYN_TOKENIZER=fastokens backend doesn't expose a special-token mutator on its Tokenizer type, so we can't replay the tokenizer_config.json merge there. Running MM-aware routing on a tokenizer that's BPE-shattering placeholders would silently emit wrong block hashes — far worse than falling back. The preprocessor now logs a single warning at init when DYN_TOKENIZER=fastokens is set and disables MM-aware routing (falls back to text-prefix routing). Tagged TODO(mm-routing) in the source; the guard goes away once fastokens upstream exposes a mutator.

Local validation

Each routed model run 3× (Phi-3 run 10× as the most-changed path); every R2 routing decision shows real [ROUTING] N/M blocks overlap matching, not load-balance luck.

The "1 missing block" in every overlap is structural: the router hashes N blocks (the trailing one is zero-padded to block boundary), but vLLM only publishes BlockStored for complete blocks of 16 tokens — so the partial-tail block sits just outside the worker's stored set. Every other block matches. The 0.9 gate threshold sits below the lowest observed floor (0.944) with margin, and well above the ~0.0 the broken-hash regression class produces.

A post-merge CI dispatch on this branch is running: https://github.com/ai-dynamo/dynamo/actions/runs/26542084108

Where reviewers should start

1. tests/utils/payloads.py + tests/utils/multimodal.py — the strong-gate kwargs (require_lightseek_init, min_routing_total_blocks) and the helper they wrap.
2. tests/serve/multimodal_profiles/vllm.py — the four model profiles opting into the gate.
3. lib/llm/src/preprocessor.rs — splice_numbered_placeholders_at_token_level (decode-roundtrip + per-segment BOS); chat_placeholder_token_id resolution; the DYN_TOKENIZER=fastokens guard.
4. lib/tokenizers/src/hf.rs — the tokenizer_config.json merge. Low-level loader used by all of dynamo; changes are additive/idempotent.
5. lib/llm/src/model_card.rs — the one-call-site update that routes the preprocessor tokenizer through the merging constructor.

A TODO(mm-routing) in read_image_token_id_from_config and the fastokens guard flag follow-ups (consolidate dual-token info into a single lightseek API; remove the fastokens guard once upstream exposes a mutator). Not blocking this PR.

Closes DIS-1977

Related issues

• relates to: feat(mm-routing): lightseek-mm Rust-frontend MM-aware KV routing #9272 (the MM-aware KV routing PR that introduced the dependency exposing these gaps)
• builds on: fix(self-host): harvest non-weight siblings into slug_dir (gh-8749) #9610 (MDC verify-and-cache slot fix for preprocessor_config.json, now merged to main)

🤖 Generated with Claude Code