- Updated ResponseGenerator to load model resources in a dedicated thread, improving responsiveness.
- Introduced a wait_until_ready method to ensure the model is fully loaded before generating responses.
- Added error handling for model loading failures, allowing for graceful degradation.
- Removed direct model loading from get_cached_model, streamlining the initialization process.

This change enhances the overall architecture by decoupling model loading from response generation, ensuring better performance and reliability.

Every VLM's decode __call__ already has an `isinstance(offset, int)` fast
path (used by plain KVCache) alongside an `isinstance(offset, mx.array)`
branch that ends in `.item()` — a blocking GPU sync on every decode step.
For uniform left-padding (the common case, including bs=1) the offset is
effectively scalar, so keeping it as a Python int lets the fast path win
for batch caches too.

Measured on Qwen2.5-VL-3B-Instruct-4bit, bs=1, 193 steady-state tokens:
  - Before: BatchGenerator / stream_generate ratio 0.80x (19.6% overhead)
  - After:  ratio ~1.0x (at parity)

- `_ScalarOffsetMixin` on upstream BatchKVCache / BatchRotatingKVCache via
  name-shadowing subclasses in mlx_vlm.models.cache — no monkey-patching.
- Same scalar-init + inflate-in-filter/extend applied inline to the local
  BatchQuantizedKVCache and BatchTurboQuantKVCache.
- `left_padding` stays as mx.array (feeds into create_causal_mask etc.).
- Non-uniform left-padding still uses mx.array offset; divergent-offset
  paths in __call__ are unchanged.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Upstream mlx-lm keeps a Python-int token counter `self._idx` on
`BatchKVCache`; it never calls `.item()` on `cache.offset` in the hot
path. Mirror that convention in mlx-vlm's VLMs so the decode step avoids
the per-step GPU sync without relying on the `_ScalarOffsetMixin` scalar
fast-path.

Affects the four VLMs that had the same 3-line `.item()` pattern:
  - qwen2_5_vl
  - qwen3_5
  - qwen2_vl
  - paddleocr_vl

Semantic note: `_idx` is the raw token count (includes padded positions).
`get_rope_index` already encodes the left-padding offset into
`mrope_position_deltas` (via `max_pos + 1 - seq_len`), and attention
masking for padded positions is handled by `cache.make_mask` →
`create_causal_mask(..., left_padding=...)`. So `_idx + rope_deltas`
yields the correct absolute rope position for both uniform and
left-padded batches — and fixes a latent bug where the previous
`max(offset, 0) + rope_deltas` would double-subtract the padding.

The divergent-offset path (continuous-batching, per-sequence offsets)
still uses `mx.maximum(cache.offset, 0)` — unchanged.

Measured on Qwen2.5-VL-3B-Instruct-4bit, bs=1, 193 steady-state tokens:
BatchGenerator / stream_generate ratio 1.01x (at parity).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Extend the previous commit to the remaining VLMs that had the same GPU-
sync-per-decode pattern. Rope positions count across the full (possibly
padded) sequence — attention masking (via cache.make_mask →
create_causal_mask with left_padding) handles per-sequence padding.

Models rewritten:
  - qwen3_vl, qwen3_vl_moe, qwen3_omni_moe
  - glm4v, glm4v_moe, glm_ocr
  - ernie4_5_moe_vl
  - falcon_ocr
  - gemma3n, mllama, hunyuan_vl, moondream3

For the last four (which had simpler `.item()` call-sites feeding
`self.rope(offset=...)` or position arithmetic), the `.item()` paths are
kept as fallbacks for caches that don't expose `_idx` — the hot path
uses `int(cache._idx)`.

Bench parity (Qwen2.5-VL-3B-Instruct-4bit, bs=1, 193 tokens):
BatchGenerator / stream_generate = 1.07x (BatchGenerator slightly ahead).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Every MRoPE VLM was rewritten in the previous two commits to use
``cache._idx`` (Python int) directly instead of ``cache.offset.item()``.
That made the scalar-int fast-path on ``cache.offset`` unreachable, so
the mixin — which collapsed uniform batch offsets to a Python int — is
dead weight.

Fix: drop the mixin + name-shadowing subclasses and restore upstream
``BatchKVCache`` / ``BatchRotatingKVCache``. Also revert the inline
scalar-offset init and int→mx.array inflation guards from the local
``BatchQuantizedKVCache`` and ``BatchTurboQuantKVCache`` for the same
reason.

Side benefit: fixes a crash on ``BatchRotatingKVCache(max_size,
left_padding)`` — the mixin's positional-argument assumption was wrong
for this class (its signature has ``max_size`` first), which surfaced
as ``TypeError: 'int' object is not iterable`` on gemma3n (and any
other model that uses a rotating KV cache) when starting the server.

Verified: gemma3n server now serves requests; Qwen2.5-VL bench stays
at rough parity with ``stream_generate`` (0.93x steady-state,
within run-to-run variance of the previous 1.01-1.07x).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Updated variable names for clarity: changed `deepstack_image_embeds` to `deepstack_visual_embeds` in both `qwen3_vl.py` and `qwen3_vl_moe.py`. This change improves code readability and consistency across the models. No functional changes were made.

Newer ``transformers`` releases (5.7+) emit ``preprocessor_config.json``
entries like ``"min_pixels": null`` / ``"max_pixels": null`` alongside
``"size": {"shortest_edge": ..., "longest_edge": ...}``. The Qwen3-VL
config loader was unconditionally overwriting ``out[...]`` with those
``None`` values after deriving sensible defaults from ``size``, which
then crashed ``_smart_resize_image`` with:

  TypeError: '>' not supported between instances of 'int' and 'NoneType'

Reproduced with ``mlx-community/Qwen3-VL-30B-A3B-Instruct-4bit`` and
``mlx-community/Qwen3-VL-2B-Instruct-4bit`` on transformers 5.7.0.dev0.
Works on transformers 5.5.4 because that release doesn't emit the
explicit ``None`` keys.

Fix: only overwrite when the incoming value is not ``None``. Same
treatment for the video preprocessor kwargs. Also harden the ``size``
lookup against a ``None`` size dict.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

``visual_embeds`` is the concatenation of visual tokens across the whole
batch (e.g. (3334 + 1734, 2048) for two images), but the per-sample
loop was passing the full tensor to ``batch_result.at[indices].add(...)``
regardless of how many visual positions that sample has. For anything
beyond a single image (bs > 1 or a single sample with >1 image) this
crashed with:

  Shapes (N, D) and (M, D) cannot be broadcast.

Fix: carry a running offset and slice ``visual_embeds[offset:offset+n]``
for each sample, where ``n`` is that sample's visual-position count.

Also fixes a latent bug in qwen3_vl_moe: the multi-image branch was
appending ``batch_hidden`` (pre-deepstack) instead of ``batch_result``,
so the deepstack add was silently dropped whenever the loop body ran.

Verified end-to-end with Qwen3-VL-2B-Instruct-4bit on two images of
different sizes (cats.jpg + graph.png).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Added assertions to ensure that the `_deepstack_process` method correctly handles multi-image batches without broadcasting errors. Also included checks to confirm that the RoPE offset is derived from `cache._idx` instead of `offset.item()`, addressing potential GPU synchronization issues. These changes improve the robustness of the model tests and guard against regressions related to recent refactors.