Skip to content

fix: normalise logits_processors None to [] to avoid mlx-lm crash (#934)#1747

Merged
jundot merged 1 commit into
jundot:mainfrom
monroewilliams:fix/logits-processors-none-to-empty-list
Jun 9, 2026
Merged

fix: normalise logits_processors None to [] to avoid mlx-lm crash (#934)#1747
jundot merged 1 commit into
jundot:mainfrom
monroewilliams:fix/logits-processors-none-to-empty-list

Conversation

@monroewilliams

@monroewilliams monroewilliams commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

Reproduction

 ERROR: POST http://127.0.0.1:8000/v1/chat/completions                                                                        
                                                                                                                                          
 Caused by:                                                                                                                               
     0: Failed to create completion message: {"error": {"message": "Cache corruption not recoverable after retries: 'NoneType' object is  
 not iterable", "type": "server_error"}}                                                                                                  
     1: {"message":"Cache corruption not recoverable after retries: 'NoneType' object is not iterable","type":"server_error"}             
  • The "Cache corruption" error message seems to be spurious, I cleared the in-memory and ssd caches and it didn't resolve.

Diagnosis

When logits_processors is None (no repetition/presence/frequency penalties configured), mlx-lm's GenerationBatch._step crashes with:

TypeError: 'NoneType' object is not iterable

This is caught by omlx's cache corruption recovery, but since the root cause persists, requests fail after 3 retries with:

Cache corruption not recoverable after retries: 'NoneType' object is not iterable

Fix

Three layers of defense to normalise None[]:

  1. _create_batch_generator — pass [] instead of None to BatchGenerator constructor
  2. _patched_generation_batch_step — normalise None[] before calling mlx-lm's _step (which crashes at any(None))
  3. _patched_ppb_split — normalise None[] when splitting batches

Consistent with the existing fix at line 6528 where per_row_lps = list(logits_processors) if logits_processors else [] was already applied.

…ndot#934)

mlx-lm's GenerationBatch._step does any(self.logits_processors) and
for p in self.logits_processors[e], both of which crash with
'NoneType' object is not iterable when logits_processors is None.

Apply three layers of defense:
1. _create_batch_generator: pass [] instead of None to BatchGenerator
2. _patched_generation_batch_step: normalise None → [] before calling
   the original mlx-lm _step (which crashes at any(None))
3. _patched_ppb_split: normalise None → [] when splitting batches
@jundot

jundot commented Jun 9, 2026

Copy link
Copy Markdown
Owner

Thanks for the defensive cleanup. The scheduler's normal insert path already passes per-row logits processor lists, and this additional normalization keeps the remaining boundaries from leaking None into mlx-lm. The focused scheduler/logits tests and CI look good, so I'm going to merge this.

@jundot jundot merged commit f0c3d3c into jundot:main Jun 9, 2026
4 checks passed
efortin added a commit to efortin/omlx that referenced this pull request Jun 10, 2026
…erge

GenerationBatch.extend() re-creates None row slots on heterogeneous
continuous-batch merges (`any([[], []])` is False), undoing the
insert-side empty-list shape from jundot#1747 whenever a request without
active processors joins a batch serving a grammar-constrained one.
The next _step then crashes with "'NoneType' object is not iterable",
and the corruption recovery re-merges the same rows, looping until the
retry budget fails every in-flight request.

Normalise every per-row slot at the _patched_generation_batch_step
chokepoint, covering both the insert and merge origins, and pin the
invariant with unit, source-level, and integration regression tests.

Fixes jundot#1798
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants