Motivation

Issue: #22025 (streaming input design and implementation)

This PR adds chunk-based streaming transcription support for Qwen3-ASR via the existing /v1/audio/transcriptions endpoint with stream=true. Audio is processed in 2-second chunks with prefix rollback, emitting partial transcripts via SSE as each chunk completes — reducing time-to-first-text compared to waiting for the full audio to be processed.

Built on top of #22073 (Qwen3-ASR model support by @adityavaid). This PR is branched from #22073 and should be merged after it. My changes are streaming_asr.py (new), streaming additions in serving_transcription.py, and a config fix in hf_transformers_utils.py.

References

• Qwen3-ASR paper: arXiv:2601.21337 (streaming algorithm: 2s chunk size, 5-token fallback, 4 unfixed chunks)
• Qwen3-ASR official streaming implementation: qwen_asr/inference/qwen3_asr.py
• vLLM streaming RFC: [RFC]: Model-specific realtime streaming abstraction vllm-project/vllm#35908 (client-side approach discussion)
• vLLM streaming quality findings: [Enhancement]: Qwen3-ASR realtime endpoint produces degraded output — stateless segments, no cross-segment context, raw format leaks vllm-project/vllm#35767 (same degradation issues)

Approach

Based on the Qwen3-ASR streaming algorithm (arXiv:2601.21337):

• Audio is split into 2-second accumulated chunks (each chunk contains all audio from 0 to current position)
• Each chunk is processed as an independent transcription request
• Previous transcript is fed as decoder prefix with rollback (last 5 words dropped) to reduce boundary jitter
• First 2 chunks are "cold start" — no prefix is injected (model doesn't have enough context yet)
• Stable text deltas are emitted via SSE word-by-word for smooth output
• Client disconnection is detected between chunks to avoid wasted computation

This is a server-side chunked streaming approach. vLLM is exploring a complementary client-side approach (vllm-project/vllm#35908) where callers control chunking via a prefix parameter. Both use the same underlying algorithm.

New files

Modified files

These are consistent with vLLM's findings (vllm-project/vllm#35767): their Qwen3-ASR realtime endpoint also produces degraded output with stateless segments and no cross-segment context. Quality improves with larger chunk_size_sec (e.g. 2s → 4s, fewer boundaries) at the cost of higher first-text latency.

Tested Cases

Test audio: https://huggingface.co/datasets/Narsil/asr_dummy/resolve/main/1.flac (11s English speech)

Derived clips generated via:

import soundfile as sf
data, sr = sf.read('/tmp/test_speech.flac')
sf.write('/tmp/short.flac', data[:int(sr*1.5)], sr)   # 1.5s, 1 chunk
sf.write('/tmp/exact.flac', data[:int(sr*2.0)], sr)   # 2.0s, exact boundary
sf.write('/tmp/mid.flac',   data[:int(sr*5.0)], sr)   # 5s, 3 chunks
Chinese test audio (gTTS)
from gtts import gTTS

import librosa, soundfile as sf

gTTS('今天天气真好，我想去公园散步，顺便买一些新鲜的水果回家做饭', lang='zh').save('/tmp/test_zh.mp3')

data, sr = librosa.load('/tmp/test_zh.mp3', sr=16000)

sf.write('/tmp/test_zh.flac', data, sr)

Architecture: Why Each Chunk Is an Independent Request

Each chunk is processed as a separate SGLang inference request. This has two consequences:

1. Repeated encoding: Each chunk re-encodes all accumulated audio from the start. For 10s audio with 2s chunks, total encoder work is 2+4+6+8+10 = 30s instead of 10s (~3x overhead). Acceptable for small models (0.6B encoder < 100ms per chunk) but becomes a bottleneck for long audio.

2. No shared state across chunks: No KV cache reuse between chunks, no encoder output caching. SGLang's scheduler treats each request independently — no concept of a "streaming session" that persists across requests.

This is the MVP architecture choice: operate purely at the serving layer, no scheduler or model code changes. Keeps the change minimal and reviewable.

Known Limitations (planned follow-ups)

• Encoder window caching: Cache completed 100-frame encoder windows across chunks to avoid re-encoding (requires model-layer changes to get_audio_feature)
• Token-level prefix: Current prefix is text-level (string concatenation); token-level prefix injection would be more precise
• Cross-chunk KV cache reuse: Requires scheduler to support persistent streaming sessions
• True real-time input: WebSocket endpoint for live audio streaming instead of chunked processing of complete audio
• Token-level output within chunks: Use stream=True per chunk for smoother output (current implementation emits word-by-word after each chunk completes)
• prefix API parameter: Add optional prefix parameter to /v1/audio/transcriptions for client-side streaming control (as discussed in [RFC]: Model-specific realtime streaming abstraction vllm-project/vllm#35908)

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Follow the SGLang code style guidance.