This line can cause a division-by-zero error if channels is 2, as the expression channels // 2 - 1 would evaluate to 0. While the default d_model (which is passed as channels) is large, a custom model configuration could set it to 2, leading to a crash. Please add a check at the beginning of the __init__ method to prevent this. For example:

if channels < 4:
    raise ValueError(f'SinusoidsPositionEmbedding needs channels to be >= 4 (got {channels})')

Shape mismatch when feature_exist_mask is partial

Medium Severity

When feature_exist_mask is not all True, hidden_states[feature_exist_mask] produces a tensor with fewer batch items than continuous_audio_hidden_states. The addition or assignment attempts to combine tensors of shape (num_valid, seq_len, hidden) with (batch_size, seq_len, hidden), causing a shape mismatch. The continuous_audio_hidden_states tensor also needs to be indexed with feature_exist_mask.

Unused avg_pooler module is dead code

Low Severity

self.avg_pooler is defined as nn.AvgPool1d(2, stride=2) but never used. The code instead uses nn.functional.avg_pool1d directly at line 389. Since nn.AvgPool1d has no learnable parameters, this module serves no purpose and can be removed.

Config defaults cause TypeError when creating model

Medium Severity

The config declares codebook_size: int | None = None and pad_token_id: int | None = None with None defaults. However, FunAudioChatDiscreteEncoder.__init__ calls int(config.pad_token_id) and int(config.codebook_size) which raises TypeError when these are None. When FunAudioChatConfig is created without an explicit audio_config, a default config with these None values is used.

Computed embed_scale variable is never used

Low Severity

self.embed_scale is computed based on config.scale_embedding but is never used in the forward method. If scale_embedding=True is configured, the expected scaling behavior (multiplying embeddings by sqrt(d_model)) will not occur, causing a mismatch with the intended HuggingFace model behavior.

Config mutation affects shared state across requests

Medium Severity

When processing long audio (speech_maxlen >= 7500), the code permanently mutates self.config._attn_implementation to "flash_attention_2". This state change persists after the forward pass completes, affecting all subsequent inference requests even for short audio that wouldn't require FlashAttention-2.

Respect text_config model_type when selecting LM arch

The FunAudioChat config explicitly defaults text_config to Qwen2 for backward compatibility (vllm/transformers_utils/configs/funaudiochat.py:97-109), but the model initialization hard-codes architectures=["Qwen3ForCausalLM"]. That forces Qwen3 even when the checkpoint/config is Qwen2, so older FunAudioChat checkpoints (or configs without an embedded text_config) will fail to load due to mismatched architecture/weights. Consider deriving architectures from config.text_config.model_type or config.text_config.architectures instead of forcing Qwen3.

Useful? React with 👍 / 👎.

You should use merge the QKV projections into QKVParallelLinear and use RowParallelLinear for out_proj. During weight loading you can use stacked_params_mapping to merge the weights, look at other models for examples.

You should use merge the QKV projections into QKVParallelLinear and use RowParallelLinear for out_proj. During weight loading you can use stacked_params_mapping to merge the weights, look at other models for examples.

now added qkvparallelLinear and rowparallellinear

This parameter initialize seems to be training specific, so it can be removed

This parameter initialize seems to be training specific, so it can be removed

done

Use vLLM's attention modules instead (refer to other models)

By the way, have you tried loading the model automatically using --model-impl transformers without needing this PR?

By the way, have you tried loading the model automatically using --model-impl transformers without needing this PR?

I believe that funaudiochat is not implemented in huggingface transformers yet. This model is fully implemented in https://github.com/FunAudioLLM/Fun-Audio-Chat.

Use vLLM's attention modules instead (refer to other models)

done using mmencoderattn

Can you move these examples into the consolidated example files (like audio_language.py)? We want to avoid having a separate example file for every model

sure, sorry these are for my debugging and attention alignment

Can you move these examples into the consolidated example files (like audio_language.py)? We want to avoid having a separate example file for every model

removed other examples and consolidate the funaudiochat examples in audio_language.py

Can you add an estimated min_transformers_version?

currently not possible, i will send the PR to transformers, but uh... not sure the version will be

Can you add an estimated min_transformers_version?

Ok no problem