/opt/conda/lib/python3.11/site-packages/peft/tuners/tuners_utils.py:212: UserWarning: Unsupported layer type '<class 'transformers.models.qwen3_5_moe.modeling_qwen3_5_moe.Qwen3_5MoeExperts'>' encountered, proceed at your own risk.
warnings.warn(f"Unsupported layer type '{type(module)}' encountered, proceed at your own risk.", UserWarning)
[Unsloth merge debug] LoRA count mismatch: modules=202, lora_A=241, lora_B=241, scaling=242
[Unsloth merge debug] Total LoRA keys: 242
key=model.language_model.embed_tokens param=None A=None B=None
key=model.language_model.layers.0.mlp.experts param=None A=(4096, 2048) B=(512, 4096)
key=model.language_model.layers.0.mlp.experts.base_layer param=None A=(4096, 1024) B=(2048, 4096)
key=model.language_model.layers.0.mlp.shared_expert.gate_proj param=None A=(16, 2048) B=(512, 16)
key=model.language_model.layers.0.mlp.shared_expert.up_proj param=None A=(16, 2048) B=(512, 16)
key=model.language_model.layers.0.mlp.shared_expert.down_proj param=None A=(16, 512) B=(2048, 16)
key=model.language_model.layers.1.mlp.experts param=None A=(4096, 2048) B=(512, 4096)
key=model.language_model.layers.1.mlp.experts.base_layer param=None A=(4096, 1024) B=(2048, 4096)
key=model.language_model.layers.1.mlp.shared_expert.gate_proj param=None A=(16, 2048) B=(512, 16)
key=model.language_model.layers.1.mlp.shared_expert.up_proj param=None A=(16, 2048) B=(512, 16)
Traceback (most recent call last):
File "/mnt/azureml/cr/j/825a13e46b4d4245bec7d53fbd0d4d8d/exe/wd/unsloth_to_vllm.py", line 49, in <module>
main()
File "/mnt/azureml/cr/j/825a13e46b4d4245bec7d53fbd0d4d8d/exe/wd/unsloth_to_vllm.py", line 43, in main
model.save_pretrained_merged(
File "/opt/conda/lib/python3.11/site-packages/unsloth/save.py", line 2714, in unsloth_generic_save_pretrained_merged
unsloth_generic_save(**arguments)
File "/opt/conda/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/unsloth/save.py", line 2662, in unsloth_generic_save
merge_and_overwrite_lora(
File "/opt/conda/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/unsloth_zoo/saving_utils.py", line 1771, in merge_and_overwrite_lora
) = prepare_saving(
^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/unsloth_zoo/saving_utils.py", line 1485, in prepare_saving
save_size = sum(get_torch_storage_size_new(x, element_size) for x in state_dict.values())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/unsloth_zoo/saving_utils.py", line 1485, in <genexpr>
save_size = sum(get_torch_storage_size_new(x, element_size) for x in state_dict.values())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/unsloth_zoo/saving_utils.py", line 1418, in get_torch_storage_size_new
shape = (x.module.in_features, x.module.out_features)
^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1964, in __getattr__
raise AttributeError(
AttributeError: 'Embedding' object has no attribute 'in_features'
After doing some continued pretraining to Qwen3.5-35B-A3B-Base with LoRA i wanted to merge the adapters into the main model with
model.save_pretrained_merged()but i got an error about Embedding.I trained the following target_modules: "q_proj, k_proj, v_proj, o_proj, down_proj, up_proj, gate_proj, embed_tokens, lm_head" and saved them all after training, then loaded the trained model and merged with this code
Besides the error i also got a warning about a LoRA count mismatch and an unsupported layer in Qwen3_5MoeExperts
Full stack trace
Details
Pip freeze
Details