System Info

• transformers version: 4.45.0.dev0
• Platform: Linux-5.15.0-119-generic-x86_64-with-glibc2.35
• Python version: 3.10.12
• Huggingface_hub version: 0.24.6
• Safetensors version: 0.4.5
• Accelerate version: 0.33.0
• Accelerate config: not found
• PyTorch version (GPU?): 2.4.0+cu121 (True)
• Tensorflow version (GPU?): not installed (NA)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Using distributed or parallel set-up in script?:
• Using GPU in script?:
• GPU type: NVIDIA A100 80GB PCIe

Who can help?

@zucchini-nlp

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

from transformers import AutoImageProcessor, AutoTokenizer


vision_model_name_or_path = "Bingsu/clip-vit-large-patch14-ko"
language_model_name_or_path = "beomi/llama-2-ko-7b"
output_dir = "test"

tokenizer = AutoTokenizer.from_pretrained(language_model_name_or_path, chat_template="test")

processor = LlavaProcessor(
    tokenizer=tokenizer,
    image_processor=AutoImageProcessor.from_pretrained(vision_model_name_or_path),
    chat_template=tokenizer.chat_template,
)

processor.save_pretrained(output_dir)
print(processor.from_pretrained(output_dir).chat_template)

Expected behavior

Hi huggingface!
As the title says, chat_template.json is not generated properly when I do LlavaProcessor.save_pretrained().

So when I do from_pretrained, the chat_template is not loaded properly.

Although there is a section in ProcessorMixin.save_pretrained() to save chat_template,

ProcessorMixin.to_dict() method is removing the chat_template.

Looking at the history, I see that this syntax was added when I added LLava-OneVision.

My guess is that chat_template.json should be saved.