System Info

Error when loading the model "meta-llama/Llama-2-7b-chat-hf" using the following code:

from transformers import AutoModelForCausalLM
chk = 'meta-llama/Llama-2-7b-chat-hf'

if __name__ == '__main__':
    model = AutoModelForCausalLM.from_pretrained(chk, load_in_4bit=True, device_map='auto')
    print("Loaded Ok")

The error message was:

`do_sample` is set to `False`. However, temperature is set to 0.9 -- this flag is only used in sample-based generation modes. Set `do_sample=True` or unset temperature to continue.

This is because the method GenerationConfig.validate() raises a ValueError and that Error is not controlled in modeling_utils.py file.
One possible solution is to add the the ValueError to the except clause in that file:

Who can help?

@gante

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

Using the main branch (install from source code)

from transformers import AutoModelForCausalLM
chk = 'meta-llama/Llama-2-7b-chat-hf'

if __name__ == '__main__':
    model = AutoModelForCausalLM.from_pretrained(chk, load_in_4bit=True, device_map='auto')
    print("Loaded Ok")

Expected behavior

To be able to load the model