The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

yes, sounds good to do it at the warning level - important to fix soon as this is breaking all transformers usage of llama 2 (and perhaps other models).

Thanks for your quick PR! It seems weird to intercept this error here. The problem seems to stem from an invalid generation config (or the exception is badly chosen) so maybe do another except with a different log: probably at warning level, telling the user the generation config is invalid so we go back to a default config.

What do you think!

@sgugger Absolutely, that makes sense. I've made the necessary changes. I've added another except block with a warning level.

Uhmmm there are way more models out there with generation config issues than I thought 💔

It seems validation needs more thought, namely:

1. We should be able to load incorrect files, but perhaps keep strictness at save time (to avoid perpetuating bad practices)
2. At least during a transition phase, we must downgrade these exceptions to warnings.

@dbuos This change is something I'd like to avoid -- if we are not throwing exceptions, we should keep doing things the way we were using before, and not reset the generation config. I'm working on validation this week, so I'd like to take this one :) As such, I'm closing this PR.

(cc @sgugger)

ok @gante , will you roll back the latest update then to reverse the breaking changes for models like Llama 2? I was just running the model and same issue. I'm moving to use transformers 4.31 but it's not ideal having to hard code that.

@RonanKMcGovern Yeah, you will be able to run

from transformers import AutoModelForCausalLM
chk = 'meta-llama/Llama-2-7b-hf'

if __name__ == '__main__':
    model = AutoModelForCausalLM.from_pretrained(chk, load_in_4bit=True, device_map='auto')
    print("Loaded Ok")

and use the model without modifications or changes in behavior. You may see a bunch of new warnings guiding you towards correct generate parameterization, though ;)

@RonanKMcGovern apologies if my reaction was perceived as abrupt when closing this PR! Your prompt reaction to fix this issue was appreciated 🤗