Have you tested this? In my local testing previously, implementing this exact change, I ran into an issue where the trainers didn't like not having a pad token in the tokenizer.

@zoidbb does adding:

tokenizer.pad_token = tokenizer.eos_token

helps solving your issue?

Have you tested this? In my local testing previously, implementing this exact change, I ran into an issue where the trainers didn't like not having a pad token in the tokenizer.

Yes, but I tested with my own data pipeline/data collator.

@zoidbb does adding:

tokenizer.pad_token = tokenizer.eos_token

helps solving your issue?

Oddly not, I tried this and it still complained. I'll provide more details this evening when I'm done with work.

@zoidbb you may also need:

tokenizer.pad_token_id = tokenizer.eos_token_id