What does this PR do?

This PR fixes two things:

• Incorrect input for liger: The input_ids, attention_mask and loss_mask were being created differently in the liger and regular implementations. This PR fixes that in the _compute_loss_liger method.
• Incorrect handling of loss_mask: This PR correctly masks the labels with the ignore_index to make sure that liger is correctly ignoring the tokens that are not required for loss calculation.
• Changes all_logps = per_token_logps.sum(-1) to all_logps = per_token_logps[:,1:].sum(-1). This ensures that the metrics are same for liger and non-liger case. NOTE: This is just for metrics and can be removed.

Testing

The changes were tested on a single GPU setup.

# train_dpo.py
from datasets import load_dataset
from trl.trainer.dpo_trainer import DPOTrainer
from trl.trainer.dpo_config import DPOConfig
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2-0.5B-Instruct")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2-0.5B-Instruct")
train_dataset = load_dataset("trl-lib/ultrafeedback_binarized", split="train")

training_args = DPOConfig(
    output_dir="Qwen2-0.5B-DPO",
    logging_steps=1,
    max_steps=10,
    use_liger_loss=False,
    per_device_train_batch_size=8,
    save_strategy="no"
    )

trainer = DPOTrainer(model=model, args=training_args, processing_class=tokenizer, train_dataset=train_dataset)
trainer.train()

Here is the output of the above code with use_liger_loss=False:

Output with use_liger_loss=True:

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a GitHub issue? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes?
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.