Feature request

For optimisation with reference model, in most cases the reference model is the same as the trained model. We should allow the user to specify the ref model only when they don't want to use the trained model.

Currently this is possible, but only when using PEFT, which is very counter-intuitive. And even using this situation, if you want to provide a ref model that is different from the trained model, you have to define force_use_model. Even more counter-intuitive.

Currently

1. model = ref_model and no peft

DPOTrainer(model=model, ref_model= ref_model)  # where ref_model should be another instance

2. model = ref_model and peft

DPOTrainer(model=model)

3. model != ref_model and no peft

DPOTrainer(model=model, ref_model=ref_model)

4. model != ref_model and peft

args = DPOConfig(force_use_ref_model=True)
DPOTrainer(model=model, ref_model=ref_model, args=args)

Proposed

1. model = ref_model

DPOTrainer(model=model)

2. model != ref_model

DPOTrainer(model=model, ref_model=ref_model)

Motivation

Make the lib use more intuitive.

Your contribution

For sure ;)