Is your feature request related to a problem? Please describe.
Recent DPO models being so good shows the impact of having good rlhf datasets.

Describe the solution you'd like

Similar to lymsys's arena battle. The ability to make multiple draft generations per prompt generations (from the same model, or different models/endpoints) and then the user would be able to click which one is better(possibly even edit the answer and fix). This would provide very rich and useful conversation trees along with rlhf data.

Describe alternatives you've considered
Lymsys ui kind of does this. With the battle mode but limited to one prompt, no follow up. I was looking for the ability to make multiple drafts every turn in the conversation.