Robust Preference Optimization through Reward Model Distillation
CoRR(2024)
Abstract
Language model (LM) post-training (or alignment) involves maximizing a reward
function that is derived from preference annotations. Direct Preference
Optimization (DPO) is a popular offline alignment method that trains a policy
directly on preference data without the need to train a reward model or apply
reinforcement learning. However, typical preference datasets have only a
single, or at most a few, annotation per preference pair, which causes DPO to
overconfidently assign rewards that trend towards infinite magnitude. This
frequently leads to degenerate policies, sometimes causing even the
probabilities of the preferred generations to go to zero. In this work, we
analyze this phenomenon and propose distillation to get a better proxy for the
true preference distribution over generation pairs: we train the LM to produce
probabilities that match the distribution induced by a reward model trained on
the preference data. Moreover, to account for uncertainty in the reward model
we are distilling from, we optimize against a family of reward models that, as
a whole, is likely to include at least one reasonable proxy for the preference
distribution. Our results show that distilling from such a family of reward
models leads to improved robustness to distribution shift in preference
annotations, while preserving the simple supervised nature of DPO.
MoreTranslated text
AI Read Science
Must-Reading Tree
Example
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined