Comparing Bad Apples to Good Oranges: Aligning Large Language Models via Joint Preference Optimization
CoRR(2024)
Abstract
A common technique for aligning large language models (LLMs) relies on
acquiring human preferences by comparing multiple generations conditioned on a
fixed context. This only leverages the pairwise comparisons when the
generations are placed in an identical context. However, such conditional
rankings often fail to capture the complex and multidimensional aspects of
human preferences. In this work, we revisit the traditional paradigm of
preference acquisition and propose a new axis that is based on eliciting
preferences jointly over the instruction-response pairs. While prior preference
optimizations are designed for conditional ranking protocols (e.g., DPO), our
proposed preference acquisition protocol introduces DOVE, a new preference
optimization objective that upweights the joint probability of the chosen
instruction-response pair over the rejected instruction-response pair.
Interestingly, we find that the LLM trained with joint instruction-response
preference data using DOVE outperforms the LLM trained with DPO by 5.2
3.3
respectively. Our findings reveal that joint preferences over instruction and
response pairs can significantly enhance the alignment of LLMs by tapping into
a broader spectrum of human preference elicitation. The data and code is
available at https://github.com/Hritikbansal/dove.
MoreTranslated text
AI Read Science
Must-Reading Tree
Example
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined