ICDPO: Effectively Borrowing Alignment Capability of Others via In-context Direct Preference Optimization
CoRR(2024)
摘要
Large Language Models (LLMs) rely on Human Preference Alignment (HPA) to
ensure the generation of safe content. Due to the heavy cost associated with
fine-tuning, fine-tuning-free methods have emerged, typically modifying LLM
decoding with external auxiliary methods. However, these methods do not
essentially enhance the LLM itself. In this paper, we rethink the derivation
procedures of DPO, based on which we conversely build an instant scorer using
the states of the LLM before and after In-context Learning (ICL). Accordingly,
we propose a novel approach called In-Context Direct Preference Optimization
(ICDPO). It enables LLMs to borrow the HPA capabilities from superior LLMs with
ICL, generating well-aligned responses as estimated by the aforementioned
instant scorer, thereby enhancing the final performance. ICDPO can be further
enhanced with a two-stage retriever and an upgraded scorer, both offering
benefits. Extensive experiments show its effectiveness, particularly in
outperforming two fine-tuning-free baselines, and it exhibits competitiveness
with SFT + LoRA. We also conduct detailed analyses to offer comprehensive
insights into ICDPO.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要