Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHF
CoRR(2024)
摘要
Reinforcement Learning from Human Feedback (RLHF) is a pivotal technique that
aligns language models closely with human-centric values. The initial phase of
RLHF involves learning human values using a reward model from ranking data. It
is observed that the performance of the reward model degrades after one epoch
of training, and optimizing too much against the learned reward model
eventually hinders the true objective. This paper delves into these issues,
leveraging the theoretical insights to design improved reward learning
algorithm termed 'Iterative Data Smoothing' (IDS). The core idea is that during
each training epoch, we not only update the model with the data, but also
update the date using the model, replacing hard labels with soft labels. Our
empirical findings highlight the superior performance of this approach over the
traditional methods.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要