CURATRON: Complete Robust Preference Data for Robust Alignment of Large Language Models
arxiv(2024)
摘要
This paper addresses the challenges of aligning large language models (LLMs)
with human values via preference learning (PL), with a focus on the issues of
incomplete and corrupted data in preference datasets. We propose a novel method
for robustly and completely recalibrating values within these datasets to
enhance LLMs resilience against the issues. In particular, we devise a
guaranteed polynomial time ranking algorithm that robustifies several existing
models, such as the classic Bradley–Terry–Luce (BTL) (Bradley and Terry,
1952) model and certain generalizations of it. To the best of our knowledge,
our present work is the first to propose an algorithm that provably recovers an
ϵ-optimal ranking with high probability while allowing as large as
O(n) perturbed pairwise comparison results per model response. Furthermore, we
show robust recovery results in the partially observed setting. Our experiments
confirm that our algorithms handle adversarial noise and unobserved comparisons
well in both general and LLM preference dataset settings. This work contributes
to the development and scaling of more reliable and ethically aligned AI models
by equipping the dataset curation pipeline with the ability to handle missing
and maliciously manipulated inputs.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要