SoFA: Shielded On-the-fly Alignment via Priority Rule Following
CoRR(2024)
摘要
The alignment problem in Large Language Models (LLMs) involves adapting them
to the broad spectrum of human values. This requirement challenges existing
alignment methods due to diversity of preferences and regulatory standards.
This paper introduces a novel alignment paradigm, priority rule following,
which defines rules as the primary control mechanism in each dialog,
prioritizing them over user instructions. Our preliminary analysis reveals that
even the advanced LLMs, such as GPT-4, exhibit shortcomings in understanding
and prioritizing the rules. Therefore, we present PriorityDistill, a
semi-automated approach for distilling priority following signals from LLM
simulations to ensure robust rule integration and adherence. Our experiments
show that this method not only effectively minimizes misalignments utilizing
only one general rule but also adapts smoothly to various unseen rules,
ensuring they are shielded from hijacking and that the model responds
appropriately.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要