AI Control: Improving Safety Despite Intentional Subversion
CoRR(2023)
摘要
As large language models (LLMs) become more powerful and are deployed more
autonomously, it will be increasingly important to prevent them from causing
harmful outcomes. Researchers have investigated a variety of safety techniques
for this purpose, e.g. using models to review the outputs of other models, or
red-teaming techniques to surface subtle failure modes. However, researchers
have not evaluated whether such techniques still ensure safety if the model is
itself intentionally trying to subvert them. In this paper, we develop and
evaluate pipelines of safety techniques ("protocols") that are robust to
intentional subversion.
We investigate a scenario in which we want to solve a sequence of programming
problems, using access to a powerful but untrusted model (in our case, GPT-4),
access to a less powerful trusted model (in our case, GPT-3.5), and limited
access to human contractors who provide high-quality trusted labor. We
investigate protocols that aim to never submit solutions containing backdoors,
which we operationalize here as logical errors that are not caught by test
cases.
We investigate a range of protocols and test each against strategies that the
untrusted model could use to subvert them. One protocol is what we call trusted
editing. This protocol first asks GPT-4 to write code, and then asks GPT-3.5 to
rate the suspiciousness of that code. If the code is below some suspiciousness
threshold, it is submitted. Otherwise, GPT-3.5 edits the solution to remove
parts that seem suspicious and then submits the edited code. Another protocol
is untrusted monitoring. This protocol asks GPT-4 to write code, and then asks
another instance of GPT-4 whether the code is backdoored, using various
techniques to prevent the GPT-4 instances from colluding. These protocols
improve substantially on simple baselines.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要