Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Models
CoRR(2023)
摘要
The integration of large language models (LLMs) with external content has
enabled more up-to-date and wide-ranging applications of LLMs, such as
Microsoft Copilot. However, this integration has also exposed LLMs to the risk
of indirect prompt injection attacks, where an attacker can embed malicious
instructions within external content, compromising LLM output and causing
responses to deviate from user expectations. To investigate this important but
underexplored issue, we introduce the first benchmark for indirect prompt
injection attacks, named BIPIA, to evaluate the risk of such attacks. Based on
the evaluation, our work makes a key analysis of the underlying reason for the
success of the attack, namely the inability of LLMs to distinguish between
instructions and external content and the absence of LLMs' awareness to not
execute instructions within external content. Building upon this analysis, we
develop two black-box methods based on prompt learning and a white-box defense
method based on fine-tuning with adversarial training accordingly. Experimental
results demonstrate that black-box defenses are highly effective in mitigating
these attacks, while the white-box defense reduces the attack success rate to
near-zero levels. Overall, our work systematically investigates indirect prompt
injection attacks by introducing a benchmark, analyzing the underlying reason
for the success of the attack, and developing an initial set of defenses.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要