Acquiring Clean Language Models from Backdoor Poisoned Datasets by Downscaling Frequency Space
CoRR(2024)
摘要
Despite the notable success of language models (LMs) in various natural
language processing (NLP) tasks, the reliability of LMs is susceptible to
backdoor attacks. Prior research attempts to mitigate backdoor learning while
training the LMs on the poisoned dataset, yet struggles against complex
backdoor attacks in real-world scenarios. In this paper, we investigate the
learning mechanisms of backdoor LMs in the frequency space by Fourier analysis.
Our findings indicate that the backdoor mapping presented on the poisoned
datasets exhibits a more discernible inclination towards lower frequency
compared to clean mapping, resulting in the faster convergence of backdoor
mapping. To alleviate this dilemma, we propose Multi-Scale Low-Rank Adaptation
(MuScleLoRA), which deploys multiple radial scalings in the frequency space
with low-rank adaptation to the target model and further aligns the gradients
when updating parameters. Through downscaling in the frequency space,
MuScleLoRA encourages the model to prioritize the learning of relatively
high-frequency clean mapping, consequently mitigating backdoor learning.
Experimental results demonstrate that MuScleLoRA outperforms baselines
significantly. Notably, MuScleLoRA reduces the average success rate of diverse
backdoor attacks to below 15% across multiple datasets and generalizes to
various backbone LMs, including BERT, RoBERTa, and Llama2. The codes are
available at https://github.com/ZrW00/MuScleLoRA.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要