See the Unseen: Better Context-Consistent Knowledge-Editing by Noises
CoRR(2024)
Abstract
Knowledge-editing updates knowledge of large language models (LLMs) and
contributes to the interpretability and application of LLMs. However, knowledge
applying is context-consistent: LLMs can recall the same knowledge in different
contexts. Existing works ignore this property and the editing lacks
generalization. In this paper, we empirically find that the effects of
different contexts upon LLMs in recalling the same knowledge follow a
Gaussian-like distribution. We then sample Gaussian noises to simulate the
effects of different contexts when updating LLMs. By such, we can make LLMs see
the unseen contexts where the edited knowledge will be applied, therefore
improving the editing generalization. Experimental results on three LLMs
demonstrate the effectiveness of our methods and also distinguish our methods
from the others of fine-tuning LLMs by noises.
MoreTranslated text
AI Read Science
Must-Reading Tree
Example
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined