G-MAP: General Memory-Augmented Pre-trained Language Model for Domain Tasks
emnlp 2022(2022)
摘要
Recently, domain-specific PLMs have been proposed to boost the task
performance of specific domains (e.g., biomedical and computer science) by
continuing to pre-train general PLMs with domain-specific corpora. However,
this Domain-Adaptive Pre-Training (DAPT; Gururangan et al. (2020)) tends to
forget the previous general knowledge acquired by general PLMs, which leads to
a catastrophic forgetting phenomenon and sub-optimal performance. To alleviate
this problem, we propose a new framework of General Memory Augmented
Pre-trained Language Model (G-MAP), which augments the domain-specific PLM by a
memory representation built from the frozen general PLM without losing any
general knowledge. Specifically, we propose a new memory-augmented layer, and
based on it, different augmented strategies are explored to build the memory
representation and then adaptively fuse it into the domain-specific PLM. We
demonstrate the effectiveness of G-MAP on various domains (biomedical and
computer science publications, news, and reviews) and different kinds (text
classification, QA, NER) of tasks, and the extensive results show that the
proposed G-MAP can achieve SOTA results on all tasks.
更多查看译文
关键词
language model,g-map,memory-augmented,pre-trained
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要