Adaptation of Biomedical and Clinical Pretrained Models to French Long Documents: A Comparative Study
CoRR(2024)
摘要
Recently, pretrained language models based on BERT have been introduced for
the French biomedical domain. Although these models have achieved
state-of-the-art results on biomedical and clinical NLP tasks, they are
constrained by a limited input sequence length of 512 tokens, which poses
challenges when applied to clinical notes. In this paper, we present a
comparative study of three adaptation strategies for long-sequence models,
leveraging the Longformer architecture. We conducted evaluations of these
models on 16 downstream tasks spanning both biomedical and clinical domains.
Our findings reveal that further pre-training an English clinical model with
French biomedical texts can outperform both converting a French biomedical BERT
to the Longformer architecture and pre-training a French biomedical Longformer
from scratch. The results underscore that long-sequence French biomedical
models improve performance across most downstream tasks regardless of sequence
length, but BERT based models remain the most efficient for named entity
recognition tasks.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要