Improving Transformer Performance for French Clinical Notes Classification Using Mixture of Experts on a Limited Dataset
arXiv (Cornell University)(2023)
摘要
Transformer-based models have shown outstanding results in natural language
processing but face challenges in applications like classifying small-scale
clinical texts, especially with constrained computational resources. This study
presents a customized Mixture of Expert (MoE) Transformer models for
classifying small-scale French clinical texts at CHU Sainte-Justine Hospital.
The MoE-Transformer addresses the dual challenges of effective training with
limited data and low-resource computation suitable for in-house hospital use.
Despite the success of biomedical pre-trained models such as CamemBERT-bio,
DrBERT, and AliBERT, their high computational demands make them impractical for
many clinical settings. Our MoE-Transformer model not only outperforms
DistillBERT, CamemBERT, FlauBERT, and Transformer models on the same dataset
but also achieves impressive results: an accuracy of 87%, precision of 87%,
recall of 85%, and F1-score of 86%. While the MoE-Transformer does not
surpass the performance of biomedical pre-trained BERT models, it can be
trained at least 190 times faster, offering a viable alternative for settings
with limited data and computational resources. Although the MoE-Transformer
addresses challenges of generalization gaps and sharp minima, demonstrating
some limitations for efficient and accurate clinical text classification, this
model still represents a significant advancement in the field. It is
particularly valuable for classifying small French clinical narratives within
the privacy and constraints of hospital-based computational resources.
更多查看译文
关键词
narratives classification,clinical,small-scale,nlp-based
AI 理解论文
溯源树
样例
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要