MeDSLIP: Medical Dual-Stream Language-Image Pre-training for Fine-grained Alignment
CoRR(2024)
摘要
Vision-language pre-training (VLP) models have shown significant advancements
in the medical domain. Yet, most VLP models align raw reports to images at a
very coarse level, without modeling fine-grained relationships between
anatomical and pathological concepts outlined in reports and the corresponding
semantic counterparts in images. To address this problem, we propose a Medical
Dual-Stream Language-Image Pre-training (MeDSLIP) framework. Specifically,
MeDSLIP establishes vision-language fine-grained alignments via disentangling
visual and textual representations into anatomy-relevant and pathology-relevant
streams. Moreover, a novel vision-language Prototypical Contr-astive Learning
(ProtoCL) method is adopted in MeDSLIP to enhance the alignment within the
anatomical and pathological streams. MeDSLIP further employs cross-stream
Intra-image Contrastive Learning (ICL) to ensure the consistent coexistence of
paired anatomical and pathological concepts within the same image. Such a
cross-stream regularization encourages the model to exploit the synchrony
between two streams for a more comprehensive representation learning. MeDSLIP
is evaluated under zero-shot and supervised fine-tuning settings on three
public datasets: NIH CXR14, RSNA Pneumonia, and SIIM-ACR Pneumothorax. Under
these settings, MeDSLIP outperforms six leading CNN-based models on
classification, grounding, and segmentation tasks.
更多查看译文
AI 理解论文
溯源树
样例
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要