PathM3: A Multimodal Multi-Task Multiple Instance Learning Framework for Whole Slide Image Classification and Captioning
arxiv(2024)
摘要
In the field of computational histopathology, both whole slide images (WSIs)
and diagnostic captions provide valuable insights for making diagnostic
decisions. However, aligning WSIs with diagnostic captions presents a
significant challenge. This difficulty arises from two main factors: 1)
Gigapixel WSIs are unsuitable for direct input into deep learning models, and
the redundancy and correlation among the patches demand more attention; and 2)
Authentic WSI diagnostic captions are extremely limited, making it difficult to
train an effective model. To overcome these obstacles, we present PathM3, a
multimodal, multi-task, multiple instance learning (MIL) framework for WSI
classification and captioning. PathM3 adapts a query-based transformer to
effectively align WSIs with diagnostic captions. Given that histopathology
visual patterns are redundantly distributed across WSIs, we aggregate each
patch feature with MIL method that considers the correlations among instances.
Furthermore, our PathM3 overcomes data scarcity in WSI-level captions by
leveraging limited WSI diagnostic caption data in the manner of multi-task
joint learning. Extensive experiments with improved classification accuracy and
caption generation demonstrate the effectiveness of our method on both WSI
classification and captioning task.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要