Fine-Grained Image-Text Alignment in Medical Imaging Enables Cyclic Image-Report Generation
CoRR(2023)
摘要
To address these issues, we propose a novel Adaptive patch-word Matching
(AdaMatch) model to correlate chest X-ray (CXR) image regions with words in
medical reports and apply it to CXR-report generation to provide explainability
for the generation process. AdaMatch exploits the fine-grained relation between
adaptive patches and words to provide explanations of specific image regions
with corresponding words. To capture the abnormal regions of varying sizes and
positions, we introduce the Adaptive Patch extraction (AdaPatch) module to
acquire the adaptive patches for these regions adaptively. In order to provide
explicit explainability for CXR-report generation task, we propose an
AdaMatch-based bidirectional large language model for Cyclic CXR-report
generation (AdaMatch-Cyclic). It employs the AdaMatch to obtain the keywords
for CXR images and `keypatches' for medical reports as hints to guide
CXR-report generation. Extensive experiments on two publicly available CXR
datasets prove the effectiveness of our method and its superior performance to
existing methods.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要