MedRG: Medical Report Grounding with Multi-modal Large Language Model
arxiv(2024)
摘要
Medical Report Grounding is pivotal in identifying the most relevant regions
in medical images based on a given phrase query, a critical aspect in medical
image analysis and radiological diagnosis. However, prevailing visual grounding
approaches necessitate the manual extraction of key phrases from medical
reports, imposing substantial burdens on both system efficiency and physicians.
In this paper, we introduce a novel framework, Medical Report Grounding
(MedRG), an end-to-end solution for utilizing a multi-modal Large Language
Model to predict key phrase by incorporating a unique token, BOX, into the
vocabulary to serve as an embedding for unlocking detection capabilities.
Subsequently, the vision encoder-decoder jointly decodes the hidden embedding
and the input medical image, generating the corresponding grounding box. The
experimental results validate the effectiveness of MedRG, surpassing the
performance of the existing state-of-the-art medical phrase grounding methods.
This study represents a pioneering exploration of the medical report grounding
task, marking the first-ever endeavor in this domain.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要