Revolutionizing Disease Diagnosis with simultaneous functional PET/MR and Deeply Integrated Brain Metabolic, Hemodynamic, and Perfusion Networks
arxiv(2024)
摘要
Simultaneous functional PET/MR (sf-PET/MR) presents a cutting-edge multimodal
neuroimaging technique. It provides an unprecedented opportunity for
concurrently monitoring and integrating multifaceted brain networks built by
spatiotemporally covaried metabolic activity, neural activity, and cerebral
blood flow (perfusion). Albeit high scientific/clinical values, short in
hardware accessibility of PET/MR hinders its applications, let alone modern
AI-based PET/MR fusion models. Our objective is to develop a clinically
feasible AI-based disease diagnosis model trained on comprehensive sf-PET/MR
data with the power of, during inferencing, allowing single modality input
(e.g., PET only) as well as enforcing multimodal-based accuracy. To this end,
we propose MX-ARM, a multimodal MiXture-of-experts Alignment and Reconstruction
Model. It is modality detachable and exchangeable, allocating different
multi-layer perceptrons dynamically ("mixture of experts") through learnable
weights to learn respective representations from different modalities. Such
design will not sacrifice model performance in uni-modal situation. To fully
exploit the inherent complex and nonlinear relation among modalities while
producing fine-grained representations for uni-modal inference, we subsequently
add a modal alignment module to line up a dominant modality (e.g., PET) with
representations of auxiliary modalities (MR). We further adopt multimodal
reconstruction to promote the quality of learned features. Experiments on
precious multimodal sf-PET/MR data for Mild Cognitive Impairment diagnosis
showcase the efficacy of our model toward clinically feasible precision
medicine.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要