Deep Learning on Multimodal Chemical and Whole Slide Imaging Data for Predicting Prostate Cancer Directly from Tissue Images

Prostate cancer is one of the most common cancers globally and is the second most common cancer in the male population in the US. Here we develop a study based on correlating the H&E-stained biopsy data with MALDI mass-spectrometric imaging of the corresponding tissue to determine the cancerous regions and their unique chemical signatures, and variation of the predicted regions with original pathological annotations. We spatially register features obtained through deep learning from high-resolution optical micrographs of whole slide H&E stained data with MSI data to correlate the chemical signature with the tissue anatomy of the data, and then use the learned correlation to predict prostate cancer from observed H&E images using trained co-registered MSI data. We found that this system is more robust than predicting from a single imaging modality and can predict cancerous regions with ∼ 80% accuracy. Two chemical biomarkers were also found to be predicting the ground truth cancerous regions. This will improve on generating patient treatment trajectories by more accurately predicting prostate cancer directly from H&E-stained biopsy images. ### Competing Interest Statement The authors have declared no competing interest.