Construction of a Multi-modal Model of Pancreatic Tumors by Integration of MRI and Pathological Images using Conditional Cycle α-GAN

In this study, we constructed the multi-modal model of a pancreas tumor by learning the correspondence between MRI voxels and pathology image patches with a conditional cycle α-GAN. In this framework, we constructs encoderdecoder networks that translate MRI voxels and pathology image patches each other, and two discriminators for both modalities. When a voxel in a pancreas tumor region in an MRI image is selected, this model can generate various corresponded pathology image patches non-invasively, and vice versa. We made a training dataset by registrating between an MRI image and a 3D pathology image and trained our multi-modal model. Using trained model, anyone can observe the change of pathology image with respect to the MRI values, and we found the behaviors are closely related to the growing process of pancreas tumor.