TCGFusion: a network for PET-MRI fusion based on GAN and transformer

Modern clinical diagnosis relies heavily on medical imaging. Unimodal images contain limited information, whereas image fusion techniques can combine functional and structural information of images, thus speeding up the diagnostic and therapeutic process. However, existing methods lack global dependency and incomplete retention of image information. Therefore, we provide an end-to-end unsupervised training model called TCGFusion: a network for PET-MRI fusion based on GAN and transformer. We set the generator to a dual-path layout to ensure the gathering of both global semantic and detailed image information. The T-Path guarantees the coarse scale picture extraction and prevents the high computational cost. The C-Path’s multi-cascade structure captures image details well. Additionally, we create a dual discriminator to determine the distribution of structural similarity between the fused result and the source image and use patch-SSIM to reinforce the semantic constraints between them and promote the image details. Through thorough investigation using public datasets, our TCGFusion possesses the capability to effectively combine the structural and functional information of an image, thus guaranteeing the integrity of the fused image. On objective measures, our method is about 3