Bimodal segmentation and classification of endoscopic ultrasonography images for solid pancreatic tumor

In this paper, we propose a bimodal method based on feature fusion in the neural network, including the endoscopic ultrasonography images and clinical data, for segmentation of solid pancreatic tumors in endoscopic ultrasonography images, and classification of three types of solid pancreatic tumors: pancreatic ductal adeno-carcinoma (PDAC), neuroendocrine tumor (pNEN) and solid pseudopapillary tumor (SPN). The database of this study involves 107 cases with 12,809 images. We use Attention U-Net as the backbone with feature fusion layer for segmentation, and a backbone of ResNet50 network with feature fusion layer for classification. The overall dice score, mIOU (segmentation) precision, recall and mIOU (detection) of our best bimodal segmentation model are 0.7552, 0.6241, 0.7204, 0.8003 and 0.6033. The sensitivity, specificity and F1 score of our best bimodal classification model are 0.9903, 1.0000, 0.9951 for PDAC, 0.8348, 0.9470 and 0.8404 for pNEN, 0.8484, 0.9444, 0.8328 for SPN, and an overall accuracy of 0.9180. We also use an interpretation model to analyze the important features that influence the final classification results, and show that clinical data like Carbohydrate antigen 199, Carbohydrate antigen 125, has great influence on the classification of PDAC and pNEN, while SPN depends more on endoscopic ultrasonography image features. Using artificial intelligence to automatically segment solid pancreatic tumors can help medical workers judge their scope and boundaries, and improve the detection rate and efficiency, and the proposed methods for classifying pancreatic masses into 3-class can facilitate physicians to master the clinical and image morphological features of these three pancreatic solid tumors.