Improving deep learning based segmentation of scars using multi-view images

The utilization of deep learning for scar segmentation in photographs enables automated and non-contact quantitative analysis of skin scars. Meanwhile, multi-view photographs are commonly employed to capture the 3D information of scars. In this paper, we propose a two-stage deep learning based segmentation framework for delineating scars from surrounding skin, leveraging multi-view images to achieve enhanced segmentation results compared to single-view approaches. In the first stage, a data augmentation method based on 3D reconstruction and view interpolation is proposed. The generated images are used in a semi-supervised setting to train a single-view segmentation network. In the second stage, a multi-view co-segmentation network (MVCSNet) is proposed to exploit the mutual information between views and to further refine the segmentation. The multi-view feature interaction module (MVFI) uses the prior segmentation results from the first stage, computes feature similarities across views, and optimizes the features. The proposed method was evaluated on two multi-view image datasets containing linear scars and patchy scars, respectively. The results show that the proposed data augmentation method can improve the generalization of the model, particularly for the dataset with smaller size. Comparative analyses demonstrate the superior performance of MVCSNet over other deep learning based segmentation or co-segmentation algorithms.