Segmentation of cervical nuclei using convolutional neural network for conventional cytology

Although implementing the Pap smear has drastically reduced the mortality rates from cervical cancer, false positives and negatives are related to the quality of the analysis and the cytopathologist experience. An alternative is the insertion of digital cytology in the quality monitoring to assist the screening. However, conventional cytology is still a major challenge, as it presents a lot of cellular overlap and several epithelial structures that make it difficult to implement computational methodologies. This article compares the performance of U-net and SegNet neural networks for nuclei segmentation in cervical images. Experiments were performed with different activation functions, batch sizes, and datasets, ISBI (synthetic images from liquid cytology) and CRIC Cervix-Seg (conventional cytology real images). The models achieved a Dice coefficient of 0.9783 for ISBI2015 and 0.9429 for CRIC Cervix-Seg. These results suggest a methodology capable of segmenting real images of cervical nuclei with quality, even in situations of overlap and artefacts, advancing efforts towards the automation of tasks as part of the cytopathological analysis in the laboratory work routine.