A multi-layered segmentation method for nucleus detection in highly clustered microscopy imaging: A practical application and validation using human U2OS cytoplasm–nucleus translocation images

Fluorescent microscopy imaging is a popular and well-established method for biomedical research. However, the large number of images created in each research trial quickly eliminates the possibility of a manual annotation; thus, the need for automatic image annotation is quickly becoming an urgent need. Furthermore, the high clustering indexes and noise observed in these images contribute to a complex issue, which has attracted the attention of the scientific community. In this paper, we present a fully automated method for annotating fluorescent confocal microscopy images in highly complex conditions. The proposed method relies on a multi-layered segmentation and declustering process, which begins with an adaptive segmentation step using a two-level Otsu’s Method. The second layer is comprised of two probabilistic classifiers, responsible for determining how many components may constitute each segmented region. The first of these employs rule-based reasoning grounded on the decreasing harmonic pattern observed in the region area density function, while the second one consists of a Support Vector Machine trained with features derived from the log likelihood ratio function of Gaussian mixture models of each region. Our results indicate that the proposed method is able to perform the identification and annotation process on par with an expert human subject, thus presenting itself a viable alternative to the traditional manual approach.