2pClPr: A 2-phase Clump Profiler for Segmentation of Cancer Cells in Fluorescence Microscopic Images

Cancer cell segmentation is challenging since they grow in tightly packed colonies (clumps), causing adjacent cells to overlap. In this work, we proposed an automated vision-based analysis framework, 2pClPr: A 2-phase Clump Profiler for the segmentation of cancer cells in fluorescence microscopy images. In the first phase, we proposed a deep learning network, Multi-Scale Cell-Net, for coarse segmentation. Another framework, Multi-Scale Region Proposal Network (MS-RPN) was simultaneously trained in parallel to Multi-Scale Cell-Net to generate seeds for each cell. The coarse segmentation map was unable to segment the complex clumps. We proposed a novel metric, the Irregularity factor ( I_ftr ), to identify those complex clumps. Once, identified, we mapped them with the seed points generated by MS-RPN. These seeds served as the initialization points for our proposed level set framework: Weighing Repelling Force Embedded-Level Set Method (WRFe-LSM) which segments the identified complex clumps in the second phase of segmentation. The final segmentation map was generated with the segmented cells from the two phases. We conducted extensive experiments on our private dataset comprising of images from four complex cancer cell lines and obtained an Aggregated Jaccard Index (AJI) of 76.6%, 72.9%, 75.5%, and 69.7% on HeLa, MDA-MB-468, MDA-MB-231, and T-47D respectively. Further, to show the generalization ability of 2pClPr, we conducted comparative experiments on a publicly available Hematoxylin-Eosin (H&E) stained histopathological images dataset (MoNuSeg) and obtained an AJI of 66.2%. A detailed evaluation of segmentation performance on both datasets shows that 2pClPr is robust and effective.