Segmentation of Overlapped Cervical Cells Using Asymmetric Mixture Model and Shape Constraint Level Set Method

Abstract

Accuracy segmentation of the nuclei and cytoplasm in Pap smear images is challenging in cervix cytological analysis. In this paper, a new fusion algorithm combining the asymmetric generalized Gaussian and Cauchy mixture model (GGCMM) with a shape constraint level set method to segment overlapping cervical smear cells is put forward. The proposed approach starts by separating nuclei and cytoplasm cluster through asymmetric GGCMM, where each component is a mixture of generalized Gaussian distribution and Cauchy distribution. The proposed asymmetric GGCMM takes into account the asymmetry of generalized Gaussian distribution and the heavier tail of Cauchy distribution. New probability distribution fits different shapes of observed data more flexibly. Then, we apply the morphological operation to remove fake nuclei which is usually much smaller than real nuclei. After that, the improved level set energy function with a distance map and a new shape prior term are applied to extract the contours of overlapping cervical cells. Due to this new level set energy function, the segmentation of every individual cell worked well, especially in overlapping areas. We evaluate the proposed method by using the ISBI 2014 Challenge Dataset. The results demonstrate that our approach outperforms existing methods in extracting overlapping cervical cells and obtains accurate cell contours.