Evaluation of cell segmentation methods without reference segmentations

Abstract

Cell segmentation is a cornerstone of many bioimage informatics studies. Inaccurate segmentation introduces computational error in downstream cellular analysis. Evaluating the segmentation results is thus a necessary step for developing the segmentation methods as well as choosing the most appropriate one for a certain kind of tissue or image. The evaluation process has typically involved comparison of segmentations to those generated by humans, which can be expensive and subject to unknown bias. We present here an approach that seeks to evaluate cell segmentation methods without relying upon comparison to results from humans. For this, we defined a number of segmentation quality metrics that can be applied to multichannel fluorescence images. We calculated these metrics for 11 previously-described segmentation methods applied to datasets from 5 multiplexed microscope modalities covering 5 tissues. Using principal component analysis to combine the metrics we defined an overall cell segmentation quality score and ranked the segmentation methods. A Reproducible Research Archive containing all data and code will be made available upon publication at http://hubmap.scs.cmu.edu.

SEG: Segmentation Evaluation in absence of Ground truth labels

“…Thus, challenges arise: how can users objectively evaluate individual segmentation methods for their own dataset without ground truth labels and choose the most suitable segmentation? Previous studies have suggested segmentation evaluation in absence of ground truth labels [15][16][17][18] . For example, reverse classification accuracy 15 uses segmentation results as pseudo-ground truth labels to train a new segmentation model.…”

“…Scores are computed purely probabilistically without thresholding the ensemble average to estimate the ground truth. Recent study 18 proposed an approach that seeks to evaluate cell segmentation methods by providing an objective evaluation approach based on assumptions about the desired characteristics of good cell segmentation methods. Evaluation metrics rely on the similarity between two segmentation methods and an overall segmentation quality score for each method uses the metrics for all methods with and without perturbation (i.e., added noise and downsampling).…”