A novel measure and significance testing in data analysis of cell image segmentation

Abstract

BackgroundCell image segmentation (CIS) is an essential part of quantitative imaging of biological cells. Designing a performance measure and conducting significance testing are critical for evaluating and comparing the CIS algorithms for image-based cell assays in cytometry. Many measures and methods have been proposed and implemented to evaluate segmentation methods. However, computing the standard errors (SE) of the measures and their correlation coefficient is not described, and thus the statistical significance of performance differences between CIS algorithms cannot be assessed.ResultsWe propose the total error rate (TER), a novel performance measure for segmenting all cells in the supervised evaluation. The TER statistically aggregates all misclassification error rates (MER) by taking cell sizes as weights. The MERs are for segmenting each single cell in the population. The TER is fully supported by the pairwise comparisons of MERs using 106 manually segmented ground-truth cells with different sizes and seven CIS algorithms taken from ImageJ. Further, the SE and 95% confidence interval (CI) of TER are computed based on the SE of MER that is calculated using the bootstrap method. An algorithm for computing the correlation coefficient of TERs between two CIS algorithms is also provided. Hence, the 95% CI error bars can be used to classify CIS algorithms. The SEs of TERs and their correlation coefficient can be employed to conduct the hypothesis testing, while the CIs overlap, to determine the statistical significance of the performance differences between CIS algorithms.ConclusionsA novel measure TER of CIS is proposed. The TER’s SEs and correlation coefficient are computed. Thereafter, CIS algorithms can be evaluated and compared statistically by conducting the significance testing.

Introduction to Digital Image Analysis in Whole-slide Imaging: A White Paper from the Digital Pathology Association

“…While it is not necessary for an algorithm to perform with 100% precision and accuracy, a reasonable level of performance should be met that correlates with the general goals of the analysis. [103104] For example, it may be acceptable for a study which is designed to rank a large number of samples or tissue microarray (TMA) cores by staining to be less accurate, whereas a clinical study to inform treatment decisions or prognosis may be required to meet a more stringent predetermined expectation of accurate classification. [105] Specific samples that do not meet these predetermined criteria should be failed upon review, which should trigger either reworking of the algorithm, reanalysis, and re-review or exclusion from analysis.…”

Standard errors and significance testing in data analysis for testing classifiers

“…The conventional ROC analysis involves two score distributions, i.e., the distributions of target scores and non-target scores, as depicted in Figure 1 (A) [1][2][3][4][5][6]. Target scores are created by comparing two different objects (e.g., images, speech segments, etc.)…”

“…It is assumed that the random samples drawn from a population are independent and identically distributed (i.i.d.). The statistics of interest may be the true accept rate at a specified false accept rate [3], or a weighted sum of the probabilities of type I (miss) and type II (false alarm) errors determined at a given decision threshold t as shown in Figure 1 (A) [4], or other measures [5].…”

Monte Carlo studies of bootstrap variability in ROC analysis with data dependency

“…The analytical approach cannot take account of how genuine scores and impostor scores are distributed, nor can it take data dependency into consideration (see below). It usually underestimates the SE of the measure (Wu et al 2017a(Wu et al , 2017b. Thus, the SE and CI of such a measure is estimated using the bootstrap method (Wu et al 2017b;Wu, Martin, and Kacker 2011;Wu et al 2017a;Efron 1979;Efron and Tibshirani 1993;Hall 1986;Efron 1987).…”