Quantitative imaging biomarkers: A review of statistical methods for computer algorithm comparisons

Abstract

Quantitative biomarkers from medical images are becoming important tools for clinical diagnosis, staging, monitoring, treatment planning, and development of new therapies. While there is a rich history of the development of quantitative imaging biomarker (QIB) techniques, little attention has been paid to the validation and comparison of the computer algorithms that implement the QIB measurements. In this paper we provide a framework for QIB algorithm comparisons. We first review and compare various study designs, including designs with the true value (e.g. phantoms, digital reference images, and zero-change studies), designs with a reference standard (e.g. studies testing equivalence with a reference standard), and designs without a reference standard (e.g. agreement studies and studies of algorithm precision). The statistical methods for comparing QIB algorithms are then presented for various study types using both aggregate and disaggregate approaches. We propose a series of steps for establishing the performance of a QIB algorithm, identify limitations in the current statistical literature, and suggest future directions for research.

Quantitative assessment of nonsolid pulmonary nodule volume with computed tomography in a phantom study

“…Prior to statistical analysis, all volume estimates were log-transformed (natural log) to make the data better suited for subsequent analyses such as ANOVA which assumes homoscedasticity (26,34). Analyses included the extraction of metrics to describe bias and repeatability, as recommended by the QIBA metrology group (27,34).…”

“…Toward that goal, we conducted a comprehensive phantom CT study that addressed the limitations of related work mentioned above by: (I) including a combination of nodule characteristics (densities as low as −800 HU, sizes of 10-and 5-mm in diameter, and spherical and spiculated shapes) and imaging protocols (radiation dose ranging from standard dose of 4.1 mGy to as low as 0.3 mGy and different state-ofthe-art reconstruction methods) with a range of parameters that test the boundaries of reliable measurement; (II) extracting fifteen repeat acquisitions per each protocol to allow for analysis of precision with improved statistical power; (III) conducting analyses based on metrics of accuracy and precision recommended by the Quantitative Imaging Biomarkers Alliance (QIBA) consortium metrology group (26,27) in an effort to standardize such analyses and enable comparisons with other studies.…”

A Comparison of Lung Nodule Segmentation Algorithms: Methods and Results from a Multi-institutional Study

“…This was performed considering all 52 nodules. Commonly used measures for repeatability include the repeatability coefficient (RC), the within-subject coefficient of variation (wCV), and the concordance correlation coefficient (CCC) [31][32][33]. The RC is defined as: Bias An estimate of systematic measurement error; it is the difference between the mean of measurements made on the same object and the measurement's true value.…”

“…The goal of this study was to evaluate the performance of the algorithms in terms of their bias, repeatability, and reproducibility [31][32][33] as well as to obtain insights into the underlying reasons for differences between algorithms on a voxel level.…”

“…As described [31][32][33], we estimated the population bias utilizing a repeated measures analysis to estimate the sample mean difference. Proportional bias was computed as the bias divided by the true value [31][32][33]. Two-way ANOVA (algorithm, nodule) was used to test the null hypothesis that there is no difference in bias between algorithms.…”