Practical no-gold-standard evaluation framework for quantitative imaging methods: application to lesion segmentation in positron emission tomography

Abstract

Abstract. Recently, a class of no-gold-standard (NGS) techniques have been proposed to evaluate quantitative imaging methods using patient data. These techniques provide figures of merit (FoMs) quantifying the precision of the estimated quantitative value without requiring repeated measurements and without requiring a gold standard. However, applying these techniques to patient data presents several practical difficulties including assessing the underlying assumptions, accounting for patient-sampling-related uncertainty, and assessing the reliability of the estimated FoMs. To address these issues, we propose statistical tests that provide confidence in the underlying assumptions and in the reliability of the estimated FoMs. Furthermore, the NGS technique is integrated within a bootstrap-based methodology to account for patient-sampling-related uncertainty. The developed NGS framework was applied to evaluate four methods for segmenting lesions from F-Fluoro-2-deoxyglucose positron emission tomography images of patients with head-and-neck cancer on the task of precisely measuring the metabolic tumor volume. The NGS technique consistently predicted the same segmentation method as the most precise method. The proposed framework provided confidence in these results, even when gold-standard data were not available. The bootstrap-based methodology indicated improved performance of the NGS technique with larger numbers of patient studies, as was expected, and yielded consistent results as long as data from more than 80 lesions were available for the analysis.

No-gold-standard evaluation of image-acquisition methods using patient data

“…This linearity has been observed in realistic simulation and phantom studies in quantitative SPECT 20 , diffusion MRI 26 , DaTScan imaging 27 , PET imaging 28 , CT imaging 29,30 . Also, we have observed in clinical FDG-PET data of patients with head-and-neck cancer that the tumor-volume measurements using different imaging methods are consistent with the linearity assumption 24 . Linearity between true and measured quantitative values is highly desirable 25,29,30 since it ensures that any changes in the true value are proportionately reflected in the measured value.…”

“…The intuition is very similar to the intuition provided for the RWT and existing NGS techniques in Jha et al 24 . First note, as illustrated in Fig.…”

“…We then extended the NGS technique to a larger range of quantitative imaging tasks and extensively validated the method, both using numerical experiments and evaluating reconstruction methods for quantitative SPECT 20,21 and segmentation methods for diffusion MRI 22,23 . Further, to address the practical difficulties in applying the NGS technique to patient data, statistical methods were developed, yielding an NGS framework 24 . This framework was applied to evaluate tumor-segmentation methods for PET images of patients with head-and-neck cancer 24 .…”

“…Further, to address the practical difficulties in applying the NGS technique to patient data, statistical methods were developed, yielding an NGS framework 24 . This framework was applied to evaluate tumor-segmentation methods for PET images of patients with head-and-neck cancer 24 . However, each of these existing NGS techniques require that, for each evaluated method, the data be available from all set of patients.…”

Lesion Synthesis Toolbox: Development and Validation of Dedicated Software for Synthesis of Lesions in Raw PET and CT Patient Images

“…non-homogenous, asymmetric structures, variability in structure size, orientation, shape). Patient images, conversely, rarely have reliable ground truths associated with them [2], [3], against which image derived metrics can be compared. Deriving ground truth information from the images themselves is conceptually flawed, especially for faint structures around the spatial resolution of the system, but comparison between alternative image generating techniques may be appropriate as demonstrated in Figure 3, where changes in reconstruction parameters lead to varying lesion conspicuousness.…”