Simulation of Unequal-Variance Binormal Multireader ROC Decision Data

Hillis, Stephen L.

doi:10.1016/j.acra.2012.09.011

Cited by 17 publications

(37 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Fourth, our ROC curve analysis was performed by five readers with 30 positive and 30 negative images for each protocol, using DBM MRMC software. In an earlier study, it was reported that the software used in the current study requires at least five readers and 25 positive and 25 negative images, to acquire more reliable ROC curves 33. Therefore, our statistical analysis of the ROC curve was likely reliable.…”

Section: Discussionmentioning

confidence: 74%

“…Each of the 60 images was displayed in a randomized order on the monitor and analyzed with a continuously distributed test. DBM MRMC software (Department of Radiology, University of Chicago, Chicago, IL, USA) was used to calculate the mean area under curve (AUC) and 95% confidence intervals for each protocol, and to calculate the difference in the average AUC and P ‐values between each protocol 28, 29, 30, 31, 32, 33. The significance level for all evaluations was 5%.…”

Section: Methodsmentioning

confidence: 99%

See 1 more Smart Citation

Quantitative and qualitative evaluation of hybrid iterative reconstruction, with and without noise power spectrum models: A phantom study

Minamishima

Sugisawa

Yamada

et al. 2018

J Applied Clin Med Phys

View full text Add to dashboard Cite

The purpose of this phantom study was to investigate the feasibility of dose reduction with hybrid iterative reconstruction, with and without a noise power spectrum (NPS) model, using both quantitative and qualitative evaluations. Standard dose (SD), three‐quarter dose (TQD), and half‐dose (HD) of radiation were used. Images were reconstructed with filtered back projection (FBP), adaptive iterative dose reduction 3D (AIDR 3D) (MILD, STR), and AIDR 3D enhanced (eAIDR 3D) (eMILD, eSTR). An NPS analysis, task‐based modulation transfer function (MTF task) analysis, and comparisons of low‐contrast detectability and image texture were performed. Although the eAIDR 3D had a higher NPS value in the high‐frequency range and improved image texture and resolution as compared with AIDR 3D at the same radiation dose and iteration levels, it yielded higher noise than AIDR 3D. Additionally, although there was no statistically significant difference between SD‐FBP and the TQD series in the comparison of the mean area under the curve (AUC), the mean AUC was statistically significantly different between SD‐FBP and the HD series. NPS values in the high‐frequency range, 10% MTF task values, low‐contrast detectability, and image textures of TQD‐eMILD were comparable to those of SD‐FBP. Our findings suggested that using eMILD can reduce the radiation dose by 25%, while potentially maintaining diagnostic performance, spatial resolution, and image texture; this could support selecting the appropriate protocol in a clinical setting.

show abstract

Section: Discussionmentioning

confidence: 74%

Section: Methodsmentioning

confidence: 99%

Quantitative and qualitative evaluation of hybrid iterative reconstruction, with and without noise power spectrum models: A phantom study

Minamishima

Sugisawa

Yamada

et al. 2018

J Applied Clin Med Phys

View full text Add to dashboard Cite

show abstract

“…This creates a problem for the parametric ROC methods that have largely relied on using the same distribution for both diseased and non-diseased populations. The most popular of these parametric models, namely the binormal ROC curve (19), often falls short in capturing the heterogeneity of the diseased population. This has been long recognized in the ROC field.…”

Section: Introductionmentioning

confidence: 99%

Mixtures of Receiver Operating Characteristic Curves

Gönen

2013

Academic Radiology

View full text Add to dashboard Cite

Rationale and Objectives ROC curves are ubiquitous in the analysis of imaging metrics as markers of both diagnosis and prognosis. While empirical estimation of ROC curves remains the most popular method, there are several reasons to consider smooth estimates based on a parametric model. Materials and Methods A mixture model is considered for modeling the distribution of the marker in the diseased population motivated by the biological observation that there is more heterogeneity in the diseased population than there is in the normal one. It is shown that this model results in an analytically tractable ROC curve which is itself a mixture of ROC curves. Results The use of CK-BB isoenzyme in diagnosis of severe head trauma is used as an example. ROC curves are fit using the direct binormal method, ROCKIT and the Box-Cox transformation as well as the proposed mixture model. The mixture model generates an ROC curve that is much closer to the empirical one than the other methods considered. Conclusions Mixtures of ROC curves can be helpful in fitting smooth ROC curves in datasets where the diseased population has higher variability than can be explained by a single distribution.

show abstract

“…As discussed by Hillis,19 when the variance components are assumed to be the same across truth states and modalities, the R&M model has the following interpretations: (1) ROC ratings for each reader are generated from an equal-variance binormal model (i.e, a binormal model such that variances of the nondiseased and diseased ROC ratings are equal); and (2) the expected differences (or separations) between the nondiseased and diseased ROC ratings vary across readers, with the separations having the same variance for each modality. This last result implies that for a simulation study that assumes equal AUCs across modalities (i.e., a null-hypothesis study), the resulting AUC estimates will have the same variance for each modality.…”

Section: Introductionmentioning

confidence: 99%

“…Such unsymmetric ROC curves have been seen as far back as the early psychophysical experiments of the 1960s 21,22 and in recent studies evaluating medical imaging modalities. 19,[23][24][25][26] Unsymmetric ROC curves have motivated other models of ROC ratings [27][28][29] and are sometimes characterized by a mean-to-sigma ratio defined as the difference of the binormal means divided by the difference of the binormal standard deviations across truth states in an unequal-variance binormal model. 19,21,26 For this reason, Hillis 19 introduced an unequal-variance binormal model by allowing some of the variance components to depend on truth but with some additional constraints.…”

Section: Introductionmentioning

confidence: 99%

Generalized Roe and Metz receiver operating characteristic model: analytic link between simulated decision scores and empirical AUC variances and covariances

Gallas¹,

Hillis

2014

J. Med. Imag

Self Cite

View full text Add to dashboard Cite

Abstract. Modeling and simulation are often used to understand and investigate random quantities and estimators. In 1997, Roe and Metz introduced a simulation model to validate analysis methods for the popular endpoint in reader studies to evaluate medical imaging devices, the reader-averaged area under the receiver operating characteristic (ROC) curve. Here, we generalize the notation of the model to allow more flexibility in recognition that variances of ROC ratings depend on modality and truth state. We also derive and validate equations for computing population variances and covariances for reader-averaged empirical AUC estimates under the generalized model. The equations are one-dimensional integrals that can be calculated using standard numerical integration techniques. This work provides the theoretical foundation and validation for a Java application called iRoeMetz that can simulate multireader multicase ROC studies and numerically calculate the corresponding variances and covariances of the empirical AUC. The iRoeMetz application and source code can be found at the "iMRMC" project on the google code project hosting site. These results and the application can be used by investigators to investigate ROC endpoints, validate analysis methods, and plan future studies.

show abstract

Simulation of Unequal-Variance Binormal Multireader ROC Decision Data

Cited by 17 publications

References 22 publications

Quantitative and qualitative evaluation of hybrid iterative reconstruction, with and without noise power spectrum models: A phantom study

Quantitative and qualitative evaluation of hybrid iterative reconstruction, with and without noise power spectrum models: A phantom study

Mixtures of Receiver Operating Characteristic Curves

Generalized Roe and Metz receiver operating characteristic model: analytic link between simulated decision scores and empirical AUC variances and covariances

Contact Info

Product

Resources

About