A Bayesian hierarchical approach to multirater correlated ROC analysis

Abstract

In a common ROC study design, several readers are asked to rate diagnostics of the same cases processed under different modalities. We describe a Bayesian hierarchical model that facilitates the analysis of this study design by explicitly modeling the three sources of variation inherent to it. In so doing, we achieve substantial reductions in the posterior uncertainty associated with estimates of the differences in areas under the estimated ROC curves and corresponding reductions in the mean squared error (MSE) of these estimates. Based on simulation studies, both the widths of confidence intervals and MSE of estimates of differences in the area under the curves appear to be reduced by a factor that often exceeds two. Thus, our methodology has important implications for increasing the power of analyses based on ROC data collected from an available study population. STATISTICS IN MEDICINE SUMMARYIn a common ROC study design, several readers are asked to rate diagnostics of the same cases processed under different modalities. We describe a Bayesian hierarchical model that facilitates the analysis of this study design by explicitly modeling the three sources of variation inherent to it. In so doing, we achieve substantial reductions in the posterior uncertainty associated with estimates of the differences in areas under the estimated ROC curves and corresponding reductions in the mean squared error (MSE) of these estimates. Based on simulation studies, both the widths of confidence intervals and MSE of estimates of differences in the area under the curves appear to be reduced by a factor that often exceeds two. Thus, our methodology has important implications for increasing the * Correspondence to:

A novel regression method for the analysis of multireader multicase‐free‐response receiver operating characteristics studies

“…The validity of this generalized DBM method, however, has not yet been justified. It also carries some concerns, including that it lacks a valid theoretical basis, makes strong assumptions, and does not directly deal with the intra‐subject correlations 2,15‐17 . Moreover, the generalized DBM method cannot include continuous covariates in the model 18 .…”

“…It also carries some concerns, including that it lacks a valid theoretical basis, makes strong assumptions, and does not directly deal with the intra-subject correlations. 2,[15][16][17] Moreover, the generalized DBM method cannot include continuous covariates in the model. 18 Overall, the aforementioned statistical issues and the lack of suitable statistical methods motivate us to propose a new approach for analyzing MRMC-FROC studies.…”

Small area estimation of receiver operating characteristic curves for ordinal data under stochastic ordering

“…Ishwaran and Gatsonis developed the Bayesian hierarchical ordinal regression model for multilevel clustered data in diagnostic imaging studies. Johnson and Johnson developed a Bayesian hierarchical latent variable model for analyzing multirater correlated ordinal data when several radiologists rate multiple exams are collected from the same individual. Wang et al proposed a Bayesian nonparametric estimation method for the ROC curve in the absence of the perfect reference test.…”

A correlated Bayesian rank likelihood approach to multiple ROC curves for endometriosis

“…More recently, Gu and Ghosal have proposed a Markov chain Monte Carlo (MCMC) approach that samples from the exact posterior distributions. Johnson and Johnson modeled correlated ROCs but used normality assumption. Dass and Kim used rank likelihood and considered multivariate rating data but did not consider model comparison.…”