Receiver Operating Characteristic Rating Analysis

Dorfman, Donald D.; Berbaum, Kevin S.; Metz, Charles E.

doi:10.1097/00004424-199209000-00015

Cited by 636 publications

(79 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Arguably the most widely used method to analyze mulitrater correlated ROC data is that due to DBM [10]. In this approach, reader ratings are jackknifed [16] to obtain pseudo-values for the A Z for each case and modality one rater at a time.…”

Section: The Jackknife Methods Of Dbmmentioning

confidence: 99%

“…This difficulty is caused by the absence of a model component for rater variability, although several methods have now been proposed to account for this source of variability (see, for example [10,11,12]). Each of these methods require two stages of modeling.…”

Section: Introductionmentioning

confidence: 99%

“…In the next section we review, arguably, the most widely used ROC model for the analysis of mulitrater correlated data, that of Dorfman, Berbaum and Metz [10] (henceforth referred to as DBM). In Section we present a Bayesian hierarchical model for the analysis of multirater correlated ROC data and highlight its connection to the standard bivariate-binormal model (e.g., [15]).…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

A Bayesian hierarchical approach to multirater correlated ROC analysis

Johnson¹,

Johnson²

2006

Statist. Med.

View full text Add to dashboard Cite

In a common ROC study design, several readers are asked to rate diagnostics of the same cases processed under different modalities. We describe a Bayesian hierarchical model that facilitates the analysis of this study design by explicitly modeling the three sources of variation inherent to it. In so doing, we achieve substantial reductions in the posterior uncertainty associated with estimates of the differences in areas under the estimated ROC curves and corresponding reductions in the mean squared error (MSE) of these estimates. Based on simulation studies, both the widths of confidence intervals and MSE of estimates of differences in the area under the curves appear to be reduced by a factor that often exceeds two. Thus, our methodology has important implications for increasing the power of analyses based on ROC data collected from an available study population. STATISTICS IN MEDICINE SUMMARYIn a common ROC study design, several readers are asked to rate diagnostics of the same cases processed under different modalities. We describe a Bayesian hierarchical model that facilitates the analysis of this study design by explicitly modeling the three sources of variation inherent to it. In so doing, we achieve substantial reductions in the posterior uncertainty associated with estimates of the differences in areas under the estimated ROC curves and corresponding reductions in the mean squared error (MSE) of these estimates. Based on simulation studies, both the widths of confidence intervals and MSE of estimates of differences in the area under the curves appear to be reduced by a factor that often exceeds two. Thus, our methodology has important implications for increasing the * Correspondence to:

show abstract

Section: The Jackknife Methods Of Dbmmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

A Bayesian hierarchical approach to multirater correlated ROC analysis

Johnson¹,

Johnson²

2006

Statist. Med.

View full text Add to dashboard Cite

show abstract

“…The primary analysis for the clinical study was based on the Dorfman-Berbaum-Metz (DBM) ANOVA-after-jackknife approach 6 . Using this approach, 90 leave-one-out samples were generated, each time leaving out one case (i.e., leaving out all 4 quadrants in the case), and the A Z values computed from the leave-one-out samples were used to compute the A Z pseudo-values that form the ANOVA-after-jackknife analysis dataset.…”

Section: Methodsmentioning

confidence: 99%

Gold standards and expert panels: a pulmonary nodule case study with challenges and solutions

Miller

O’Shaughnessy²,

Wood³

et al. 2004

SPIE Proceedings

View full text Add to dashboard Cite

Introduction: Comparative evaluations of reader performance using different modalities, e.g. CT with computer-aided detection (CAD) vs. CT without CAD, generally require a "truth" definition based on a gold standard. There are many situations in which a true invariant gold standard is impractical or impossible to obtain. For instance, small pulmonary nodules are generally not assessed by biopsy or resection. In such cases, it is common to use a unanimous consensus or majority agreement from an expert panel as a reference standard for actionability in lieu of the unknown gold standard for disease. Nonetheless, there are three major concerns about expert panel reference standards: (1) actionability is not synonymous with disease (2) it may be possible to obtain different conclusions about which modality is better using different rules (e.g. majority vs. unanimous consensus), and (3) the variability associated with the panelists is not formally captured in the p-values or confidence intervals that are generally produced for estimating the extent to which one modality is superior to the other. Methods:A multi-reader-multi-case (MRMC) receiver operating characteristic (ROC) study was performed using 90 cases, 15 readers, and a reference truth based on 3 experienced panelists. The primary analyses were conducted using a reference truth of unanimous consensus regarding actionability (3 out of 3 panelists). To assess the three concerns noted above: (1) additional data from the original radiology reports were compared to the panel (2) the complete analysis was repeated using different definitions of truth, and (3) bootstrap analyses were conducted in which new truth panels were constructed by picking 1, 2, or 3 panelists at random. Conclusions:The definition of the reference truth affected the results for each modality (CT with CAD and CT without CAD) considered by itself, but the effects were similar, so the primary analysis comparing the modalities was robust to the choice of the reference truth.

show abstract

“…As an index of detection performance, we used the area under the receiver operating characteristic curve (AUC), which was calculated by the trapezoidal rule. We performed the analysis of variance (ANOVA) of pseudovalues of the AUC computed by the jackknife analysis method proposed by Dorfman et al 25 and post hoc pairwise comparisons with Bonferroni correction, which is designated the Dorfman-Berbaum-Metz method. The DSs for GGNs were compared among the 12 acquisition methods by using a one-way ANOVA in the total 168 image sets in the following 6 subgroups classified by the kind of simulated a Significantly higher than the images acquired using IR at 120 kV/10 mA and using FBP at 120 kV/10, 20, 80 and 160 mA.…”

Section: Objective Image Quality Measurementmentioning

confidence: 99%