A comparison of the Dorfman–Berbaum–Metz and Obuchowski–Rockette methods for receiver operating characteristic (ROC) data

Abstract

There are several different statistical methods for analysing multireader ROC studies, with the Dorfman-Berbaum-Metz (DBM) method being the most frequently used. Another method is the corrected F method proposed by Obuchowski and Rockette (OR). The DBM and OR procedures at first appear quite different: DBM is a three-way ANOVA analysis of pseudovalues while OR is a two-way ANOVA analysis of accuracy estimates with correlated errors. We show that the original DBM and OR F statistics for testing the null hypothesis of equal treatments have the same form and will typically have similar values; however, differences in the denominator degrees of freedom will result in differences in p-values even when the F statistics are identical. We show how the methods can be generalized to include variations in the accuracy measure, covariance method, and degrees of freedom. Identical results are obtained when the methods agree with respect to all three of these procedure parameters; hence for a particular choice of procedure parameters the choice of method appears to depend mainly on software preference and availability. The methods are compared using data from a factorial study with two modalities, five readers, and 114 patients.

The Effects of Ambient Lighting in Chest Radiology Reading Rooms

“…Selection of an image location, if a nodule was present in the image, was considered a true positive image identification. To compare the detection performance of the radiologists, a receiver operating characteristic (ROC) curve analysis based on the multiple reader multiple case (MRMC) method developed by Dorfman, Berbaum, and Metz and implemented in the software DBM MRMC 2.2 was used [16][17][18][19][20][21][22]. The analysis considered readers as fixed effects and the cases as random effects.…”

Verification of DICOM GSDF in Complex Backgrounds

“…Readers were made aware that contrast disks were randomly located in the image on approximately 66 % of the images. To identify a difference in detection performance based on different median background intensities, the reader data were analyzed using the Trapezoidal/Wilcoxon method in DBM-MRMC [20][21][22][23][24][25][26] 2.2 software (University of Chicago, Kurt Rossmann Laboratories for Radiologic Image Research, Chicago, IL, USA), by comparing the area under the ROC curve (AUC). Using JMP®, Version 9 statistical software (SAS Institute Inc., Cary, NC, USA), contingency tables were generated and used to determine the contrast threshold over all median backgrounds and to perform comparisons of reader scores at each background level.…”

Monte Carlo Validation of the Dorfman-Berbaum-Metz Method Using Normalized Pseudovalues and Less Data-Based Model Simplification1

“…Hillis et al [8] and Hillis and Berbaum [12] suggest a similar approach but with less model simplification: model (1) is simplified by omitting the treatment× case variance component if its estimate is not positive, but the treatment × reader variance component is never omitted. Specifically, they suggest that the denominator in equation (2) be changed according to the following rules:…”

“…For example, if the binormal maximum likelihood estimate (MLE) for the AUC is .97 and the jackknife AUC estimate is 1.02, clearly most investigators would prefer to report the MLE. For trapezoidal-rule (trapezoid) AUC estimates this is not a problem, since the trapezoid and corresponding jackknife AUC estimates are equal [8].…”

“…Recently Hillis et al [8] generalize the DBM method by showing how it can be used with normalized pseudovalues, which allow the procedure to be based on the original accuracy estimates rather than the jackknife accuracy estimates. Although use of the normalized pseudovalues eliminates the problem of jackknife accuracy estimates outside the parameter space, they do not empirically investigate the performance of the DBM procedure using normalized pseudovalues.…”