Monte Carlo validation of a multireader method for receiver operating characteristic discrete rating data: Factorial experimental design

Dorfman, Donald D.; Berbaum, Kevin S.; Lenth, Russell V.; Chen, Yeh-Fong; Donaghy, Brenda A.

doi:10.1016/s1076-6332(98)80294-8

Cited by 89 publications

(77 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Selection of an image location, if a nodule was present in the image, was considered a true positive image identification. To compare the detection performance of the radiologists, a receiver operating characteristic (ROC) curve analysis based on the multiple reader multiple case (MRMC) method developed by Dorfman, Berbaum, and Metz and implemented in the software DBM MRMC 2.2 was used [16][17][18][19][20][21][22]. The analysis considered readers as fixed effects and the cases as random effects.…”

Section: Performance Evaluationmentioning

confidence: 99%

The Effects of Ambient Lighting in Chest Radiology Reading Rooms

et al. 2012

View full text Add to dashboard Cite

Under typical dark chest radiography reading room conditions, a radiologist's pupils contract and dilate as their visual focus intermittently shifts between the high luminance monitor and the darker background wall, resulting in increased visual fatigue and degradation of diagnostic performance. A controlled increase of ambient lighting may minimize these visual adjustments and potentially improve comfort and accuracy. This study was designed to determine the effect of a controlled increase of ambient lighting on chest radiologist nodule detection performance. Four chest radiologists read 100 radiographs (50 normal and 50 containing a subtle nodule) under low (E01 lx) and elevated (E050 lx) ambient lighting levels on a DICOM-calibrated, medicalgrade liquid crystal display. Radiologists were asked to identify nodule locations and rate their detection confidence. A receiver operating characteristic (ROC) analysis of radiologist results was performed and area under ROC curve (AUC) values calculated for each ambient lighting level. Additionally, radiologist selection times under both illuminance conditions were determined. Average AUC values did not significantly differ (p>0.05) between ambient lighting levels (estimated mean difference0−0.03; 95% CI, (−0.08, 0.03)). Average selection times decreased or remained constant with increased illuminance. The most considerable decreases occurred for false positive identification times (35.4±18.8 to 26.2±14.9 s) and true positive identification times (29.7± 18.3 to 24.5±15.5 s). No performance differences were statistically significant. Study findings suggest that a controlled increase of ambient lighting within darkly lit chest radiology reading rooms, to a level more suitable for performance of common radiological tasks, does not appear to have a statistically significant effect on nodule detection performance.

show abstract

Section: Performance Evaluationmentioning

confidence: 99%

The Effects of Ambient Lighting in Chest Radiology Reading Rooms

et al. 2012

View full text Add to dashboard Cite

show abstract

“…Readers were made aware that contrast disks were randomly located in the image on approximately 66 % of the images. To identify a difference in detection performance based on different median background intensities, the reader data were analyzed using the Trapezoidal/Wilcoxon method in DBM-MRMC [20][21][22][23][24][25][26] 2.2 software (University of Chicago, Kurt Rossmann Laboratories for Radiologic Image Research, Chicago, IL, USA), by comparing the area under the ROC curve (AUC). Using JMP®, Version 9 statistical software (SAS Institute Inc., Cary, NC, USA), contingency tables were generated and used to determine the contrast threshold over all median backgrounds and to perform comparisons of reader scores at each background level.…”

Section: Methodsmentioning

confidence: 99%

Verification of DICOM GSDF in Complex Backgrounds

et al. 2012

View full text Add to dashboard Cite

While previous research has determined the contrast detection threshold in medical images, it has focused on uniform backgrounds, has not used calibrated monitors, or has involved a low number of readers. With complex clinical images, how the Grayscale Standard Display Function (GSDF) affects the detection threshold and whether the median background intensity shift has been minimized by GSDF remains unknown. We set out to determine if the median background affected the detection of a low-contrast object in a clustered lumpy background, which simulated a mammography image, and to define the contrast detection threshold for these complex images. Clustered lumpy background images were created of different median intensities and disks of varying contrasts were inserted. A reader study was performed with 17 readers of varying skill level who scored with a five-point confidence scale whether a disk was present. The results were analyzed using reader operating characteristic (ROC) methodology. Contingency tables were used to determine the contrast detection threshold. No statistically significant difference was seen in the area under the ROC curve across all of the backgrounds. Contrast detection fell below 50 % between +3 and +2 gray levels. Our work supports the conclusion that Digital Imaging and Communications in Medicine GSDF calibrated monitors do perceptually linearize detection performance across shifts in median background intensity. The contrast detection threshold was determined to be +3 gray levels above the background for an object of 1°visual angle.

show abstract

“…There are several different statistical methods for analyzing multireader ROC studies, with the Dorfman-Berbaum-Metz (DBM) method [1][2][3] being the most frequently used. The DBM method involves an analysis of variance (ANOVA) of pseudovalues computed with the Quenouille-Tukey jackknife [4][5][6].…”

Section: Introductionmentioning

confidence: 99%

“…Roe and Metz [2] evaluated the performance of the DBM method for continuous rating data in a Monte Carlo study, while Dorfman et al [3], using the same simulation framework as Roe and Metz, evaluated the performance for discrete rating data. In both studies the empirical type I error rates for the DBM method are usually either in close agreement with the nominal significance levels or conservative.…”

Section: Introductionmentioning

confidence: 99%

“…In both studies the empirical type I error rates for the DBM method are usually either in close agreement with the nominal significance levels or conservative. Dorfman et al [3] conclude that the DBM method provides a "moderately conservative statistical test of modality differences", with the degree of conservatism greatest with very large ROC areas and decreasing as the number of cases increases. The downside of a conservative test is that the power is diminished as compared to the same test with the critical value adjusted to yield significance levels closer to the nominal level.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Monte Carlo Validation of the Dorfman-Berbaum-Metz Method Using Normalized Pseudovalues and Less Data-Based Model Simplification1

Hillis

Berbaum

2005

Academic Radiology

Self Cite

View full text Add to dashboard Cite

Rationale and Objectives-Two problems of the Dorfman-Berbaum-Metz (DBM) method for analyzing multireader ROC studies are that it tends to be conservative and can produce AUC estimates outside the parameter space -i.e., greater than one or less than zero. Recently it has been shown that the problem of AUC (or other accuracy) estimates outside the parameter space can be eliminated by using normalized pseudovalues, and it has been suggested that less data-based model simplification be used. Our purpose is to empirically investigate if these two modificationsnormalized pseudovalues and less data-based model simplification -result in improved performance.Materials and Methods-We examine the performance of the DBM procedure using the two proposed modifications for discrete and continuous ratings in a null simulation study comparing modalities with respect to the ROC area. The simulation study includes 144 different combinations of reader and case sample sizes, normal/abnormal case sample ratios, and variance components. The ROC area is estimated using parametric and nonparametric estimation.Results-The DBM procedure with both modifications performs better than either the original DBM procedure or the DBM procedure with only one of the modifications. For parametric estimation with discrete rating data, use of both modifications resulted in the mean type I error (0.043) closest to the nominal .05 level and the smallest range (0.050) and standard deviation (0.0108) across the 144 type I error rates.Conclusions-We recommend that normalized pseudovalues and less data-based model simplification be used with the DBM procedure.

show abstract

Monte Carlo validation of a multireader method for receiver operating characteristic discrete rating data: Factorial experimental design

Cited by 89 publications

References 23 publications

The Effects of Ambient Lighting in Chest Radiology Reading Rooms

The Effects of Ambient Lighting in Chest Radiology Reading Rooms

Verification of DICOM GSDF in Complex Backgrounds

Monte Carlo Validation of the Dorfman-Berbaum-Metz Method Using Normalized Pseudovalues and Less Data-Based Model Simplification1

Contact Info

Product

Resources

About