Recent Developments in the Dorfman-Berbaum-Metz Procedure for Multireader ROC Study Analysis

Hillis, Stephen L.; Berbaum, Kevin S.; Metz, Charles E.

doi:10.1016/j.acra.2007.12.015

Cited by 175 publications

(120 citation statements)

References 17 publications

Supporting

Mentioning

120

Contrasting

Order By: Relevance

“…We used McNemar's test to compare the sensitivity and specificity of the iPad 2 and the LCD monitor [20]. The areas under the receiver operating characteristic (ROC) curves for the displaying devices were compared by the Dorfman-Berbaum-Metz method, using RSCORE and semi-parametric estimation of ROC indices with DBM-MRMC software (version 2.3, available from http://perception.radiology.uiowa.edu) [21]. In assessing the differences between overall reviewer's performances, the areas under the ROC curves were calculated by the above same method.…”

Section: Discussionmentioning

confidence: 99%

An Assessment of the iPad 2 as a CT Teleradiology Tool Using Brain CT with Subtle Intracranial Hemorrhage Under Conventional Illumination

Park¹,

Lee²,

Kang³

2013

J Digit Imaging

View full text Add to dashboard Cite

We examined the potential of the iPad 2 as a teleradiologic tool for evaluating brain computed tomography (CT) with subtle hemorrhage in the conventional lighting conditions which are common situations in the remote CT reading. The comparison of the clinician's performance was undertaken through detecting hemorrhage by the iPad 2 and the clinical liquid crystal display (LCD) monitor. We selected 100 brain CT exams performed for head trauma or headache. Fifty had subtle radiological signs of intracranial hemorrhage (ICH), while the other 50 showed no significant abnormality. Five emergency medicine physicians reviewed these brain CT scans using the iPad 2 and the LCD monitor, scoring the probability of ICH on each exam on a five-point scale. Result showed high sensitivities and specificities in both devices. We generated receiver operating characteristic curves and calculated the average area under the curve of the iPad 2 and the LCD (0.935 and 0.900). Using the iPad 2 and reliable internet connectivity, clinicians can provide remote evaluation of brain CT with subtle hemorrhage under suboptimal viewing condition. Considering the distinct advantages of the iPad 2, the popular out-of-hospital use of mobile CT teleradiology would be anticipated soon.

show abstract

Section: Discussionmentioning

confidence: 99%

An Assessment of the iPad 2 as a CT Teleradiology Tool Using Brain CT with Subtle Intracranial Hemorrhage Under Conventional Illumination

Park¹,

Lee²,

Kang³

2013

J Digit Imaging

View full text Add to dashboard Cite

show abstract

“…Selection of an image location, if a nodule was present in the image, was considered a true positive image identification. To compare the detection performance of the radiologists, a receiver operating characteristic (ROC) curve analysis based on the multiple reader multiple case (MRMC) method developed by Dorfman, Berbaum, and Metz and implemented in the software DBM MRMC 2.2 was used [16][17][18][19][20][21][22]. The analysis considered readers as fixed effects and the cases as random effects.…”

Section: Performance Evaluationmentioning

confidence: 99%

The Effects of Ambient Lighting in Chest Radiology Reading Rooms

et al. 2012

View full text Add to dashboard Cite

Under typical dark chest radiography reading room conditions, a radiologist's pupils contract and dilate as their visual focus intermittently shifts between the high luminance monitor and the darker background wall, resulting in increased visual fatigue and degradation of diagnostic performance. A controlled increase of ambient lighting may minimize these visual adjustments and potentially improve comfort and accuracy. This study was designed to determine the effect of a controlled increase of ambient lighting on chest radiologist nodule detection performance. Four chest radiologists read 100 radiographs (50 normal and 50 containing a subtle nodule) under low (E01 lx) and elevated (E050 lx) ambient lighting levels on a DICOM-calibrated, medicalgrade liquid crystal display. Radiologists were asked to identify nodule locations and rate their detection confidence. A receiver operating characteristic (ROC) analysis of radiologist results was performed and area under ROC curve (AUC) values calculated for each ambient lighting level. Additionally, radiologist selection times under both illuminance conditions were determined. Average AUC values did not significantly differ (p>0.05) between ambient lighting levels (estimated mean difference0−0.03; 95% CI, (−0.08, 0.03)). Average selection times decreased or remained constant with increased illuminance. The most considerable decreases occurred for false positive identification times (35.4±18.8 to 26.2±14.9 s) and true positive identification times (29.7± 18.3 to 24.5±15.5 s). No performance differences were statistically significant. Study findings suggest that a controlled increase of ambient lighting within darkly lit chest radiology reading rooms, to a level more suitable for performance of common radiological tasks, does not appear to have a statistically significant effect on nodule detection performance.

show abstract

“…devchakraborty.com). The algorithm used for significance testing, applicable to all FOMs in this study and implemented as a subroutine in JAFROC software, was developed at the University of Iowa (45)(46)(47). JAFROC software was appropriately modified to allow this algorithm to be used with nonarea-based FOMs, such as maximum LLF and others.…”

Section: Discussionmentioning

confidence: 99%

“…Statistical analysis for the case management question involved the use of standard multiple reader and multiple case analysis by using the Dorfman-Berbaum-Metz procedure (45) as updated by Hillis et al (46,47) at an a level of .05. The levels of significance of reader and modality-reader effects were calculated by using the analysis of variance table output of JAFROC.…”

Section: Thoracic Imaging: Multi-institutional Evaluation Of Pulmonarmentioning

confidence: 99%

Multi-Institutional Evaluation of Digital Tomosynthesis, Dual-Energy Radiography, and Conventional Chest Radiography for the Detection and Management of Pulmonary Nodules

Dobbins

McAdams

Sabol³

et al. 2017

Radiology

View full text Add to dashboard Cite

Purpose:To conduct a multi-institutional, multireader study to compare the performance of digital tomosynthesis, dual-energy (DE) imaging, and conventional chest radiography for pulmonary nodule detection and management. Materials and Methods:In this binational, institutional review board-approved, HIPAAcompliant prospective study, 158 subjects (43 subjects with normal findings) were enrolled at four institutions. Informed consent was obtained prior to enrollment. Subjects underwent chest computed tomography (CT) and imaging with conventional chest radiography (posteroanterior and lateral), DE imaging, and tomosynthesis with a flat-panel imaging device. Three experienced thoracic radiologists identified true locations of nodules (n = 516, 3-20-mm diameters) with CT and recommended case management by using Fleischner Society guidelines. Five other radiologists marked nodules and indicated case management by using images from conventional chest radiography, conventional chest radiography plus DE imaging, tomosynthesis, and tomosynthesis plus DE imaging. Sensitivity, specificity, and overall accuracy were measured by using the free-response receiver operating characteristic method and the receiver operating characteristic method for nodule detection and case management, respectively. Results were further analyzed according to nodule diameter categories (3-4 mm, .4 mm to 6 mm, .6 mm to 8 mm, and .8 mm to 20 mm). Results:Maximum lesion localization fraction was higher for tomosynthesis than for conventional chest radiography in all nodule size categories (3.55-fold for all nodules, P , .001; 95% confidence interval [CI]: 2.96, 4.15). Case-level sensitivity was higher with tomosynthesis than with conventional chest radiography for all nodules (1.49-fold, P , .001; 95% CI: 1.25, 1.73). Case management decisions showed better overall accuracy with tomosynthesis than with conventional chest radiography, as given by the area under the receiver operating characteristic curve (1.23-fold, P , .001; 95% CI: 1.15, 1.32). There were no differences in any specificity measures. DE imaging did not significantly affect nodule detection when paired with either conventional chest radiography or tomosynthesis. Conclusion:Tomosynthesis outperformed conventional chest radiography for lung nodule detection and determination of case management; DE imaging did not show significant differences over conventional chest radiography or tomosynthesis alone. These findings indicate performance likely achievable with a range of reader expertise.Online supplemental material is available for this article. Imaging data were collected from subjects at four tertiary care sitesthree in the United States and one in Sweden. The study (ClinicalTrials.gov no. NCT00963651) was approved by the institutional review board of each institution, and informed consent was obtained from each subject prior to enrollment. The study was compliant with the Health Insurance Portability and Accountability Act. Subjects enrolled were referred for, or had recently undergone,...

show abstract

Recent Developments in the Dorfman-Berbaum-Metz Procedure for Multireader ROC Study Analysis

Cited by 175 publications

References 17 publications

An Assessment of the iPad 2 as a CT Teleradiology Tool Using Brain CT with Subtle Intracranial Hemorrhage Under Conventional Illumination

An Assessment of the iPad 2 as a CT Teleradiology Tool Using Brain CT with Subtle Intracranial Hemorrhage Under Conventional Illumination

The Effects of Ambient Lighting in Chest Radiology Reading Rooms

Multi-Institutional Evaluation of Digital Tomosynthesis, Dual-Energy Radiography, and Conventional Chest Radiography for the Detection and Management of Pulmonary Nodules

Contact Info

Product

Resources

About