Generalized Roe and Metz receiver operating characteristic model: analytic link between simulated decision scores and empirical AUC variances and covariances

Gallas, Brandon D.; Hillis, Stephen L.

doi:10.1117/1.jmi.1.3.031006

Cited by 18 publications

(20 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Roe and Metz (RM) proposed a model for the simulation of MRMC data that emulate confidence‐of‐disease data collected from such studies. Studies that have used this model for evaluating various MRMC analysis and sample size methods include those given in the Reference List . To account for the MRMC study design, the RM model generates continuous decision variable (DV) data according to a binormal model for which the nondiseased and diseased DV variances are equal for each test‐reader combination, while allowing the differences between the diseased and nondiseased DV means to vary across test‐reader combinations.…”

Section: Introductionmentioning

confidence: 99%

Relationship between Roe and Metz simulation model for multireader diagnostic data and Obuchowski‐Rockette model parameters

Hillis

2018

Statistics in Medicine

Self Cite

View full text Add to dashboard Cite

For the typical diagnostic radiology study design, each case (ie, patient) undergoes several diagnostic tests (or modalities) and the resulting images are interpreted by several readers. Often, each reader is asked to assign a confidence-of-disease rating to each case for each test, and the diagnostic tests are compared with respect to reader-performance outcomes that are functions of the reader receiver operating characteristic (ROC) curves, such as the area under the ROC curve. These reader-performance outcomes are frequently analyzed using the Obuchowski and Rockette method, which allows conclusions to generalize to both the reader and case populations. The simulation model proposed by Roe and Metz (RM) in 1997 emulates confidence-of-disease data collected from such studies and has been an important tool for empirically evaluating various reader-performance analysis methods. However, because the RM model parameters are expressed in terms of a continuous decision variable rather than in terms of reader-performance outcomes, it has not been possible to evaluate the realism of the RM model. I derive the relationships between the RM and Obuchowski-Rockette model parameters for the empirical area under the ROC curve reader-performance outcome. These relationships make it possible to evaluate the realism of the RM parameter models and to assess the performance of Obuchowski-Rockette parameter estimates. An example illustrates the application of the relationships for assessing the performance of a proposed upper one-sided confidence bound for the Obuchowski-Rockette test-by-reader variance component, which is useful for sample size estimation.

show abstract

Section: Introductionmentioning

confidence: 99%

Relationship between Roe and Metz simulation model for multireader diagnostic data and Obuchowski‐Rockette model parameters

Hillis

2018

Statistics in Medicine

Self Cite

View full text Add to dashboard Cite

show abstract

“…A mixed effects ANOVA model (modalities and readers as fixed effects, cases as a random effect) was used to evaluate the significance of AUC for CE-bCT. As a check, we also conducted an analysis of each reader separately using a non-parametric ROC model implemented in the iMRMC software 19 .…”

Section: Methodsmentioning

confidence: 99%

Differentiation of ductal carcinoma in-situ from benign micro-calcifications by dedicated breast computed tomography

Aminololama-Shakeri

Abbey

Gazi

et al. 2016

European Journal of Radiology

View full text Add to dashboard Cite

Purpose Compare conspicuity of ductal carcinoma in-situ (DCIS) to benign calcifications on unenhanced (bCT), contrast-enhanced dedicated breast CT (CEbCT) and mammography (DM). Methods and Materials The institutional review board approved this HIPAA-compliant study. 42 women with Breast Imaging Reporting and Data System 4 or 5 category micro-calcifications had breast CT before biopsy. Three subjects with invasive disease at surgery were excluded. Two breast radiologists independently compared lesion conspicuity scores (CS) for CEbCT, to bCT and DM. Enhancement was measured in Hounsfield units (HU). Mean CS ± standard deviations are shown. Receiver operating characteristic analysis (ROC) measured radiologists’ discrimination performance by comparing CS to enhancement alone. Statistical measurements were made using ANOVA F-test, Wilcoxon rank-sum test and robust linear regression analyses. Results 39 lesions (17 DCIS, 22 benign) were analyzed. DCIS (8.5±0.9, n=17) was more conspicuous than benign micro-calcifications (3.6±2.9, n=22; p<0.0001) on CEbCT. DCIS was equally conspicuous on CEbCT and DM (8.5±0.9, 8.7±0.8, n=17; p=0.85) and more conspicuous when compared to bCT (5.3±2.6, n=17; p<0.001). All DCIS enhanced; mean enhancement (90HU ±53HU, n=17) was higher compared to benign lesions (33 ±30HU, n=22)(p<0.0001). ROC analysis of the radiologists’ CS showed high discrimination performance (AUC=0.94) compared to enhancement alone (AUC=0.85) (p<0.026). Conclusion DCIS is more conspicuous than benign micro-calcifications on CEbCT. DCIS visualization on CEbCT is equal to mammography but improved compared to bCT. Radiologists’ discrimination performance using CEBCT is significantly higher than enhancement values alone. CEbCT may have an advantage over mammography by reducing false positive examinations when calcifications are analyzed.

show abstract

“…OSEL scientists share their models and data with the public to facilitate the use and broader adoption of these computational tools and approaches. For example, anatomical models of the head (the MIDA model) and whole body models (the Virtual Family) can be downloaded from the IT'IS Foundation's website 12 , 13 The experimental and computational data from an FDA-led multi-laboratory study for fluid dynamics on generic medical devices can be found here; 14 the simulation and statistical tools for the VICTRE project are here 15 and here ( 65 , 66 ); the HIFU simulator here 16 Other software applications are being shared through Github 17 , such as the design of a generic inferior vena cava filter 18 and a risk assessment tool for assessing color additives 19…”

Section: Computational Modeling Researchmentioning

confidence: 99%

Advancing Regulatory Science With Computational Modeling for Medical Devices at the FDA's Office of Science and Engineering Laboratories

et al. 2018

View full text Add to dashboard Cite

Protecting and promoting public health is the mission of the U.S. Food and Drug Administration (FDA). FDA's Center for Devices and Radiological Health (CDRH), which regulates medical devices marketed in the U.S., envisions itself as the world's leader in medical device innovation and regulatory science–the development of new methods, standards, and approaches to assess the safety, efficacy, quality, and performance of medical devices. Traditionally, bench testing, animal studies, and clinical trials have been the main sources of evidence for getting medical devices on the market in the U.S. In recent years, however, computational modeling has become an increasingly powerful tool for evaluating medical devices, complementing bench, animal and clinical methods. Moreover, computational modeling methods are increasingly being used within software platforms, serving as clinical decision support tools, and are being embedded in medical devices. Because of its reach and huge potential, computational modeling has been identified as a priority by CDRH, and indeed by FDA's leadership. Therefore, the Office of Science and Engineering Laboratories (OSEL)—the research arm of CDRH—has committed significant resources to transforming computational modeling from a valuable scientific tool to a valuable regulatory tool, and developing mechanisms to rely more on digital evidence in place of other evidence. This article introduces the role of computational modeling for medical devices, describes OSEL's ongoing research, and overviews how evidence from computational modeling (i.e., digital evidence) has been used in regulatory submissions by industry to CDRH in recent years. It concludes by discussing the potential future role for computational modeling and digital evidence in medical devices.

show abstract

Generalized Roe and Metz receiver operating characteristic model: analytic link between simulated decision scores and empirical AUC variances and covariances

Cited by 18 publications

References 34 publications

Relationship between Roe and Metz simulation model for multireader diagnostic data and Obuchowski‐Rockette model parameters

Relationship between Roe and Metz simulation model for multireader diagnostic data and Obuchowski‐Rockette model parameters

Differentiation of ductal carcinoma in-situ from benign micro-calcifications by dedicated breast computed tomography

Advancing Regulatory Science With Computational Modeling for Medical Devices at the FDA's Office of Science and Engineering Laboratories

Contact Info

Product

Resources

About