Relationship between Roe and Metz simulation model for multireader diagnostic data and Obuchowski‐Rockette model parameters

Abstract

For the typical diagnostic radiology study design, each case (ie, patient) undergoes several diagnostic tests (or modalities) and the resulting images are interpreted by several readers. Often, each reader is asked to assign a confidence-of-disease rating to each case for each test, and the diagnostic tests are compared with respect to reader-performance outcomes that are functions of the reader receiver operating characteristic (ROC) curves, such as the area under the ROC curve. These reader-performance outcomes are frequently analyzed using the Obuchowski and Rockette method, which allows conclusions to generalize to both the reader and case populations. The simulation model proposed by Roe and Metz (RM) in 1997 emulates confidence-of-disease data collected from such studies and has been an important tool for empirically evaluating various reader-performance analysis methods. However, because the RM model parameters are expressed in terms of a continuous decision variable rather than in terms of reader-performance outcomes, it has not been possible to evaluate the realism of the RM model. I derive the relationships between the RM and Obuchowski-Rockette model parameters for the empirical area under the ROC curve reader-performance outcome. These relationships make it possible to evaluate the realism of the RM parameter models and to assess the performance of Obuchowski-Rockette parameter estimates. An example illustrates the application of the relationships for assessing the performance of a proposed upper one-sided confidence bound for the Obuchowski-Rockette test-by-reader variance component, which is useful for sample size estimation.

Multireader sample size program for diagnostic studies: demonstration and methodology

“…We note that this result has previously been given by Hillis, 28 who refers to as the reader-specific expected accuracy.…”

“…Typically, conjectured values for the error correlations, , , and , are used instead of the error covariances, , , and , because they have been shown in simulations to be relatively stable across different case and reader sample sizes when rating data are generated from the same probabilistic statistical model. 28 In contrast, the covariances are dependent on the case sample sizes. The correlation is the within-reader between-test correlation of accuracy measurement errors for a fixed reader when reading random samples of cases.…”

Determining Roe and Metz model parameters for simulating multireader multicase confidence-of-disease rating data based on read-data or conjectured Obuchowski-Rockette parameter estimates

“…which result in the same DV distributions for both tests 1 and 2. Under this constraint, it can be shown 3 that the mean and median separation of the nondiseased and diseased DV distributions across the reader population is given by μ + and the median reader-specific AUC is given by A z = Φ μ + / 2 .…”

“…2 Numerous studies have used this model for evaluating MRMC analysis and sample size methods. As discussed by Hillis, 3 the RM model generates continuous confidence-of-diseases ratings based on an underlying equal-variance binormal model for each reader, with the separation between the normal and abnormal rating distributions varying across readers.…”

Paired split-plot designs of multireader multicase studies

“…It is known that some of the Roe and Metz simulation parameters may not be realistic. 18 Thus, it is useful to further compare different designs with realistic variance parameters estimated from real data. We used the iMRMC software 8 to analyze a real dataset, and, based on the estimated parameters, we compared the PSP design to the FC design.…”