Receiver Operator characteristic (ROC) Analysis without Truth

Henkelman, R. Mark; Kay, Ian; Bronskill, Michael

doi:10.1177/0272989x9001000105

Cited by 95 publications

(61 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The variance of the noise σ m was fixed for each modality in this experiment. In the limit of large patient numbers, the three different curves (each representing a different modality) tend to a minimum value σ m / a m (see Eqq [1] and [5]) in accordance with ML theory. Figure 3b compares the performance of conventional regression analysis with that of RWT.…”

Section: Analysis Of Rwtsupporting

confidence: 73%

“…We use a similar figure of merit to characterize the performance of a single application of RWT. The RMSE for a given modality m is (5) This figure of merit was chosen because it measures the difference between the gold standard, Θ p , and the values found through adjusting the data, θ pm , by the estimated linear model parameters, â m and b̂m. Note that this figure of merit cannot be used in practice, however, because of the lack of a gold standard, but it provides an excellent technique to evaluate the method in a simulation.…”

Section: Figure Of Meritmentioning

confidence: 99%

“…This type of analysis required a gold standard until Henkelman et al (5) developed a technique to compute ROC curves for multiple imaging modalities without use of a gold standard.…”

mentioning

confidence: 99%

See 2 more Smart Citations

Estimation in Medical Imaging without a Gold Standard

Kupinski¹,

Hoppin²,

Clarkson³

et al. 2002

Academic Radiology

View full text Add to dashboard Cite

Rationale and Objectives-In medical imaging, physicians often estimate a parameter of interest (eg, cardiac ejection fraction) for a patient to assist in establishing a diagnosis. Many different estimation methods may exist, but rarely can one be considered a gold standard. Therefore, evaluation and comparison of different estimation methods are difficult. The purpose of this study was to examine a method of evaluating different estimation methods without use of a gold standard.Materials and Methods-This method is equivalent to fitting regression lines without the x axis. To use this method, multiple estimates of the clinical parameter of interest for each patient of a given population were needed. The authors assumed the statistical distribution for the true values of the clinical parameter of interest was a member of a given family of parameterized distributions. Furthermore, they assumed a statistical model relating the clinical parameter to the estimates of its value. Using these assumptions and observed data, they estimated the model parameters and the parameters characterizing the distribution of the clinical parameter.Results-The authors applied the method to simulated cardiac ejection fraction data with varying numbers of patients, numbers of modalities, and levels of noise. They also tested the method on both linear and nonlinear models and characterized the performance of this method compared to that of conventional regression analysis by using x-axis information. Results indicate that the method follows trends similar to that of conventional regression analysis as patients and noise vary, although conventional regression analysis outperforms the method presented because it uses the gold standard which the authors assume is unavailable. Conclusion-The method accurately estimates model parameters. These estimates can be used to rank the systems for a given estimation task. KeywordsEstimation; gold standard; image-quality assessment; maximum likelihood Much of the recent research in medical imaging has dealt with the development of imaging systems or of image-processing techniques to produce "better" images. Thus, regardless of the imaging modality involved, a definition of what constitutes a "better" image is required. One common approach to the assessment of image quality is visual comparison by human observers. This method, however, is both subjective and often irreproducible. A more

show abstract

Section: Analysis Of Rwtsupporting

confidence: 73%

Section: Figure Of Meritmentioning

confidence: 99%

See 1 more Smart Citation

Estimation in Medical Imaging without a Gold Standard

Kupinski¹,

Hoppin²,

Clarkson³

et al. 2002

Academic Radiology

View full text Add to dashboard Cite

show abstract

“…Uebersax ( 1988) and Uebersax andGrove ( 1989, 1993) discussed a latent distribution model for multiple rater ordered category data. Quinn ( 1989) and Henkelman, Kay, and Bronskill ( 1990) derived similar approaches from signal detection theory. These models regard the characteristic that ratings assess as a continuous latent trait.…”

Section: Introductionmentioning

confidence: 99%

Statistical Modeling of Expert Ratings on Medical Treatment Appropriateness

Uebersax

1993

Journal of the American Statistical Association

View full text Add to dashboard Cite

This article uses latent structure analysis to model ordered category ratings by multiple experts on the appropriateness of indications for the medical procedure carotid endarterectomy. The statistical method used is a form of located latent class analysis, which combines elements of latent class and latent trait analysis. It assumes that treatment indications fall into distinct latent classes, with each latent class corresponding to a different level of appropriateness. The appropriateness rating of a treatment indication by a rater is assumed determined by the latent class membership of the indication, rating category thresholds of the rater, and random measurement error. The located latent class model has two alternative forms: a normal ogive form, which derives from the assumption of normally distributed measurement error, and a logistic approximation to the normal form. The approach has the following advantages for the analysis of ordered category ratings by multiple experts: ( 1 ) it assesses whether different raters base ratings on the same or different criteria; ( 2 ) it assesses rater bias-the tendency of some raters to make higher or lower ratings than others; ( 3 ) it characterizes rater differences in rating category definitions; ( 4 ) it provides theoretically based methods for combining the ratings of different raters; and ( 5 ) it provides a description of the distribution of the latent trait. The data examined are appropriateness ratings on 848 indications for carotid endarterectomy made by nine medical experts. The located latent class approach provides unique insights concerning the data. It identifies what appears to be a set of clear nonindications for carotid endarterectomy, but a corresponding set of clear indications is not evident. The results indicate that all raters measured a common latent trait of treatment appropriateness, but that some measured the trait better than others. Rater differences in overall bias and rating category definitions are evident. Two methods are used to combine raters' ratings. One uses ratings to calculate a continuous appropriateness score for each indication. The other uses ratings to assign indications to discrete outcome categories, each corresponding to a specific level of appropriateness. The located latent class approach for ordered category measures has possible applications besides the analysis of expert ratings, such as item analysis. Potential extensions of the model are discussed.KEY WORDS: Expert ratings; Health care; Latent class analysis; Latent trait analysis; Observer agreement; Ordered category data. I. INTRODUCTION1986), who had panels of national experts rate the appropriateness of several medical treatments. There is much recent concern about ensuring the approBecause of space limitations, only a brief synopsis of stapriateness of medical treatment. One proposed strategy for tistical methods for the analysis of multiple expert rating establishing treatment standards is to have expert panels re-data with ordered categories is provided. More...

show abstract

“…However, the literature on modeling continuous test scores without a gold standard is comparatively small. Frequentist parametric approaches have been developed by Henkelman, Kay, and Bronskill [26], Beiden et al [27], and Kupinski et al [28]. A Bayesian parametric approach was developed by Choi et al [29], who used a bivariate two-group normal model for correlated tests in ROC analysis, while Choi, Johnson, and Thurmond [30] developed methods for risk prediction based on parametric models without a gold-standard test.…”

Section: Introductionmentioning

confidence: 99%

Flexible regression models for ROC and risk analysis, with or without a gold standard

Branscum

Johnson

Hanson

et al. 2015

Statistics in Medicine

View full text Add to dashboard Cite

A novel semiparametric regression model is developed for evaluating the covariate-specific accuracy of a continuous medical test or biomarker. Ideally, studies designed to estimate or compare medical test accuracy will use a separate, flawless gold-standard procedure to determine the true disease status of sampled individuals. We treat this as a special case of the more complicated and increasingly common scenario in which disease status is unknown because a gold-standard procedure does not exist or is too costly or invasive for widespread use. To compensate for missing data on disease status, covariate information is used to discriminate between diseased and healthy units. We thus model the probability of disease as a function of 'disease covariates'. In addition, we model test/biomarker outcome data to depend on 'test covariates', which provides researchers the opportunity to quantify the impact of covariates on the accuracy of a medical test. We further model the distributions of test outcomes using flexible semiparametric classes. An important new theoretical result demonstrating model identifiability under mild conditions is presented. The modeling framework can be used to obtain inferences about covariate-specific test accuracy and the probability of disease based on subject-specific disease and test covariate information. The value of the model is illustrated using multiple simulation studies and data on the age-adjusted ability of soluble epidermal growth factor receptor -a ubiquitous serum protein -to serve as a biomarker of lung cancer in men. sas code for fitting the model is provided.

show abstract

Receiver Operator characteristic (ROC) Analysis without Truth

Cited by 95 publications

References 9 publications

Estimation in Medical Imaging without a Gold Standard

Estimation in Medical Imaging without a Gold Standard

Statistical Modeling of Expert Ratings on Medical Treatment Appropriateness

Flexible regression models for ROC and risk analysis, with or without a gold standard

Contact Info

Product

Resources

About