Methods for Evaluating Prediction Performance of Biomarkers and Tests

Pepe, Margaret S.; Janes, Holly

doi:10.1007/978-1-4614-8981-8_7

Cited by 33 publications

(51 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This work extends existing approaches for evaluating markers for risk prediction (see Pepe and Janes (2012), Huang et al (2007), Gu and Pepe (2009)). It also unifies existing methodology for evaluating treatment selection markers.…”

Section: Discussionmentioning

confidence: 76%

“…Still others have focused on the specific problem of optimizing marker combinations for treatment selection (Lu, Zhang, and Zeng (2011), Foster, Taylor, and Ruberg (2011), Gunter, Zhu, and Murphy (2011), Qian and Murphy (2011), McKeague and Qian (2013), Zhang, Tsiatis, Laber, and Davidian (2012)). A complete framework for marker evaluation, on par with those developed for evaluating markers for classification (Pepe (2003), Zhou, McClish, and Obuchowski (2002)) or risk prediction (Pepe and Janes (2012)), is still forthcoming.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

An Approach to Evaluating and Comparing Biomarkers for Patient Treatment Selection

Janes

Brown

Huang

et al. 2014

The International Journal of Biostatistics

Self Cite

View full text Add to dashboard Cite

Despite the heightened interest in developing biomarkers predicting treatment response that are used to optimize patient treatment decisions, there has been relatively little development of statistical methodology to evaluate these markers. There is currently no unified statistical framework for marker evaluation. This paper proposes a suite of descriptive and inferential methods designed to evaluate individual markers and to compare candidate markers. An R software package has been developed which implements these methods. Their utility is illustrated in the breast cancer treatment context, where candidate markers are evaluated for their ability to identify a subset of women who do not benefit from adjuvant chemotherapy and can therefore avoid its toxicity.

show abstract

Section: Discussionmentioning

confidence: 76%

Section: Introductionmentioning

confidence: 99%

An Approach to Evaluating and Comparing Biomarkers for Patient Treatment Selection

Janes

Brown

Huang

et al. 2014

The International Journal of Biostatistics

Self Cite

View full text Add to dashboard Cite

show abstract

“…A variety of statistics have been proposed and there has been some debate in the literature about which measures are most appropriate [14,22]. Arguments have centered on the interpretations and clinical relevance of various measures.…”

Section: Discussionmentioning

confidence: 99%

“…The ROC( f , risk) measure is the proportion of cases classified as high risk when the high risk threshold is chosen as that exceeded by no more than a proportion f of controls. This is closely related to the PCF statistic proposed by Pfeiffer and Gail [17,14] that is defined as the proportion of cases classified as high risk when the high risk threshold is chosen so that a fixed proportion of the population is classified as high risk. The standardized net benefit, S NB( t ), is a weighted average of the true and false positive rates associated with use of the risk threshold t to classify subjects as high risk.…”

Section: Illustration With Simulated Datamentioning

confidence: 99%

The Net Reclassification Index (NRI): A Misleading Measure of Prediction Improvement Even with Independent Test Data Sets

et al. 2014

View full text Add to dashboard Cite

The Net Reclassification Index (NRI) is a very popular measure for evaluating the improvement in prediction performance gained by adding a marker to a set of baseline predictors. However, the statistical properties of this novel measure have not been explored in depth. We demonstrate the alarming result that the NRI statistic calculated on a large test dataset using risk models derived from a training set is likely to be positive even when the new marker has no predictive information. A related theoretical example is provided in which an incorrect risk function that includes an uninformative marker is proven to erroneously yield a positive NRI. Some insight into this phenomenon is provided. Since large values for the NRI statistic may simply be due to use of poorly fitting risk models, we suggest caution in using the NRI as the basis for marker evaluation. Other measures of prediction performance improvement, such as measures derived from the ROC curve, the net benefit function and the Brier score, cannot be large due to poorly fitting risk functions.

show abstract

“…Calibration is one of the most important model performance characteristics because a miscalibrated model produces invalid risk estimates [5] and can introduce errors into decision-making. The Hosmer-Lemeshow goodness-of-fit test is often used as a test of calibration [6][7].…”

Section: Introductionmentioning

confidence: 99%

Tests of calibration and goodness‐of‐fit in the survival setting

Demler

Paynter

Cook

2015

Statistics in Medicine

229

201

View full text Add to dashboard Cite

Summary To access the calibration of a predictive model in a survival analysis setting, several authors have extended the Hosmer and Lemeshow goodness of fit test to survival data. Grønnesby and Borgan developed a test under the assumption of proportional hazards, and Nam and D'Agostino developed a nonparametric test that is applicable in a more general survival setting for data with limited censoring. We analyze the performance of the two tests and show that the Grønnesby-Borgan test attains appropriate size in a variety of settings, whereas the D'Agostino-Nam method has a higher than nominal Type 1 error when there is more than trivial censoring. Both tests are sensitive to small cell sizes. We develop a modification of the D'Agostino-Nam test to allow for higher censoring rates. We show that this modified D'Agostino-Nam test has appropriate control of Type 1 error and comparable power to the Grønnesby-Borgan test, and is applicable to settings other than proportional hazards. We also discuss the application to small cell sizes.

show abstract

Methods for Evaluating Prediction Performance of Biomarkers and Tests

Cited by 33 publications

References 32 publications

An Approach to Evaluating and Comparing Biomarkers for Patient Treatment Selection

An Approach to Evaluating and Comparing Biomarkers for Patient Treatment Selection

The Net Reclassification Index (NRI): A Misleading Measure of Prediction Improvement Even with Independent Test Data Sets

Tests of calibration and goodness‐of‐fit in the survival setting

Contact Info

Product

Resources

About