Evaluating a New Marker for Risk Prediction Using the Test Tradeoff: An Update

Abstract

Most of the methodological literature on evaluating an additional marker for risk prediction involves purely statistical measures of classification performance. A disadvantage of a purely statistical measure is the difficulty in deciding the improvement in the measure that would make inclusion of the additional marker worthwhile. In contrast, a medical decision making approach can weigh the cost or harm of ascertaining an additional marker against the benefit of a higher true positive rate for a given false positive rate that may be associated with risk prediction involving the additional marker. An appealing form of the medical decision making approach involves the risk threshold, which is the risk at which the expected utility of treatment and no treatment is the same. In this framework, a readily interpretable evaluation of the net benefit of an additional marker is the test tradeoff corresponding to the risk threshold. The test tradeoff is the minimum number of tests for a new marker that need to be traded for a true positive to yield an increase in the net benefit of risk prediction with the additional marker. For a sensitivity analysis the test tradeoff is computed over multiple risk thresholds. This article updates the theory and estimation of the test tradeoff. An example is provided.

A model to predict risk of postpartum infection after Caesarean delivery*

“…Finally, a decision curve was plotted to inform clinicians about the range of threshold probabilities for which the prediction model would be of clinical value [22]. Decision curve analysis offers insight into clinical consequences of using the model by determining the relationship between a chosen predicted probability threshold and the relative value of falsepositive and false-negative results to obtain a value of net benefit of using the model at that threshold [23][24][25][26]. All analyses were performed using R version 3.2.3 (SPSS Inc., Chicago, IL) and methods of Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) were followed [18].…”

A new approach to test validity and clinical usefulness of the 2013 ACC/AHA guideline on statin therapy: A population-based study

“…For this reason we used the test tradeoff defined by Baker SG, et al as the minimum number of tests that have to be traded for a true-positive to yield an increase in the net benefit [20,21]; it equals to the inverse of net benefit. The test tradeoff multiplied by the proportion of subjects who supposed to be treated yields a treatment tradeoff which is equal to NNT, the numbers that should be treated to prevent one more event (true-positive).…”

Letters to the Editor

“…where C is the utility of a true negative minus the utility of a false positive, B is the utility of a true positive minus the utility of a false negative, and π = Pr(G = 1) (e.g., Baker, Van Calster, and Steyerberg 2012). In this framework q M (x) equals a monotonic function of Pr(J (x) = j |G = 1)/ Pr(J (x) = j |G = 0).…”

“…The set of false and true positive rates over values of k gives the Receiver Operating Characteristic (ROC) curve. The optimal classification boundary q M (x) = k in the external validation sample occurs at the value of k when the slope of the aforementioned ROC curve equals the right side of Equation (5) (e.g., Baker, Van Calster, and Steyerberg 2012).…”