Quantifying discrimination of Framingham risk functions with different survival C statistics

Abstract

Cardiovascular risk prediction functions offer an important diagnostic tool for clinicians and patients themselves. They are usually constructed with the use of parametric or semi-parametric survival regression models. It is essential to be able to evaluate the performance of these models, preferably with summaries that offer natural and intuitive interpretations. The concept of discrimination, popular in the logistic regression context, has been extended to survival analysis. However, the extension is not unique. In this paper, we define discrimination in survival analysis as the model’s ability to separate those with longer event-free survival from those with shorter event-free survival within some time horizon of interest. This definition remains consistent with that used in logistic regression, in the sense that it assesses how well the model-based predictions match the observed data. Practical and conceptual examples and numerical simulations are employed to examine four C statistics proposed in the literature to evaluate the performance of survival models. We observe that they differ in the numerical values and aspects of discrimination that they capture. We conclude that the index proposed by Harrell is the most appropriate to capture discrimination described by the above definition. We suggest researchers report which C statistic they are using, provide a rationale for their selection, and be aware that comparing different indices across studies may not be meaningful.

Comparing two correlated C indices with right‐censored survival outcome: a one‐shot nonparametric approach

“…Various popular extensions of the C index have been proposed in the literature since then [19, 24, 25]. Pencina et al [26] studied these different C statistics systematically and concluded that the C index proposed by Harrell et al [22, 23] is the most appropriate in capturing the discriminating ability of a predictive variable to separate those with longer event-free survival from those with shorter event-free survival within some time horizon of interest.…”

Predictive accuracy of novel risk factors and markers: A simulation study of the sensitivity of different performance measures for the Cox proportional hazards regression model

“…10,11 Different adaptations of the c-statistic have been proposed for use with time-to-event outcomes in which censoring may occur. 8,9,12 Let P i denote the model-based predicted probability of the occurrence of an event prior to time τ and let D i be an event indicator at time τ (i.e. D i  = 1 if the i th subject experienced the event prior to time τ and D i  = 0 otherwise).…”

“…An estimator of this concordance index is $\text{AUC}(\text{CD})(\tau )=\frac{\text{E}[(1-S(\tau |P_j))·S(\tau |P_i)·I(P_i<P_j)]}{\text{E}[(1-S(\tau |P_i)]·\text{E}[S(\tau |P_i)]}$. 8 …”

Statistical controversies in clinical research: prognostic gene signatures are not (yet) useful in clinical practice

“…This is often the case in applications in medicine as only very strong independent prognostic factors can lead to large increases in predictive accuracy. For survival outcomes, there exist different generalization of the AUC [28, 29] and an alternative measure is an R 2 -type statistic to compare the extra variation in clinical outcome explained by the gene signature [30]. Of note, also the batch and laboratory effects typically observed play a role in the lack of applicability of many gene signatures in the clinic, for which a fully specified algorithm is needed for a single patient from a random batch or laboratory.…”