Validation of prediction models in the presence of competing risks: a guide through modern methods

Abstract

Thorough validation is pivotal for any prediction model before it can be advocated for use in medical practice. For time-to-event outcomes such as breast cancer recurrence, death from other causes is a competing risk. Model performance measures must account for such competing events. In this article, we present a comprehensive yet accessible overview of performance measures for this competing event setting, including the calculation and interpretation of statistical measures for calibration, discrimination, overall prediction error, and clinical usefulness by decision curve analysis. All methods are illustrated for patients with breast cancer, with publicly available data and R code.

Derivation and validation of the CFracture competing risk fracture prediction tool compared with QFracture in older people and people with comorbidity: a population cohort study

“…However, in CFracture, we also accounted for non-fracture death as a second (competing) outcome and included the CCI score as a validated predictor of mortality. 17 These models allow the cumulative incidence function or probability of a fracture outcome occurring over time to be directly estimated. The proportional hazards assumption was assessed by plots of Schoenfeld residuals against time…”

Interpretable prognostic modeling of endometrial cancer

“…IBS reports both model discrimination and calibration, i.e. the extent of an agreement between observed outcomes and model predictions 44 . Brier score is defined as a mean squared difference between event indicators and predicted survival probabilities at a time t 28 .…”

Bias by censoring for competing events in survival analysis

“…Based on a single centre study, for example, El Ters et al9 observed an over-estimation of 13 percentage points (34% minus 21%) in older recipients 20 years after transplantation. Given the increasing interest in implementing prognostic models in medicine,10 also specifically in kidney transplantation (eg, finding surrogate endpoints for clinical trials11 and for individual patient monitoring), the accuracy of such prognostications becomes important 12. We therefore exemplify the bias in a large European registry of kidney transplantations (see also box 1).…”

“…Given the increasing interest in implementing prognostic models in medicine, 10 also specifically in kidney transplantation (eg, finding surrogate endpoints for clinical trials 11 and for individual patient monitoring), the accuracy of such prognostications becomes important. 12 We therefore exemplify the bias in a large European registry of kidney transplantations (see also box 1). More specifically, we show these biases after transplantation in the short term up to 10 years after transplantation, in both high and low risk groups for recipient death, and according to nonparametric (Kaplan-Meier versus Aalen-Johansen) and semi-parametric (Cox versus Fine and Gray) statistical methods.…”