Practical experiences on the necessity of external validation

Abstract

The validity of prognostic models is an important prerequisite for their applicability in practical clinical settings. Here, we report on a specific prognostic study on stroke patients and describe how we explored the prediction performance of our model. We considered two practically highly relevant generalization aspects, namely, the model's performance in patients recruited at a later time point (temporal transportability) and in medical centers different from those used for model building (geographic transportability). To estimate the accuracy of the model, we investigated classical internal validation techniques and leave-one-center-out cross validation (CV). Prognostic models predicting functional independence of stroke patients were developed in a training set using logistic regression, support vector machines, and random forests (RFs). Tenfold CV and leave-one-center-out CV were employed to estimate temporal and geographic transportability of the models. For temporal and external validation, the resulting models were used to classify patients from a later time point and from different clinics. When applying the regression model or the RFs, accuracy in the temporal validation data was well predicted from classical internal validation. However, when predicting geographic transportability all approaches had difficulties. We observed that the leave-one-center-out CV yielded better estimates than classical CV. On the basis of our results, we conclude that external validation in patients from different clinics is required before a prognostic model can be applied in practice. Even validating the model in patients recruited merely at a later time point does not suffice to predict how it may fare with regard to another clinic.

Overview of random forest methodology and practical guidance with emphasis on computational biology and bioinformatics

“…The (combination of) parameter value(s) yielding the lowest error is then selected. However, it needs to be kept in mind that this tuning of parameters increases the necessity of externally validating the resulting prediction rule [43].…”

A noninvasive algorithm to exclude pre-capillary pulmonary hypertension

“…To overcome the problem of overoptimistic results, both an internal and a temporal validation step were added [29]. Internal validation was based on the bootstrap approach [30], reporting an average of 30 bootstrap samples.…”

Comments on ‘Practical experiences on the necessity of external validation’ by I. R. König, J. D. Malley, C. Weimar, H.‐C. Diener and A. Ziegler, Statistics in Medicine, DOI: 10.1002/sim.3069

“…In their comments on our recent publication [1], Björk et al [2] raise two important discussion points. We welcome their comments and concerns as well as the opportunity to reply adequately.…”

“…In a very interesting empirical study, König et al investigated the ability of two cross validation (CV) methods, tenfold CV and leave-one-center-out CV, to predict the generalizability of prognostic models [1]. Three different prognostic models, based on logistic regression, support vector machines and random forests, were established and validated.…”