Handling missing predictor values when validating and applying a prediction model to new patients

Abstract

Missing data present challenges for development and real-world application of clinical prediction models. While these challenges have received considerable attention in the development setting, there is only sparse research on the handling of missing data in applied settings. The main unique feature of handling missing data in these settings is that missing data methods have to be performed for a single new individual, precluding direct application of mainstay methods used during model development. Correspondingly, we propose that it is desirable to perform model validation using missing data methods that transfer to practice in single new patients. This article compares existing and new methods to account for missing data for a new individual in the context of prediction. These methods are based on (i) submodels based on observed data only, (ii) marginalization over the missing variables, or (iii) imputation based on fully conditional specification (also known as chained equations). They were compared in an internal validation setting to highlight the use of missing data methods that transfer to practice while validating a model. As a reference, they were compared to the use of multiple imputation by chained equations in a set of test patients, because this has been used in validation studies in the past. The methods were evaluated in a simulation study where performance was measured by means of optimism corrected C-statistic and mean squared prediction error. Furthermore, they were applied in data from a large Dutch cohort of prophylactic implantable cardioverter defibrillator patients. K E Y W O R D S clinical prediction modeling, missing data, real-world application, validation Jeroen Hoogland and Marit van Barreveld have contributed equally This is an open access article under the terms of the Creative Commons Attribution-NonCommercial License, which permits use, distribution and reproduction in any medium, provided the original work is properly cited and is not used for commercial purposes.

Real-time imputation of missing predictor values improved the application of prediction models in daily practice

“…The three methods under evaluation are mean imputation, joint modeling imputation (JMI), and conditional modeling imputation (CMI) [13,14,17]. All methods were implemented in R and facilitate live imputation of missing values in individual patients.…”

“…2). It is assumed that all predictor variables of the training sample are normally distributed, such that imputations for an individual patient can directly be generated from the mean and covariance of the training sample and the observed predictor values [14,17]. In contrast to overall mean imputation, the use of covariances between all predictors incorporates the relation between the predictors, and therefore, allows imputations to be tailored to an individual patient's own characteristics.…”

“…This approach, however, is not directly applicable when applying a prediction model in real-time to a single patient in the consulting room. In particular, the models used for imputation cannot be generated ''live'' in clinical practice, and therefore, need to be derived elsewhere and beforehand [14].…”

“…To address these issues, we expand on two well-known methods that may also be used in real-time imputation of missing predictor values [14]: joint modeling imputation (JMI) [17] and conditional modeling imputation (CMI, known for its common use in multiple imputations by chained equations) [13]. As opposed to mean imputation, these methods are able to incorporate the relationship between multiple patient characteristics, and therefore, allow imputations to be adjusted for observed patient specific characteristics.…”

Evaluation of predictive model performance of an existing model in the presence of missing data

“…Alternatively, the user of the model may have access to their own dataset, with information on both the covariates and outcomes for people in this dataset, and if the individual subject can be considered as coming from the same population as this dataset, then the question again is how to make use of these data. These different challenges have received limited attention in the statistical literature, 16,17 but have been expounded upon in a recent publication 18 …”