A three-step approach for the derivation and validation of high-performing predictive models using an operational dataset: congestive heart failure readmission case study

Abstract

BackgroundThe aim of this study was to propose an analytical approach to develop high-performing predictive models for congestive heart failure (CHF) readmission using an operational dataset with incomplete records and changing data over time.MethodsOur analytical approach involves three steps: pre-processing, systematic model development, and risk factor analysis. For pre-processing, variables that were absent in >50% of records were removed. Moreover, the dataset was divided into a validation dataset and derivation datasets which were separated into three temporal subsets based on changes to the data over time. For systematic model development, using the different temporal datasets and the remaining explanatory variables, the models were developed by combining the use of various (i) statistical analyses to explore the relationships between the validation and the derivation datasets; (ii) adjustment methods for handling missing values; (iii) classifiers; (iv) feature selection methods; and (iv) discretization methods. We then selected the best derivation dataset and the models with the highest predictive performance. For risk factor analysis, factors in the highest-performing predictive models were analyzed and ranked using (i) statistical analyses of the best derivation dataset, (ii) feature rankers, and (iii) a newly developed algorithm to categorize risk factors as being strong, regular, or weak.ResultsThe analysis dataset consisted of 2,787 CHF hospitalizations at University of Utah Health Care from January 2003 to June 2013. In this study, we used the complete-case analysis and mean-based imputation adjustment methods; the wrapper subset feature selection method; and four ranking strategies based on information gain, gain ratio, symmetrical uncertainty, and wrapper subset feature evaluators. The best-performing models resulted from the use of a complete-case analysis derivation dataset combined with the Class-Attribute Contingency Coefficient discretization method and a voting classifier which averaged the results of multi-nominal logistic regression and voting feature intervals classifiers. Of 42 final model risk factors, discharge disposition, discretized age, and indicators of anemia were the most significant. This model achieved a c-statistic of 86.8%.ConclusionThe proposed three-step analytical approach enhanced predictive model performance for CHF readmissions. It could potentially be leveraged to improve predictive model performance in other areas of clinical medicine.

Predictive models for identifying risk of readmission after index hospitalization for heart failure: A systematic review

“…The CASP tool qualified the remaining 25 studies as multivariate predictive models and they are examined in detail in this paper. [5][6][7][15][16][17][18][19][21][22][23][24][25][26][27][28][29][30][31][32][33][34][35][36][37] The CASP tool conducts the assessment in three parts, A (six questions), B (two questions), and C (three questions), and they broadly focus on validity, results, and general applicability of each study. The questions are answered either "yes" or "no" or "can't tell."…”

“…The multivariate predictive models were published from 1997 to 2017 and the majority of them (21 out of 25, or 84%) were built in the USA. [5][6][7][15][16][17][18][19][21][22][23][24][25]27,[29][30][31][33][34][35][36] One study was done in Australia, the UK, and the Netherlands each. 28,32,37 One study did secondary analysis on RCT data gathered from patients in the Netherlands, Germany, and the UK.…”

“…A majority of the studies (17 out of 25, or 68%) are observational [5][6][7][15][16][17]21,22,24,25,[27][28][29][30][31]35,36 -either prospective 15,28 or retrospective [5][6][7]16,17,21,22,24,25,27,[29][30][31]35,36 -and eight studies 18,19,23,26,[32][33][34]37 (32%) are secondary analysis of the RCTs, as shown in Table 4. One prospective study 15 provided detailed statistics of attrition but the other 28 did not and hence it might have been at the risk of attrition bias.…”

“…Due to the near ubiquity of EHR systems and their tendency to gather detailed patient-and provider-level data, these systems have recently become the data source of choice for building predictive models. 6,7,21,25,[27][28][29][30][31][35][36][37] Outcome and statistical model characteristics The earlier studies are inconsistent in terms of rehospitalization timeframe, which varied from seven days to one year, 5,[16][17][18][19]22 and some studies reported all-cause readmission while others reported only heart failure readmission, 16,23,32 as shown in Table 5. We now notice readmission as a single outcome and 30 days as a standard readmission timeframe in the recent studies.…”

“…One study 34 used the Random Forests algorithm in conjunction with the survival function to estimate readmission days survived from index discharge. Five studies 6,[25][26][27]34 used some version of non-parametric techniques such as Decision Trees, Random Forests, or multiple classifier schemes. Two studies used the Random Forests method for comparing two different models either with another predictor set 34 or with another modeling technique 7 .…”