A novel interpretable machine learning system to generate clinical risk scores: An application for predicting early mortality or unplanned readmission in a retrospective cohort study

Abstract

Risk scores are widely used for clinical decision making and commonly generated from logistic regression models. Machine-learning-based methods may work well for identifying important predictors to create parsimonious scores, but such ‘black box’ variable selection limits interpretability, and variable importance evaluated from a single model can be biased. We propose a robust and interpretable variable selection approach using the recently developed Shapley variable importance cloud (ShapleyVIC) that accounts for variability in variable importance across models. Our approach evaluates and visualizes overall variable contributions for in-depth inference and transparent variable selection, and filters out non-significant contributors to simplify model building steps. We derive an ensemble variable ranking from variable contributions across models, which is easily integrated with an automated and modularized risk score generator, AutoScore, for convenient implementation. In a study of early death or unplanned readmission after hospital discharge, ShapleyVIC selected 6 variables from 41 candidates to create a well-performing risk score, which had similar performance to a 16-variable model from machine-learning-based ranking. Our work contributes to the recent emphasis on interpretability of prediction models for high-stakes decision making, providing a disciplined solution to detailed assessment of variable importance and transparent development of parsimonious clinical risk scores.

Identifying clinical features and blood biomarkers associated with mild cognitive impairment in Parkinson disease using machine learning

“…In the first step, the overall importance of all 41 variables to logistic regression analysis of PD‐MCI was assessed using ShapleyVIC to screen out variables less useful to the analysis. ShapleyVIC is a recently developed interpretable machine learning method for comprehensive variable importance analysis [19], and its clinical application has been illustrated in a study of adverse outcomes after hospital discharge [25]. Compared to conventional inference, it extends the analysis to a group of nearly optimal models and pools information across these good models using a meta‐analysis approach to provide a robust measure of overall variable importance.…”

Using an Interpretable Amino Acid-Based Machine Learning Method to Enhance the Diagnosis of Major Depressive Disorder

“…Logistic regression was selected because of its interpretability in understanding which features are the most important for the prediction task [5,[30][31][32][33][34], compared to more complex models such as deep neural networks [34,35]. This is a crucial characteristic of the logistic regression model for this research, given that the main objective of this paper is to determine which amino acids are most significant for MDD prediction.…”

A universal AutoScore framework to develop interpretable scoring systems for predicting common types of clinical outcomes

“… 3 The modularized structure allows AutoScore to be integrated with more advanced interpretable machine learning methods (e.g., the Shapley variable importance cloud 28 ) for improved robustness, interpretability and transparency in the risk score development. 29 …”