Don’t Dismiss Logistic Regression: The Case for Sensible Extraction of Interactions in the Era of Machine Learning

Abstract

Background:Machine learning approaches have become increasingly popular modeling techniques, relying on data-driven heuristics to arrive at its solutions. Recent comparisons between these algorithms and traditional statistical modeling techniques have largely ignored the superiority gained by the former approaches due to involvement of model-building search algorithms. This has led to alignment of statistical and machine learning approaches with different types of problems and the under-development of procedures that combine their attributes. In this context, we hoped to understand the domains of applicability for each approach and to identify areas where a marriage between the two approaches is warranted. We then sought to develop a hybrid statistical-machine learning procedure with the best attributes of each. Methods:We present three simple examples to illustrate when to use each modeling approach and posit a general framework for combining them into an enhanced logistic regression model building procedure that aids interpretation. We study 556 benchmark machine learning datasets to uncover when machine learning techniques outperformed rudimentary logistic regression models and so are potentially well-equipped to enhance them. We illustrate a software package, InteractionTransformer, which embeds logistic regression with advanced model building capacity by using machine learning algorithms to extract candidate interaction features from a random forest model for inclusion in the model. Finally, we apply our enhanced logistic regression analysis to two real-word biomedical examples, one where predictors vary linearly with the outcome and another with extensive second-order interactions. Results:Preliminary statistical analysis demonstrated that across 556 benchmark datasets, the random forest approach significantly outperformed the logistic regression approach. We found a statistically significant increase in predictive performance when using hybrid procedures and greater clarity in the association with the outcome of terms acquired compared to directly interpreting the random forest output. Conclusions:When a random forest model is closer to the true model, hybrid statistical-machine learning procedures can substantially enhance the performance of statistical procedures in an automated manner while preserving easy interpretation of the results. Such hybrid methods may help facilitate widespread adoption of machine learning techniques in the biomedical setting.

Identification of Spatial Proteomic Signatures of Colon Tumor Metastasis: A Digital Spatial Profiling Approach

“…Salient cross-level interactions were identified from the MEML method using the interactiontransformer package in Python v3.7, which scores interactions using SHAP and selects the top interactions using a knee locator method. Interactions were applied to a Bayesian hierarchical logistic regression model to report pertinent effect modifiers (e.g., effect of CD20, conditional on age; interactions encapsulated in 𝑥 ⃗) 27,37 : 𝑙𝑜𝑔𝑖𝑡(𝑝 " ) = 𝛽 ⃗ ⋅ 𝑥 ⃗ + 𝜃 <+,=> ["] . As many interactions were initially selected, features were selected using the Horseshoe LASSO method and the remaining features were fit with weakly informative priors 38,39 .…”

Investigating the potential for machine learning prediction of patient outcomes: a retrospective study of hospital acquired pressure injuries

“…The model graphics were generated using the SHAP library. We tested for possible interaction effects using the InteractionTransformer package [28]. Code is available upon request.…”

What Makes You Hold on to That Old Car? Joint Insights From Machine Learning and Multinomial Logit on Vehicle-Level Transaction Decisions

“…Recent advances in "Explainable AI" (Lundberg et al, 2020) have improved the interpretability of tree-based ML models exploring high-dimensional feature spaces. Behavioral insights from ML models, such as individual feature importance, directional influences, and feature interactions, have been incorporated into the logistic model building process to improve model specification and prediction performance (Zhao et al, 2019), (Levy and O'Malley, 2020). Travel behavior insights jointly determined from ML and logistic models can also improve the robustness of derived conclusions.…”