Hollow-tree super: A directional and scalable approach for feature importance in boosted tree models

Abstract

Purpose Current limitations in methodologies used throughout machine-learning to investigate feature importance in boosted tree modelling prevent the effective scaling to datasets with a large number of features, particularly when one is investigating both the magnitude and directionality of various features on the classification into a positive or negative class. This manuscript presents a novel methodology, “Hollow-tree Super” (HOTS), designed to resolve and visualize feature importance in boosted tree models involving a large number of features. Further, this methodology allows for accurate investigation of the directionality and magnitude various features have on classification and incorporates cross-validation to improve the accuracy and validity of the determined features of importance. Methods Using the Iris dataset, we first highlight the characteristics of HOTS by comparing it to other commonly used techniques for feature importance, including Gini Importance, Partial Dependence Plots, and Permutation Importance, and explain how HOTS resolves the weaknesses present in these three strategies for investigating feature importance. We then demonstrate how HOTS can be utilized in high dimensional spaces such as neuroscientific setting, by taking 60 Schizophrenic subjects from the publicly available SchizConnect database and applying the method to determine which regions of the brain were most important for the positive and negative classification of schizophrenia as determined by the positive and negative syndrome scale (PANSS). Results HOTS effectively replicated and supported the findings of feature importance for classification of the Iris dataset when compared to Gini importance, Partial Dependence Plots and Permutation importance, determining ‘petal length’ as the most important feature for positive and negative classification. When applied to the Schizconnect dataset, HOTS was able to resolve from 379 independent features, the top 10 most important features for classification, as well as their directionality for classification and magnitude compared to other features. Cross-validation supported that these same 10 features were consistently used in the decision-making process across multiple trees, and these features were localised primarily to the occipital and parietal cortices, commonly disturbed brain regions in those afflicted with Schizophrenia. Conclusion HOTS effectively overcomes previous challenges of identifying feature importance at scale, and can be utilized across a swathe of disciplines. As computational power and data quantity continues to expand, it is imperative that a methodology is developed that is able to handle the demands of working with large datasets that contain a large number of features. This approach represents a unique way to investigate both the directionality and magnitude of feature importance when working at scale within a boosted tree model that can be easily visualized within commonly used software.

12 Plagues of AI in Healthcare: A Practical Guide to Current Issues With Using Machine Learning in a Medical Context

“…Ultimately, ML tools are capable of taking highly dimensional data and quickly making accurate decisions in highly time-critical medical scenarios, a feat that humans may never physically nor cognitively be capable of performing ( 54 ). However, if we could explain the various decisions being executed by a certain model and the specific features being analyzed to produce a certain outcome ( 24 ), physicians can better interpret these results based on logic and previous knowledge. Then, healthcare providers may not only be able to better trust these algorithms, but providers may also continually improve the model's performance when the system presents an error that is likely based on a specific wrong answer possibly being executed in a portion of a decision tree.…”

“…Movement toward white-box, also called glass-box, models provides a solution to address concerns of explainability. These models can often be seen with linear ( 58 ) and decision-tree based models ( 24 ), although a number of other applications are increasingly being developed ( 53 ). In fact, DL based networks make up the majority of the highly sought after radiological-AI applications for the medical field ( 1 ), such as the systems that can diagnose brain cancer during surgery.…”

“…Research and Development goals should be grounded in what has already been suggested in the medical field as important avenues of future work for clinical improvements. For instance, rather than attempting to predict an illness which can already be clearly identified in clinical practice, ML tools can reduce the complexity of patient information presented in a specific pathological states ( 24 , 25 ) and then present statistical irregularities that can be used by physicians to make more informed decisions for clinical treatment ( 26 , 27 ). Ultimately, it must be remembered that AI is a powerful tool that can be leveraged to answer a difficult questions, the usefulness of the tool is a function of the appropriateness of the question being asked.…”

Machine Learning Decomposition of the Anatomy of Neuropsychological Deficit in Alzheimer’s Disease and Mild Cognitive Impairment

“…The black box problem in machine learning generally limits the ability to utilize machine learning techniques in clinical practice, as there is generally a need-to-know which parts of the brain contribute to a given pathology. In order to address this, we used a boosted trees approach, called Hollow-tree Super (HoTS) ( Doyen et al, 2021 ), to determine which features of each machine learning model, in this case the functional connectivity among brain regions, were contributing most to the model’s prediction of performance on each neuropsychological test. Performance in each test was classified by a tertile split, with the upper and lower tertiles classed as poor and good performance, respectively.…”

“…In this study, we performed functional connectivity-based analysis and utilized a recently described machine learning approach ( Doyen et al, 2021 ) to identify commonalities and differences in brain regions across neuropsychological domains in a cohort of MCI and AD patients, and age-matched cognitively unimpaired subjects. We sought to explore patterns among these regions to identify potential markers which may be used in future studies to develop better disease classifiers.…”