Comparison of feature importance measures as explanations for classification models

Saarela, Mirka; Jauhiainen, Susanne

doi:10.1007/s42452-021-04148-9

Cited by 199 publications

(125 citation statements)

References 38 publications

(49 reference statements)

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…By now, however, “Explainable AI” is a dedicated branch in ML research, and numerous model-specific and model-agnostic methods are available that can partially explain ML prediction outcomes ( 32 ). Two common ways to explain model performance is to analyze the distribution of input samples ( 4 , 33 ), and to analyze feature importance ( 34 ), especially in a clinical setting ( 35 ).…”

Section: Methodsmentioning

confidence: 99%

“…Obviously, not every classification algorithm yields the same ranking for feature importances. It is argued that a combination of several feature importance rankings can provide more reliable and trustworthy ( 34 ). Therefore, for our report to the expert, we aim at presenting a single table with the top 10 most important features for the given classification problem.…”

Section: Methodsmentioning

confidence: 99%

See 1 more Smart Citation

Using Base-ml to Learn Classification of Common Vestibular Disorders on DizzyReg Registry Data

et al. 2021

View full text Add to dashboard Cite

Background: Multivariable analyses (MVA) and machine learning (ML) applied on large datasets may have a high potential to provide clinical decision support in neuro-otology and reveal further avenues for vestibular research. To this end, we build base-ml, a comprehensive MVA/ML software tool, and applied it to three increasingly difficult clinical objectives in differentiation of common vestibular disorders, using data from a large prospective clinical patient registry (DizzyReg).Methods: Base-ml features a full MVA/ML pipeline for classification of multimodal patient data, comprising tools for data loading and pre-processing; a stringent scheme for nested and stratified cross-validation including hyper-parameter optimization; a set of 11 classifiers, ranging from commonly used algorithms like logistic regression and random forests, to artificial neural network models, including a graph-based deep learning model which we recently proposed; a multi-faceted evaluation of classification metrics; tools from the domain of “Explainable AI” that illustrate the input distribution and a statistical analysis of the most important features identified by multiple classifiers.Results: In the first clinical task, classification of the bilateral vestibular failure (N = 66) vs. functional dizziness (N = 346) was possible with a classification accuracy ranging up to 92.5% (Random Forest). In the second task, primary functional dizziness (N = 151) vs. secondary functional dizziness (following an organic vestibular syndrome) (N = 204), was classifiable with an accuracy ranging from 56.5 to 64.2% (k-nearest neighbors/logistic regression). The third task compared four episodic disorders, benign paroxysmal positional vertigo (N = 134), vestibular paroxysmia (N = 49), Menière disease (N = 142) and vestibular migraine (N = 215). Classification accuracy ranged between 25.9 and 50.4% (Naïve Bayes/Support Vector Machine). Recent (graph-) deep learning models classified well in all three tasks, but not significantly better than more traditional ML methods. Classifiers reliably identified clinically relevant features as most important toward classification.Conclusion: The three clinical tasks yielded classification results that correlate with the clinical intuition regarding the difficulty of diagnosis. It is favorable to apply an array of MVA/ML algorithms rather than a single one, to avoid under-estimation of classification accuracy. Base-ml provides a systematic benchmarking of classifiers, with a standardized output of MVA/ML performance on clinical tasks. To alleviate re-implementation efforts, we provide base-ml as an open-source tool for the community.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

Using Base-ml to Learn Classification of Common Vestibular Disorders on DizzyReg Registry Data

et al. 2021

View full text Add to dashboard Cite

show abstract

“…The variable importance [or feature importance; (Saarela and Jauhiainen, 2021)] represents the relative influence of each input variable in the model predictions and can be estimated using both RF and XGB models. Because the previous results indicate the best performance is obtained using the combined flow data set (with flow regime as a categorical input variable), we focus here on the base configuration (C0) with all flows combined.…”

Section: Variable Importance-base Configuration With Combined Flowsmentioning

confidence: 99%

“…Here, those key variables identified as most important differ depending on the ML technique used, and while they represent physical attributes or processes that are well known to influence HEFs and TTDs, it is not yet clear to what degree these measures are indicative of actual physical controls. While understanding feature importance remains an active research area in explainable AI, it has been suggested that combining multiple ML methods is necessary to increase interpretability of the predictions (Saarela and Jauhiainen, 2021). In this work, the importance of several input variables was identified by both ML models and is consistent with understanding from previous studies, including river bathymetry features (Cardenas and Wilson, 2007;Tonina and Buffington, 2007;Stonedahl et al, 2013), flow regime (Sawyer FIGURE 5 | ML model prediction R 2 on testing dataset for single-ML and flow regime-specific ML models, the top panel is RF model, the bottom panel is based on XGB model.…”

Section: Variable Importance-base Configuration With Combined Flowsmentioning

confidence: 99%

Machine Learning Analysis of Hydrologic Exchange Flows and Transit Time Distributions in a Large Regulated River

Ren

Song

Fang

et al. 2021

Front. Artif. Intell.

View full text Add to dashboard Cite

Hydrologic exchange between river channels and adjacent subsurface environments is a key process that influences water quality and ecosystem function in river corridors. High-resolution numerical models were often used to resolve the spatial and temporal variations of exchange flows, which are computationally expensive. In this study, we adopt Random Forest (RF) and Extreme Gradient Boosting (XGB) approaches for deriving reduced order models of hydrologic exchange flows and associated transit time distributions, with integrated field observations (e.g., bathymetry) and hydrodynamic simulation data (e.g., river velocity, depth). The setup allows an improved understanding of the influences of various physical, spatial, and temporal factors on the hydrologic exchange flows and transit times. The predictors also contain those derived using hybrid clustering, leveraging our previous work on river corridor system hydromorphic classification. The machine learning-based predictive models are developed and validated along the Columbia River Corridor, and the results show that the top parameters are the thickness of the top geological formation layer, the flow regime, river velocity, and river depth; the RF and XGB models can achieve 70% to 80% accuracy and therefore are effective alternatives to the computational demanding numerical models of exchange flows and transit time distributions. Each machine learning model with its favorable configuration and setup have been evaluated. The transferability of the models to other river reaches and larger scales, which mostly depends on data availability, is also discussed.

show abstract

“…This is what was done in Figures 4 and 5. For other measures of feature importance, see Saarela and Jauhiainen, 2021. For the Hedonic feature set (as defined in Section 5.1), we obtained the rankings described in Figures 4 and 5, for the Random Forests and the Gradient Boosting regressors, respectively.…”

Section: Feature Selectionmentioning

confidence: 99%

Developing Real Estate Automated Valuation Models by Learning from Heterogeneous Data Sources

Bergadano

Bertilone

Paolotti

et al. 2021

INTREST

View full text Add to dashboard Cite

In this paper we propose a data acquisition methodology, and a Machine Learning solution for the partially automated evaluation of real estate properties. The novelty and importance of the approach lies in two aspects: (1) when compared to Automated Valuation Models (AVMs) as available to real estate operators, it is highly adaptive and non-parametric, and integrates diverse data sources; (2) when compared to Machine Learning literature that has addressed real estate applications, it is more directly linked to the actual business processes of appraisal companies: in this context prices that are advertised online are normally not the most relevant source of information, while an appraisal document must be proposed by an expert and approved by a validator, possibly with the help of technological tools. We describe a case study using a set of 7988 appraisal documents for residential properties in Turin, Italy. Open data were also used, including location, nearby points of interest, comparable property prices, and the Italian revenue service area code. The observed mean error as measured on an independent test set was around 21 K€, for an average property value of about 190 K€. The AVM described here can help the stakeholders in this process (experts, appraisal company) to provide a reference price to be used by the expert, to allow the appraisal company to validate their evaluations in a faster and cheaper way, to help the expert in listing a set of comparable properties, that need to be included in the appraisal document.

show abstract

Comparison of feature importance measures as explanations for classification models

Cited by 199 publications

References 38 publications

Using Base-ml to Learn Classification of Common Vestibular Disorders on DizzyReg Registry Data

Using Base-ml to Learn Classification of Common Vestibular Disorders on DizzyReg Registry Data

Machine Learning Analysis of Hydrologic Exchange Flows and Transit Time Distributions in a Large Regulated River

Developing Real Estate Automated Valuation Models by Learning from Heterogeneous Data Sources

Contact Info

Product

Resources

About