Towards a More Reliable Interpretation of Machine Learning Outputs for Safety-Critical Systems Using Feature Importance Fusion

Abstract

When machine learning supports decision-making in safety-critical systems, it is important to verify and understand the reasons why a particular output is produced. Although feature importance calculation approaches assist in interpretation, there is a lack of consensus regarding how features’ importance is quantified, which makes the explanations offered for the outcomes mostly unreliable. A possible solution to address the lack of agreement is to combine the results from multiple feature importance quantifiers to reduce the variance in estimates and to improve the quality of explanations. Our hypothesis is that this leads to more robust and trustworthy explanations of the contribution of each feature to machine learning predictions. To test this hypothesis, we propose an extensible model-agnostic framework divided in four main parts: (i) traditional data pre-processing and preparation for predictive machine learning models, (ii) predictive machine learning, (iii) feature importance quantification, and (iv) feature importance decision fusion using an ensemble strategy. Our approach is tested on synthetic data, where the ground truth is known. We compare different fusion approaches and their results for both training and test sets. We also investigate how different characteristics within the datasets affect the quality of the feature importance ensembles studied. The results show that, overall, our feature importance ensemble framework produces 15% less feature importance errors compared with existing methods. Additionally, the results reveal that different levels of noise in the datasets do not affect the feature importance ensembles’ ability to accurately quantify feature importance, whereas the feature importance quantification error increases with the number of features and number of orthogonal informative features. We also discuss the implications of our findings on the quality of explanations provided to safety-critical systems.

Generic Diagnostic Framework for Anomaly Detection—Application in Satellite and Spacecraft Systems

“…If it is set too high, anomalies will be missed, and if it is set too low, the rate of false positives will become high. Typically used methodologies for thresholding are Area Under Curve Percentage (AUCP) [16], Median Absolute Deviation (MAD) [17], Modified Thompson Tau Test (MTT) [18], Variational Autoencoders (VAE) [19], Z-Score [20] or Clustering-based techniques [21].…”

Investigation Of Diabetes Data with Permutation Feature Importance Based Deep Learning Methods

“…The purpose of the PFI is to calculate how much the performance measure of the model has decreased by randomly extracting the features from the data set. The amount of increase in the RMSE (Ibrahim and Jafari 2019) or MAE (Rengasamy, Rothwell, and Figueredo 2021) values can be determined by the effects of the used features in the model on the classification. The bigger the change, the more important that feature is.…”

Uso de dados administrativos hospitalares para o desenvolvimento de modelos preditivos de readmissão hospitalar não planejadas de pacientes idosos em um hospital público terciário na cidade de São Paulo, Brasil

“…Considerada uma das mais importantes fases na construção de um modelo de aprendizado de máquina, técnicas de seleção de variáveis vem sendo destaques em várias literaturas (Chen et al, 2020;Rengasamy et al, 2021), pois ao selecionar as variáveis que realmente são relevantes para serem aplicados ao modelo, ocorrerá uma interpretação melhor de como cada uma delas afetam as predições. Embora a identificação e seleção de variáveis possa ser feita de forma empírica (por meio do conhecimento de especialistas, popularidade na literatura e sucesso preditivo em pesquisas anteriores), tendo a oportunidade de melhorar ainda mais o presente modelo com recursos correlacionados e não redundantes, além de diminuir a complexidade, facilitar a compreensão e ajuda a melhorar o desempenho das métricas em exatidão, precisão e recuperação.…”