Enhancing Explainability of Neural Networks Through Architecture Constraints

Yang, Zebin; Zhang, Aijun; Sudjianto, Agus

doi:10.1109/tnnls.2020.3007259

Cited by 69 publications

(21 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Whereas SHAP and LIME seek to explain complex models using a regression-like paradigm (i.e., a linear additive function), Explainable Neural Networks (XNNs) [144] use a more general formulation based on an "additive index model" [127]. Here, the algorithm seeks to return a function that describes how model predictions vary with changes to individual parameters (or, more recently, pairs of parameters [148]). As in LIME and SHAP, these models can help data scientists with the appropriate training to understand how changing a specific feature might change the model's prediction, albeit at the risk of inferring spurious correlations.…”

Section: Local Feature Importancementioning

confidence: 99%

Psychological foundations of explainability and interpretability in artificial intelligence

Broniatowski¹

2021

View full text Add to dashboard Cite

In this paper, we make the case that interpretability and explainability are distinct requirements for machine learning systems. To make this case, we provide an overview of the literature in experimental psychology pertaining to interpretation (especially of numerical stimuli) and comprehension. We find that interpretation refers to the ability to contextualize a model's output in a manner that relates it to the system's designed functional purpose, and the goals, values, and preferences of end users. In contrast, explanation refers to the ability to accurately describe the mechanism, or implementation, that led to an algorithm's output, often so that the algorithm can be improved in some way. Beyond these definitions, our review shows that humans differ from one another in systematic ways, that affect the extent to which they prefer to make decisions based on detailed explanations versus less precise interpretations. These individual differences, such as personality traits and skills, are associated with their abilities to derive meaningful interpretations from precise explanations of model output. This implies that system output should be tailored to different types of users.

show abstract

Section: Local Feature Importancementioning

confidence: 99%

Psychological foundations of explainability and interpretability in artificial intelligence

Broniatowski¹

2021

View full text Add to dashboard Cite

show abstract

“…By contrast, other researches have tried to improve interpretability by changing the structure of the neural networks (NN). Yang et al [ 22 ] proposed the use of an explainable NN (xNN) subject to interpretability constraints in terms of the additivity, sparsity, orthogonality, and smoothness. A complex function is decomposed into sparse additive subnetworks and the projection indexes are forced to be mutually orthogonal such that the resulting subnetworks tend to be less confounded with each other.…”

Section: Related Studiesmentioning

confidence: 99%

“…Studies on the development and testing of an IML learning model have been conducted to improve transparency while maintaining a high-level learning ability by modifying the existing machine learning technologies or developing new ones. The technical approach for IML can be divided into the following: (a) explaining a decision of the learning model (ELM) [ 10 , 11 , 12 , 13 , 14 ] and (b) interpreting the learning model (ILM) [ 15 , 16 , 17 , 18 , 19 , 20 , 21 , 22 , 23 ].…”

Section: Introductionmentioning

confidence: 99%

Model Simplification of Deep Random Forest for Real-Time Applications of Various Sensor Data

Kim

Nam

2021

Sensors

View full text Add to dashboard Cite

The deep random forest (DRF) has recently gained new attention in deep learning because it has a high performance similar to that of a deep neural network (DNN) and does not rely on a backpropagation. However, it connects a large number of decision trees to multiple layers, thereby making analysis difficult. This paper proposes a new method for simplifying a black-box model of a DRF using a proposed rule elimination. For this, we consider quantifying the feature contributions and frequency of the fully trained DRF in the form of a decision rule set. The feature contributions provide a basis for determining how features affect the decision process in a rule set. Model simplification is achieved by eliminating unnecessary rules by measuring the feature contributions. Consequently, the simplified and transparent DRF has fewer parameters and rules than before. The proposed method was successfully applied to various DRF models and benchmark sensor datasets while maintaining a robust performance despite the elimination of a large number of rules. A comparison with state-of-the-art compressed DNNs also showed the proposed model simplification’s higher parameter compression and memory efficiency with a similar classification accuracy.

show abstract

“…Neural Interaction Transparency (NIT) (Tsang et al, 2018) is a framework that produces the same model as GAMI-Net, but by disentangling interactions within a FFNN. The Explainable Neural Network (xNN) (Vaughan et al, 2018), Adaptive xNN (AxNN) (Chen et al, 2020) and Enhanced xNN (ExNN) (Yang et al, 2020a) are all based on the Generalised Additive Index Model (GAIM):…”

Section: Related Workmentioning

confidence: 99%

Pairwise networks for feature ranking of a geomagnetic storm model

Beukes

Davel

Lotz

2020

SACJ

View full text Add to dashboard Cite

Feedforward neural networks provide the basis for complex regression models that produce accurate predictions in a variety of applications. However, they generally do not explicitly provide any information about the utility of each of the input parameters in terms of their contribution to model accuracy. With this in mind, we develop the pairwise network, an adaptation to the fully connected feedforward network that allows the ranking of input parameters according to their contribution to model output. The application is demonstrated in the context of a space physics problem. Geomagnetic storms are multi-day events characterised by significant perturbations to the magnetic field of the Earth, driven by solar activity. Previous storm forecasting efforts typically use solar wind measurements as input parameters to a regression problem tasked with predicting a perturbation index such as the 1-minute cadence symmetric-H (Sym-H) index. We re-visit the task of predicting Sym-H from solar wind parameters, with two ‘twists’: (i) Geomagnetic storm phase information is incorporated as model inputs and shown to increase prediction performance. (ii) We describe the pairwise network structure and training process – first validating ranking ability on synthetic data, before using the network to analyse the Sym-H problem.

show abstract

Enhancing Explainability of Neural Networks Through Architecture Constraints

Cited by 69 publications

References 21 publications

Psychological foundations of explainability and interpretability in artificial intelligence

Psychological foundations of explainability and interpretability in artificial intelligence

Model Simplification of Deep Random Forest for Real-Time Applications of Various Sensor Data

Pairwise networks for feature ranking of a geomagnetic storm model

Contact Info

Product

Resources

About