Opportunities and Challenges in Explainable Artificial Intelligence (XAI): A Survey

Das, Arun; Rad, Paul

doi:10.48550/arxiv.2006.11371

Cited by 177 publications

(238 citation statements)

References 61 publications

Supporting

Mentioning

187

Contrasting

Unclassified

Order By: Relevance

“…In addition to seeking for a similarity index, previous works also explored other ways to interpret DNN representations [7,26,42,43], e.g., visualizing the hidden layer representations [34,45], reasoning explanation [11,14], and gradient-based attribution methods [2,35]. Visualizationbased interpretation [45] aims to explain the model decision by depicting the correlation between input space to the final output.…”

Section: Interpreting Dnn Representationsmentioning

confidence: 99%

Graph-Based Similarity of Neural Network Representations

Chen¹,

Lu²,

Yang³

et al. 2021

Preprint

View full text Add to dashboard Cite

Understanding the black-box representations in Deep Neural Networks (DNN) is an essential problem in deep learning. In this work, we propose Graph-Based Similarity (GBS) to measure the similarity of layer features. Contrary to previous works that compute the similarity directly on the feature maps, GBS measures the correlation based on the graph constructed with hidden layer outputs. By treating each input sample as a node and the corresponding layer output similarity as edges, we construct the graph of DNN representations for each layer. The similarity between graphs of layers identifies the correspondences between representations of models trained in different datasets and initializations. We demonstrate and prove the invariance property of GBS, including invariance to orthogonal transformation and invariance to isotropic scaling, and compare GBS with CKA. GBS shows state-of-the-art performance in reflecting the similarity and provides insights on explaining the adversarial sample behavior on the hidden layer space. 1 .

show abstract

Section: Interpreting Dnn Representationsmentioning

confidence: 99%

Graph-Based Similarity of Neural Network Representations

Chen¹,

Lu²,

Yang³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…For linear or piecewise linear models, such as linear regression, softmax regression, decision trees, and k-nearest neighbors, their simple classification mechanics make them intrinsically interpretable. Their expressive ability is quite limited so that they cannot achieve mundane classification performance when the features have complex interactions (Molnar 2020;Das and Rad 2020;Linardatos, Papastefanopoulos, and Kotsiantis 2021). Most complex machine learning models are not easily interpretable in themselves.…”

Section: Related Workmentioning

confidence: 99%

“…However, most deep neural networks only implicitly learn and use the patterns, and do not explicitly explain the reasons for which a sample belongs to a class. This causes concerns about applying deep learning in some critical fields, such as healthcare and automatic pilot systems (Choi et al 2016;Molnar 2020;Das and Rad 2020;Linardatos, Papastefanopoulos, and Kotsiantis 2021).…”

Section: Introductionmentioning

confidence: 99%

On the Effectiveness of Interpretable Feedforward Neural Network

Li¹,

Fung²,

Abusitta³

2021

Preprint

View full text Add to dashboard Cite

Deep learning models have achieved state-of-the-art performance in many classification tasks. However, most of them cannot provide an interpretation for their classification results. Machine learning models that are interpretable are usually linear or piecewise linear and yield inferior performance. Non-linear models achieve much better classification performance, but it is hard to interpret their classification results. This may have been changed by an interpretable feedforward neural network (IFFNN) proposed that achieves both high classification performance and interpretability for malware detection. If the IFFNN can perform well in a more flexible and general form for other classification tasks while providing meaningful interpretations, it may be of great interest to the applied machine learning community. In this paper, we propose a way to generalize the interpretable feedforward neural network to multi-class classification scenarios and any type of feedforward neural networks, and evaluate its classification performance and interpretability on intrinsic interpretable datasets. We conclude by finding that the generalized IFFNNs achieve comparable classification performance to their normal feedforward neural network counterparts and provide meaningful interpretations. Thus, this kind of neural network architecture has great practical use.

show abstract

“…XAI aims to develop and study methodologies for explaining the predictions made by advanced learning machines such as DNNs. Recent advances in XAI have led to a variety of novel methods [20], [21], [22]. These can be grouped into global and local explanation methods.…”

Section: Introductionmentioning

confidence: 99%

Explaining Bayesian Neural Networks

Bykov,

Höhne,

Creosteanu

et al. 2021

Preprint

View full text Add to dashboard Cite

To make advanced learning machines such as Deep Neural Networks (DNNs) more transparent in decision making, explainable AI (XAI) aims to provide interpretations of DNNs' predictions. These interpretations are usually given in the form of heatmaps, each one illustrating relevant patterns regarding the prediction for a given instance. Bayesian approaches such as Bayesian Neural Networks (BNNs) so far have a limited form of transparency (model transparency) already built-in through their prior weight distribution, but notably, they lack explanations of their predictions for given instances. In this work, we bring together these two perspectives of transparency into a holistic explanation framework for explaining BNNs. Within the Bayesian framework, the network weights follow a probability distribution. Hence, the standard (deterministic) prediction strategy of DNNs extends in BNNs to a predictive distribution, and thus the standard explanation extends to an explanation distribution. Exploiting this view, we uncover that BNNs implicitly employ multiple heterogeneous prediction strategies. While some of these are inherited from standard DNNs, others are revealed to us by considering the inherent uncertainty in BNNs. Our quantitative and qualitative experiments on toy/benchmark data and realworld data from pathology show that the proposed approach of explaining BNNs can lead to more effective and insightful explanations.

show abstract

Opportunities and Challenges in Explainable Artificial Intelligence (XAI): A Survey

Cited by 177 publications

References 61 publications

Graph-Based Similarity of Neural Network Representations

Graph-Based Similarity of Neural Network Representations

On the Effectiveness of Interpretable Feedforward Neural Network

Explaining Bayesian Neural Networks

Contact Info

Product

Resources

About