From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI

Abstract

The rising popularity of explainable artificial intelligence (XAI) to understand high-performing black boxes, also raised the question of how to evaluate explanations of machine learning (ML) models. While interpretability and explainability are often presented as a subjectively validated binary property, we consider it a multi-faceted concept. We identify 12 conceptual properties, such as Compactness and Correctness, that should be evaluated for comprehensively assessing the quality of an explanation. Our so-called Co-12 properties serve as categorization scheme for systematically reviewing the evaluation practice of more than 300 papers published in the last 7 years at major AI and ML conferences that introduce an XAI method. We find that 1 in 3 papers evaluate exclusively with anecdotal evidence, and 1 in 5 papers evaluate with users. We also contribute to the call for objective, quantifiable evaluation methods by presenting an extensive overview of quantitative XAI evaluation methods. This systematic collection of evaluation methods provides researchers and practitioners with concrete tools to thoroughly validate, benchmark and compare new and existing XAI methods. This also opens up opportunities to include quantitative metrics as optimization criteria during model training in order to optimize for accuracy and interpretability simultaneously.

Sources of Risk of AI Systems

“…For this reason, these methods need to be analysed in detail to obtain a list of risk sources that have an impact on the safety of AI systems. For example, the ongoing scientific discussion on the topic of XAI (explainable artificial intelligence) shows that this is one of the core problems of deep learning [ 59 , 60 , 61 ]. This problem is a direct result of model complexity, which in turn results from the application of artificial intelligence to complex tasks in complex environments [ 62 , 63 ].…”

A Comprehensive Survey on Trustworthy Graph Neural Networks: Privacy, Robustness, Fairness, and Explainability

“…DNNs typically stack multiple complex nonlinear layers [226], resulting in predictions difficult to understand. To expose the black box of these highly complex deep models in a systematic and interpretable manner, explainable DNNs [134], have been explored recently. However, most of these works focus on images or texts, which cannot be directly applied to GNNs due to the discreteness of graph topology and the message-passing of GNNs.…”

“…Whereas a gold standard exists for comparing predictive models, there is no agreed-upon evaluation strategy for explainable AI methods. As argued in [134], evaluating the plausibility and convincingness of an explanation to humans is different from evaluating its correctness, and those criteria should not be conflated. In this part, we systematically analyze certain properties that good explanations should satisfy.…”

“…Ideal explanations should contain "the whole truth" [100]. High completeness is desired to provide enough details, and it should be balanced with correctness [134]. • Consistency: The obtained explanations should be consistent concerning to inputs.…”

Evaluating Feature Attribution: An Information-Theoretic Perspective

“…There is a plethora of works on different explanation techniques (Tjoa & Guan, 2020), especially attribution methods that assign importance scores to each input features. With the growing numbers of attribution methods, various scholars have presented desiderata that explanations, should fulfill (Bhatt et al, 2020a;Nguyen & Martínez, 2020;Fel et al, 2021;Afchar et al, 2021;Nauta et al, 2022 2018) also proposed to perturb the pixels in the input image according to the importance scores. However, Hooker et al (2019) showed that the perturbation introduces artifacts and results in a distribution shift, putting these no-retraining approaches in question.…”