From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI

Abstract

The rising popularity of explainable artificial intelligence (XAI) to understand high-performing black boxes raised the question of how to evaluate explanations of machine learning (ML) models. While interpretability and explainability are often presented as a subjectively validated binary property, we consider it a multi-faceted concept. We identify 12 conceptual properties, such as Compactness and Correctness, that should be evaluated for comprehensively assessing the quality of an explanation. Our so-called Co-12 properties serve as categorization scheme for systematically reviewing the evaluation practices of more than 300 papers published in the last 7 years at major AI and ML conferences that introduce an XAI method. We find that 1 in 3 papers evaluate exclusively with anecdotal evidence, and 1 in 5 papers evaluate with users. This survey also contributes to the call for objective, quantifiable evaluation methods by presenting an extensive overview of quantitative XAI evaluation methods. Our systematic collection of evaluation methods provides researchers and practitioners with concrete tools to thoroughly validate, benchmark and compare new and existing XAI methods. The Co-12 categorization scheme and our identified evaluation methods open up opportunities to include quantitative metrics as optimization criteria during model training in order to optimize for accuracy and interpretability simultaneously.

Barriers and enabling factors for error analysis in NLG research

“…Recent work in the NLG community has aimed to provide an overview of our evaluation practices, and move towards standardising our terminology and assessment materials (Belz et al, 2020;Howcroft et al, 2020). There have been similar efforts in the areas of Explainable AI (Nauta et al, 2022) and Intelligent Virtual Agents (Fitrianie et al, 2019(Fitrianie et al, , 2020. The majority of our respondents indicated that they would be more likely to carry out an error analysis if there were an existing taxonomy of errors that they could use.…”

Streamlining models with explanations in the learning loop

“…By definition, algorithm-generated explanation of ML models have to be understandable by humans. This concept, tightly coupled with the amount of information included in an explanation, has been referred to as comprehensibility [14], conciseness [3], and compactness [22] -amongst others. Local linear explanations provide an insight on how a black-box classifier weights its input features to achieve the classification outcome of a given sample.…”

Accurate and Reliable Classification of Unstructured Reports on Their Diagnostic Goal Using BERT Models

“…We test the resulting explanations by measuring the classifier’s prediction with only (Fidelity) and without the important words (Deletion Check) [ 46 ]. If relevant words for the classifier were selected by the explanation method, one would expect that the accuracy stayed similar to the original model with only the important words and is significantly reduced if the important words are removed.…”