On the Interaction of Belief Bias and Explanations

Abstract

A myriad of explainability methods have been proposed in recent years, but there is little consensus on how to evaluate them. While automatic metrics allow for quick benchmarking, it isn't clear how such metrics reflect human interaction with explanations. Human evaluation is of paramount importance, but previous protocols fail to account for belief biases affecting human performance, which may lead to misleading conclusions. We provide an overview of belief bias, its role in human evaluation, and ideas for NLP practitioners on how to account for it. For two experimental paradigms, we present a case study of gradientbased explainability introducing simple ways to account for humans' prior beliefs: models of varying quality and adversarial examples. We show that conclusions about the highest performing methods change when introducing such controls, pointing to the importance of accounting for belief bias in evaluation.

Human Interpretation of Saliency-based Explanation Over Text

“…By assigning human-like reasoning to the model behavior being explained [39], the explainee may fill any incompleteness in the explanation with assumptions from their own priors about what is plausible to them [9,22].…”

“…Additional research shows this effect manifests in practice in AI settings [11,14,22,25,40]. This means, for example, that the explainee may underestimate the influence of a punctuation token, even if the explanation reports that this token is highly significant (Figure 1), because the explainee is attempting to understand how the model reasons by analogy to the explainee's own mind which is an instance of anthropomorphic bias [8,29,61] and belief bias [16,22].We identify three different such biases which may influence the explainee's interpretation: (i) anthropomorphic bias and belief bias: influence by the explainee's self projection onto the model; (ii) visual perception bias: influence by the explainee's visual affordances for comprehending information; (iii) learning effects: observable temporal changes in the explainee's interpretation as a result of interacting with the explanation over multiple instances.We thus address the following question in this paper: When a human explainee observes feature-attribution explanations, does their comprehended information differ from what the explanation "objectively" attempts to communicate? If so, how?We propose a methodology to investigate whether explainees exhibit biases when interpreting feature-attribution explanations in NLP, which effectively distort the objective attribution into a subjective interpretation of it (Section 4).We conduct user studies in which we show an input sentence and a feature-attribution explanation (i.e., saliency map) to explainees, ask them to report their subjective interpretation, and analyze their responses for statistical significance across multiple factors, such as word length, total input length, or dependency relation, using GAMMs (Section 5).We find that word length, sentence length, the position of the sentence in the temporal course of the experiment, the saliency rank, capitalization, dependency relation, word position, word frequency as well as sentiment can significantly affect user perception.…”

“…While much research exists on developing and verifying such explanations [1,4,32,35,50,51],less is known about the information that human explainees actually understand from them [2,12,19,39].In the explainable NLP literature, it is generally (implicitly) assumed that the explainee interprets the information "correctly", as it is communicated [4,17,20]: e.g., when one word is explained to be influential in the model's decision process, or more influential than another word, it is assumed that the explainee understands this relationship [28].We question this assumption: research in the social sciences describes modes in which the human explainee may be biased-via some cognitive habit-in their interpretation of processes [15,37,39,52]. Additional research shows this effect manifests in practice in AI settings [11,14,22,25,40]. This means, for example, that the explainee may underestimate the influence of a punctuation token, even if the explanation reports that this token is highly significant (Figure 1), because the explainee is attempting to understand how the model reasons by analogy to the explainee's own mind which is an instance of anthropomorphic bias [8,29,61] and belief bias [16,22].We identify three different such biases which may influence the explainee's interpretation: (i) anthropomorphic bias and belief bias: influence by the explainee's self projection onto the model; (ii) visual perception bias: influence by the explainee's visual affordances for comprehending information; (iii) learning effects: observable temporal changes in the explainee's interpretation as a result of interacting with the explanation over multiple instances.We thus address the following question in this paper: When a human explainee observes feature-attribution explanations, does their comprehended information differ from what the explanation "objectively" attempts to communicate?…”

“…Additional research shows this effect manifests in practice in AI settings [11,14,22,25,40]. This means, for example, that the explainee may underestimate the influence of a punctuation token, even if the explanation reports that this token is highly significant (Figure 1), because the explainee is attempting to understand how the model reasons by analogy to the explainee's own mind which is an instance of anthropomorphic bias [8,29,61] and belief bias [16,22].…”

Diagnosing AI Explanation Methods with Folk Concepts of Behavior

“…(3) Additional social science sources on theory of XAI communication with humans: discourse theory [82] and collaboration theory [101]. (4) Other cognitive habits in comprehending explanations of behavior: e.g., the least effort principle [130], confirmation bias [92], belief bias [42].…”