Attention is not not Explanation

Abstract

Attention mechanisms play a central role in NLP systems, especially within recurrent neural network (RNN) models. Recently, there has been increasing interest in whether or not the intermediate representations offered by these modules may be used to explain the reasoning for a model's prediction, and consequently reach insights regarding the model's decision-making process. A recent paper claims that 'Attention is not Explanation' (Jain and Wallace, 2019). We challenge many of the assumptions underlying this work, arguing that such a claim depends on one's definition of explanation, and that testing it needs to take into account all elements of the model. We propose four alternative tests to determine when/whether attention can be used as explanation: a simple uniform-weights baseline; a variance calibration based on multiple random seed runs; a diagnostic framework using frozen weights from pretrained models; and an end-to-end adversarial attention training protocol. Each allows for meaningful interpretation of attention mechanisms in RNN models. We show that even when reliable adversarial distributions can be found, they don't perform well on the simple diagnostic, indicating that prior work does not disprove the usefulness of attention mechanisms for explainability.

BERTology Meets Biology: Interpreting Attention in Protein Language Models

“…The present analysis identifies associations between attention and various properties of proteins. It does not attempt to establish a causal link between attention and model behavior [28,84], nor to explain model predictions [35,87]. While the focus of this paper is reconciling attention patterns with known properties of proteins, one could also leverage attention to discover novel types of properties and processes.…”

“…Interpreting attention on natural language sequences is a well-established area of research [12,30,87,90]. In some cases, it has been shown that attention correlates with syntactic and semantic relationships in natural language [15,32,83].…”

Attention in Natural Language Processing

“…It then becomes hard if not impossible to pinpoint the reasons behind the wrong output of a neural architecture. Interestingly, attention could provide a key to partially interpret and explain neural network behavior [5]- [9], even if it cannot be considered a reliable means of explanation [10], [11]. For instance, the weights computed by attention could point us to relevant information discarded by the neural network or to irrelevant elements of the input source that have been factored in and could explain a surprising output of the neural network.…”

Topics to Avoid: Demoting Latent Confounds in Text Classification

“…Recently, there has been a debate on whether attention can be used to explain model decisions (Serrano and Smith, 2019;Jain and Wallace, 2019;Wiegreffe and Pinter, 2019), we thus present additional analysis of our proposed method based on saliency maps (Ding et al, 2019). Saliency maps have been shown to better capture word alignment than attention probabilities in neural machine translation.…”