Evaluating Recurrent Neural Network Explanations

Abstract

Recently, several methods have been proposed to explain the predictions of recurrent neural networks (RNNs), in particular of LSTMs. The goal of these methods is to understand the network's decisions by assigning to each input variable, e.g., a word, a relevance indicating to which extent it contributed to a particular prediction. In previous works, some of these methods were not yet compared to one another, or were evaluated only qualitatively. We close this gap by systematically and quantitatively comparing these methods in different settings, namely (1) a toy arithmetic task which we use as a sanity check, (2) a five-class sentiment prediction of movie reviews, and besides (3) we explore the usefulness of word relevances to build sentence-level representations. Lastly, using the method that performed best in our experiments, we show how specific linguistic phenomena such as the negation in sentiment analysis reflect in terms of relevance patterns, and how the relevance visualization can help to understand the misclassification of individual samples.

Towards Explainable Artificial Intelligence

“…A recent study [8] proposes to objectively evaluate explanation for sequential data using ground truth information in a toy task. The idea of this evaluation metric is to add or subtract two numbers within an input sequence and measure the correlation between the relevances assigned to the elements of the sequence and the two input numbers.…”

The elephant in the interpretability room: Why use attention as explanation when we have saliency methods?

“…In this section we discuss various input saliency methods for NLP as alternatives to attention: gradient-based ( §3.1), propagation-based ( §3.2), and occlusion-based methods ( §3.3), following Arras et al (2019). We do not endorse any specific method 1 , but rather try to give an overview of methods and how they differ.…”

“…Relevance is redistributed until we arrive at the input layers. While LRP requires implementing a custom backward pass, it does allow precise control to preserve relevance, and it has been shown to work better than using gradient-based methods on text classification (Arras et al, 2019).…”

“…For an evaluation of methods for explaining LSTM-based models, see e.g.,Poerner et al (2018) andArras et al (2019).…”

Evaluating Attribution Methods using White-Box LSTMs

“…For example, Zhang et al (2018) propose the pointing game task, in which the highest-relevance pixel for an image classifier input must belong to the object described by the target output class. Within this framework, ), Poerner et al (2018, Arras et al (2019), and Yang and Kim (2019) construct datasets in which input features exhibit experimentally controlled notions of importance, yielding "ground truth" attributions against which heatmaps can be evaluated.…”