Transformer Interpretability Beyond Attention Visualization

Abstract

Self-attention techniques, and specifically Transformers, are dominating the field of text processing and are becoming increasingly popular in computer vision classification tasks. In order to visualize the parts of the image that led to a certain classification, existing methods either rely on the obtained attention maps, or employ heuristic propagation along the attention graph. In this work, we propose a novel way to compute relevancy for Transformer networks. The method assigns local relevance based on the deep Taylor decomposition principle and then propagates these relevancy scores through the layers. This propagation involves attention layers and skip connections, which challenge existing methods. Our solution is based on a specific formulation that is shown to maintain the total relevancy across layers. We benchmark our method on very recent visual Transformer networks, as well as on a text classification problem, and demonstrate a clear advantage over the existing explainability methods. Our code is available at: https://github.com/hila-chefer/ Transformer-Explainability.

Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformers

“…This, however, neglects the intermediate attention scores, as well as the other components of the Transformers. As noted by Chefer et al [5], the computation in each attention head mixes queries, keys, and values and cannot be fully captured by considering only the inner products of queries and keys, which is what is referred to as attention.…”

“…What is common to all of these is that the mapping from the two inputs to the prediction contains interaction between the two modalities. These interactions often challenge the existing explainability methods that are aimed at attention-based models, since, as far as we can ascertain, all existing Transformer explainability methods (e.g., [5,1]) heavily rely on self-attention, and do not provide adaptations to any other form of attention, which is commonly used in multi-modal Transformers.…”

“…We use an exemplar model from each architecture, and prove our method's superiority over existing Transformer explainability methods, adapted from their single modality origin. Our explainability prescription is easier to implement than existing methods, such as [5], and can be readily applied to any attention-based architecture.…”

Vision Transformer using Low-level Chest X-ray Feature Corpus for COVID-19 Diagnosis and Severity Quantification

“…For the interpretability of the classification model, we adopted a visualization method of saliency map tailored for ViT suggested by (Chefer et al, 2020), which computes relevancy for Transformer network. Specifically, unlike the traditional approaches of gradient propagation methods (Selvaraju et al, 2017;Smilkov et al, 2017;Srinivas and Fleuret, 2019) or attribution propagation methods (Bach et al, 2015;Gu et al, 2018), which rely on the heuristic propagation along attention graph or the obtained attention maps, the method in Chefer et al (2020) calculate the local relevance with deep Taylor decomposition, which is then propagated throughout the layers. This relevance propagation method is especially useful for models based on Transformer architecture, as it overcomes the problem of selfattention operations and skip connections.…”

IA-RED$^2$: Interpretability-Aware Redundancy Reduction for Vision Transformers

“…As the attention module in the vision transformer computes the fullyconnected relations among all of the input patches, the computational cost is then quadratic with regard to the length of the input sequence. On the other hand, previous works [6,8] have already shown the vulnerable interpretability of the original vision transformer, where the raw attention coming from the architecture sometimes fails to perceive the informative region of the input images.…”