Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformers

Chefer, Hila; Gur, Shir; Wolf, Lior

doi:10.1109/iccv48922.2021.00045

Cited by 112 publications

(54 citation statements)

References 42 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…'Generic Attention Explainability' (GAE) by Chefer et al [2021a] propagates attention gradients together with gradients from other parts of the network, resulting in state-of-the art performance in explaining Transformer architectures.…”

Section: Benchmark Methodsmentioning

confidence: 99%

XAI for Transformers: Better Explanations through Conservative Propagation

Ameen¹,

Schnake²,

Eberle³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Transformers have become an important workhorse of machine learning, with numerous applications. This necessitates the development of reliable methods for increasing their transparency. Multiple interpretability methods, often based on gradient information, have been proposed. We show that the gradient in a Transformer reflects the function only locally, and thus fails to reliably identify the contribution of input features to the prediction. We identify Attention Heads and LayerNorm as main reasons for such unreliable explanations and propose a more stable way for propagation through these layers. Our proposal, which can be seen as a proper extension of the well-established LRP method to Transformers, is shown both theoretically and empirically to overcome the deficiency of a simple gradient-based approach, and achieves state-of-the-art explanation performance on a broad range of Transformer models and datasets.

show abstract

Section: Benchmark Methodsmentioning

confidence: 99%

XAI for Transformers: Better Explanations through Conservative Propagation

Ameen¹,

Schnake²,

Eberle³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…We use the pretrained vision transformer [22] ViT-B/32 model of CLIP, that performs global context modeling using self-attention between patches of a given image to capture meaningful features. We use the recent transformer interpretability method by Chefer et al [7] Input Attention Distribution Proposed Random to extract a relevancy map from the self-attention heads, without any text supervision.…”

Section: Strokes Initializationmentioning

confidence: 99%

CLIPasso: Semantically-Aware Object Sketching

Vinker¹,

Pajouheshgar²,

Bo³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Recently, a first attempt at explaining predictions by a VL transformer was proposed in [6]. There the authors constructed a relevancy map using the model's attention layers to track the interactions between modalities.…”

Section: Related Workmentioning

confidence: 99%

VL-InterpreT: An Interactive Visualization Tool for Interpreting Vision-Language Transformers

Aflalo¹,

Du²,

Tseng³

et al. 2022

Preprint

View full text Add to dashboard Cite

Breakthroughs in transformer-based models have revolutionized not only the NLP field, but also vision and multimodal systems. However, although visualization and interpretability tools have become available for NLP models, internal mechanisms of vision and multimodal transformers remain largely opaque. With the success of these transformers, it is increasingly critical to understand their inner workings, as unraveling these black-boxes will lead to more capable and trustworthy models. To contribute to this quest, we propose VL-InterpreT, which provides novel interactive visualizations for interpreting the attentions and hidden representations in multimodal transformers. VL-InterpreT is a task agnostic and integrated tool that (1) tracks a variety of statistics in attention heads throughout all layers for both vision and language components, (2) visualizes crossmodal and intra-modal attentions through easily readable heatmaps, and (3) plots the hidden representations of vision and language tokens as they pass through the transformer layers. In this paper, we demonstrate the functionalities of VL-InterpreT through the analysis of KD-VLP, an end-toend pretraining vision-language multimodal transformerbased model, in the tasks of Visual Commonsense Reasoning (VCR) and WebQA, two visual question answering benchmarks. Furthermore, we also present a few interesting findings about multimodal transformer behaviors that were learned through our tool. * Equal Contributions1 A screencast and live demo of our app is available at https:// youtu.be/2HZ2IjzG5_4 and http://vlinterpretenv4envenv.eba-vmhhefup.us-east-2.elasticbeanstalk.com/. Source code will be released shortly.

show abstract

Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformers

Cited by 112 publications

References 42 publications

XAI for Transformers: Better Explanations through Conservative Propagation

XAI for Transformers: Better Explanations through Conservative Propagation

CLIPasso: Semantically-Aware Object Sketching

VL-InterpreT: An Interactive Visualization Tool for Interpreting Vision-Language Transformers

Contact Info

Product

Resources

About