Transformer Interpretability Beyond Attention Visualization

Chefer, Hila; Gur, Shir; Wolf, Lior

doi:10.1109/cvpr46437.2021.00084

Cited by 388 publications

(261 citation statements)

References 22 publications

Supporting

Mentioning

211

Contrasting

Order By: Relevance

“…Finally, the underlying deep models are black-boxes, making it hard to understand why specific images are identified and described as inappropriate. Combining Q16 with explainable AI methods such as [6] to explain the reasons is likely to improve the datasheet.…”

Section: Discussionmentioning

confidence: 99%

Can Machines Help Us Answering Question 16 in Datasheets, and In Turn Reflecting on Inappropriate Content?

Schramowski¹,

Tauchmann²,

Kersting³

2022

Preprint

View full text Add to dashboard Cite

This paper contains images and descriptions that are offensive in nature.Large datasets underlying much of current machine learning raise serious issues concerning inappropriate content such as offensive, insulting, threatening, or might otherwise cause anxiety. This calls for increased dataset documentation, e.g., using datasheets. They, among other topics, encourage to reflect on the composition of the datasets. So far, this documentation, however, is done manually and therefore can be tedious and error-prone, especially for large image datasets. Here we ask the arguably "circular" question of whether a machine can help us reflect on inappropriate content, answering Question 16 in Datasheets. To this end, we propose to use the information stored in pre-trained transformer models to assist us in the documentation process. Specifically, prompt-tuning based on a dataset of socio-moral values steers CLIP to identify potentially inappropriate content, therefore reducing human labor. We then document the inappropriate images found using word clouds, based on captions generated using a vision-language model. The documentations of two popular, large-scale computer vision datasets-ImageNet and OpenImages-produced this way suggest that machines can indeed help dataset creators to answer Question 16 on inappropriate image content.

show abstract

Section: Discussionmentioning

confidence: 99%

Can Machines Help Us Answering Question 16 in Datasheets, and In Turn Reflecting on Inappropriate Content?

Schramowski¹,

Tauchmann²,

Kersting³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…While this does not provide a deep understanding on the kind of relationships the model has learned [205], it yields some insights as to what it deems important for specific samples [206]. Few works have tried to interpret Transformers further than this for vision [207], and so far within the literature of VTs we only find a limited subset of works that visualize these attention activations for specific samples [66], [69], [99], [130]. The work of [14] consistently finds 6 different patterns in their cross-modal attention, showing, for instance, how some heads learn local, modality specific or cross-modal attention.…”

Section: The Road Aheadmentioning

confidence: 99%

Video Transformers: A Survey

Selva¹,

Johansen²,

Escalera³

et al. 2022

Preprint

View full text Add to dashboard Cite

Transformer models have shown great success modeling long-range interactions. Nevertheless, they scale quadratically with input length and lack inductive biases. These limitations can be further exacerbated when dealing with the high dimensionality of video. Proper modeling of video, which can span from seconds to hours, requires handling long-range interactions. This makes Transformers a promising tool for solving video related tasks, but some adaptations are required. While there are previous works that study the advances of Transformers for vision tasks, there is none that focus on in-depth analysis of video-specific designs. In this survey we analyse and summarize the main contributions and trends for adapting Transformers to model video data. Specifically, we delve into how videos are embedded and tokenized, finding a very widspread use of large CNN backbones to reduce dimensionality and a predominance of patches and frames as tokens. Furthermore, we study how the Transformer layer has been tweaked to handle longer sequences, generally by reducing the number of tokens in single attention operation. Also, we analyse the self-supervised losses used to train Video Transformers, which to date are mostly constrained to contrastive approaches. Finally, we explore how other modalities are integrated with video and conduct a performance comparison on the most common benchmark for Video Transformers (i.e., action classification), finding them to outperform 3D CNN counterparts with equivalent FLOPs and no significant parameter increase.

show abstract

“…Compared with dense attention, the k-NN attention filters out most irrelevant information from background regions which are similar to the foreground, and successfully concentrates on the most informative foreground regions. Images from different classes are visualized in Figure 4 using Transformer Attribution method [5] on DeiT-Tiny. It can be seen that the k-NN attention is more concentrated and accurate, especially in the situations of cluttered background and occlusion.…”

Section: Visualizationmentioning

confidence: 99%

KVT: k-NN Attention for Boosting Vision Transformers

Wang

et al. 2021

Preprint

View full text Add to dashboard Cite

Convolutional Neural Networks (CNNs) have dominated computer vision for years, due to its ability in capturing locality and translation invariance. Recently, many vision transformer architectures have been proposed and they show promising performance. A key component in vision transformers is the fully-connected selfattention which is more powerful than CNNs in modelling long range dependencies. However, since the current dense self-attention uses all image patches (tokens) to compute attention matrix, it may neglect locality of images patches and involve noisy tokens (e.g., clutter background and occlusion), leading to a slow training process and potential degradation of performance. To address these problems, we propose a sparse attention scheme, dubbed k-NN attention, for boosting vision transformers. Specifically, instead of involving all the tokens for attention matrix calculation, we only select the top-k similar tokens from the keys for each query to compute the attention map. The proposed k-NN attention naturally inherits the local bias of CNNs without introducing convolutional operations, as nearby tokens tend to be more similar than others. In addition, the k-NN attention allows for the exploration of long range correlation and at the same time filters out irrelevant tokens by choosing the most similar tokens from the entire image. Despite its simplicity, we verify, both theoretically and empirically, that k-NN attention is powerful in distilling noise from input tokens and in speeding up training. Extensive experiments are conducted by using ten different vision transformer architectures to verify that the proposed k-NN attention can work with any existing transformer architectures to improve its prediction performance.

show abstract

Transformer Interpretability Beyond Attention Visualization

Cited by 388 publications

References 22 publications

Can Machines Help Us Answering Question 16 in Datasheets, and In Turn Reflecting on Inappropriate Content?

Can Machines Help Us Answering Question 16 in Datasheets, and In Turn Reflecting on Inappropriate Content?

Video Transformers: A Survey

KVT: k-NN Attention for Boosting Vision Transformers

Contact Info

Product

Resources

About