Recurrent Vision Transformer for Solving Visual Reasoning Problems

Abstract

Although convolutional neural networks (CNNs) showed remarkable results in many vision tasks, they are still strained by simple yet challenging visual reasoning problems. Inspired by the recent success of the Transformer network in computer vision, in this paper, we introduce the Recurrent Vision Transformer (RViT) model. Thanks to the impact of recurrent connections and spatial attention in reasoning tasks, this network achieves competitive results on the same-different visual reasoning problems from the SVRT dataset. The weight-sharing both in spatial and depth dimensions regularizes the model, allowing it to learn using far fewer free parameters, using only 28k training samples. A comprehensive ablation study confirms the importance of a hybrid CNN + Transformer architecture and the role of the feedback connections, which iteratively refine the internal representation until a stable prediction is obtained. In the end, this study can lay the basis for a deeper understanding of the role of attention and recurrent connections for solving visual abstract reasoning tasks. The code for reproducing our results is publicly available here: [hidden for double-blind review].

Understanding the Computational Demands Underlying Visual Reasoning

“…As an initial experiment, we attempted to train and test a Vision Transformer 1 (ViT) (Dosovitskiy et al, 2020) constrained to have a similar number of parameters (21 million) to the ResNet-50 used here. We were not able to get these architectures to do well on most of the tasks that are difficult for ResNets even with 100,000 samples (also shown in Messina, Amato, Carrara, Gennaro, & Falchi, 2021a). It is worth noting that even 100,000 samples remain a relatively small data set size by modern standards since ViT was trained from scratch.…”