Keeping Your Eye on the Ball: Trajectory Attention in Video Transformers

Patrick, Mandela; Campbell, Dylan; Asano, Yuki; Misra, Ishan; Metze, Florian; Feichtenhofer, Christoph; Vedaldi, Andrea; Henriques, João F.

doi:10.48550/arxiv.2106.05392

Cited by 5 publications

(8 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Neverthe-less, it is witnessed that the attention process shall be done with each frame independently, so that the model will attend to locations following the movement of instance through the video. This observation aligns with the conclusion drawn in action recognition [17,28], in which the 1D time domain and 2D space domain have different characteristics and should be handled in a different fashion.…”

Section: Introductionsupporting

confidence: 89%

SeqFormer: Sequential Transformer for Video Instance Segmentation

Wu¹,

Jiang²,

Bai³

et al. 2021

Preprint

View full text Add to dashboard Cite

In this work, we present SeqFormer, a frustratingly simple model for video instance segmentation. SeqFormer follows the principle of vision transformer that models instance relationships among video frames. Nevertheless, we observe that a stand-alone instance query suffices for capturing a time sequence of instances in a video, but attention mechanisms should be done with each frame independently. To achieve this, SeqFormer locates an instance in each frame and aggregates temporal information to learn a powerful representation of a video-level instance, which is used to predict the mask sequences on each frame dynamically. Instance tracking is achieved naturally without tracking branches or post-processing. On the YouTube-VIS dataset, SeqFormer achieves 47.4 AP with a ResNet-50 backbone and 49.0 AP with a ResNet-101 backbone without bells and whistles. Such achievement significantly exceeds the previous state-of-the-art performance by 4.6 and 4.4, respectively. In addition, integrated with the recentlyproposed Swin transformer, SeqFormer achieves a much higher AP of 59.3. We hope SeqFormer could be a strong baseline that fosters future research in video instance segmentation, and in the meantime, advances this field with a more robust, accurate, neat model. The code and the pre-trained models are publicly available at https:// github.com/wjf5203/SeqFormer.

show abstract

Section: Introductionsupporting

confidence: 89%

SeqFormer: Sequential Transformer for Video Instance Segmentation

Wu¹,

Jiang²,

Bai³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Due to the local nature of 2D and 3D convolutions, most of these models typically operate on short video clips of a few seconds. Inspired by the success of Transformer models in natural language processing (NLP), recently the transformerbased models have been successfully used for video recognition tasks [5,2,35,15,38]. However, due to the quadratic cost of the self-attention operation, these models are very computationally costly and, thus, only applied to short-range video segments.…”

Section: Related Workmentioning

confidence: 99%

“…The majority of modern video recognition models [8,16,41,17,46,25,5,2,35,15,38] are unfortunately not equipped to solve these tasks as they are designed for short-range videos (e.g., 5-10 seconds in duration). Furthermore, extending these models to the long-range video setting by simply adding more input video frames is impractical due to excessive computational cost and GPU memory consumption.…”

Section: Introductionmentioning

confidence: 99%

Long Movie Clip Classification with State-Space Video Models

Islam¹,

Bertasius²

2022

Preprint

View full text Add to dashboard Cite

Most modern video recognition models are designed to operate on short video clips (e.g., 5-10s in length). Because of this, it is challenging to apply such models to long movie understanding tasks, which typically require sophisticated long-range temporal reasoning capabilities. The recently introduced video transformers partially address this issue by using long-range temporal self-attention. However, due to the quadratic cost of self-attention, such models are often costly and impractical to use. Instead, we propose ViS4mer, an efficient long-range video model that combines the strengths of self-attention and the recently introduced structured state-space sequence (S4) layer. Our model uses a standard Transformer encoder for short-range spatiotemporal feature extraction, and a multi-scale temporal S4 decoder for subsequent long-range temporal reasoning. By progressively reducing the spatiotemporal feature resolution and channel dimension at each decoder layer, ViS4mer learns complex long-range spatiotemporal dependencies in a video. Furthermore, ViS4mer is 2.63× faster and requires 8× less GPU memory than the corresponding pure self-attention-based model. Additionally, ViS4mer achieves state-of-the-art results in 7 out of 9 long-form movie video classification tasks on the LVU benchmark. Furthermore, we also show that our approach successfully generalizes to other domains, achieving competitive results on the Breakfast and the COIN procedural activity datasets. The code will be made publicly available. 1 Recently, several Transformer models [6,50] have been shown to perform well on long-range video understanding tasks. However, due to the quadratic cost of standard self-attention, these models are 1 https://github.com/md-mohaiminul/ViS4merPreprint. Under review.

show abstract

“…Recently, transformers are applied to model spatiotemporal dependencies for video recognition [1,3,4,13,38,40,41,62,63] by virtue of their great power in capturing long-range dependencies [9,12,36,48]. With pretraining on a large-scale image dataset, video transformers achieve the best reported accuracy on video benchmarks [1,3,38], such as Kinetics-400/600 [28].…”

Section: Transformer-based Video Recognitionmentioning

confidence: 99%

DualFormer: Local-Global Stratified Transformer for Efficient Video Recognition

Liang¹,

Zhou²,

Zimmermann³

et al. 2021

Preprint

View full text Add to dashboard Cite

While transformers have shown great potential on video recognition tasks with their strong capability of capturing long-range dependencies, they often suffer high computational costs induced by self-attention operation on the huge number of 3D tokens in a video. In this paper, we propose a new transformer architecture, termed DualFormer, which can effectively and efficiently perform space-time attention for video recognition. Specifically, our DualFormer stratifies the full space-time attention into dual cascaded levels, i.e., to first learn fine-grained local space-time interactions among nearby 3D tokens, followed by the capture of coarsegrained global dependencies between the query token and the coarse-grained global pyramid contexts. Different from existing methods that apply space-time factorization or restrict attention computations within local windows for improving efficiency, our local-global stratified strategy can well capture both short-and long-range spatiotemporal dependencies, and meanwhile greatly reduces the number of keys and values in attention computation to boost efficiency. Experimental results show the superiority of DualFormer on five video benchmarks against existing methods. In particular, DualFormer sets new state-of-the-art 82.9%/85.2% top-1 accuracy on Kinetics-400/600 with ∼1000G inference FLOPs which is at least 3.2× fewer than existing methods with similar performances.

show abstract

Keeping Your Eye on the Ball: Trajectory Attention in Video Transformers

Cited by 5 publications

References 37 publications

SeqFormer: Sequential Transformer for Video Instance Segmentation

SeqFormer: Sequential Transformer for Video Instance Segmentation

Long Movie Clip Classification with State-Space Video Models

DualFormer: Local-Global Stratified Transformer for Efficient Video Recognition

Contact Info

Product

Resources

About