Long Movie Clip Classification with State-Space Video Models

Abstract

Most modern video recognition models are designed to operate on short video clips (e.g., 5-10s in length). Because of this, it is challenging to apply such models to long movie understanding tasks, which typically require sophisticated long-range temporal reasoning capabilities. The recently introduced video transformers partially address this issue by using long-range temporal self-attention. However, due to the quadratic cost of self-attention, such models are often costly and impractical to use. Instead, we propose ViS4mer, an efficient long-range video model that combines the strengths of self-attention and the recently introduced structured state-space sequence (S4) layer. Our model uses a standard Transformer encoder for short-range spatiotemporal feature extraction, and a multi-scale temporal S4 decoder for subsequent long-range temporal reasoning. By progressively reducing the spatiotemporal feature resolution and channel dimension at each decoder layer, ViS4mer learns complex long-range spatiotemporal dependencies in a video. Furthermore, ViS4mer is 2.63× faster and requires 8× less GPU memory than the corresponding pure self-attention-based model. Additionally, ViS4mer achieves state-of-the-art results in 7 out of 9 long-form movie video classification tasks on the LVU benchmark. Furthermore, we also show that our approach successfully generalizes to other domains, achieving competitive results on the Breakfast and the COIN procedural activity datasets. The code will be made publicly available. 1 Recently, several Transformer models [6,50] have been shown to perform well on long-range video understanding tasks. However, due to the quadratic cost of standard self-attention, these models are 1 https://github.com/md-mohaiminul/ViS4merPreprint. Under review.

Simple Hardware-Efficient Long Convolutions for Sequence Modeling

“…Recently, a new class of sequence models based on state space models (SSMs) [30,34,37,46] has emerged as a powerful general-purpose sequence modeling framework. SSMs scale nearly linearly in sequence length and have shown state-of-the-art performance on a range of sequence modeling tasks, from long range modeling [68] to language modeling [17,50], computer vision [39,53], and medical analysis [70].…”

Incorporating Domain Knowledge Graph into Multimodal Movie Genre Classification with Self-Supervised Attention and Contrastive Learning

“…Movie genre classification is a fundamental task for certain downstream tasks such as movie recommendation [12], understanding [25], editing [9], description [39], etc. Previous studies [16,53] have achieved unparalleled results in movie genre classification with a single modality such as posters, plot summaries, movie trailers, audio, or metadata.…”