STSM: Spatio-Temporal Shift Module for Efficient Action Recognition

Abstract

The modeling, computational complexity, and accuracy of spatio-temporal models are the three major foci in the field of video action recognition. The traditional 2D convolution has low computational complexity, but it cannot capture the temporal relationships. Although the 3D convolution can obtain good performance, it is with both high computational complexity and a large number of parameters. In this paper, we propose a plug-and-play Spatio-Temporal Shift Module (STSM), which is a both effective and high-performance module. STSM can be easily inserted into other networks to increase or enhance the ability of the network to learn spatio-temporal features, effectively improving performance without increasing the number of parameters and computational complexity. In particular, when 2D CNNs and STSM are integrated, the new network may learn spatio-temporal features and outperform networks based on 3D convolutions. We revisit the shift operation from the perspective of matrix algebra, i.e., the spatio-temporal shift operation is a convolution operation with a sparse convolution kernel. Furthermore, we extensively evaluate the proposed module on Kinetics-400 and Something-Something V2 datasets. The experimental results show the effectiveness of the proposed STSM, and the proposed action recognition networks may also achieve state-of-the-art results on the two action recognition benchmarks.

Temporal superimposed crossover module for effective continuous sign language

“…Lin et al [27] proposed a time shift module for hardware efficient video recognition, which moves part of the channel along the time dimension to exchange information with adjacent frames. Yang et al [28] proposed a spatial-temporal displacement module for efficient video recognition. This module moves some channels in the time dimension and space dimension of different channels, enabling the network to learn its spatial-temporal characteristics.…”

Efficient Aggressive Behavior Recognition of Pigs Based on Temporal Shift Module

“…This module can be inserted in existing 2D CNNs to achieve time modeling of zero computation and zero parameters. Although TSM was widely used in tasks such as video classification and action recognition [ 22 , 23 ], no researcher applied TSM to the field of pig farming. In this study, we aimed to insert TSM into four widely used 2D CNN models, which enhance the model’s learning ability on time features, while maintaining the model’s performance in handling spatial features.…”