TSM: Temporal Shift Module for Efficient Video Understanding

Abstract

The explosive growth in video streaming gives rise to challenges on performing video understanding at high accuracy and low computation cost. Conventional 2D CNNs are computationally cheap but cannot capture temporal relationships; 3D CNN based methods can achieve good performance but are computationally intensive, making it expensive to deploy. In this paper, we propose a generic and effective Temporal Shift Module (TSM) that enjoys both high efficiency and high performance. Specifically, it can achieve the performance of 3D CNN but maintain 2D CNN's complexity. TSM shifts part of the channels along the temporal dimension; thus facilitate information exchanged among neighboring frames. It can be inserted into 2D CNNs to achieve temporal modeling at zero computation and zero parameters. We also extended TSM to online setting, which enables real-time low-latency online video recognition and video object detection. TSM is accurate and efficient: it ranks the first place on the Something-Something leaderboard upon publication; on Jetson Nano and Galaxy Note8, it achieves a low latency of 13ms and 35ms for online video recognition. The code is available at: https://github. com/mit-han-lab/temporal-shift-module.

A Spatio-temporal Hybrid Network for Action Recognition

“…As shown in Table 2 Other SOTA Methods. We also compare our method with other recentlyproposed approaches, including TSM [20], STM [14], and ABM [40]. As Table 2 shows, STH with 8-frame input already outperforms ABM with (16 × 3)frame input.…”

“…TRN [38] can learn temporal reasoning relationship from the features of the last layer, but its performance is still inferior to ours. The recently-proposed TSM [20] method performs better than other 2D based methods including TRN [38] and MFNet [18] as it has stronger temporal modeling ability across all levels. Compared to TSM, our proposed STH network achieves new state-of-the-art performance with 46.8% top-1 accuracy at T = 8 and 48.3% top-1 accuracy at T = 16, with even lower computational complexity.…”

“…Compared to the 2D-Conv models including TSN [33] and TRN [38], our approach shows clear performance improvement on both datasets. For Something-Something V2 dataset, our model outperforms the current state-ofthe-art efficient method TSM [20] with fewer FLOPs and parameters, using the same sampling strategy. In addition, our method sampling with only 8 frames performs better than the latest model ABM [40].…”

“…However, it sacrifices the temporal modeling capability at some 2D layers. The latest method TSM (Temporal Shift Module) [20] shift part of the channels along the temporal dimension for efficient temporal modeling. STM (Spatio-Temporal and Motion Encoding) [14] designs two module, specifically, a channel-wise spatio-temporal module to encode spatiotemporal feature and a channel-wise motion module to learn motion feature.…”

“…Generally speaking, spatial information focus on static appearance features such as actors and objects from a video, while temporal information can be regarded as an indicator for recognizing motion and action. State-of-the-art approaches leverage both spatial and temporal information to enhance the performance of action recognition [22,20,2,29,6,5]. Temporal modeling is of key importance for recognizing actions.…”