Motion Feature Network: Fixed Motion Filter for Action Recognition

Abstract

Spatio-temporal representations in frame sequences play an important role in the task of action recognition. Previously, a method of using optical flow as a temporal information in combination with a set of RGB images that contain spatial information has shown great performance enhancement in the action recognition tasks. However, it has an expensive computational cost and requires two-stream (RGB and optical flow) framework. In this paper, we propose MFNet (Motion Feature Network) containing motion blocks which make it possible to encode spatiotemporal information between adjacent frames in a unified network that can be trained end-to-end. The motion block can be attached to any existing CNN-based action recognition frameworks with only a small additional cost. We evaluated our network on two of the action recognition datasets (Jester and Something-Something) and achieved competitive performances for both datasets by training the networks from scratch.

Multi-Agent Reinforcement Learning Based Frame Sampling for Effective Untrimmed Video Recognition

“…For example, TLE [7], ShuttleNet [35], AttentionClusters [29] and NetVlad [1,16] are proposed for better local feature integration instead of directly AVG-Pooling as used in TSN. OFF [37] and motion feature network [26] are proposed for integrating motion information modeling into a spatial CNN network, instead of using two streams. I3D [3] inflates deeper networks than C3D for spatial-temporal modeling.…”

Representation Flow for Action Recognition

“…Table 8 shows the results. MFNet [15] captures motion by spatially shifting CNN feature maps, then summing the results, TVNet [5] applies a convolutional optical flow method to RGB inputs, and ActionFlowNet [15] 52.5 56.8 TVNet [5] 39.4 57.5 RGB-OFF [21] 55.6 56.9 Ours 61.1 65.4 [16] trains a CNN to jointly predict optical flow and activity classes. We also compare to OFF [21] using only RGB inputs.…”

STM: SpatioTemporal and Motion Encoding for Action Recognition

“…The upper part presents the 3D CNN based methods, including S3D-G [37], ECO [42] and I3D+GCN models [35]. The lower part is 2D CNN based methods, including TRN [40], MFNet [18] and TSM [19]. It is clear that even STM with 8 RGB frames Table 4.…”

“…However, calculating optical flow with TV-L1 method [38] is expensive in both time and space. Recently many approaches have been proposed to estimate optical flow with CNN [5,14,6,21] or explored alternatives of optical flow [33,39,26,18]. TSN frameworks [33] involved RGB difference between two frames to represent motion in videos.…”

“…Optical Flow guided Feature (OFF) [26] contains a set of operators including sobel and element-wise subtraction for OFF generation. MFNet [18] adopted five fixed motion filters as a motion block to find feature-level temporal features between two adjacent time steps. Our proposed CMM branch is also designed for finding better yet lightweight alternative motion representation.…”