Attentional Fused Temporal Transformation Network for Video Action Recognition

Abstract

Effective spatiotemporal feature representation is crucial to the video-based action recognition task. Focusing on discriminate spatiotemporal feature learning, we propose Information Fused Temporal Transformation Network (IF-TTN) for action recognition on top of popular Temporal Segment Network (TSN) framework. In the network, Information Fusion Module (IFM) is designed to fuse the appearance and motion features at multiple ConvNet levels for each video snippet, forming a short-term video descriptor. With fused features as inputs, Temporal Transformation Networks (TTN) are employed to model middle-term temporal transformation between the neighboring snippets following a sequential order. As TSN itself depicts longterm temporal structure by segmental consensus, the proposed network comprehensively considers multiple granularity temporal features. Our IF-TTN achieves the stateof-the-art results on two most popular action recognition datasets: UCF101 and HMDB51. Empirical investigation reveals that our architecture is robust to the input motion map quality. Replacing optical flow with the motion vectors from compressed video stream, the performance is still comparable to the flow-based methods while the testing speed is 10x faster.

Using Artificial Intelligence for Assistance Systems to Bring Motor Learning Principles into Real World Motor Tasks

“…More information, e.g., sound can be added via new streams [ 166 , 167 , 168 ]. The architecture was further investigated by trying different ways of fusing the layers and deeper networks [ 169 , 170 , 171 , 172 ]. To facilitate the high computational costs of 3D convolutional layers, Lin et al [ 173 ] introduced the Temporal Shift Module (TSM) that can be incorporated into 2D CNNs to model the exchanges among neighboring frames while maintaining the lower computational costs of 2D CNNs.…”

An Improved Attention-Based Spatiotemporal-Stream Model for Action Recognition in Videos

“…In order to integrate spatial and temporal information comprehensively, we concatenate spatial featureF s , motion FeatureF t and the output F st of FCL. The fusion and concatenation process is depicted in (7).…”

“…In this paper, to validate the performance of the proposed iCBAM-based method, we compare our proposed method to recent popular and related approaches, including IDT [25], two-stream [3], TSN [6], C3D [28], two-stream + IDT [4], IF-TTN [7], DTPP [5] and attention-based models [32]- [34]. Since the proposed iCBAM-based spatiotemporal-stream network is trained from scratch, we do not compare it with those are pre-trained on the large dataset.…”

“…Zhu et al [5] suggest a Deep network with Temporal Pyramid Pooling (DTPP) to realize an end-to-end video-level representation learning approach. In order to realize the long-range temporal structure modeling, Wang et al [6] propose Temporal Segment Networks (TSN) to recognize video actions and on top of TSN, an Information Fused Temporal Transformation Network (IF-TTN) is reported to learn spatiotemporal feature representation [7]. Additionally, Zang et al [8] suggest utilizing attention-based temporal weighted CNN to learn action features.…”