Learning Spatio-Temporal Representation with Pseudo-3D Residual Networks

Abstract

Convolutional Neural Networks (CNN) have been regarded as a powerful class of models for image recognition problems. Nevertheless, it is not trivial when utilizing a CNN for learning spatio-temporal video representation. A few studies have shown that performing 3D convolutions is a rewarding approach to capture both spatial and temporal dimensions in videos. However, the development of a very deep 3D CNN from scratch results in expensive computational cost and memory demand. A valid question is why not recycle off-the-shelf 2D networks for a 3D CNN. In this paper, we devise multiple variants of bottleneck building blocks in a residual learning framework by simulating 3 × 3 × 3 convolutions with 1 × 3 × 3 convolutional filters on spatial domain (equivalent to 2D CNN) plus 3 × 1 × 1 convolutions to construct temporal connections on adjacent feature maps in time. Furthermore, we propose a new architecture, named Pseudo-3D Residual Net (P3D ResNet), that exploits all the variants of blocks but composes each in different placement of ResNet, following the philosophy that enhancing structural diversity with going deep could improve the power of neural networks. Our P3D ResNet achieves clear improvements on Sports-1M video classification dataset against 3D CNN and frame-based 2D CNN by 5.3% and 1.8%, respectively. We further examine the generalization performance of video representation produced by our pre-trained P3D ResNet on five different benchmarks and three different tasks, demonstrating superior performances over several state-of-the-art techniques.

A Spatio-temporal Hybrid Network for Action Recognition

“…However, 3D convolution increases computational complexity substantially and greatly affects inference efficiency. Some works [21,37,29] factorize the spatial-temporal 3D convolution into one 2D spatial convolution and one 1D temporal convolution, and than ensemble them in a sequential or parallel manner. However, the spatial and temporal information are modeled independently.…”

“…3D-Conv [13,27,2] directly applies 3D convolutional kernels (e.g., 3 × 3 × 3), resulting into a heavy computational cost. (2+1)D-Conv [41,39,29,37,21] decomposes the 3D kernels into 2D spatial kernels and 1D temporal kernels, which are sequentially stacked. Different from the above convolutions which regard all input channels as a whole, STH constructs spatio-temporal hybrid kernels by mixing the basic spatial and temporal kernels along the input channels in one convolutional layer, resulting in deeper integration of spatio-temporal information in one layer.…”

“…As illustrated above, 2D CNNs lack temporal modeling capability while 3D CNNs are computationally intensive. Some researchers attempt to reach a com-promise between performance and complexity by mixing 2D and 3D CNNs (Mixed 2D&3D CNNs) [41,39] or decomposing 3D convolutions into 2D spatial convolutions followed by 1D temporal convolutions ((2+1)D CNNs) [37,21,29]. However, among these approaches, the mixed 2D&3D CNNs reduce computational complexity at the expense of restricting temporal modeling capability, while the spatio-temporal separation convolutions learn spatial and temporal information using two sequentially-stacked layers.…”

A convolutional neural network algorithm for automatic segmentation of head and neck organs at risk using deep lifelong learning

“…Recently, 3D CNNs have shown promise in 3D shape recognition, 3D object detection, CAD classification, hand gesture recognition, and diagnostic imaging predictions . 3D CNNs can reach into 3D space and aggregate authentically three‐dimensional information about a structure.…”

“…Recently, 3D CNNs have shown promise in 3D shape recognition, 3D object detection, CAD classification, hand gesture recognition, and diagnostic imaging predictions. 16,[25][26][27][28][29] 3D CNNs can reach into 3D space and aggregate authentically three-dimensional information about a structure. 3D CNNs have the ability to train on a richer and more holistic learning environment, but are much more memory-intensive than 2D models.…”