Learning Spatiotemporal Features with 3D Convolutional Networks

Abstract

We propose a simple, yet effective approach for spatiotemporal feature learning using deep 3-dimensional convolutional networks (3D ConvNets) trained on a large scale supervised video dataset. Our findings are three-fold: 1) 3D ConvNets are more suitable for spatiotemporal feature learning compared to 2D ConvNets; 2) A homogeneous architecture with small 3 × 3 × 3 convolution kernels in all layers is among the best performing architectures for 3D ConvNets; and 3) Our learned features, namely C3D (Convolutional 3D), with a simple linear classifier outperform state-of-the-art methods on 4 different benchmarks and are comparable with current best methods on the other 2 benchmarks. In addition, the features are compact: achieving 52.8% accuracy on UCF101 dataset with only 10 dimensions and also very efficient to compute due to the fast inference of ConvNets. Finally, they are conceptually very simple and easy to train and use.

Detecting and Removing Visual Distractors for Video Aesthetic Enhancement

“…By jointly encoding spatio-temporal information in the learning process, 3D convolutional networks [50] have achieved good performance in semantic video short classification. Other works use 2D CNN structures to perform recognition and detection tasks in video by fine-tuning the networks using video frames [51], or by combining video frames and optical flow maps as the input layer [42] [52].…”

“…Tran et al [50] proposed a 3D convolutional network for video classification. However, this structure does not produce the pixel-level labeling that is required in our task.…”

DAPs: Deep Action Proposals for Action Understanding

“…Here, we adopt the recent approach of Xu et al [38], which encodes features learned by a conv-net model using VLAD. Here, we use the activations from the fc7 layer from a 3D conv-net [34] as our features. We first learn a codebook using k-means with k = 256.…”

“…Our network integrates the following modules: Visual encoder: It encodes a small video volume into a meaningful low dimensional feature vector. In practice, we use activations from the top layer of a 3D convolutional network trained for action classification (C3D network [34]). Sequence encoder: It encodes the sequence of visual codes as a discriminative sequence of hidden states.…”

“…More details about the optimization problem are provided in the supplementary material. Implementation Details: for our visual encoder, we use the publicly available pre-trained C3D model [34] which has a temporal resolution of 16 frames. To shorten the training time of our implementation, we reduce the dimensionality of the activations from the second fully-connected layer (fc7 ) of our visual encoder from 4096 to 500 dimensions using PCA.…”