Deep Manifold Learning Combined With Convolutional Neural Networks for Action Recognition

Abstract

Learning deep representations have been applied in action recognition widely. However, there have been a few investigations on how to utilize the structural manifold information among different action videos to enhance the recognition accuracy and efficiency. In this paper, we propose to incorporate the manifold of training samples into deep learning, which is defined as deep manifold learning (DML). The proposed DML framework can be adapted to most existing deep networks to learn more discriminative features for action recognition. When applied to a convolutional neural network, DML embeds the previous convolutional layer's manifold into the next convolutional layer; thus, the discriminative capacity of the next layer can be promoted. We also apply the DML on a restricted Boltzmann machine, which can alleviate the overfitting problem. Experimental results on four standard action databases (i.e., UCF101, HMDB51, KTH, and UCF sports) show that the proposed method outperforms the state-of-the-art methods.

Deep Manifold Structure Transfer for Action Recognition

“…Action recognition has attracted much attention in the past decade [22], [29]. The targets of action recognition evolute from scimple background to real-world video sequences, while the recognition methods shift from hand-crafted to learning based.…”

“…Action classification in video had been one of the most challenging problems next to the image classification [10]. Recent deep learning approaches including 3D CNN [11], two-stream CNNs [2], C3D [12], TDD [13], TSN [14], ST-ResNet+iDT [15], L 2 STM [16], ST-VLMPF [17], P3D ResNet [18], I3D [19], 3D ResNeXt [20], R(2+1)D-TwoStream [7], CO2FI+ASYN [21], and DML [22] have shown state-ofthe-art performances in action recognition. The recent development of CNNs with spatio-temporal 3D convolutional kernels (3D CNNs) rapidly grows and contributes to significant advances in video recognition [7], [18]- [20] because 3D CNNs can be used to directly extract spatio-temporal features from raw videos.…”

“…The recent development of CNNs with spatio-temporal 3D convolutional kernels (3D CNNs) rapidly grows and contributes to significant advances in video recognition [7], [18]- [20] because 3D CNNs can be used to directly extract spatio-temporal features from raw videos. However, most of these approaches aim at distinguishing the inter-class variability, but often ignore the intra-class distribution [23], and therefore could suffer from the over-fitting problem [22], particularly when the video datasets are relatively small compared to immense number of parameters in 3D CNNs [20] and intra-class variations in the training data are significant [22].…”

“…To alleviate the over-fitting problem, regularization techniques [8], [22]- [24] and prior knowledge, e.g., 2D topological structure of input data [25], graph-based embedding [26], are explored in deep feature learning. Nonlinear structures, e.g., Riemanian manifold [27], Grassmann manifold [28], [29] have been incorporated as constraints to balance the learned model [30], [31].…”

Toward Compact ConvNets via Structure-Sparsity Regularized Filter Pruning

“…It has resulted in structured sparse patterns 2 that can accelerate the online inference. However, such a magnitude-based measurement (e.g., 1 -norm) is too simple and inefficient to determine the importance of each filter, due to the existence of nonlinear activation functions (e.g., rectifier linear unit (ReLU) [40]) and other complex operations (e.g., pooling and batch normalization [41]). To explain, filters with small 1norm values may have large responses in the output.…”