Exploiting Image-trained CNN Architectures for Unconstrained Video Classification

Abstract

In this paper, we propose an efficient approach to exploit off-the-shelf image-trained CNN architectures for video classification and evaluate on the challenging TRECVID MED'14 dataset and UCF-101 dataset. Our work is closely related to other research efforts towards the efficient use of CNN for video classification. While it is now clear that CNN-based approaches outperform most state-of-the-art handcrafted features for image classification, it is not yet obvious that this holds true for video classification. Moreover, there seems to be mixed conclusions regarding the benefit of training a spatiotemporal vs. applying an image-trained CNN architecture on videos. Although the specificity of the considered video datasets might play a role, the way the 2D CNN architecture is exploited for video classification is certainly the main reason behind these contradictory observations. The additional computational cost of training on videos is also an element that should be taken into account when comparing the two options. Prior to training a spatiotemporal CNN architecture, it thus seems legitimate to fully exploit the potential of image-trained CNN architectures. Obtained on a highly heterogeneous video dataset, we believe that our results can serve as a strong 2D CNN baseline against which to compare CNN architectures specifically trained on videos.We conduct an in-depth exploration of different strategies for doing event detection in videos using convolutional neural networks (CNNs) trained for image classification (Figure 1, 2). We study different ways of performing spatial and temporal pooling, feature normalization, choice of CNN layers as well as choice of classifiers. Making judicious choices along these dimensions led to a very significant increase in performance over more naive approaches that have been used till now. The modality fusion of image-trained CNN features and motion-based Fisher vectors shows considerably improvement in classification performance. On TRECVID MED'14 dataset, our methods, based entirely on image-trained CNN features, can outperform several state-of-the-art non-CNN models. Our proposed late fusion of CNN-and motion-based features can further increase the mean average precision (mAP) on MED'14 from 34.95% to 38.

Dynamic Image Networks for Action Recognition

“…Considering deep learning methods, our method performs on par and is only outperformed from [33]. [33] makes use of the very deep VGGnet [24], which is a more competitive network than that the Alexnet architecture we rely on. Hence a direct comparison is not possible.…”

Using Spatial Pooler of Hierarchical Temporal Memory for object classification in noisy video streams

“…video classification in noisy video streams). Nevertheless, we examined the following papers : [15], [16], [17] which presents results of video classification using UCF-101 dataset. The best systems presented in those papers are based on various architectures of Convolutional Neural Networks (CNNs) and achieve accuracy of 80% and more.…”

Modeling and Analysis of a Deep Learning Pipeline for Cloud based Video Analytics

“…Zhongwen Xu et al [19] proposed discriminative CNN video representation to perform event detection from video dataset. Andrej Karpathy et al [6], Joe Yue-Hei Ng [8] and Shengxin Zha [20] used CNN architectures to perform video classification. They also retrained the top layers of their systems to study the generalization performance of their models and reported performance improvements from 88.6 percent to 88.0 percent.…”