Combining Very Deep Convolutional Neural Networks and Recurrent Neural Networks for Video Classification

A novel keyframe extraction method for video classification using deep neural networks

“…We used the pre-trained neural network VGG-16 to generalize the pre-learnt feature representations using transfer learning. In our previous work [18], ConvLSTM and LSTM with local features extracted using VGG-16 outperformed those using global features. Thus, this paper uses the ConvLSTM(1) and LSTM(1) architectures used in [18].…”

“…In our previous work [18], ConvLSTM and LSTM with local features extracted using VGG-16 outperformed those using global features. Thus, this paper uses the ConvLSTM(1) and LSTM(1) architectures used in [18]. Apart from the newly proposed keyframe extraction method, we also conducted the experiments using not only 20 but also 101 categories of the UCF-101 dataset and evaluating the proposed methods on the KTH action recognition dataset as well.…”

“…It is noteworthy that some related work on deep neural networks for video classification has been presented in our previous work [18].…”

“…The UCF-101 dataset has defined three training-testing splits, aiming to facilitate benchmarking algorithms. In our previous experiment [18], only the first 20 categories of the dataset were used due to limited time and computing facility, and the first training-testing split was adopted to generate training and testing data. However, we conducted the experiments in this paper using all categories with the three training-testing splits of the UCF-101 and the KTH datasets.…”

“…The main contributions of this work are the identification of the best architecture for combining CNNs and RNNs for video classification and the proposal of the action template-based keyframe extraction, which aims to extract more informative frames by calculating the similarity only between action regions, rather than whole frames. The former was partly presented in [18], which has been substantially extended and serves as a baseline to test the newly proposed method. In this paper, extensive experiments have shown that the action template-based keyframe extraction method significantly outperformed the frame selection methods used in our experiments for comparison purposes.…”