RPCA-KFE: Key Frame Extraction for Video Using Robust Principal Component Analysis

Abstract

Key frame extraction algorithms consider the problem of selecting a subset of the most informative frames from a video to summarize its content. Several applications, such as video summarization, search, indexing, and prints from video, can benefit from extracted key frames of the video under consideration. Most approaches in this class of algorithms work directly with the input video data set, without considering the underlying low-rank structure of the data set. Other algorithms exploit the low-rank component only, ignoring the other key information in the video. In this paper, a novel key frame extraction framework based on robust principal component analysis (RPCA) is proposed. Furthermore, we target the challenging application of extracting key frames from unstructured consumer videos. The proposed framework is motivated by the observation that the RPCA decomposes an input data into: 1) a low-rank component that reveals the systematic information across the elements of the data set and 2) a set of sparse components each of which containing distinct information about each element in the same data set. The two information types are combined into a single l1-norm-based non-convex optimization problem to extract the desired number of key frames. Moreover, we develop a novel iterative algorithm to solve this optimization problem. The proposed RPCA-based framework does not require shot(s) detection, segmentation, or semantic understanding of the underlying video. Finally, experiments are performed on a variety of consumer and other types of videos. A comparison of the results obtained by our method with the ground truth and with related state-of-the-art algorithms clearly illustrates the viability of the proposed RPCA-based framework.

On the Applications of Robust PCA in Image and Video Processing

“…This application is the most investigated one. Indeed, numerous authors used RPCA problem formulations in applications such as background/foreground separation [4], [208], [223], background initialization [255], [258], moving target detection [241], motion saliency detection [47], [300], [332], motion estimation [238], visual object tracking [168] [276], action recognition [126], key frame extraction [60], video object segmentation [130], [153], [197], [317], [319], video coding [45], [46], [110], [331], video restoration and denoising [142], [334], [109], [318], [176], video inpainting [142], hyperspectral video processing [96], [42], and video stabilization [68].…”

“…Other methods exploit the low-rank component only but they ignored the other key information in the video. On the other hand, Dang et al [60] developed a Key Frame Extraction (KFE) algorithm based on RPCA which decomposes the input video data into a low-rank component which reveals the information across the elements of the dataset, and a set of sparse components each of which containing distinct information about each element. Then, Dang et al [60] combined the two information types into a single 1 -norm based non-convex optimization problem to extract the desired number of key frames.…”

“…On the other hand, Dang et al [60] developed a Key Frame Extraction (KFE) algorithm based on RPCA which decomposes the input video data into a low-rank component which reveals the information across the elements of the dataset, and a set of sparse components each of which containing distinct information about each element. Then, Dang et al [60] combined the two information types into a single 1 -norm based non-convex optimization problem to extract the desired number of key frames. Extensive experiments on a variety of consumer and other types of videos show that RPCA-KFE with the ground truth and with related state-of-the-art algorithms clearly illustrates its viability.…”

Algorithms for Stable PCA

“…• Video processing: Numerous authors used the RPCA framework in applications such as action recognition [71], motion estimation [149], motion saliency detection [191], video coding [214] [56] [27] [28], key frame extraction [32], hyperspectral video processing [44], video restoration [78], and in background and foreground separation [3] [124] [128].…”

Low-Rank Decomposition and Total Variation Regularization of Hyperspectral Video Sequences

“…In addition, the total variation (TV) regularization for video can explicitly describe the gas plume continuity in both the spatial and temporal directions. In traditional moving object detection methods [23][24][25][26][27][28], one column of the matrix is the vectorized image of a frame, hence only one 2-D matrix is required to represent the video. However, in HVS, there are hundreds of bands.…”