Multi-View Surveillance Video Summarization via Joint Embedding and Sparse Optimization

Abstract

Most traditional video summarization methods are designed to generate effective summaries for single-view videos, and thus they cannot fully exploit the complicated intra and inter-view correlations in summarizing multi-view videos in a camera network. In this paper, with the aim of summarizing multi-view videos, we introduce a novel unsupervised framework via joint embedding and sparse representative selection. The objective function is two-fold. The first is to capture the multiview correlations via an embedding, which helps in extracting a diverse set of representatives. The second is to use a 2,1norm to model the sparsity while selecting representative shots for the summary. We propose to jointly optimize both of the objectives, such that embedding can not only characterize the correlations, but also indicate the requirements of sparse representative selection. We present an efficient alternating algorithm based on half-quadratic minimization to solve the proposed non-smooth and non-convex objective with convergence analysis.A key advantage of the proposed approach with respect to the state-of-the-art is that it can summarize multi-view videos without assuming any prior correspondences/alignment between them, e.g., uncalibrated camera networks. Rigorous experiments on several multi-view datasets demonstrate that our approach clearly outperforms the state-of-the-art methods.

Cloud-Assisted Multiview Video Summarization Using CNN and Bidirectional LSTM

“…We also compared our framework with other approaches based on MVS. A gap in the value of precision can be observed in Table IV where our system is lagging behind from the state-of-the-art [25] due to the presence of some redundant frames in the final summary.…”

“…For instance, [18] performs activity based shot segmentation while an another approach presented in [27] detects shots boundaries based on the motion in video data. Panda et al [24,25] utilizes Spatiotemporal C3D features for video representation in shots. Similarly, Muhammad et al [9] exploits deep features for shots segmentation.…”

“…Similarly, it can be observed from Table IV that we compared our framework with seven existing approaches. The most recent candidate paper for comparison is published in 2017 and we compared our results with their method [25]. We also compared our framework with other approaches based on MVS.…”

“…The proposed approach outruns all the existing methods and F1score is improved 0.1 compared to [25], which is considered as the best among all MVS approaches. The last column indicates the event recall value which is increased by 1.2 from [43].…”

“…They finally used sparse representative selection over learned embedding for summary generation. In another MVS research [25], inter-and intra-view correlations are computed via sparse coefficients. It contains a coefficient matrix, which represents relative importance of segmented shots.…”