Unsupervised Video Summarization via Attention-Driven Adversarial Learning

Abstract

This paper presents a new video summarization approach that integrates an attention mechanism to identify the significant parts of the video, and is trained unsupervisingly via generative adversarial learning. Starting from the SUM-GAN model, we first develop an improved version of it (called SUM-GAN-sl) that has a significantly reduced number of learned parameters, performs incremental training of the model's components, and applies a stepwise label-based strategy for updating the adversarial part. Subsequently, we introduce an attention mechanism to SUM-GAN-sl in two ways: i) by integrating an attention layer within the variational auto-encoder (VAE) of the architecture (SUM-GAN-VAAE), and ii) by replacing the VAE with a deterministic attention auto-encoder (SUM-GAN-AAE). Experimental evaluation on two datasets (SumMe and TVSum) documents the contribution of the attention auto-encoder to faster and more stable training of the model, resulting in a significant performance improvement with respect to the original model and demonstrating the competitiveness of the proposed SUM-GAN-AAE against the state of the art.

AC-SUM-GAN: Connecting Actor-Critic and Generative Adversarial Networks for Unsupervised Video Summarization

“…Most of them utilize Long Short-Term Memory (LSTM) units [3] to learn how to assess the importance of each video frame. However, experimentation with some of these methods (dppLSTM [4], DR-DSN [5], SUM-GAN-sl [6], SUM-GAN-AAE [7]) resulted in findings that are consistent with the claims in [8] about the low variation of the computed frame-level importance scores by LSTMs. As a consequence, the selections made by the trained LSTM seem to have a limited impact in summarization; the latter is mainly affected by factors such as the video fragmentation, or the approach used for fragment selection given a target summary length (such as the Knapsack algorithm).…”

Performance over Random

“…The utilized evaluation protocols in these methods targeted the assessment of the created keyframe-based summaries. A typical approach, applied in [8], involved independent users that assess both the relevance of each individual keyframe (using a [1][2][3][4][5] scale) and the quality of the entire summary w.r.t. redundant or missing information.…”

“…Our aim is to assess the representativeness of results when the evaluation is based on a small set of randomly-created splits and the reliability of performance comparisons that use different data splits for each algorithm. In this context we evaluate five publicly-available video summarization algorithms (two supervised: dppLSTM [41], VASNet [14]; and three unsupervised: DR-DSN [46], SUM-GANsl [3], SUM-GAN-AAE [2]) using the established protocol and a fixed set of 5 randomly-generated data splits of the SumMe and TVSum datasets (that simulates the evaluation conditions of most SoA works). These methods are, to our knowledge, the only ones Table 1: Comparison (F-Score (%)) of five publicly-available video summarization approaches in SumMe and TVSum datasets, using 5 and 50 randomly-generated splits.…”

“…To quantify the degree of correlation between each method's ( [2,3,14,41,46]) performance and the performance of the random and the average human summarizer, we report the "Covariance" and the "Pearson correlation coefficient". The results presented in Table 2 show that, according to both measures and in both datasets, the performance correlation among the tested methods and the random summarizer is stronger than the correlation with the average human summarizer.…”

“…In the following we review the relevant literature, focusing on the utilized evaluation approaches (Section 2). Then, we perform a large-scale experiment using the SumMe and TVSum datasets and the publicly-available software implementations of five summarization methods ( [2,3,14,41,46]), to assess the established evaluation protocol according to the aspects discussed above. This study indicates limitations of this protocol w.r.t.…”