A Stepwise, Label-based Approach for Improving the Adversarial Training in Unsupervised Video Summarization

Abstract

In this paper we present our work on improving the efficiency of adversarial training for unsupervised video summarization. Our starting point is the SUM-GAN model, which creates a representative summary based on the intuition that such a summary should make it possible to reconstruct a video that is indistinguishable from the original one. We build on a publicly available implementation of a variation of this model, that includes a linear compression layer to reduce the number of learned parameters and applies an incremental approach for training the different components of the architecture. After assessing the impact of these changes to the model's performance, we propose a stepwise, label-based learning process to improve the training efficiency of the adversarial part of the model. Before evaluating our model's efficiency, we perform a thorough study with respect to the used evaluation protocols and we examine the possible performance on two benchmarking datasets, namely SumMe and TVSum. Experimental evaluations and comparisons with the state of the art highlight the competitiveness of the proposed method. An ablation study indicates the benefit of each applied change on the model's performance, and points out the advantageous role of the introduced stepwise, labelbased training strategy on the learning efficiency of the adversarial part of the architecture.

Performance over Random

“…The utilized evaluation protocols in these methods targeted the assessment of the created keyframe-based summaries. A typical approach, applied in [8], involved independent users that assess both the relevance of each individual keyframe (using a [1][2][3][4][5] scale) and the quality of the entire summary w.r.t. redundant or missing information.…”

“…Our aim is to assess the representativeness of results when the evaluation is based on a small set of randomly-created splits and the reliability of performance comparisons that use different data splits for each algorithm. In this context we evaluate five publicly-available video summarization algorithms (two supervised: dppLSTM [41], VASNet [14]; and three unsupervised: DR-DSN [46], SUM-GANsl [3], SUM-GAN-AAE [2]) using the established protocol and a fixed set of 5 randomly-generated data splits of the SumMe and TVSum datasets (that simulates the evaluation conditions of most SoA works). These methods are, to our knowledge, the only ones Table 1: Comparison (F-Score (%)) of five publicly-available video summarization approaches in SumMe and TVSum datasets, using 5 and 50 randomly-generated splits.…”

“…In the following we review the relevant literature, focusing on the utilized evaluation approaches (Section 2). Then, we perform a large-scale experiment using the SumMe and TVSum datasets and the publicly-available software implementations of five summarization methods ( [2,3,14,41,46]), to assess the established evaluation protocol according to the aspects discussed above. This study indicates limitations of this protocol w.r.t.…”

Unsupervised Video Summarization via Attention-Driven Adversarial Learning

“…39.9 (−) 53.9 (−) Tessellation [14] 41.4 (−) 64.1 (+) DR-DSN [32] 41.4 (−) 57.6 (−) Online Motion-AE [29] 37.7 (−) 51.5 (−) UnpairedVSN [20] 47.5 (−) 55.6 (−) SUM-GAN-sl [1] 47.3 (−) 58.0 (−) SUM-GAN-VAAE 45.7 (−) 57.6 (−) SUM-GAN-AAE 48.9 58.3…”

“…we developed a new model (called SUM-GAN-sl and presented in[1]) Incremental training of SUM-GAN-sl. Adversarial learning follows a stepwise label-based approach.…”