Dilated temporal relational adversarial network for generic video summarization

Abstract

The large amount of videos popping up every day, make it more and more critical that key information within videos can be extracted and understood in a very short time. Video summarization, the task of finding the smallest subset of frames, which still conveys the whole story of a given video, is thus of great significance to improve efficiency of video understanding. We propose a novel Dilated Temporal Relational Generative Adversarial Network (DTR-GAN) to achieve frame-level video summarization. Given a video, it selects the set of key frames, which contain the most meaningful and compact information. Specifically, DTR-GAN learns a dilated temporal relational generator and a discriminator with three-player loss in an adversarial manner. A new dilated temporal relation (DTR) unit is introduced to enhance temporal representation capturing. The generator uses this unit to effectively exploit global multi-scale temporal context to select key frames and to complement the commonly used Bi-LSTM. To ensure that summaries capture enough key video representation from a global perspective rather than a trivial randomly shorten sequence, we present a discriminator that learns to enforce both the information completeness and compactness of summaries via a three-player loss. The loss includes the generated summary loss, the random summary loss, and the real summary (ground-truth) loss, 2 Yujia Zhang et al.which play important roles for better regularizing the learned model to obtain useful summaries. Comprehensive experiments on three public datasets show the effectiveness of the proposed approach.

Video Summarization by Learning From Unpaired Data

“…Supervised methods [8,10,11,19,24,31,32,38,39,40,41,42,47] learn video summarization from labeled data consisting of raw videos and their corresponding groundtruth summary videos. Supervised methods tend to outperform unsupervised ones, since they can learn useful cues from ground truth summaries that are hard to capture with hand-crafted heuristics.…”

Comprehensive Video Understanding: Video Summarization with Content-Based Video Recommender Design

“…A significant number of deep learning based frameworks have been explored recently in solving video summarization [3,22,24,26,28]. K. Zhang et al creatively applied LSTM in supervised video sequence labelling to model video temporal information with good performance [24].…”

“…K. Zhou et al showed that fully unsupervised learning can outperform many supervised methods by considering diversity and representativeness in reinforcement learning-based framework [28]. Y. Zhang et al introduced adversarial loss to video summarization which learns a dilated temporal relational generator and a discriminator with three-player loss [26].…”

“…Currently, as video summarization problem appeals to a lot of researchers, many state-of-the-art approaches have been presented in solving this problem [3,[24][25][26]. Generally, they treat video summarization as sequence-tosequence learning problem.…”

Goal-Driven Sequential Data Abstraction

“…Video/text summarization Existing models are either supervised or unsupervised. Unsupervised summarization models in video [9,31,40,41,43,50,52,54,55] and text [10,26,25,3] domains aim to identify a small subset of key units (video-segments/sentences) that preserve the global content of the input, e.g., using criteria like diversity and representativeness. In contrast, supervised video [13,15,16,42,49,51] and text [37,6,28,30,46] summarization methods solve the same problem by employing ground-truth summaries as training targets.…”