Move Forward and Tell: A Progressive Generator of Video Descriptions

Abstract

We present an efficient framework that can generate a coherent paragraph to describe a given video. Previous works on video captioning usually focus on video clips. They typically treat an entire video as a whole and generate the caption conditioned on a single embedding. On the contrary, we consider videos with rich temporal structures and aim to generate paragraph descriptions that can preserve the story flow while being coherent and concise. Towards this goal, we propose a new approach, which produces a descriptive paragraph by assembling temporally localized descriptions. Given a video, it selects a sequence of distinctive clips and generates sentences thereon in a coherent manner. Particularly, the selection of clips and the production of sentences are done jointly and progressively driven by a recurrent network -what to describe next depends on what have been said before. Here, the recurrent network is learned via self-critical sequence training with both sentencelevel and paragraph-level rewards. On the ActivityNet Captions dataset, our method demonstrated the capability of generating high-quality paragraph descriptions for videos. Compared to those by other methods, the descriptions produced by our method are often more relevant, more coherent, and more concise.

Grounded Video Description

“…For a fair comparison, we use exactly the same frame-wise feature from this work for our temporal attention module. For video paragraph description, we compare our methods against the SotA method MFT [31] with the evaluation script provided by the authors [31]. For image captioning, we compare against two SotA methods, Neural Baby Talk (NBT) [16] and BUTD [1].…”

Streamlined Dense Video Captioning

“…Recall (@tIoU) Precision (@tIoU) @0.3 @0.5 @0.7 @0.9 Average @0.3 @0.5 @0.7 @0.9 Average MFT [30] 46. 18 respectively.…”

“…Proposal the results of MFT [30], which is originally proposed for video paragraph generation but its event selection module is also able to generate an event sequence from the candidate event proposals; it makes a choice between selecting each proposal for caption generation and skipping it, and constructs an event sequence implicitly. For MFT, we compare performances in both event detection and dense captioning.…”

“…Despite their impressive performances, they are limited to describing a video using a single sentence and can be applied only to a short video containing a single event. Thus, Yu et al [35] propose a hierarchical recurrent neural network to generate a paragraph for a long video, while Xiong et al [30] introduce a paragraph generation method based on event proposals, where an event selection module determines which proposals need to be utilized for caption generation in a progressive way. Contrary to these tasks, which simply generate a sentence or paragraph for an input video, dense video captioning requires localizing and describing events at the same time.…”

“…Understanding video contents is an important topic in computer vision. Through the introduction of large-scale datasets [9,31] and the recent advances of deep learning technology, research towards video content understanding is no longer limited to activity classification or detection and addresses more complex tasks including video caption generation [1,4,13,14,15,22,23,26,28,30,33,35,36].…”