Learning Semantic Concepts and Temporal Alignment for Narrated Video Procedural Captioning

Shi, Botian; Ji, Lei; Niu, Zhendong; Duan, Nan; Zhou, Ming; Chen, Xilin

doi:10.1145/3394171.3413498

Cited by 19 publications

(14 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Comparing to CNN, the attention mechanism learns more global dependencies, therefore, transformer also shows great performance in low-level tasks [3]. Transformer has also been proved effectiveness in multi-modal area, including multi-modal representations [45] and applications [13,19,31]. Inspired by the extensive applications of transformer, we integrate the transformer encoder-decoder into the document image rectification problem.…”

Section: Transformer In Language and Visionmentioning

confidence: 99%

DocTr: Document Image Transformer for Geometric Unwarping and Illumination Correction

Feng¹,

Wang²,

Zhou³

et al. 2021

Proceedings of the 29th ACM International Conference on Multimedia

View full text Add to dashboard Cite

show abstract

Section: Transformer In Language and Visionmentioning

confidence: 99%

DocTr: Document Image Transformer for Geometric Unwarping and Illumination Correction

Feng¹,

Wang²,

Zhou³

et al. 2021

Proceedings of the 29th ACM International Conference on Multimedia

View full text Add to dashboard Cite

show abstract

“…Recent researches [3,33,38,42,47] mainly focus on modeling the relationship between fixed video representations and the output textual descriptions via an encoder-decoder framework for video captioning. Specifically, these methods [12,27,30,33,59] employ an encoder to refine video representations from a set of fixed video frame features, and a language decoder operates on top of these refined video representations to learn visual-textual alignment for caption generation.…”

Section: Related Workmentioning

confidence: 99%

“…Video captioning [3,12,27,30,38,42,47,48,59] is the task of describing the visual content of a given video in natural language. As such, it requires an algorithm to understand Different from prior works that use offline-extracted 2D/3D features, we propose to adopt the video transformer as our video encoder, and present an end-to-end fully Transformer-based model for video captioning.…”

Section: Introductionmentioning

confidence: 99%

SwinBERT: End-to-End Transformers with Sparse Attention for Video Captioning

Lin¹,

Fu²,

Lin³

et al. 2021

Preprint

View full text Add to dashboard Cite

The canonical approach to video captioning dictates a caption generation model to learn from offline-extracted dense video features. These feature extractors usually operate on video frames sampled at a fixed frame rate and are often trained on image/video understanding tasks, without adaption to video captioning data. In this work, we present SWINBERT, an end-to-end transformer-based model for video captioning, which takes video frame patches directly as inputs, and outputs a natural language description. Instead of leveraging multiple 2D/3D feature extractors, our method adopts a video transformer to encode spatialtemporal representations that can adapt to variable lengths of video input without dedicated design for different frame rates. Based on this model architecture, we show that video captioning can benefit significantly from more densely sampled video frames as opposed to previous successes with sparsely sampled video frames for video-and-language understanding tasks (e.g., video question answering). Moreover, to avoid the inherent redundancy in consecutive video frames, we propose adaptively learning a sparse attention mask and optimizing it for task-specific performance improvement through better long-range video sequence modeling. Through extensive experiments on 5 video captioning datasets, we show that SWINBERT achieves across-theboard performance improvements over previous methods, often by a large margin. The learned sparse attention masks in addition push the limit to new state of the arts, and can be transferred between different video lengths and between different datasets.

show abstract

“…Although retrieval-based methods can find human-like sentences with similar semantics to the video, it is challenging to generate an entirely correct description due to limited retrieval samples. With the advent of the encoder-decoder framework, most of the current work is studying how to better use visual features [35,23,1,36,22,24] and design elaborate models [34,2,28,13] to generate sentences directly. However, the diversity and controllability of sentences generated in this way are not satisfactory.…”

Section: Related Workmentioning

confidence: 99%

“…Video captioning is one of the most important visionlanguage tasks, and it seeks to automatically describe what has happened in the video according to the visual content. Recently, many promising methods [36,22,24,34,2] have been proposed to address this task. These methods mainly focus on learning the spatial-temporal representations of videos to fully tap visual information and devising novel decoders to achieve visual-textual alignment or controllable decoding.…”

Section: Introductionmentioning

confidence: 99%

Open-book Video Captioning with Retrieve-Copy-Generate Network

Zhang¹,

Qi²,

Yuan³

et al. 2021

Preprint

View full text Add to dashboard Cite

In this paper, we convert traditional video captioning task into a new paradigm, i.e., Open-book Video Captioning, which generates natural language under the prompts of video-content-relevant sentences, not limited to the video itself. To address the open-book video captioning problem, we propose a novel Retrieve-Copy-Generate network, where a pluggable video-to-text retriever is constructed to retrieve sentences as hints from the training corpus effectively, and a copy-mechanism generator is introduced to extract expressions from multi-retrieved sentences dynamically. The two modules can be trained end-to-end or separately, which is flexible and extensible. Our framework coordinates the conventional retrieval-based methods with orthodox encoder-decoder methods, which can not only draw on the diverse expressions in the retrieved sentences but also generate natural and accurate content of the video. Extensive experiments on several benchmark datasets show that our proposed approach surpasses the state-of-the-art performance, indicating the effectiveness and promising of the proposed paradigm in the task of video captioning.

show abstract

Learning Semantic Concepts and Temporal Alignment for Narrated Video Procedural Captioning

Cited by 19 publications

References 23 publications

DocTr: Document Image Transformer for Geometric Unwarping and Illumination Correction

DocTr: Document Image Transformer for Geometric Unwarping and Illumination Correction

SwinBERT: End-to-End Transformers with Sparse Attention for Video Captioning

Open-book Video Captioning with Retrieve-Copy-Generate Network

Contact Info

Product

Resources

About