Reconstruct and Represent Video Contents for Captioning via Reinforcement Learning

Zhang, Wei; Wang, Bairui; Ma, Lin; Liu, Wei

doi:10.1109/tpami.2019.2920899

Cited by 61 publications

(19 citation statements)

References 62 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…J. Mun et al [74] proposed a framework where an event sequence generation network is used to monitor the series of events for generated captions from the video. W. Zhang et al [117] proposed a reconstruction network for a description of visual contents, which operates on both forward flow (from video to sentence) and backward flow (from sentence to video). W. Xu et al [108] proposed a polishing network that utilizes the RL technique to refine the generated captions.…”

Section: Deep Reinforcement Learning (Drl) Architecturesmentioning

confidence: 99%

A Review on Methods and Applications in Multimodal Deep Learning

Summaira¹,

Li²,

Shoib³

et al. 2022

Preprint

View full text Add to dashboard Cite

Deep Learning has implemented a wide range of applications and has become increasingly popular in recent years. The goal of multimodal deep learning (MMDL) is to create models that can process and link information using various modalities. Despite the extensive development made for unimodal learning, it still cannot cover all the aspects of human learning. Multimodal learning helps to understand and analyze better when various senses are engaged in the processing of information. This paper focuses on multiple types of modalities, i.e., image, video, text, audio, body gestures, facial expressions, and physiological signals. Detailed analysis of the baseline approaches and an in-depth study of recent advancements during the last five years (2017 to 2021) in multimodal deep learning applications has been provided. A fine-grained taxonomy of various multimodal deep learning methods is proposed, elaborating on different applications in more depth. Lastly, main issues are highlighted separately for each domain, along with their possible future research directions.CCS Concepts: • Computing methodologies → Machine learning; • Information systems → Multimedia and multimodal retrieval.

show abstract

Section: Deep Reinforcement Learning (Drl) Architecturesmentioning

confidence: 99%

A Review on Methods and Applications in Multimodal Deep Learning

Summaira¹,

Li²,

Shoib³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…This framework is trained in a supervised manner, while RL further enhances the model for better context modeling. W. Zhang et al [177] proposed a reconstruction network for a description of visual contents, which operates on both forward flow (from video to sentence) and backward flow (from sentence to video). Encoder-Decoder utilizes the forward flow for text description, and reconstructor (local and global) utilizes backward flow.…”

Section: Deep Reinforcement Learning (Drl) Architecturesmentioning

confidence: 99%

Recent Advances and Trends in Multimodal Deep Learning: A Review

Summaira¹,

Li²,

Shoib³

et al. 2021

Preprint

View full text Add to dashboard Cite

Deep Learning has implemented a wide range of applications and has become increasingly popular in recent years. The goal of multimodal deep learning is to create models that can process and link information using various modalities. Despite the extensive development made for unimodal learning, it still cannot cover all the aspects of human learning. Multimodal learning helps to understand and analyze better when various senses are engaged in the processing of information. This paper focuses on multiple types of modalities, i.e., image, video, text, audio, body gestures, facial expressions, and physiological signals. Detailed analysis of past and current baseline approaches and an in-depth study of recent advancements in multimodal deep learning applications has been provided. A fine-grained taxonomy of various multimodal deep learning applications is proposed, elaborating on different applications in more depth. Architectures and datasets used in these applications are also discussed, along with their evaluation metrics. Last, main issues are highlighted separately for each domain along with their possible future research directions.CCS Concepts: • Computing methodologies → Machine learning; • Information systems → Multimedia and multimodal retrieval.

show abstract

“…Early works [10,14,25,26,34,39,45] on video captioning adopted template-based methods, which first define some specific rules for language grammar, and then generate captions by associating words detected from visual contents with predefined sentence templates. Currently, with the development of deep neural networks, sequence learning methods with an encoder-decoder architecture [31] have been widely adopted for video captioning.…”

Section: Related Work 21 Video Captioningmentioning

confidence: 99%

Controllable Video Captioning with an Exemplar Sentence

Yuan

Ma²,

Wang

et al. 2020

Proceedings of the 28th ACM International Conference on Multimedia

Self Cite

View full text Add to dashboard Cite

In this paper, we investigate a novel and challenging task, namely controllable video captioning with an exemplar sentence. Formally, given a video and a syntactically valid exemplar sentence, the task aims to generate one caption which not only describes the semantic contents of the video, but also follows the syntactic form of the given exemplar sentence. In order to tackle such an exemplar-based video captioning task, we propose a novel Syntax Modulated Caption Generator (SMCG) incorporated in an encoder-decoder-reconstructor architecture. The proposed SMCG takes video semantic representation as an input, and conditionally modulates the gates and cells of long short-term memory network with respect to the encoded syntactic information of the given exemplar sentence. Therefore, SMCG is able to control the states for word prediction and achieve the syntax customized caption generation. We conduct experiments by collecting auxiliary exemplar sentences for two public video captioning datasets. Extensive experimental results demonstrate the effectiveness of our approach on generating syntax controllable and semantic preserved video captions. By providing different exemplar sentences, our approach is capable of producing different captions with various syntactic structures, thus indicating a promising way to strengthen the diversity of video captioning.

show abstract

Reconstruct and Represent Video Contents for Captioning via Reinforcement Learning

Cited by 61 publications

References 62 publications

A Review on Methods and Applications in Multimodal Deep Learning

A Review on Methods and Applications in Multimodal Deep Learning

Recent Advances and Trends in Multimodal Deep Learning: A Review

Controllable Video Captioning with an Exemplar Sentence

Contact Info

Product

Resources

About