Attentive Semantic Video Generation Using Captions

Marwah, Tanya; Mittal, Gaurav; Balasubramanian, Vineeth N

doi:10.1109/iccv.2017.159

Cited by 55 publications

(33 citation statements)

References 17 publications

(29 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Mittal et al [23] first introduced this task and proposed a VAE-based framework, called Sync-DRAW, to encode simple captions and generate semantically consistent videos. A concurrent work [21] performs variable-length semantic video generation from captions. The model relies on VAE and recurrent neural networks (RNNs) to learn the longterm and short-term context of the video.…”

Section: Text-to-video Generationmentioning

confidence: 99%

Make It Move: Controllable Image-to-Video Generation with Text Descriptions

Hu¹,

Luo²,

Chen³

2021

Preprint

View full text Add to dashboard Cite

Generating controllable videos conforming to user intentions is an appealing yet challenging topic in computer vision. To enable maneuverable control in line with user intentions, a novel video generation task, named Text-Imageto-Video generation (TI2V), is proposed. With both controllable appearance and motion, TI2V aims at generating videos from a static image and a text description. The key challenges of TI2V task lie both in aligning appearance and motion from different modalities, and in handling uncertainty in text descriptions. To address these challenges, we propose a Motion Anchor-based video GEnerator (MAGE) with an innovative motion anchor (MA) structure to store appearance-motion aligned representation. To model the uncertainty and increase the diversity, it further allows the injection of explicit condition and implicit randomness. Through three-dimensional axial transformers, MA is interacted with given image to generate next frames recursively with satisfying controllability and diversity. Accompanying the new task, we build two new video-text paired datasets based on MNIST and CATER for evaluation. Experiments conducted on these datasets verify the effectiveness of MAGE and show appealing potentials of TI2V task. Source code for model and datasets will be available soon.* This work was done while Yaosi Hu was an intern at MSRA.

show abstract

Section: Text-to-video Generationmentioning

confidence: 99%

Make It Move: Controllable Image-to-Video Generation with Text Descriptions

Hu¹,

Luo²,

Chen³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Prior work has explored supplementing relevant visual content to text stories based on the similarity of the story text to the annotation of images or video clips [16,17,44]. Similarly, built on large training datasets of visual content with text annotations, research has explored the generation of images [13], 3D scenes [2], and video sequence [22] from text with neural nets. Leake et al explored leveraging word concreteness, which measures how closely a word is related some perceptible concepts, to automatically create slideshows from Session 10A: Bridging Visual and Audio Content UIST '20, October 20-23, 2020, Virtual Event, USA text input, by composing images of the most concrete words [19] .…”

Section: Automatic Generation Of Visual Content From Textmentioning

confidence: 99%

Crosscast: Adding Visuals to Audio Travel Podcasts

Xia

Jacobs

Agrawala

2020

Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology

View full text Add to dashboard Cite

show abstract

“…Several more studies [13], [25], [15] adopted convolutional LSTM to take spatial and temporal contexts into account. [26] proposed a video generation framework which utilized the Conv-LSTM to encode short-term and long-term spatial-temporal context for semantic video generation using captions. [27] combined the ConvLSTM into a deep generative model which modeled the factorization of the joint likelihood of inputs in the form of video data.…”

Section: Related Workmentioning

confidence: 99%

Long-Term Video Prediction via Criticization and Retrospection

Chen

Yang

et al. 2020

IEEE Trans. on Image Process.

View full text Add to dashboard Cite

Video prediction refers to predicting and generating future video frames by observed a set of consecutive frames. Conventional video prediction methods continuously criticize the discrepancy between predicted and ground truth frames in a recursive generation process. As the prediction error accumulates recursively, these methods would easily become out of control and are often confined to the short-term horizon. In this paper, we introduce a retrospection process to rectify the prediction errors beyond criticizing the future prediction. The introduced retrospection process is designed to look back what have been learned from the past and rectify the prediction deficiencies. To this end, we build a retrospection network to reconstruct the past frames given the currently predicted frames. A retrospection loss is introduced to push the retrospection frames being consistent with the observed frames, so that the prediction error is alleviated. On the other hand, an auxiliary route is built by reversing the flow of time and executing a similar retrospection. These two routes interact with each other to boost the performance of retrospection network and enhance the understanding of dynamics across frames, especially for the longterm horizon. An adversarial loss is employed to generate more realistic results in both prediction and retrospection process. Extensive experiments on natural video dataset demonstrate the advantage of introducing retrospection process in long-term video prediction.

show abstract

Attentive Semantic Video Generation Using Captions

Cited by 55 publications

References 17 publications

Make It Move: Controllable Image-to-Video Generation with Text Descriptions

Make It Move: Controllable Image-to-Video Generation with Text Descriptions

Crosscast: Adding Visuals to Audio Travel Podcasts

Long-Term Video Prediction via Criticization and Retrospection

Contact Info

Product

Resources

About