Controllable Video Generation with Sparse Trajectories

Hao, Zekun; Huang, Xun; Belongie, Serge

doi:10.1109/cvpr.2018.00819

Cited by 79 publications

(83 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Meanwhile, Reda et al [34] propose to model moving appearances with both convolutional kernels as in [10] and vectors as optical flow. Our closest prior work is [11] which also composes the pixel-and flow-based predictions through occlusion maps. However, our proposed method can differentiate in three aspects: (1) our pixel and flow prediction tasks are separately trained; (2) we employ an occlusion inpainter for our pixel generation so that more contextual information after warping can be utilized; (3) instead of predicting occlusion as another side task, we directly refer to predicted flows as the proxy for post warping confidence.…”

Section: High-fidelity Video Predictionmentioning

confidence: 99%

Disentangling Propagation and Generation for Video Prediction

Gao

Cai³

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

View full text Add to dashboard Cite

A dynamic scene has two types of elements: those that move fluidly and can be predicted from previous frames, and those which are disoccluded (exposed) and cannot be extrapolated. Prior approaches to video prediction typically learn either to warp or to hallucinate future pixels, but not both. In this paper, we describe a computational model for high-fidelity video prediction which disentangles motion-specific propagation from motion-agnostic generation. We introduce a confidence-aware warping operator which gates the output of pixel predictions from a flow predictor for non-occluded regions and from a context encoder for occluded regions. Moreover, in contrast to prior works where confidence is jointly learned with flow and appearance using a single network, we compute confidence after a warping step, and employ a separate network to inpaint exposed regions. Empirical results on both synthetic and real datasets show that our disentangling approach provides better occlusion maps and produces both sharper and more realistic predictions compared to strong baselines.

show abstract

Section: High-fidelity Video Predictionmentioning

confidence: 99%

Disentangling Propagation and Generation for Video Prediction

Gao

Cai³

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

View full text Add to dashboard Cite

show abstract

“…A second closely related task to story visualization is video generation, especially that of text-to-video [24,13] or image-to-video generation [1,31,32]. Existing approaches only generate short video clips [13,5,12] without scene changes. The biggest challenge in video generation is how to ensure a smooth motion transition across successive video frames.…”

Section: Related Workmentioning

confidence: 99%

“…The biggest challenge in video generation is how to ensure a smooth motion transition across successive video frames. A trajectory, skeleton or simple landmark is used in existing work, to help model the motion feature [12,37,33]. To this end, researchers disentangle dynamic and static features for motion and background, respectively [32,24,31,6].…”

Section: Related Workmentioning

confidence: 99%

StoryGAN: A Sequential Conditional GAN for Story Visualization

Gan

Shen

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

127

133

View full text Add to dashboard Cite

We propose a new task, called Story Visualization. Given a multi-sentence paragraph, the story is visualized by generating a sequence of images, one for each sentence. In contrast to video generation, story visualization focuses less on the continuity in generated images (frames), but more on the global consistency across dynamic scenes and characters -a challenge that has not been addressed by any singleimage or video generation methods. We therefore propose a new story-to-image-sequence generation model, StoryGAN, based on the sequential conditional GAN framework. Our model is unique in that it consists of a deep Context Encoder that dynamically tracks the story flow, and two discriminators at the story and image levels, to enhance the image quality and the consistency of the generated sequences. To evaluate the model, we modified existing datasets to create the CLEVR-SV and Pororo-SV datasets. Empirically, Story-GAN outperforms state-of-the-art models in image quality, contextual consistency metrics, and human evaluation.

show abstract

“…To make predictions occur more realistic, others tackled the problem by learning separate representations for the static and dynamic components of a video. This is done either by incorporating motion conditions, such as optical flow information [12], [34], [15], [17], [25], or by learning sparse features that represent pixel dynamics [26]. Decomposing the video into static and non-static components allows the network to simply reproduce the values of the static part for the majority of pixels.…”

Section: Related Workmentioning

confidence: 99%

Enhancing Traffic Scene Predictions with Generative Adversarial Networks

König

Aigner

Körner

2019

2019 IEEE Intelligent Transportation Systems Conference (ITSC)

View full text Add to dashboard Cite

We present a new two-stage pipeline for predicting frames of traffic scenes where relevant objects can still reliably be detected. Using a recent video prediction network, we first generate a sequence of future frames based on past frames. A second network then enhances these frames in order to make them appear more realistic. This ensures the quality of the predicted frames to be sufficient to enable accurate detection of objects, which is especially important for autonomously driving cars. To verify this two-stage approach, we conducted experiments on the Cityscapes dataset. For enhancing, we trained two image-to-image translation methods based on generative adversarial networks, one for blind motion deblurring and one for image super-resolution. All resulting predictions were quantitatively evaluated using both traditional metrics and a state-of-the-art object detection network showing that the enhanced frames appear qualitatively improved. While the traditional image comparison metrics, i.e., MSE, PSNR, and SSIM, failed to confirm this visual impression, the object detection evaluation resembles it well. The best performing prediction-enhancement pipeline is able to increase the average precision values for detecting cars by about 9% for each prediction step, compared to the non-enhanced predictions.

show abstract

Controllable Video Generation with Sparse Trajectories

Cited by 79 publications

References 12 publications

Disentangling Propagation and Generation for Video Prediction

Disentangling Propagation and Generation for Video Prediction

StoryGAN: A Sequential Conditional GAN for Story Visualization

Enhancing Traffic Scene Predictions with Generative Adversarial Networks

Contact Info

Product

Resources

About