End-To-End Time-Lapse Video Synthesis From a Single Outdoor Image

Nam, Seonghyeon; Ma, Chongyang; Chai, Menglei; Brendel, William; Xu, Ning; Kim, Seon Joo Joo

doi:10.1109/cvpr.2019.00150

Cited by 30 publications

(18 citation statements)

References 41 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We have intentionally selected challenging samples to show the robustness to complex lighting conditions and even, as in the case of the avatar, to non-real data distributions which were not previously seen by the model. These results are encouraging to further extend the model to video generation [12,30,42,47,48] in future works. Fig.…”

Section: Simultaneous Edition Of Multiple Ausmentioning

confidence: 70%

GANimation: One-Shot Anatomically Consistent Facial Animation

et al. 2019

View full text Add to dashboard Cite

Recent advances in Generative Adversarial Networks (GANs) have shown impressive results for the task of facial expression synthesis. The most successful architecture is StarGAN [5], that conditions GANs' generation process with images of a specific domain, namely a set of images of people sharing the same expression. While effective, this approach can only generate a discrete number of expressions, determined by the content and granularity of the dataset. To address this limitation, in this paper, we introduce a novel GAN conditioning scheme based on Action Units (AU) annotations, which describes in a continuous manifold the anatomical facial movements defining a human expression. Our approach allows controlling the magnitude of activation of each AU and combining several of them. Additionally, we propose a weakly supervised strategy to train the model, that only requires images annotated with their activated AUs, and exploit a novel self-learned attention mechanism that makes our network robust to changing backgrounds, lighting conditions and occlusions. Extensive evaluation shows that our approach goes beyond competing conditional generators both in the capability to synthesize a much wider range of expressions ruled by anatomically feasible muscle movements, as in the capacity of dealing with images in the wild.

show abstract

Section: Simultaneous Edition Of Multiple Ausmentioning

confidence: 70%

GANimation: One-Shot Anatomically Consistent Facial Animation

et al. 2019

View full text Add to dashboard Cite

show abstract

“…This task can be decomposed into two subtasks: color animation and motion animation [2]. Color animation is animating the color of a landscape at different hours in one day (e.g., time-varying colors in the sky) [1,15]. Motion animation is animating the scene at a smaller time scale (e.g., moving clouds) [2,14,27,32].…”

Section: Related Workmentioning

confidence: 99%

Learning Fine-Grained Motion Embedding for Landscape Animation

Xue

Liu

Yang

et al. 2021

Proceedings of the 29th ACM International Conference on Multimedia

View full text Add to dashboard Cite

In this paper we focus on landscape animation, which aims to generate time-lapse videos from a single landscape image. Motion is crucial for landscape animation as it determines how objects move in videos. Existing methods are able to generate appealing videos by learning motion from real time-lapse videos. However, current methods suffer from inaccurate motion generation, which leads to unrealistic video results. To tackle this problem, we propose a model named FGLA to generate high-quality and realistic videos by learning Fine-Grained motion embedding for Landscape Animation. Our model consists of two parts: (1) a motion encoder which embeds time-lapse motion in a fine-grained way. (2) a motion generator which generates realistic motion to animate input images. To train and evaluate on diverse time-lapse videos, we build the largest high-resolution Time-lapse video dataset with Diverse scenes, namely Time-lapse-D, which includes 16,874 video clips with over 10 million frames. Quantitative and qualitative experimental results demonstrate the superiority of our method. In particular, our method achieves relative improvements by 19% on LIPIS and 5.6% on FVD compared with state-of-the-art methods on our dataset. A user study carried out with 700 human subjects shows that our approach visually outperforms existing methods by a large margin.

show abstract

“…Nam et al . [20] learns the correlation between the illumination change of an outdoor scene and the time of the day by a multi-frame joint conditional generation framework. Yang et al .…”

Section: Related Workmentioning

confidence: 99%

DTVNet: Dynamic Time-Lapse Video Generation via Single Still Image

Zhang

Liu

et al. 2020

Computer Vision – ECCV 2020

View full text Add to dashboard Cite

This paper presents a novel end-to-end dynamic time-lapse video generation framework, named DTVNet, to generate diversified time-lapse videos from a single landscape image, which are conditioned on normalized motion vectors. The proposed DTVNet consists of two submodules: Optical Flow Encoder (OFE) and Dynamic Video Generator (DVG). The OFE maps a sequence of optical flow maps to a normalized motion vector that encodes the motion information inside the generated video. The DVG contains motion and content streams that learn from the motion vector and the single image respectively, as well as an encoder and a decoder to learn shared content features and construct video frames with corresponding motion respectively. Specifically, the motion stream introduces multiple adaptive instance normalization (AdaIN) layers to integrate multi-level motion information that are processed by linear layers. In the testing stage, videos with the same content but various motion information can be generated by different normalized motion vectors based on only one input image. We further conduct experiments on Sky Time-lapse dataset, and the results demonstrate the superiority of our approach over the state-of-the-art methods for generating high-quality and dynamic videos, as well as the variety for generating videos with various motion information. 1

show abstract

End-To-End Time-Lapse Video Synthesis From a Single Outdoor Image

Cited by 30 publications

References 41 publications

GANimation: One-Shot Anatomically Consistent Facial Animation

GANimation: One-Shot Anatomically Consistent Facial Animation

Learning Fine-Grained Motion Embedding for Landscape Animation

DTVNet: Dynamic Time-Lapse Video Generation via Single Still Image

Contact Info

Product

Resources

About