DTVNet: Dynamic Time-Lapse Video Generation via Single Still Image

Zhang, Jiangning; Xu, Chao; Liu, Liang; Wang, Mengmeng; Wu, Xia; Liu, Yong; Jiang, Yunliang

doi:10.1007/978-3-030-58558-7_18

Cited by 20 publications

(25 citation statements)

References 32 publications

(56 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…When the motion clue is not provided at all, videos are generated in a stochastic manner constrained by the spatial information provided by the input image [2,5,8,18,45,47,48]. The models used to generate videos can be generative adversarial network (GAN) [48] or variational autoencoder (VAE) [8]. This kind of stochastic video generation can only handle short dynamic patterns in distribution.…”

Section: Image-to-video Generationmentioning

confidence: 99%

Make It Move: Controllable Image-to-Video Generation with Text Descriptions

Hu¹,

Luo²,

Chen³

2021

Preprint

View full text Add to dashboard Cite

Generating controllable videos conforming to user intentions is an appealing yet challenging topic in computer vision. To enable maneuverable control in line with user intentions, a novel video generation task, named Text-Imageto-Video generation (TI2V), is proposed. With both controllable appearance and motion, TI2V aims at generating videos from a static image and a text description. The key challenges of TI2V task lie both in aligning appearance and motion from different modalities, and in handling uncertainty in text descriptions. To address these challenges, we propose a Motion Anchor-based video GEnerator (MAGE) with an innovative motion anchor (MA) structure to store appearance-motion aligned representation. To model the uncertainty and increase the diversity, it further allows the injection of explicit condition and implicit randomness. Through three-dimensional axial transformers, MA is interacted with given image to generate next frames recursively with satisfying controllability and diversity. Accompanying the new task, we build two new video-text paired datasets based on MNIST and CATER for evaluation. Experiments conducted on these datasets verify the effectiveness of MAGE and show appealing potentials of TI2V task. Source code for model and datasets will be available soon.* This work was done while Yaosi Hu was an intern at MSRA.

show abstract

Section: Image-to-video Generationmentioning

confidence: 99%

Make It Move: Controllable Image-to-Video Generation with Text Descriptions

Hu¹,

Luo²,

Chen³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Therefore, we compare with Monkey-Net. Also, we compare with DTV-Net [22] and MD-GAN [12]. DTV-Net uses optical flow to extract a motion vector.…”

Section: ) Video Reconstructionmentioning

confidence: 99%

“…Each optical flow was extracted using the Farneback method, except for DTV-Net. For DTV-Net, we use optical flow extracted by ARFLow [23], following the original method [22].…”

Section: ) Optical Flow Evaluationmentioning

confidence: 99%

Animating Cloud Images With Flow Style Transfer

Kurisaki

Kawamoto

2021

IEEE Access

View full text Add to dashboard Cite

We propose a method for animating static images using a generative adversarial network (GAN). Given a source image depicting a cloud image and a driving video sequence depicting a moving cloud image, our framework generates a video in which the source image is animated according to the driving sequence. By inputting the source image and optical flow of the driving video into the generator, a video is generated that is conditioned by the optical flow. The optical flow enables the application of the captured motion of clouds in the source image. Further, we experimentally show that the proposed method is more effective than the existing methods for animating a keypoint-less video (in which the keypoints cannot be explicitly determined) such as a moving cloud image. Furthermore, we show an improvement in the quality of the generated video due to the use of optical flow in the video reconstruction.INDEX TERMS Image animation, video generation, generative adversarial networks, optical flow.

show abstract

“…This topic has drawn great attention in recent years [6,34,36], as video representation has become an evolving trend in content consumption that makes people more engaged. More specifically, landscape animation [2,27,32] can benefit a broad range of applications, such as video footage production, social engagement boosting, virtual background animation for online video conferences. Thus, we mainly focus on landscape animation in this paper.…”

Section: Introductionmentioning

confidence: 99%

“…Besides, the lack of motion embedding leads to a lack of diverse video generation capabilities, which limits the practicality of these methods. To solve these problems, some works use an encoder-decoder paradigm to explicitly learn motion by using a convolutional encoder to embed estimated optical flows into a latent space [2,32]. Relying on optical flows as the representation of motion, these works are able to well learn motion then generate vivid videos.…”

Section: Introductionmentioning

confidence: 99%

Learning Fine-Grained Motion Embedding for Landscape Animation

Xue

Liu

Yang

et al. 2021

Proceedings of the 29th ACM International Conference on Multimedia

View full text Add to dashboard Cite

In this paper we focus on landscape animation, which aims to generate time-lapse videos from a single landscape image. Motion is crucial for landscape animation as it determines how objects move in videos. Existing methods are able to generate appealing videos by learning motion from real time-lapse videos. However, current methods suffer from inaccurate motion generation, which leads to unrealistic video results. To tackle this problem, we propose a model named FGLA to generate high-quality and realistic videos by learning Fine-Grained motion embedding for Landscape Animation. Our model consists of two parts: (1) a motion encoder which embeds time-lapse motion in a fine-grained way. (2) a motion generator which generates realistic motion to animate input images. To train and evaluate on diverse time-lapse videos, we build the largest high-resolution Time-lapse video dataset with Diverse scenes, namely Time-lapse-D, which includes 16,874 video clips with over 10 million frames. Quantitative and qualitative experimental results demonstrate the superiority of our method. In particular, our method achieves relative improvements by 19% on LIPIS and 5.6% on FVD compared with state-of-the-art methods on our dataset. A user study carried out with 700 human subjects shows that our approach visually outperforms existing methods by a large margin.

show abstract

DTVNet: Dynamic Time-Lapse Video Generation via Single Still Image

Cited by 20 publications

References 32 publications

Make It Move: Controllable Image-to-Video Generation with Text Descriptions

Make It Move: Controllable Image-to-Video Generation with Text Descriptions

Animating Cloud Images With Flow Style Transfer

Learning Fine-Grained Motion Embedding for Landscape Animation

Contact Info

Product

Resources

About