Animating Arbitrary Objects via Deep Motion Transfer

Abstract

This paper introduces a novel deep learning framework for image animation. Given an input image with a target object and a driving video sequence depicting a moving object, our framework generates a video in which the target object is animated according to the driving sequence. This is achieved through a deep architecture that decouples appearance and motion information. Our framework consists of three main modules: (i) a Keypoint Detector unsupervisely trained to extract object keypoints, (ii) a Dense Motion prediction network for generating dense heatmaps from sparse keypoints, in order to better encode motion information and (iii) a Motion Transfer Network, which uses the motion heatmaps and appearance information extracted from the input image to synthesize the output frames. We demonstrate the effectiveness of our method on several benchmark datasets, spanning a wide variety of object appearances, and show that our approach outperforms stateof-the-art image animation and video generation methods. Our source code is publicly available 1 .

3D Guided Fine-Grained Face Manipulation

“…Several video generation methods for face animation were proposed. Given a face image, such methods perform video prediction [26] or motion transfer [22,28] to manipulate faces. Recently, Pumarola et al [21] presented a work performing anatomically-aware face animation.…”

Multi-Channel Attention Selection GAN With Cascaded Semantic Guidance for Cross-View Image Translation

“…To generate domain-specific images, Conditional GAN (CGAN) [27] has been proposed. CGAN usually combines a vanilla GAN and some external information, such as class labels or tags [29,30,4,40,37], text descriptions [33,50], human pose [8,38,28,22,35] and reference images [25,16]. Image-to-Image Translation frameworks adopt inputoutput data to learn a parametric mapping between inputs and outputs.…”

Attribute-Driven Spontaneous Motion in Unpaired Image Translation

“…To deform input to the target image, backward motion is usually considered as a vector from the target to source points [14,6,33]. However, such motion representation may not be suitable for convolution/deconvolution layer networks with aligned operators, since the representation is aligned to the unknown deformed image rather than the input one.…”