Photo Wake-Up: 3D Character Animation From a Single Photo

Abstract

Figure 1: Given a single photo as input (far left), we create a 3D animatable version of the subject, which can now walk towards the viewer (middle). The 3D result can be experienced in augmented reality (right); in the result above the user has virtually hung the artwork with a HoloLens headset and can watch the character run out of the painting from different views. Please see all results in the supplementary video: https://youtu.be/G63goXc5MyU. AbstractWe present a method and application for animating a human subject from a single photo. E.g., the character can walk out, run, sit, or jump in 3D. The key contributions of this paper are: 1) an application of viewing and animating humans in single photos in 3D, 2) a novel 2D warping method to deform a posable template body model to fit the person's complex silhouette to create an animatable mesh, and 3) a method for handling partial self occlusions. We compare to state-of-the-art related methods and evaluate results with human studies. Further, we present an interactive interface that allows re-posing the person in 3D, and an augmented reality setup where the animated 3D person can emerge from the photo into the real world. We demonstrate the method on photos, posters, and art. The project page is at https

SiCloPe: Silhouette-Based Clothed People

“…More recent works predict body parameters of the popular SMPL model [6] by either minimizing the silhouette matching error [42], joint error based on the silhouette and 2D joints [43], or an adversarial loss that can distinguish unrealistic reconstruction output [23]. Concurrent to our work, Weng et al [52] present a method to animate a person in 3D from a single image based on the SMPL model and 2D warping.…”

The Rocketbox Library and the Utility of Freely Available Rigged Avatars

“…Recent deep learning methods also using retrieved data can be used at many stages of the avatar creation process. Some methods have been successfully used to create avatars from pictures by recreating full 3D meshes from a photo (Hu et al, 2017;Saito et al, 2019), meshes from multiple cameras Collet et al (2015); Guo et al (2019), reduce the generated artifacts (Blanz and Vetter, 1999;Ichim et al, 2015), as well as to improve rigging (Weng et al, 2019). Deep learning methods can also generate completely new avatars that are not representations of existing people, by using adversarial networks (Karras et al, 2019).…”

A State-of-the-Art Review on Image Synthesis With Generative Adversarial Networks

“…Although there have been some studies involving video processing, such as video generation [115], video colorization [116], [117], video inpainting [118], motion transfer [119], and facial animation synthesis [120]- [123], the research on video using GANs is limited. In addition, although GANs have been applied to the generation and synthesis of 3D models, such as 3D colorization [124], 3D face reconstruction [125], [126], 3D character animation [127], and 3D textured object generation [128], the results are far from perfect. At present, GANs are still based on large amounts of training data.…”