Pose with style

Abstract

We present an algorithm for re-rendering a person from a single image under arbitrary poses. Existing methods often have difficulties in hallucinating occluded contents photo-realistically while preserving the identity and fine details in the source image. We first learn to inpaint the correspondence field between the body surface texture and the source image with a human body symmetry prior. The inpainted correspondence field allows us to transfer/warp local features extracted from the source to the target view even under large pose changes. Directly mapping the warped local features to an RGB image using a simple CNN decoder often leads to visible artifacts. Thus, we extend the StyleGAN generator so that it takes pose as input (for controlling poses) and introduces a spatially varying modulation for the latent space using the warped local features (for controlling appearances). We show that our method compares favorably against the state-of-the-art algorithms in both quantitative evaluation and visual comparison.

Generating Texture for 3D Human Avatar from a Single Image using Sampling and Refinement Networks

“…Similar to previous methods [LIP19; APTM19; GSVL19; ALY*21], we map the source image into the UV space of the SMPL model. We denote the mapped partial texture as T source .…”

“…These methods reported better outcomes in retaining the local details presented in the source image compared to the methods [NGK18] that directly utilize the color pixels of the source image. Similar to these approaches [GSVL19; ALY*21], we use a sampler network to sample the visible regions in the source image to create the texture map. Because the texture map is used for intermediate representation, it is prone to artifacts such as stretching‐out or distortion when directly used for rendering as shown in Figure 5.…”

“…Example results of the generated texture map and rendered images are shown in Figure 1. We conducted a set of experiments to show that our approach outperforms the various baseline methods [2,19,24,43,46] in reproducing the details present in the source image and aligning textures to the surface of the target 3D human mesh.…”

State‐of‐the‐Art in the Architecture, Methods and Applications of StyleGAN

“…Albahar et al [ALY*21] suggest spatial control through the initial input constant, while leveraging the inherent semantic understanding StyleGAN naturally develops, reinforced by human pose labeling. Unlike most editing works, which manipulate the behavior of a pretrained StyleGAN, this work proposes architectural changes to the generator, to adapt it to human pose inputs.…”

“…While all the aforementioned works showed incredible results and promise in real‐world scenarios, they are limited in the domains they operate over. Some works have explored going beyond the facial domain and have explored applying StyleGAN for full‐body synthesis in various applications such as virtual try‐on and portrait reposing [LVKS21,ALY*21].…”