High-Fidelity Neural Human Motion Transfer from Monocular Video

Style and Pose Control for Image Synthesis of Humans from a Single Monocular View

“…To address the limitations of classical rendering methods, recent works integrated deep learning techniques into the classical rendering pipelines. Some methods [Kappel et al 2020;Liu et al 2020bMeshry et al 2019;Thies et al 2019;Yoon et al 2020] first render 2D/3D skeleton, 2D joint heat maps, or a coarse surface geometry with explicit or neural textures into coarse RGB images or feature maps which are then translated into high-quality images using image translation networks, such as pix2pix . Another line of works learn scene representations for novel view synthesis from 2D images.…”

“…The human pose transfer problem is defined as transferring person appearance from one pose to another [Ma et al 2017]. Most approaches formulate it as an image-to-image mapping problem, i.e., given a reference image of the target person, mapping the body pose in the format of renderings of a skeleton [Chan et al 2019;Kratzwald et al 2017;Pumarola et al 2018;Siarohin et al 2018;, dense mesh [Grigor'ev et al 2019;Kappel et al 2020;Liu et al 2020bWang et al 2018a;Yoon et al 2020] or joint position heatmaps [Aberman et al 2019;Ma et al 2017 to real images. Ma et al [2017] design a two-stage framework, which first generates a coarse image of the person in the reference image with the target pose and refines the coarse image with a UNet trained in an adversarial way.…”

Neural Actor: Neural Free-view Synthesis of Human Actors with Pose Control

“…In the context of entire human bodies, many of the approaches formulate this task as an image-to-image mapping problem. Given an appearance, these methods maps the body pose in the form of renderings of a skeleton [Chan et al 2019;Kappel et al 2020;Kratzwald et al 2017;Li et al 2019;Pumarola et al 2018;Shysheya et al 2019;Siarohin et al 2018;Zhu et al 2019], projection of a dense human model [Grigor'ev et al 2019;Liu et al 2020bLiu et al , 2019bNeverova et al 2018;Prokudin et al 2021;Raj et al 2021a;Sarkar et al 2020;, or joint position heatmaps [Aberman et al 2019;Ma et al 2017Ma et al , 2018 to realistic human images. To better preserve the appearance from the reference image to the generated image, some methods [Liu et al 2020b;Sarkar et al 2020] first map the person's appearance from screen space to UV space and feed the rendering of the person in the target pose with the UV texture map into an image-to-image translation network.…”