Human Motion Transfer With 3D Constraints and Detail Enhancement

Sun, Yang-Tian; Fu, Qiancheng; Jiang, Yue-Ren; Liu, Zitao; Lai, Yuekun; Fu, Hongbo; Gao, Lin

doi:10.1109/tpami.2022.3201904

Cited by 9 publications

(2 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…2D human neural rendering approaches [4], [7], [31], [42] appeared as effective approaches for human appearance synthesis. However, these methods still suffer in creating fine texture details, notably in some body parts as the face and hands.…”

Section: Methods I: Image-based Renderingmentioning

confidence: 99%

Transferring Human Motion and Appearance in Monocular Videos

Gomes¹,

Martins

Nascimento³

2022

Anais Estendidos Do XXXV Conference on Graphics, Patterns and Images (SIBGRAPI Estendido 2022)

View full text Add to dashboard Cite

This thesis investigates the problem of transferring human motion and appearance from video to video preserving motion features, body shape, and visual quality. In other words, given two input videos, we investigate how to synthesize a new video, where a target person from the first video is placed into a new context performing different motions from the second video. Possible application domains are on graphics animations and entertainment media that rely on synthetic characters and virtual environments to create visual content. We introduce two novel methods for transferring appearance and retargeting human motion from monocular videos, and by consequence, increase the creative possibilities of visual content. Differently from recent appearance transferring methods, our approaches take into account 3D shape, appearance, and motion constraints. Specifically, our first method is based on a hybrid image-based rendering technique that exhibits competitive visual retargeting quality compared to state-of-the-art neural rendering approaches, even without computationally intensive training. Then, inspired by the advantages of the first method, we designed an end-to-end learning-based transferring strategy. Taking advantages of both differentiable rendering and the 3D parametric model, our second data-driven method produces a fully 3D controllable human model, i.e., the user can control the human pose and rendering parameters. Experiments on different videos show that our methods preserve specific features of the motion that must be maintained (e.g., feet touching the floor, hands touching a particular object) while holding the best values for appearance in terms of Structural Similarity (SSIM), Learned Perceptual Image Patch Similarity (LPIPS), Mean Squared Error (MSE), and Fréchet Video Distance (FVD). We also provide to the community a new dataset composed of several annotated videos with motion constraints for retargeting applications and paired motion sequences from different characters to evaluate transferring approaches.

show abstract

Section: Methods I: Image-based Renderingmentioning

confidence: 99%

Transferring Human Motion and Appearance in Monocular Videos

Gomes¹,

Martins

Nascimento³

2022

Anais Estendidos Do XXXV Conference on Graphics, Patterns and Images (SIBGRAPI Estendido 2022)

View full text Add to dashboard Cite

show abstract

“…The first is based on a general model; the model is trained for an unseen target, or the source pose is transferred to a given target within find adjustments [13,14,15,16]. The second is based on an individualized model; the focus is placed on learning the appearance of a specific person, and new poses are generated for the same person [17,18,19,20,21].…”

Section: Related Workmentioning

confidence: 99%

Study on 2D Sprite *3.Generation Using the Impersonator Network

2023

KSII TIIS

View full text Add to dashboard Cite

This study presents a method for capturing photographs of users as input and converting them into 2D character animation sprites using a generative adversarial network-based artificial intelligence network. Traditionally, 2D character animations have been created by manually creating an entire sequence of sprite images, which incurs high development costs. To address this issue, this study proposes a technique that combines motion videos and sample 2D images. In the 2D sprite generation process that uses the proposed technique, a sequence of images is extracted from real-life images captured by the user, and these are combined with character images from within the game. Our research aims to leverage cutting-edge deep learning-based image manipulation techniques, such as the GAN-based motion transfer network (impersonator) and background noise removal (U 2 -Net), to generate a sequence of animation sprites from a single image. The proposed technique enables the creation of diverse animations and motions just one image. By utilizing these advancements, we focus on enhancing productivity in the game and animation industry through improved efficiency and streamlined production processes. By employing state-of-the-art techniques, our research enables the generation of 2D sprite images with various motions, offering significant potential for boosting productivity and creativity in the industry.

show abstract

SE-ORNet: Self-Ensembling Orientation-Aware Network for Unsupervised Point Cloud Shape Correspondence

Deng,

Wang,

et al. 2023

2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

Human Motion Transfer With 3D Constraints and Detail Enhancement

Cited by 9 publications

References 35 publications

Transferring Human Motion and Appearance in Monocular Videos

Transferring Human Motion and Appearance in Monocular Videos

Study on 2D Sprite *3.Generation Using the Impersonator Network

SE-ORNet: Self-Ensembling Orientation-Aware Network for Unsupervised Point Cloud Shape Correspondence

Contact Info

Product

Resources

About