Creating and Reenacting Controllable 3D Humans with Differentiable Rendering

Gomes, Thiago L.; Coutinho, Thiago M.; Azevedo, Rafael Alves de; Martins, Renato; Nascimento, Erickson R.

doi:10.1109/wacv51458.2022.00079

Cited by 4 publications

(2 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To reconstruct a 3D human, early works (Collet et al, 2015;Guo et al, 2019;Su et al, 2020) count on off-the-shelf tools to predict the camera depth. As mitigating the costly hardware requirement, they estimate a 3D human texture (Xu and Loy, 2021;Gomes et al, 2022) via the UV mapping (Shysheya et al, 2019;Yoon et al, 2021). With the promising success of NeRF, recent works (Peng et al, 2021b,a;Su et al, 2021) adopt volume rendering for 3D humans from multi-view videos (Weng et al, 2022;.…”

Section: Related Workmentioning

confidence: 99%

Text-guided 3D Human Generation from 2D Collections

Fu,

Xiong,

Nie

et al. 2023

Findings of the Association for Computational Linguistics: EMNLP 2023

View full text Add to dashboard Cite

3D human modeling has been widely used for engaging interaction in gaming, film, and animation. The customization of these characters is crucial for creativity and scalability, which highlights the importance of controllability. In this work, we introduce Text-guided 3D Human Generation (T3H), where a model is to generate a 3D human, guided by the fashion description. There are two goals: 1) the 3D human should render articulately, and 2) its outfit is controlled by the given text. To address this T3H task, we propose Compositional Cross-modal Human (CCH). CCH adopts cross-modal attention to fuse compositional human rendering with the extracted fashion semantics. Each human body part perceives relevant textual guidance as its visual patterns. We incorporate the human prior and semantic discrimination to enhance 3D geometry transformation and fine-grained consistency, enabling this to learn from 2D collections for data efficiency. We conduct evaluations on DeepFashion and SHHQ with diverse fashion attributes covering the shape, fabric, and color of upper and lower clothing. Extensive experiments demonstrate that CCH achieves superior results for T3H with high efficiency.

show abstract

Section: Related Workmentioning

confidence: 99%

Text-guided 3D Human Generation from 2D Collections

Fu,

Xiong,

Nie

et al. 2023

Findings of the Association for Computational Linguistics: EMNLP 2023

View full text Add to dashboard Cite

show abstract

“…The results of this dissertation were published in the International Journal of Computer Vision (IJCV'21) [47], and in two international conferences on applications of computer vision (WACV'20 [48] and WACV'22 [49]). The student also contributed as co-author to one related publication on human motion generation in the international journal Computers & Graphics [50].…”

Section: Publications and Awardsmentioning

confidence: 99%

Transferring Human Motion and Appearance in Monocular Videos

Gomes¹,

Martins

Nascimento³

2022

Anais Estendidos Do XXXV Conference on Graphics, Patterns and Images (SIBGRAPI Estendido 2022)

View full text Add to dashboard Cite

This thesis investigates the problem of transferring human motion and appearance from video to video preserving motion features, body shape, and visual quality. In other words, given two input videos, we investigate how to synthesize a new video, where a target person from the first video is placed into a new context performing different motions from the second video. Possible application domains are on graphics animations and entertainment media that rely on synthetic characters and virtual environments to create visual content. We introduce two novel methods for transferring appearance and retargeting human motion from monocular videos, and by consequence, increase the creative possibilities of visual content. Differently from recent appearance transferring methods, our approaches take into account 3D shape, appearance, and motion constraints. Specifically, our first method is based on a hybrid image-based rendering technique that exhibits competitive visual retargeting quality compared to state-of-the-art neural rendering approaches, even without computationally intensive training. Then, inspired by the advantages of the first method, we designed an end-to-end learning-based transferring strategy. Taking advantages of both differentiable rendering and the 3D parametric model, our second data-driven method produces a fully 3D controllable human model, i.e., the user can control the human pose and rendering parameters. Experiments on different videos show that our methods preserve specific features of the motion that must be maintained (e.g., feet touching the floor, hands touching a particular object) while holding the best values for appearance in terms of Structural Similarity (SSIM), Learned Perceptual Image Patch Similarity (LPIPS), Mean Squared Error (MSE), and Fréchet Video Distance (FVD). We also provide to the community a new dataset composed of several annotated videos with motion constraints for retargeting applications and paired motion sequences from different characters to evaluate transferring approaches.

show abstract