Riggable 3D Face Reconstruction via In-Network Optimization

Bai, Ziqian; Cui, Zhaopeng; Liu, Xiaoming; Tan, Ping

doi:10.1109/cvpr46437.2021.00615

Cited by 42 publications

(45 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The method is based on the FLAME face model. Another approach that has been designed with animation in mind is INORig [ 47 ]. The approach is built on an end-to-end trainable network that first parameterizes the face rig as a compact latent code with a neural decoder, and then estimates the latent code as well as per-image parameters via a learnable optimization.…”

Section: Related Workmentioning

confidence: 99%

“…Among the possible approaches, we considered those that have made their code publicly available: DECA [ 46 ], 3DDFAV2 [ 45 ], 3DSFMFace [ 53 ], Extreme3D [ 37 ], RingNet [ 42 ], Deep3DFace [ 41 ], INORig [ 47 ], and PRNet [ 50 ].…”

Section: Comparing the 3d Face Reconstruction Approachesmentioning

confidence: 99%

“…For DECA, the figure shows the coarse mesh that, opposed to the detailed one, provides a texture. ( a ) Input image; ( b ) ground truth; ( c ) 3DDFAv2 [ 45 ]; ( d ) DECA [ 46 ]; ( e ) Deep3DFace [ 41 ]; ( f ) Extreme3D [ 37 ]; ( g ) INORig [ 47 ]; ( h ) PRNet [ 50 ]; ( i ) RingNet [ 42 ].…”

Section: Figurementioning

confidence: 99%

“… Some samples used in the texture quality experiment. ( a ) Input image; ( b ) 3DDFAv2 [ 45 ]; ( c ) DECA [ 46 ]; ( d ) Deep3DFace [ 41 ]; ( e ) INORig [ 47 ]; ( f ) PRNet [ 50 ]; ( g ) RingNet [ 42 ]. …”

Section: Figurementioning

confidence: 99%

See 3 more Smart Citations

Designing an AI-Based Virtual Try-On Web Application

Marelli

Bianco

Ciocca

2022

Sensors

View full text Add to dashboard Cite

In the last few years, Augmented Reality, Virtual Reality, and Artificial Intelligence (AI) have been increasingly employed in different application domains. Among them, the retail market presents the opportunity to allow people to check the appearance of accessories, makeup, hairstyle, hair color, and clothes on themselves, exploiting virtual try-on applications. In this paper, we propose an eyewear virtual try-on experience based on a framework that leverages advanced deep learning-based computer vision techniques. The virtual try-on is performed on a 3D face reconstructed from a single input image. In designing our system, we started by studying the underlying architecture, components, and their interactions. Then, we assessed and compared existing face reconstruction approaches. To this end, we performed an extensive analysis and experiments for evaluating their design, complexity, geometry reconstruction errors, and reconstructed texture quality. The experiments allowed us to select the most suitable approach for our proposed try-on framework. Our system considers actual glasses and face sizes to provide a realistic fit estimation using a markerless approach. The user interacts with the system by using a web application optimized for desktop and mobile devices. Finally, we performed a usability study that showed an above-average score of our eyewear virtual try-on application.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Comparing the 3d Face Reconstruction Approachesmentioning

confidence: 99%

Section: Figurementioning

confidence: 99%

Section: Figurementioning

confidence: 99%

See 2 more Smart Citations

Designing an AI-Based Virtual Try-On Web Application

Marelli

Bianco

Ciocca

2022

Sensors

View full text Add to dashboard Cite

show abstract

“…3D representations of the human head along with a set of controls that can be used to animate the geometry. This has been traditionally addressed by recovering a personalized set of expression bases known as blendshapes, obtained through deformation transfer [7,19,21,22,46] or deep neural networks [1,9,58]. Other rigging techniques have also been investigated, such as joint-based representations [17,21,54] and muscle-based systems [2].…”

Section: Related Workmentioning

confidence: 99%

I M Avatar: Implicit Morphable Head Avatars from Videos

Zheng¹,

Abrevaya²,

Bühler³

et al. 2021

Preprint

View full text Add to dashboard Cite

Traditional morphable face models provide fine-grained control over expression but cannot easily capture geometric and appearance details. Neural volumetric representations approach photo-realism but are hard to animate and do not generalize well to unseen expressions. To tackle this problem, we propose IMavatar (Implicit Morphable avatar), a novel method for learning implicit head avatars from monocular videos. Inspired by the fine-grained control mechanisms afforded by conventional 3DMMs, we represent the expression-and pose-related deformations via learned blendshapes and skinning fields. These attributes are pose-independent and can be used to morph the canonical geometry and texture fields given novel expression and pose parameters. We employ ray tracing and iterative rootfinding to locate the canonical surface intersection for each pixel. A key contribution is our novel analytical gradient formulation that enables end-to-end training of IMavatars from videos. We show quantitatively and qualitatively that our method improves geometry and covers a more complete expression space compared to state-of-the-art methods.

show abstract

PR3D: Precise and realistic 3D face reconstruction from a single image

Huang,

2024

Computer Animation & Virtual

View full text Add to dashboard Cite

Reconstructing the three‐dimensional (3D) shape and texture of the face from a single image is a significant and challenging task in computer vision and graphics. In recent years, learning‐based reconstruction methods have exhibited outstanding performance, but their effectiveness is severely constrained by the scarcity of available training data with 3D annotations. To address this issue, we present the PR3D (Precise and Realistic 3D face reconstruction) method, which consists of high‐precision shape reconstruction based on semi‐supervised learning and high‐fidelity texture reconstruction based on StyleGAN2. In shape reconstruction, we use in‐the‐wild face images and 3D annotated datasets to train the auxiliary encoder and the identity encoder, encoding the input image into parameters of FLAME (a parametric 3D face model). Simultaneously, a novel semi‐supervised hybrid landmark loss is designed to more effectively learn from in‐the‐wild face images and 3D annotated datasets. Furthermore, to meet the real‐time requirements in practical applications, a lightweight shape reconstruction model called fast‐PR3D is distilled through teacher–student learning. In texture reconstruction, we propose a texture extraction method based on face reenactment in StyleGAN2 style space, extracting texture from the source and reenacted face images to constitute a facial texture map. Extensive experiments have demonstrated the state‐of‐the‐art performance of our method.

show abstract

Riggable 3D Face Reconstruction via In-Network Optimization

Cited by 42 publications

References 35 publications

Designing an AI-Based Virtual Try-On Web Application

Designing an AI-Based Virtual Try-On Web Application

I M Avatar: Implicit Morphable Head Avatars from Videos

PR3D: Precise and realistic 3D face reconstruction from a single image

Contact Info

Product

Resources

About