The role of prior in image based 3D modeling: a survey

Zhu, Hao; Nie, Yongming; Yue, Tao; Cao, Xun

doi:10.1007/s11704-016-5520-8

Cited by 20 publications

(11 citation statements)

References 48 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…According to the representation of the 3D modeling, traditional MVS can be categorized into volumetric method (Kutulakos and Seitz 2000;Seitz and Dyer 1997), point-based method (Furukawa and Ponce 2010;Lhuillier and Quan 2005;Chen et al 2019) and depthbased method (Campbell et al 2008;Galliani, Lasinger, and Schindler 2015;Schönberger et al 2016;Tola, Strecha, and Fua 2012;Liu et al 2009). Due to the space limitation, we recommend referring to the survey/benchmark (Seitz et al 2006;Knapitsch et al 2017;Zhu et al 2017) and the tutorial (Furukawa and Hernández 2015) for the comprehensive review for traditional MVS. Here we focus on reviewing recently proposed learning-based MVS and face-specific MVS, which are more relevant to our work.…”

Section: Related Workmentioning

confidence: 99%

Detailed Facial Geometry Recovery from Multi-View Images by Learning an Implicit Function

Xiao¹,

Zhu²,

Huang³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Recovering detailed facial geometry from a set of calibrated multi-view images is valuable for its wide range of applications. Traditional multi-view stereo (MVS) methods adopt optimization methods to regularize the matching cost. Recently, learning-based methods integrate all these into an endto-end neural network and show superiority of efficiency. In this paper, we propose a novel architecture to recover extremely detailed 3D faces in roughly 10 seconds. Unlike previous learning-based methods that regularize the cost volume via 3D CNN, we propose to learn an implicit function for regressing the matching cost. By fitting a 3D morphable model from multi-view images, the features of multiple images are extracted and aggregated in the mesh-attached UV space, which makes the implicit function more effective in recovering detailed facial shape. Our method outperforms SOTA learning-based MVS in accuracy by a large margin on the FaceScape dataset. The code and data will be released soon.

show abstract

Section: Related Workmentioning

confidence: 99%

Detailed Facial Geometry Recovery from Multi-View Images by Learning an Implicit Function

Xiao¹,

Zhu²,

Huang³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Then the human mesh will get deformed with the Laplacian deformation approach given the movements of the handles while maintaining the local geometry as much as possible. The deforming strategy has been used in multi-view shape reconstruction problem [1,39,18,24,40,26,27], while we are the first to predict the deformation from a single image with the deep neural network.…”

Section: Hierarchical Deformation Frameworkmentioning

confidence: 99%

Detailed Human Shape Estimation From a Single Image by Hierarchical Mesh Deformation

Zhu

Zuo

Wang

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

Self Cite

159

118

View full text Add to dashboard Cite

This paper presents a novel framework to recover detailed human body shapes from a single image. It is a challenging task due to factors such as variations in human shapes, body poses, and viewpoints. Prior methods typically attempt to recover the human body shape using a parametric based template that lacks the surface details. As such the resulting body shape appears to be without clothing. In this paper, we propose a novel learningbased framework that combines the robustness of parametric model with the flexibility of free-form 3D deformation. We use the deep neural networks to refine the 3D shape in a Hierarchical Mesh Deformation (HMD) framework, utilizing the constraints from body joints, silhouettes, and per-pixel shading information. We are able to restore detailed human body shapes beyond skinned models. Experiments demonstrate that our method has outperformed previous state-of-the-art approaches, achieving better accuracy in terms of both 2D IoU number and 3D metric distance. The code is available in https://github.com/ zhuhao-nju/hmd.git.

show abstract

“…Traditional depth sensors and 3D scanners suffer from the limited spatial resolution, so they can't recover detailed facial geometry. The sparse multi-view camera system suffers from the unstable and inaccurate reconstruction [18], [19], [20]. The drawbacks of these methods limit the quality of 3D face model in previous datasets.…”

Section: Related Workmentioning

confidence: 99%

FaceScape: 3D Facial Dataset and Benchmark for Single-View 3D Face Reconstruction

Zhu¹,

Yang²,

Guo³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

In this paper, we present a large-scale detailed 3D face dataset, FaceScape, and the corresponding benchmark to evaluate single-view facial 3D reconstruction. By training on FaceScape data, a novel algorithm is proposed to predict elaborate riggable 3D face models from a single image input. FaceScape dataset provides 18,760 textured 3D faces, captured from 938 subjects and each with 20 specific expressions. The 3D models contain the pore-level facial geometry that is also processed to be topologically uniformed. These fine 3D facial models can be represented as a 3D morphable model for rough shapes and displacement maps for detailed geometry. Taking advantage of the large-scale and high-accuracy dataset, a novel algorithm is further proposed to learn the expression-specific dynamic details using a deep neural network. The learned relationship serves as the foundation of our 3D face prediction system from a single image input. Different than the previous methods, our predicted 3D models are riggable with highly detailed geometry under different expressions. We also use FaceScape data to generate the in-the-wild and in-the-lab benchmark to evaluate recent methods of single-view face reconstruction. The accuracy is reported and analyzed on the dimensions of camera pose and focal length, which provides a faithful and comprehensive evaluation and reveals new challenges. The unprecedented dataset, benchmark, and code have been released to the public for research purpose 1 .

show abstract

The role of prior in image based 3D modeling: a survey

Cited by 20 publications

References 48 publications

Detailed Facial Geometry Recovery from Multi-View Images by Learning an Implicit Function

Detailed Facial Geometry Recovery from Multi-View Images by Learning an Implicit Function

Detailed Human Shape Estimation From a Single Image by Hierarchical Mesh Deformation

FaceScape: 3D Facial Dataset and Benchmark for Single-View 3D Face Reconstruction

Contact Info

Product

Resources

About