Towards a Complete 3D Morphable Model of the Human Head

Ploumpis, Stylianos; Ververas, Evangelos; Sullivan, Eimear O’; Moschoglou, Stylianos; Wang, Haoyang; Pears, Nick; Smith, William A. P.; Geçer, Barış; Zafeiriou, Stefanos

doi:10.1109/tpami.2020.2991150

Cited by 85 publications

(71 citation statements)

References 65 publications

(109 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In Ploumpis et al (2019Ploumpis et al ( , 2020, a high-definition head and face model is created by blending together the Liverpool-York Head model (LYHM) (Dai et al 2017) and the Large-Scale Face Model (LSFM) (Booth et al 2018a). While LYHM includes a facial region, replacing it with LSFM offers more details.…”

Section: Part-based Modelsmentioning

confidence: 99%

Shape My Face: Registering 3D Face Scans by Surface-to-Surface Translation

et al. 2021

Self Cite

View full text Add to dashboard Cite

Standard registration algorithms need to be independently applied to each surface to register, following careful pre-processing and hand-tuning. Recently, learning-based approaches have emerged that reduce the registration of new scans to running inference with a previously-trained model. The potential benefits are multifold: inference is typically orders of magnitude faster than solving a new instance of a difficult optimization problem, deep learning models can be made robust to noise and corruption, and the trained model may be re-used for other tasks, e.g. through transfer learning. In this paper, we cast the registration task as a surface-to-surface translation problem, and design a model to reliably capture the latent geometric information directly from raw 3D face scans. We introduce Shape-My-Face (SMF), a powerful encoder-decoder architecture based on an improved point cloud encoder, a novel visual attention mechanism, graph convolutional decoders with skip connections, and a specialized mouth model that we smoothly integrate with the mesh convolutions. Compared to the previous state-of-the-art learning algorithms for non-rigid registration of face scans, SMF only requires the raw data to be rigidly aligned (with scaling) with a pre-defined face template. Additionally, our model provides topologically-sound meshes with minimal supervision, offers faster training time, has orders of magnitude fewer trainable parameters, is more robust to noise, and can generalize to previously unseen datasets. We extensively evaluate the quality of our registrations on diverse data. We demonstrate the robustness and generalizability of our model with in-the-wild face scans across different modalities, sensor types, and resolutions. Finally, we show that, by learning to register scans, SMF produces a hybrid linear and non-linear morphable model. Manipulation of the latent space of SMF allows for shape generation, and morphing applications such as expression transfer in-the-wild. We train SMF on a dataset of human faces comprising 9 large-scale databases on commodity hardware.

show abstract

Section: Part-based Modelsmentioning

confidence: 99%

Shape My Face: Registering 3D Face Scans by Surface-to-Surface Translation

et al. 2021

Self Cite

View full text Add to dashboard Cite

show abstract

“…In [17], Ploumpis proposes a pipeline for combining current 3DMM of the human head to face and some other parts. He uses a regressor to accomplish missing parts of one model, the Gaussian Process framework is applied to blend covariance matrices from multiple models.…”

Section: Related Workmentioning

confidence: 99%

“…The component of the facial organs (including eyes, teeth, et al) have been accomplished manually. The texture of the iris is received by reshaping the largest ellipse inside the projection of the eye region to the most frontal input face image [17]. The main process of the grid expansion is shown in Fig.…”

Section: Mapping Optimizationmentioning

confidence: 99%

Full Face-and-Head 3D Model With Photorealistic Texture

Fan

Liu²,

et al. 2020

IEEE Access

View full text Add to dashboard Cite

In the recent period, significant progress has been achieved towards reconstructing the 3D face model from face image. With the support of the render engines and sufficient data, the reconstruction results are fine in detail. Nevertheless, the research on the 3D face reconstruction with texture from a single unrestricted face image is imperfect. The rebuild process lacks essential structure and texture information in the profile and the craniofacial region. To address this problem, we present a method of creating a 3D full face-and-head model with photorealistic texture from a single "in-the-wild" face image in this paper. To this end, we introduce a pipeline to integrate the highly-detailed face model into the basic model. Specifically, the basic model was built by multilinear optimization, and the highly-detailed face model which represents the facial features generated by constrained illumination distribution. Additionally, to infer the invisible region texture information corresponding to the input face image, we design an effective architecture with the generative adversarial network (GAN) for panoramic UV texture generation. The final results after UV texture mapping were visualized in the experiment, which demonstrates that the model faithfully recovers the photorealistic details in arbitrary perspective. Furthermore, compared to the state-of-the-art facial modeling techniques and existing commercial solutions, our method takes less input and performs better in surface detail. INDEX TERMS 3D face reconstruction, full face-and-head model, 3D Morphable Model, UV texture, Generative Adversarial Networks.

show abstract

“…While a number of datasets for the human ear soft tissue exist [18,27,31], there are, to the best of our knowledge, none that capture the paired relationship between ear cartilage and soft tissue. To address this, we construct a database of these paired samples.…”

Section: Datasetmentioning

confidence: 99%

“…Soft tissue meshes were rigidly aligned with a soft tissue ear template mesh. The template mesh used was a non-watertight triangular mesh, consisting of 2800 vertices, and was the same as that used in [27]. The same rigid transformation was applied to the cartilage, to maintain the spatial relationship between paired cartilage and soft tissue meshes.…”

Section: Datasetmentioning

confidence: 99%

Ear Cartilage Inference for Reconstructive Surgery with Convolutional Mesh Autoencoders

Sullivan

Lande

Osolos

et al. 2020

Medical Image Computing and Computer Assisted Intervention – MICCAI 2020

Self Cite

View full text Add to dashboard Cite

Many children born with ear microtia undergo reconstructive surgery for both aesthetic and functional purposes. This surgery is a delicate procedure that requires the surgeon to carve a "scaffold" for a new ear, typically from the patient's own rib cartilage. This is an unnecessarily invasive procedure, and reconstruction relies on the skill of the surgeon to accurately construct a scaffold that best suits the patient based on limited data. Work in stem-cell technologies and bioprinting present an opportunity to change this procedure by providing the opportunity to "bioprint" a personalised cartilage scaffold in a lab. To do so, however, a 3D model of the desired cartilage shape is first required. In this paper we optimise the standard convolutional mesh autoencoder framework such that, given only the soft tissue surface of an unaffected ear, it can accurately predict the shape of the underlying cartilage. To prevent predicted cartilage meshes from intersecting with, and protruding through, the soft tissue ear mesh, we develop a novel intersectionbased loss function. These combined efforts present a means of designing personalised ear cartilage scaffold for use in reconstructive ear surgery.

show abstract

Towards a Complete 3D Morphable Model of the Human Head

Cited by 85 publications

References 65 publications

Shape My Face: Registering 3D Face Scans by Surface-to-Surface Translation

Shape My Face: Registering 3D Face Scans by Surface-to-Surface Translation

Full Face-and-Head 3D Model With Photorealistic Texture

Ear Cartilage Inference for Reconstructive Surgery with Convolutional Mesh Autoencoders

Contact Info

Product

Resources

About