Semi-Supervised Face Frontalization in the Wild

Liang, Ruiyang; Chen, Xu; Xu, Xuexin; Hu, Guosheng; Zuo, Wangmeng; Hancock, Edwin R.

doi:10.1109/tifs.2020.3025412

Cited by 12 publications

(9 citation statements)

References 37 publications

(112 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Some of the best performing DNN-based frontalization methods use CNN-GAN architectures, e.g. Yin et al (2017); Huang et al (2017); Tran et al (2017); Zhao et al (2018); Zhang et al (2019); Rong et al (2020); Zhang et al (2021); Yin et al (2020), which outperform CNN-only models, e.g. Yim et al (2015).…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Expression-preserving face frontalization improves visually assisted speech processing

Kang,

Sadeghi,

Horaud

et al. 2022

Preprint

View full text Add to dashboard Cite

Face frontalization consists of synthesizing a frontally-viewed face from an arbitrarily-viewed one. The main contribution of this paper is a frontalization methodology that preserves non-rigid facial deformations in order to boost the performance of visually assisted speech communication. The method alternates between the estimation of (i) the rigid transformation (scale, rotation, and translation) and (ii) the nonrigid deformation between an arbitrarily-viewed face and a face model. The method has two important merits: it can deal with non-Gaussian errors in the data and it incorporates a dynamical face deformation model. For that purpose, we use the generalized Student tdistribution in combination with a linear dynamic system in order to account for both rigid head motions and time-varying facial deformations caused by speech production. We propose to use the zero-mean normalized cross-correlation (ZNCC) score to evaluate the ability of the method to preserve facial expressions. The method is thoroughly evaluated and compared with several state of the art methods, either based on traditional geometric models or on deep learning. Moreover, we show that the method, when incorporated into deep learning pipelines, namely lip reading and speech en-

show abstract

Section: Related Workmentioning

confidence: 99%

“…These methods necessitate large collections of input/output pairs of face images. For that purpose Zhang et al (2019); Yin et al (2020); Zhang et al (2021) use two datasets that contain multiple-camera recordings in a controlled setup, i.e. Gao et al (2007); Gross et al (2010).…”

Section: Related Workmentioning

confidence: 99%

Expression-preserving face frontalization improves visually assisted speech processing

Kang,

Sadeghi,

Horaud

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Some of the best performing DNN-based frontalization methods use CNN/GAN architectures, e.g. [54,27,49,59,57,43,58,55], which outperform CNNonly ones, e.g. [53].…”

Section: Related Workmentioning

confidence: 99%

“…These methods necessitate large collections of input/output pairs of face images. For that purpose [57,55,58] use two datasets that contain multiplecamera recordings in a controlled setup, i.e. [20,21].…”

Section: Related Workmentioning

confidence: 99%

“…[57] proposed to learn dense pixel-to-pixel correspondences between the two faces. Subsequently, [58] proposed a semisupervised GAN-based method that augments the paired face images of [21] with unpaired in-the-wild faces with large variations in identity, e.g. [26]; their adversarial and identity-preserving losses enhance face recognition performance.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Robust Face Frontalization For Visual Speech Recognition

Kang

Horaud

Sadeghi

2021

2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW)

View full text Add to dashboard Cite

Face frontalization consists of synthesizing a frontallyviewed face from an arbitrarily-viewed one. The main contribution of this paper is a robust frontalization method that preserves non-rigid facial deformations, i.e. expressions, to perform lip reading. The method iteratively estimates the rigid transformation (scale, rotation, and translation) and the non-rigid deformation between 3D landmarks extracted from an arbitrarily-viewed face, and 3D vertices parameterized by a deformable shape model. An important merit of the method is its ability to deal with large Gaussian and non-Gaussian errors in the data. For that purpose, we use the generalized Student-t distribution. The associated EM algorithm assigns a weight to each observed landmark, the higher the weight the more important the landmark, thus favoring landmarks that are only affected by rigid head movements. We propose to use the zero-mean normalized cross-correlation (ZNCC) score to evaluate the ability to preserve facial expressions. We show that the method, when incorporated into a deep lipreading pipeline, considerably improves the word classification score on an in-the-wild benchmark. An extended version with supplemental materials can be found at https: //team.inria.fr/robotlearn/rff-vsr/ * This work has been partially supported by MIAI@Grenoble Alpes (ANR-19-P3IA-0003).

show abstract