Learning Flow-Based Feature Warping for Face Frontalization with Illumination Inconsistent Supervision

Wei, Yuxiang; Liu, Ming; Wang, Haolin; Zhu, Ruifeng; Hu, Guosheng; Zuo, Wangmeng

doi:10.1007/978-3-030-58610-2_33

Cited by 24 publications

(13 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Image animation: Supervised methods [23,24,11,27,29,55,60,19,5,30,56,36,26] focus on the animation of a specific object type. Among these, the human body [22,29,13,54,31,6,1,33,52,14] and human face [11,12,28,51,46,48,3,41,47,16] are the most popular animation objects. Methods of this kind rely on object-specific landmark detectors, 3D models or other forms of supervision, which are usually pre-trained on a large amount of labeled data.…”

Section: Related Workmentioning

confidence: 99%

Motion Transformer for Unsupervised Image Animation

Tao¹,

Wang²,

Ge³

et al. 2022

Preprint

View full text Add to dashboard Cite

Image animation aims to animate a source image by using motion learned from a driving video. Current state-of-the-art methods typically use convolutional neural networks (CNNs) to predict motion information, such as motion keypoints and corresponding local transformations. However, these CNN based methods do not explicitly model the interactions between motions; as a result, the important underlying motion relationship may be neglected, which can potentially lead to noticeable artifacts being produced in the generated animation video. To this end, we propose a new method, the motion transformer, which is the first attempt to build a motion estimator based on a vision transformer. More specifically, we introduce two types of tokens in our proposed method: i) image tokens formed from patch features and corresponding position encoding; and ii) motion tokens encoded with motion information. Both types of tokens are sent into vision transformers to promote underlying interactions between them through multi-head self attention blocks. By adopting this process, the motion information can be better learned to boost the model performance. The final embedded motion tokens are then used to predict the corresponding motion keypoints and local transformations. Extensive experiments on benchmark datasets show that our proposed method achieves promising results to the state-of-the-art baselines. Our source code will be public available.

show abstract

Section: Related Workmentioning

confidence: 99%

Motion Transformer for Unsupervised Image Animation

Tao¹,

Wang²,

Ge³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…With the publication of TP-GAN (Two Pathways Generative Adversarial Networks) and CR-GAN (Complete Representations Generative Adversarial Networks), the research on face frontalization shifted into conditional-GANs [10], [22]. Since the high achievement in image translation by Pix2Pix (Pixel to Pixel Generative Adversarial Networks) and CycleGAN (Cycle-Consistent Adversarial Networks) [4], [5], recent publications in face frontalization were based on these two frameworks [3], [6], [11], [28]. The current state-of-the-art model in face frontaliztaion using the Color FRET database is Pairwise-GAN (Pairwise Generative Adversarial Networks), constructed by pair generators and a PatchGAN as the discriminator [3].…”

Section: B Face Frontalizationmentioning

confidence: 99%

“…Two base frameworks in facial frontalization from previous research are Pix2Pix (Pixel to Pixel generative adversarial networks) and Cycle-GAN (Cycle-Consistent generative adversarial networks) [4], [5]. The current state-of-the-art models are Pairwise-GAN and FFWM (Flow-based Feature Warping Models), and use different databases [3], [6]. We selected Pix2Pix and Pairwise-GAN for our analysis.…”

Section: Introductionmentioning

confidence: 99%

Exploring Biases and Prejudice of Facial Synthesis via Semantic Latent Space

Shen

Plested

Caldwell

et al. 2021

Preprint

View full text Add to dashboard Cite

Deep learning (DL) models are widely used to provide a more convenient and smarter life. However, biased algorithms will negatively influence us. For instance, groups targeted by biased algorithms will feel unfairly treated and even fearful of negative consequences of these biases. This work targets biased generative models' behaviors, identifying the cause of the biases and eliminating them. We can (as expected) conclude that biased data causes biased predictions of face frontalization models. Varying the proportions of male and female faces in the training data can have a substantial effect on behavior on the test data: we found that the seemingly obvious choice of 50:50 proportions was not the best for this dataset to reduce biased behavior on female faces, which was 71% unbiased as compared to our top unbiased rate of 84%. Failure in generation and generating incorrect gender faces are two behaviors of these models. In addition, only some layers in face frontalization models are vulnerable to biased datasets. Optimizing the skipconnections of the generator in face frontalization models can make models less biased. We conclude that it is likely to be impossible to eliminate all training bias without an unlimited size dataset, and our experiments show that the bias can be reduced and quantified. We believe the next best to a perfect unbiased predictor is one that has minimized the remaining known bias.

show abstract

“…II. RELATED WORK Face Frontalization: Recent face frontalization methods utilize either 2D/3D warping [12], [51], [20], [51], [50], stochastic modeling [38], [2], [1] or the generative models [3], [39], [16], [17], [41], [42], [49], [40], [48], [21]. For instance, Hassner et al [12] proposed a single unmodified 3D surface model for frontalization.…”

Section: Introductionmentioning

confidence: 99%

“…Yin et al [42] proposed a selfattention-based generator is to integrate local features with their long-range dependencies for obtaining better frontalized faces. Wei et al [40] proposed the FFWM model to overcome the illumination issue via flow-based feature warping. Thermal-to-visible Synthesis: Various approaches have been proposed in the literature for thermal-to-visible face synthesis and matching [7], [14], [13], [43], [44], [47], [29], [27], [5], [37], [36], [6], [34], [18].…”

Section: Introductionmentioning

confidence: 99%

Heterogeneous Face Frontalization via Domain Agnostic Learning

Di,

Hu,

Patel

2021

Preprint

View full text Add to dashboard Cite

Recent advances in deep convolutional neural networks (DCNNs) have shown impressive performance improvements on thermal to visible face synthesis and matching problems. However, current DCNN-based synthesis models do not perform well on thermal faces with large pose variations. In order to deal with this problem, heterogeneous face frontalization methods are needed in which a model takes a thermal profile face image and generates a frontal visible face. This is an extremely difficult problem due to the large domain as well as large pose discrepancies between the two modalities. Despite its applications in biometrics and surveillance, this problem is relatively unexplored in the literature. We propose a domain agnostic learning-based generative adversarial network (DAL-GAN) which can synthesize frontal views in the visible domain from thermal faces with pose variations. DAL-GAN consists of a generator with an auxiliary classifier and two discriminators which capture both local and global texture discriminations for better synthesis. A contrastive constraint is enforced in the latent space of the generator with the help of a dual-path training strategy, which improves the feature vector's discrimination. Finally, a multi-purpose loss function is utilized to guide the network in synthesizing identitypreserving cross-domain frontalization. Extensive experimental results demonstrate that DAL-GAN can generate better quality frontal views compared to the other baseline methods.

show abstract

Learning Flow-Based Feature Warping for Face Frontalization with Illumination Inconsistent Supervision

Cited by 24 publications

References 31 publications

Motion Transformer for Unsupervised Image Animation

Motion Transformer for Unsupervised Image Animation

Exploring Biases and Prejudice of Facial Synthesis via Semantic Latent Space

Heterogeneous Face Frontalization via Domain Agnostic Learning

Contact Info

Product

Resources

About