Matching Thermal to Visible Face Images Using a Semantic-Guided Generative Adversarial Network

Abstract

Designing face recognition systems that are capable of matching face images obtained in the thermal spectrum with those obtained in the visible spectrum is a challenging problem. In this work, we propose the use of semantic-guided generative adversarial network (SG-GAN) to automatically synthesize visible face images from their thermal counterparts. Specifically, semantic labels, extracted by a face parsing network, are used to compute a semantic loss function to regularize the adversarial network during training. These semantic cues denote high-level facial component information associated with each pixel. Further, an identity extraction network is leveraged to generate multi-scale features to compute an identity loss function. To achieve photo-realistic results, a perceptual loss function is introduced during network training to ensure that the synthesized visible face is perceptually similar to the target visible face image. We extensively evaluate the benefits of individual loss functions, and combine them effectively to learn the mapping from thermal to visible face images. Experiments involving two multispectral face datasets show that the proposed method achieves promising results in both face synthesis and cross-spectral face matching.

Generating Thermal Human Faces for Physiological Assessment using Thermal Sensor Auxiliary Labels

“…Full symbol notation is in the Approach. 11,12,13,14,15,16]. Further, the TV GAN approaches we identified are trained on one single dataset collected by a single thermal sensor [1,2,3,4,5,6,7,11,12].…”

ET-CycleGAN: Generating Thermal Images from Images in the Visible Spectrum for Facial Emotion Recognition

“…To the best of our knowledge, none of the methods in the literature deal with the generation of synthesized thermal face images for facial emotion recognition. Our proposal is inspired by [27] and [4], in the way they include networks to compute losses during the training of the GAN.…”

“…(4) where ϕ F denotes the features extracted from multiple layers of the VGG-19 network pre-trained on the VGGFace2 [3] as used in [4]. L F ensures that the synthesized visual image in the cycle training contains facial features that are similar to the ground-truth image.…”

“…For the facial network we used a VGG-19 pre-trained with the VGGFace2 dataset. As in [4], the extracted features are obtained from the relu1-1, relu2-1, relu3-1, relu4-1, and relu5-1 layers. For the emotion network we used a ResNet-50 pre-trained with the VGGFace2, the FER2013 [8], and the USTC-NVIE-posed.…”

Explainable Thermal to Visible Face Recognition Using Latent-Guided Generative Adversarial Network

“…Recent work on thermal-to-visible face recognition has predominantly focused on synthesizing visible faces from their thermal counterparts by generative adversarial networks, in order to minimize the spectral difference [18], [20], [17], [1], [4]. Cross-spectral identity matches were implicitly enforced by minimizing the reconstruction error [1], or by using an identity extraction network [18], [1], [4]. However, to the best of our knowledge, there is no prior thermal-tovisible generative adversarial network (GAN) that explicitly encodes identity.…”