Facial landmark detection on thermal data via fully annotated visible-to-thermal data synthesis

Abstract

Thermal imaging has substantially evolved, during the recent years, to be established as a complement, or even occasionally as an alternative to conventional visible light imaging, particularly for face analysis applications. Facial landmark detection is a crucial prerequisite for facial image processing. Given the upswing of deep learning based approaches, the performance of facial landmark detection has been significantly improved. However, this uprise is merely limited to visible spectrum based face analysis tasks, as there are only few research works on facial landmark detection in thermal spectrum. This limitation is mainly due to the lack of available thermal face databases provided with full facial landmark annotations. In this paper, we propose to tackle this data shortage by converting existing face databases, designed for facial landmark detection task, from visible to thermal spectrum that will share the same provided facial landmark annotations. Using the synthesized thermal databases along with the facial landmark annotations, two different models are trained using active appearance models and deep alignment network. Evaluating the models trained on synthesized thermal data on real thermal data, we obtained facial landmark detection accuracy of 94.59% when tested on low quality thermal data and 95.63% when tested on high quality thermal data with a detection threshold of 0.15×IOD.

When Visible-to-Thermal Facial GAN Beats Conditional Diffusion

“…Examples include the favtGAN approach [13] where thermal faces are generated by modifying a PatchGAN discriminator [27] to learn auxiliary thermal sensor classes from a combination of different datasets. Mallat et al apply a Cascaded Refinement Network (CRN) [39] based on progressively upsampled feature maps [14,40]. Pavez et al generates a set of stylized thermal facial images using the GansNRoses [41] architecture for thermal facial recognition experiments [42], and [15] uses StyleGAN2 to generate random high resolution thermal faces that are not mapped to existing visible faces.…”

“…The GAN framework has been applied successfully in both directions of Visible-to-Thermal (VT) and Thermal-to-Visible (TV) for person re-identification [2,3,4,5,6,7,8,9,10,11,12]. For facial VT translation, paired methods are preferable, in order to preserve the physiological mapping which is considered a biometric [13,14,15]. Yet, few works have explored how to estimate temperature from thermogram pixels where some works use metadata supplied by the camera, that demonstrate high quality results [16,17].…”

TFLD: Thermal Face and Landmark Detection for Unconstrained Cross-spectral Face Recognition

“…In addition, face and landmark detection in the thermal spectrum was addressed by methods based on generative adversarial networks (GANs), aiming at translating facial images to the visible spectrum, then extracting facial key points and transferring the key points to the image of the original spectrum. Mallat et al [11] proposed converting existing visible face databases to the thermal spectrum. Active appearance models (AAMs) and DAN are then trained using the synthetic thermal, along with the shared landmark annotations.…”

“…We note that existing face and landmark detection algorithms, that were trained with visible face images, fail to generalize onto thermal images [7,14] due to the cross-spectral modality gap. At the same time, lack of available annotated thermal datasets is the primary cause for the scarcity of work focused on detection of thermal facial landmarks [8,11,16].…”