Generating Thermal Human Faces for Physiological Assessment using Thermal Sensor Auxiliary Labels

Abstract

Thermal images reveal medically important physiological information about human stress, signs of inflammation, and emotional mood that cannot be seen on visible images. Providing a method to generate thermal faces from visible images would be highly valuable for the telemedicine community in order to show this medical information. To the best of our knowledge, there are limited works on visible-to-thermal (VT) face translation, and many current works go the opposite direction to generate visible faces from thermal surveillance images (TV) for law enforcement applications. As a result, we introduce favtGAN, a VT GAN which uses the pix2pix image translation model with an auxiliary sensor label prediction network for generating thermal faces from visible images. Since most TV methods are trained on only one data source drawn from one thermal sensor, we combine datasets from faces and cityscapes. These combined data are captured from similar sensors in order to bootstrap the training and transfer learning task, especially valuable because visible-thermal face datasets are limited. Experiments on these combined datasets show that favtGAN demonstrates an increase in SSIM and PSNR scores of generated thermal faces, compared to training on a single face dataset alone.

When Visible-to-Thermal Facial GAN Beats Conditional Diffusion

“…For specifically facial VT translation, paired translation methods are used in order to preserve the mapping between the visible and thermal physiology of the subject, which is considered a biometric [25]. Examples include the favtGAN approach [13] where thermal faces are generated by modifying a PatchGAN discriminator [27] to learn auxiliary thermal sensor classes from a combination of different datasets. Mallat et al apply a Cascaded Refinement Network (CRN) [39] based on progressively upsampled feature maps [14,40].…”

“…2) CycleGAN [28], 3) ThermalGAN [16], and 4) favtGAN [13]). These baselines were selected based on their use in existing VT/TV studies such as [7,17,36,42].…”

“…These baselines were selected based on their use in existing VT/TV studies such as [7,17,36,42]. We combine the Eurecom and Devcom datasets together in order to train the favtgan [13] model that requires two datasets captured from similar thermal sensors. We compare the baselines against VTF-GAN and its variants VTF-GAN-FFT-P and VTF-GAN-FFT-G. Further, we trained pix2pixHD [63], FastCUT [32], GansNRoses [41], StarGAN [30], and MUNIT [31], but excluded the results due to distortion and failure to translate into the thermal modality, where results are shown in Figure 10.…”

“…The GAN framework has been applied successfully in both directions of Visible-to-Thermal (VT) and Thermal-to-Visible (TV) for person re-identification [2,3,4,5,6,7,8,9,10,11,12]. For facial VT translation, paired methods are preferable, in order to preserve the physiological mapping which is considered a biometric [13,14,15]. Yet, few works have explored how to estimate temperature from thermogram pixels where some works use metadata supplied by the camera, that demonstrate high quality results [16,17].…”

An Effective Sentiment Analysis of Social Media Data Using Deep Recurrent Neural Network Models

“…The next task is to identify COVID-19 related tweets using various machine learning algorithms. The classification algorithm used in the following article uses a sequential regression engine that predicts naming at the emotional level [6,7].Some researchers conclude that the regression problem of machine learning algorithms on Twitter data is superior to the problem of sentiment analysis [8]. Additionally, the following approach has done a lot of work to improve the results.…”