Synthesis of High-Quality Visible Faces from Polarimetric Thermal Faces using Generative Adversarial Networks

Abstract

The large domain discrepancy between faces captured in polarimetric (or conventional) thermal and visible domain makes cross-domain face verification a highly challenging problem for human examiners as well as computer vision algorithms. Previous approaches utilize either a twostep procedure (visible feature estimation and visible image reconstruction) or an input-level fusion technique, where different Stokes images are concatenated and used as a multichannel input to synthesize the visible image given the corresponding polarimetric signatures. Although these methods have yielded improvements, we argue that input-level fusion alone may not be sufficient to realize the full potential of the available Stokes images. We propose a Generative Adversarial Networks (GAN) based multi-stream featurelevel fusion technique to synthesize high-quality visible images from prolarimetric thermal images. The proposed network consists of a generator sub-network, constructed using an encoder-decoder network based on dense residual

Polarimetric Thermal to Visible Face Verification via Self-Attention Guided Synthesis

“…Figure 5 shows the evaluation performance for two different experimental settings, S0 (representing conventional thermal) and Polar separately. As can be seen from Figure 6, compared with the other state-of-the-art methods, the proposed method per-forms better and comparably to [36]. In addition, it can be observed that the performance corresponding to the Polar modality is always better than the S0 modality, which demonstrates the advantage of using the polarimetric thermal images than the conventional thermal images.…”

“…The proposed method is evaluated on the ARL Multimodal Face Database [10] which consists of polarimetric (i.e. Stokes image) and visible images from Volume I [10] and II [36]. The Volume I data consists of images corresponding to 60 subjects.…”

“…The Volume I data consists of images corresponding to 60 subjects. On the other hand, the Volume II data consists of images from 51 subjects (81 subjects in [26] 85.42% 82.49% 21.46% 26.25% AP-GAN [5] 88.93% ± 1.54% 84.16% ± 1.54% 19.02% ± 1.69% 23.90% ± 1.52% Multi-stream GAN [36] 96.03% 85.74% 11.78% 23.18% Ours 93.68% ± 0.97% 89.20% ± 1.56% 13.46% ± 1.92% 18.77% ± 1.36% I and 25 subjects' images from Volume II are used for training, the remaining 26 subjects' images from Volume II are used for evaluation. We repeat this process 5 times and report the average results.…”

“…Riggan et al [27] trained a regression network to estimate the mapping between features from both visible and thermal then reconstruct the visible face based on the estimated features. Zhang et al [35,36] leveraged generative adversarial networks (GANs) to synthesize visible images from polarimetric thermal images. Riggan et al [26] proposed a global and local region-based synthesis network to transform the thermal image into the visible spectrum.…”

“…In a recent work [5] Di et al developed an attribute preserved adversarial network called AP-GAN to enhance the quality of the synthesized images. Zhang et al [36] introduced a multi-stream dense-residual encoderdecoder network, which implemented a feature-level fusion techniques to solve the problem.…”