Nighttime Thermal Infrared Image Colorization with Dynamic Label Mining

Abstract

Stable imaging in adverse environments (e.g., total darkness) makes thermal infrared (TIR) cameras a prevalent option for night scene perception. However, the low contrast and lack of chromaticity of TIR images are detrimental to human interpretation and subsequent deployment of RGB-based vision algorithms. Therefore, it makes sense to colorize the nighttime TIR images by translating them into the corresponding daytime color images (NTIR2DC). Despite the impressive progress made in the NTIR2DC task, how to improve the translation performance of small object classes is under-explored.To address this problem, we propose a generative adversarial network incorporating feedback-based object appearance learning (FoalGAN). Specifically, an occlusion-aware mixup module and corresponding appearance consistency loss are proposed to reduce the context dependence of object translation. As a representative example of small objects in nighttime street scenes, we illustrate how to enhance the realism of traffic light by designing a traffic light appearance loss. To further improve the appearance learning of small objects, we devise a dual feedback learning strategy to selectively adjust the learning frequency of different samples. In addition, we provide pixel-level annotation for a subset of the Brno dataset, which can facilitate the research of NTIR image understanding under multiple weather conditions. Extensive experiments illustrate that the proposed FoalGAN is not only effective for appearance learning of small objects, but also outperforms other image translation methods in terms of semantic preservation and edge consistency for the NTIR2DC task.

Memory-Guided Collaborative Attention for Nighttime Thermal Infrared Image Colorization

“…PearlGAN [9] was proposed to reduce semantic encoding entanglement and geometric distortion in the NTIR2DC task. DlamGAN [10] was designed with a dynamic label mining module to predict the semantic masks of NTIR images to encourage semantically consistent colorization. Despite the impressive progress, few efforts have been made to improve the colorization performance of small sample objects.…”

“…To reduce content distortion during translation, many researchers [5], [6], [8] have introduced semantic consistency loss using the available semantic annotations. When no semantic annotation is available for both domains, DlamGAN [10] first predicts the pseudolabels of one domain using domain adaptation, and then introduces a dynamic label mining module to obtain the pseudo-labels of the other domain. Although semantic consistency loss can significantly reduce the semantic distortion during translation, the edges within or between classes of the background category are usually smoothed or disappear to enhance the realism of the image patches.…”

“…Therefore, similar to [5], [6], we introduce auxiliary segmentation networks (i.e, S A and S B ) and segmentation masks to encourage semantically consistent I2I translation. Due to the lack of manual annotation of both domains, similar to DlamGAN [10], we first obtain the pseudo-labels of domain A using the existing DC image segmentation model, and then design a label mining module to predict the pseudolabels of NTIR images online. Unlike DlamGAN, our proposed approach uses an online semantic distillation module that not only exploits different thresholds to balance the distribution bias among categories, but also utilizes the variation in temperature distribution among categories to reduce high-confidence noise.…”

“…As shown in Fig. 1, the popular NTIR2DC methods (e.g., PearlGAN [9] and DlamGAN [10]) 1 fail to generate plausible pedestrians, as shown in the white dashed boxes. Moreover, the gradients of some trunk regions in their colorization results are vanishing, as shown in the red boxes.…”

“…Moreover, the gradients of some trunk regions in their colorization results are vanishing, as shown in the red boxes. To address the above problems, we Input NTIR Image GT Mask (a) PearlGAN [9] (b) DlamGAN [10] (c) Proposed propose a Memory-guided cOllaboRative atteNtion GAN model (MornGAN).…”