Thermal infrared image colorization on nighttime driving scenes with spatially-correlated learning

Abstract

Despite recent promising advances in thermal infrared image colorization for nighttime driving scenes, the content structure preservation and fidelity of colorized images remain ineffectively addressed. Therefore, we introduce a dynamic spatial structure guidance to automatically learn the task-oriented structure representation in the unpaired thermal infrared image colorization of nighttime driving scenes. First, patch-based spatial correlation maps are constructed for the nighttime thermal infrared image and the color image. The content structure preservation during thermal infrared image colorization is realized by reflecting the patch-based spatial structural correlations between the input and output images. To flexibly utilize the spatial correlations, structural representation learning was constructed to learn task-specific structural characterization. We innovatively treat the spatial structural similarity between the corresponding image patches as an interest metric to guide contrastive learning to optimize structural representation learning. Compared with state-of-theart methods, our model achieved an advanced thermal infrared image colorization performance. Additionally, we performed semantic segmentation and object detection tasks on the colorization to further address the practicality. Extensive qualitative and quantitative experiments and evaluations demonstrate that our method achieves state-of-the-art results in the nighttime thermal infrared colorization.