Infrared and Visible Image Fusion with a Generative Adversarial Network and a Residual Network

Abstract

Infrared and visible image fusion can obtain combined images with salient hidden objectives and abundant visible details simultaneously. In this paper, we propose a novel method for infrared and visible image fusion with a deep learning framework based on a generative adversarial network (GAN) and a residual network (ResNet). The fusion is accomplished with an adversarial game and directed by the unique loss functions. The generator with residual blocks and skip connections can extract deep features of source image pairs and generate an elementary fused image with infrared thermal radiation information and visible texture information, and more details in visible images are added to the final images through the discriminator. It is unnecessary to design the activity level measurements and fusion rules manually, which are now implemented automatically. Also, there are no complicated multi-scale transforms in this method, so the computational cost and complexity can be reduced. Experiment results demonstrate that the proposed method eventually gets desirable images, achieving better performance in objective assessment and visual quality compared with nine representative infrared and visible image fusion methods.

Infrared and visible image fusion based on multi‐channel convolutional neural network

“…This is also the reason why few scholars use infrared and visible image pairs to train deep learning models. Referring to the ideas of Li et al [20] and Xu et al [34], 566 groups of images are processed by clipping operation to extend data sets. Namely, each image is divided into several image blocks with the same size (128 × 128 in our method).…”

FuseVis: Interpreting Neural Networks for Image Fusion Using Per-Pixel Saliency Visualization

“…[53] went a step further and defined a trainable fusion layer. Refs [54][55][56] applied the GAN approach for fusing infrared and visible images using end-to-end neural networks. Ref.…”

Double-channel cascade-based generative adversarial network for power equipment infrared and visible image fusion

“…According to the subjective evaluation, it can be seen that the fusion results only pay attention to the visible light information and lose the thermal target information of the infrared image. Therefore, in order to retain more effective information of the source image and the correlation between the source image and the fusion image, the content loss function designed in this paper includes gradient loss and similarity loss, namely, Since the thermal radiation information of the infrared image is characterized by its pixel intensity and the texture detail information of the visible image is characterized by its gradient [23], the image intensity and gradient are calculated and can be expressed as: In the design of content loss function, the idea of reference [24] is introduced. Image structural similarity is an image quality measure that calculates the differences between images.…”