Infrared and Visible Image Fusion Using a Deep Unsupervised Framework With Perceptual Loss

Abstract

The fusion of infrared and visible images can utilize the indication characteristics and the textural details of source images to realize the all-weather detection. The deep learning (DL) based fusion solutions can reduce the computational cost and complexity compared with traditional methods since there is no need to design complex feature extraction methods and fusion rules. There are no standard reference images and the publicly available infrared and visible image pairs are scarce. Most supervised DL-based solutions have to take pre-training on other labeled large datasets which may not behave well when testing. The few unsupervised fusion methods can hardly obtain ideal images with good visual impression. In this paper, an infrared and visible image fusion method based on unsupervised convolutional neural network is proposed. When designing the network structure, densely connected convolutional network (DenseNet) is used as the sub-network for feature extraction and reconstruction to ensure that more information of source images can be retained in the fusion images. As to loss function, the perceptual loss is creatively introduced and combined with the structure similarity loss to constrain the updating of weight parameters during the back propagation. The perceptual loss designed helps to improve the visual information fidelity (VIF) of the fusion image effectively. Experimental results show that this method can obtain fusion images with prominent targets and obvious details. Compared with other 7 traditional and deep learning methods, the fusion results of this method are better on objective evaluation and visual observation when taken together.INDEX TERMS Infrared and visible images, deep learning, unsupervised image fusion, densely connected convolutional network, perceptual loss.

Infrared and Visible Image Fusion Method Using Salience Detection and Convolutional Neural Network

“…In our earlier reading of the literature, a large number of papers [ 2 , 3 , 4 , 5 , 9 , 11 , 12 , 13 , 14 , 15 , 16 , 19 , 25 , 26 , 27 , 28 , 29 , 30 ] used the TNO dataset for the training and testing of the model. Therefore, the same TNO dataset is used in this paper.…”

“…Therefore, fusion performance analysis based on quantitative evaluation is essential and complements the subjective evaluation. Drawing on the evaluation metrics found in most fusion papers [ 3 , 4 , 5 , 11 , 12 , 14 , 15 , 16 , 25 , 26 , 27 ], four mainstream evaluation metrics have been selected for this paper, including the Peak Signal Noise Ratio (PSNR) [ 26 ], Structure Similarity Index (SSIM) [ 4 , 12 , 15 , 16 , 27 ], Spatial Frequency (SF) [ 3 , 5 , 11 , 15 , 25 ] and Mutual Information (MI) [ 3 , 4 , 5 , 14 , 16 ]. They are defined as follows: where z represents the difference between the maximum and minimum greyscale values of the ideal reference image, usually 255.…”

“…Multiscale transforms [ 8 ] can be subdivided into pyramid transforms, wavelet transforms, non-sampled contour transforms, etc. The sparse representation [ 11 ] is a super-complete dictionary learned from a large number of natural images and sparsely encoded to obtain sparse representation coefficients so that the source image is sparsely represented by the learned super-complete dictionary. Subspace is the projection of a high-dimensional input image into a low-dimensional space (inter-subspace).…”

“…However, with a wide range of infrared and visible image fusion tasks, a prominent target mask does not solve all problems once and for all. In infrared and visible image fusion tasks, people’s vision is more sensitive to salient targets, which are quickly noticed and have a very high salience [ 11 , 12 ]. With significance detection, object representations in images can be processed automatically.…”

A Multi-Stage Visible and Infrared Image Fusion Network Based on Attention Mechanism

“…In contrast to the “reconstruction training” mode, the feature fusion part is usually designed as dual-light feature concatenation, and the interaction is implicitly implemented in the subsequent image reconstruction process [ 11 , 14 , 21 ]. In addition, for the GAN commonly used in this mode, the fusion model is always trained simultaneously with the designed discriminators [ 10 , 12 , 22 , 23 , 24 , 25 ]. In the process of adversarial gaming, it simultaneously improves the respective capabilities of both sides to achieve the improvement of the fusion effect.…”