Infrared and visible image fusion with convolutional neural networks

Liu, Yu; Chen, Xun; Cheng, Juan; Peng, Hu; Wang, Zengfu

doi:10.1142/s0219691318500182

Cited by 320 publications

(169 citation statements)

References 34 publications

Supporting

Mentioning

169

Contrasting

Order By: Relevance

“…In this part, nine other fusion methods are recommended to compare the fusion results with ours, including curvelet transform (CVT) [71], dual-tree complex wavelet transform (DTCWT) [72], Laplacian pyramid (LP) [73], nonsubsampled contourlet transform (NSCT) [74], two-scale image fusion based on visual saliency (TSIFVS) [75], guided filtering based fusion (GFF) [76], and convolutional neural network based fusion (CNN) [24], dense block based fusion [55] which includes Dense-add and Dense-L1 according to the different fusion strategies, and a GAN based method (FusionGAN) [26]. We used the codes provided by the authors or a well-known toolbox to generate the fused images from source image pairs except for the last method which cannot get the ideal outputs directly.…”

Section: Fusion Resultsmentioning

confidence: 99%

“…In the article, the authors proposed a CNN based siamese network for multi-focus image fusion and extended it for infrared and visible image fusion. Later, considering the different imaging modalities, they combined the image pyramid decomposition and a local similarity strategy [24] with the former siamese network to fuse infrared and visible images. The siamese network consisted of two branches and the weights were constrained to the same.…”

Section: Dl-based Fusion Methodsmentioning

confidence: 99%

“…DL-based image fusion methods can extract deep features automatically to overcome the difficulties caused by manual measurements and assignments, making the whole fusion process become easier. Liu et al [23,24] first proposed the methods to use convolutional neural networks (CNN) of DL for infrared and visible image fusion. Last year, Li et al [2,25] presented some other DL frameworks to fuse infrared and visible images.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Infrared and Visible Image Fusion with a Generative Adversarial Network and a Residual Network

Wang

et al. 2020

Applied Sciences

View full text Add to dashboard Cite

Infrared and visible image fusion can obtain combined images with salient hidden objectives and abundant visible details simultaneously. In this paper, we propose a novel method for infrared and visible image fusion with a deep learning framework based on a generative adversarial network (GAN) and a residual network (ResNet). The fusion is accomplished with an adversarial game and directed by the unique loss functions. The generator with residual blocks and skip connections can extract deep features of source image pairs and generate an elementary fused image with infrared thermal radiation information and visible texture information, and more details in visible images are added to the final images through the discriminator. It is unnecessary to design the activity level measurements and fusion rules manually, which are now implemented automatically. Also, there are no complicated multi-scale transforms in this method, so the computational cost and complexity can be reduced. Experiment results demonstrate that the proposed method eventually gets desirable images, achieving better performance in objective assessment and visual quality compared with nine representative infrared and visible image fusion methods.

show abstract

Section: Fusion Resultsmentioning

confidence: 99%

Section: Dl-based Fusion Methodsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Infrared and Visible Image Fusion with a Generative Adversarial Network and a Residual Network

Wang

et al. 2020

Applied Sciences

View full text Add to dashboard Cite

show abstract

“…We choose six representative multimodal image fusion algorithms to implement performance comparison. The six image fusion methods are the DTCWT-based method (Lewis et al, 2007), the NSST-based method (Gao et al, 2013), the convolutional neural network (CNN)based method (Liu, Chen, Cheng, Peng, & Wang, 2018), the SFL-CTbased method (Feng et al, 2013), the CST-based method (Qiu et al, 2018), and the generative adversarial network (GAN)-based method (Tang et al, 2019). The parameters of these six fusion methods are set to the default values as reported in their original publications to conduct unbiased comparison.…”

Section: Compared Methodsmentioning

confidence: 99%

A phase congruency‐based green fluorescent protein and phase contrast image fusion method in nonsubsampled shearlet transform domain

Tang

Liu

Cheng

et al. 2020

Microscopy Res & Technique

Self Cite

View full text Add to dashboard Cite

Image fusion technique is an effective way to merge the information contained in different imaging modalities by generating a more informative composite image. Fusion of green fluorescent protein (GFP) and phase contrast images is of great significance to the subcellular localization, the functional analysis of protein, and the expression of gene. In this article, a phase congruency (PC)‐based GFP and phase contrast image fusion method in nonsubsampled shearlet transform (NSST) domain is presented. The input images are decomposed by the NSST to acquire the multiscale and multidirection representations. The high‐frequency coefficients are fused with a strategy based on PC and parameter‐adaptive pulse coupled neural network (PA‐PCNN), while the low‐frequency coefficients are integrated through a local energy (LE)‐based rule. Finally, the fused image is generated by conducting the inverse NSST on the merged high‐ and low‐frequency coefficients. Experimental results illustrate that the presented method outperforms several state‐of‐the‐art GFP and phase contrast image fusion algorithms on both qualitative and quantitative assessments.

show abstract

“…Although CNN based RGB-T salient object detection algorithms are not well investigated yet, a large number of deep neural networks with RGB-T inputs have been presented for some other computer vision or image processing tasks, such as pedestrian detection [36]- [38], image fusion [50], object tracking [51]- [53]. For example, Wagner et al [37] presented an RGB-T pedestrian detection method by fusing information with CNNs, where information from the RGB and thermal infrared images was integrated via an earlyfusion and a late-fusion based CNN architecture.…”

Section: Rgb-t Salient Object Detectionmentioning

confidence: 99%

RGB-T Salient Object Detection via Fusing Multi-Level CNN Features

Zhang

Huang

Yang

et al. 2020

IEEE Trans. on Image Process.

169

View full text Add to dashboard Cite

RGB-induced salient object detection has recently witnessed substantial progress, which is attributed to the superior feature learning capability of deep convolutional neural networks (CNNs). However, such detections suffer from challenging scenarios characterized by cluttered backgrounds, low-light conditions and variations in illumination. Instead of improving RGB based saliency detection, this paper takes advantage of the complementary benefits of RGB and thermal infrared images. Specifically, we propose a novel end-to-end network for multimodal salient object detection, which turns the challenge of RGB-T saliency detection to a CNN feature fusion problem. To this end, a backbone network (e.g., VGG-16) is first adopted to extract the coarse features from each RGB or thermal infrared image individually, and then several adjacent-depth feature combination (ADFC) modules are designed to extract multi-level refined features for each single-modal input image, considering that features captured at different depths differ in semantic information and visual details. Subsequently, a multi-branch group fusion (MGF) module is employed to capture the crossmodal features by fusing those features from ADFC modules for a RGB-T image pair at each level. Finally, a joint attention guided bi-directional message passing (JABMP) module undertakes the task of saliency prediction via integrating the multi-level fused features from MGF modules. Experimental results on several public RGB-T salient object detection datasets demonstrate the superiorities of our proposed algorithm over the state-of-the-art approaches, especially under challenging conditions, such as poor illumination, complex background and low contrast.

show abstract

Infrared and visible image fusion with convolutional neural networks

Cited by 320 publications

References 34 publications

Infrared and Visible Image Fusion with a Generative Adversarial Network and a Residual Network

Infrared and Visible Image Fusion with a Generative Adversarial Network and a Residual Network

A phase congruency‐based green fluorescent protein and phase contrast image fusion method in nonsubsampled shearlet transform domain

RGB-T Salient Object Detection via Fusing Multi-Level CNN Features

Contact Info

Product

Resources

About