Visible and Infrared Image Fusion Using Deep Learning

Zhang, Xingchen; Demiris, Yiannis

doi:10.1109/tpami.2023.3261282

Cited by 47 publications

(19 citation statements)

References 190 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…I MAGE fusion plays an important role in the field of image process [1]. The definition of image fusion is to integrate multiple images acquired from the same or different modalities into one single image which carries all complementary information from all images [2]. The fused image is more informative and accurate than any of the source images.…”

Section: Introductionmentioning

confidence: 99%

“…Infrared images capture thermal radiation from objects, which are robust to illumination and all weather conditions. Targets can usually be clearly distinguished from the background in infrared images, as indicated by the high gray value [2]. However, infrared images often have low resolution and poor textures due to the limitations of imaging sensors.On another hand, visible images provide high-resolution and enriched texture details, making them suitable for human visualization.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Two-Stream Edge-Aware Network for Infrared and Visible Image Fusion With Multi-Level Wavelet Decomposition

Wang,

Shu,

et al. 2024

IEEE Access

View full text Add to dashboard Cite

Infrared and visible image fusion (IVIF) aims to generate a fused image with both salient target and rich textures from two different complementary modality images. To better integrate valuable edge information into the fused image, we first propose a novel two-stream network based on Auto-Encoder (AE) framework, which extracts deep hierarchical detail information at coarse scale from base stream by multi-level wavelet decomposition progressively and incorporates them into detail stream for information compensation. The aggregation of edge information ranging from coarse to fine facilitates a more comprehensive representation of contours and textures. Then, we propose a new feature fusion strategy, termed as Structural Feature Map Decomposition (SFMD). The first step is to decompose local patches of feature map with each modality into three independent components by Structural Patch Decomposition (SPD). In the second step, appropriate fusion rules are carefully designed for each component and the fused patch can be derived by inverse SPD. Our extensive experiments on several benchmark datasets show that our method outperforms seven compared state-of-the-art methods, especially in human visual perception.

show abstract

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Two-Stream Edge-Aware Network for Infrared and Visible Image Fusion With Multi-Level Wavelet Decomposition

Wang,

Shu,

et al. 2024

IEEE Access

View full text Add to dashboard Cite

show abstract

“…I MAGE fusion can combine the superiority of multi-source images that captured by different sensors to produce an informative fused image [3], [4]. Infrared and visible image fusion technology has been widely applied to intelligence acquisition or analysis [5]- [9], because infrared sensors are sensitive to the thermal radiation of the heat source, that can enhance the infrared targets.…”

Section: Introductionmentioning

confidence: 99%

TFIV: Multigrained Token Fusion for Infrared and Visible Image via Transformer

Li,

Yang,

Bai

et al. 2023

IEEE Trans. Instrum. Meas.

View full text Add to dashboard Cite

Infrared and visible image fusion aims to extract complementary features to synthesize a single fused image. Many methods employ convolutional neural networks (CNNs) to extract local features due to its translation invariance and locality. However, CNNs fail to consider the image's non-local self-similarity (NLss), though it can expand the receptive field by pooling operations, it still inevitably leads to information loss. In addition, the transformer structure extracts long-range dependence by considering the correlativity among all image patches, leading to information redundancy of such transformer-based methods. However, graph representation is more flexible than grid (CNN) or sequence (transformer structure) representation to address irregular objects, and graph can also construct the relationships among the spatially repeatable details or texture with far-space distance. Therefore, to address the above issues, it is significant to convert images into the graph space and thus adopt graph convolutional networks (GCNs) to extract NLss. This is because the graph can provide a fine structure to aggregate features and propagate information across the nearest vertices without introducing redundant information. Concretely, we implement a cascaded NLss extraction pattern to extract NLss of intra-and inter-modal by exploring interactions of different image pixels in intra-and inter-image positional distance. We commence by preforming GCNs on each intra-modal to aggregate features and propagate information to extract independent intra-modal NLss. Then, GCNs are performed on the concatenate intramodal NLss features of infrared and visible images, which can explore the cross-domain NLss of inter-modal to reconstruct the fused image. We progressively expand the kennel sizes and dilations to increase the receptive field of GCNs, when we extract intra-and inter-modal NLss. Ablation studies and extensive experiments illustrates the effectiveness and superiority of the proposed method on three datasets.

show abstract

“…Infrared imaging plays a crucial role and finds extensive application in low-light even dark and smoky conditions [1], [2], [3] since they utilize the thermal radiation of objects for imaging, which can effectively alleviate adverse effects caused by challenging environmental conditions. Currently, computer vision tasks in the infrared vision community, such as object detection [4], [5], and semantic segmentation [6], [7], are explored by task-specific designed models.…”

Section: Introductionmentioning

confidence: 99%

Numerical Simulation for Water Inrush in the University City Tunnel in the Chongqing City, Southwestern China

Liu,

Wu,

Jiang

et al. 2023

Lecture Notes in Civil Engineering

View full text Add to dashboard Cite

In recent years, the foundation models have swept the computer vision field and facilitated the development of various tasks within different modalities. However, it remains an open question on how to design an infrared foundation model. In this paper, we propose InfMAE, a foundation model in infrared modality. We release an infrared dataset, called Inf30 to address the problem of lacking large-scale data for self-supervised learning in the infrared vision community. Besides, we design an information-aware masking strategy, which is suitable for infrared images. This masking strategy allows for a greater emphasis on the regions with richer information in infrared images during the self-supervised learning process, which is conducive to learning the generalized representation. In addition, we adopt a multi-scale encoder to enhance the performance of the pre-trained encoders in downstream tasks. Finally, based on the fact that infrared images do not have a lot of details and texture information, we design an infrared decoder module, which further improves the performance of downstream tasks. Extensive experiments show that our proposed method InfMAE outperforms other supervised methods and self-supervised learning methods in three downstream tasks. Our code will be made public at https://github.com/liufangcen/InfMAE.

show abstract

Visible and Infrared Image Fusion Using Deep Learning

Cited by 47 publications

References 190 publications

Two-Stream Edge-Aware Network for Infrared and Visible Image Fusion With Multi-Level Wavelet Decomposition

Two-Stream Edge-Aware Network for Infrared and Visible Image Fusion With Multi-Level Wavelet Decomposition

TFIV: Multigrained Token Fusion for Infrared and Visible Image via Transformer

Numerical Simulation for Water Inrush in the University City Tunnel in the Chongqing City, Southwestern China

Contact Info

Product

Resources

About