Image Fusion With Convolutional Sparse Representation

Liu, Yu; Chen, Xun; Ward, Rabab K.; Wang, Z. Jane

doi:10.1109/lsp.2016.2618776

Cited by 773 publications

(382 citation statements)

References 37 publications

Supporting

Mentioning

351

Contrasting

Order By: Relevance

“…Many works have studied the fusion performance in different transform domains, including for example discrete wavelet transform (DWT) [36], discrete cosine transform (DCT) [37], non-subsampled contourlet transform [38]. There are also some papers based on sparse coding [3], [39], [40], which fuse the sparse representations in the sparse domain. The most widely used fusion rules are choose-max [41] and weighted average [42].…”

Section: Multi-modal Image Fusionmentioning

confidence: 99%

Deep Convolutional Neural Network for Multi-Modal Image Restoration and Fusion

Deng

Dragotti

2021

IEEE Trans. Pattern Anal. Mach. Intell.

150

View full text Add to dashboard Cite

In this paper, we propose a novel deep convolutional neural network to solve the general multi-modal image restoration (MIR) and multi-modal image fusion (MIF) problems. Different from other methods based on deep learning, our network architecture is designed by drawing inspirations from a new proposed multi-modal convolutional sparse coding (MCSC) model. The key feature of the proposed network is that it can automatically split the common information shared among different modalities, from the unique information that belongs to each single modality, and is therefore denoted with CU-Net, i.e., Common and Unique information splitting network. Specifically, the CU-Net is composed of three modules, i.e., the unique feature extraction module (UFEM), common feature preservation module (CFPM), and image reconstruction module (IRM). The architecture of each module is derived from the corresponding part in the MCSC model, which consists of several learned convolutional sparse coding (LCSC) blocks. Extensive numerical results verify the effectiveness of our method on a variety of MIR and MIF tasks, including RGB guided depth image super-resolution, flash guided non-flash image denoising, multi-focus and multi-exposure image fusion. Index Terms-Multi-modal image restoration, image fusion, multi-modal convolutional sparse coding ! • The authors are with the

show abstract

Section: Multi-modal Image Fusionmentioning

confidence: 99%

Deep Convolutional Neural Network for Multi-Modal Image Restoration and Fusion

Deng

Dragotti

2021

IEEE Trans. Pattern Anal. Mach. Intell.

150

View full text Add to dashboard Cite

show abstract

“…In the experiments, the presented fusion strategy is compared with following five fusion algorithms, including multimodal medical image fusion based on NSCT and PCNN with modified SF (NSCT‐PCNN‐MSF), NNM based on nuclear norm minimization (NNM), GFF based on guided filtering (GFF), image fusion metric with Laplace transform and sparse representation (LP‐SR), CSR based on convolutional sparse representation (CSR). For fair comparison, we use the parameters that were reported by the authors to yield the best fusion results.…”

Section: Experimental and Analysismentioning

confidence: 99%

“…SR fusion methods have emerged as an attractive direction in sparse domain methods over the last few years, which provide better performances. However, SR‐based fusion methods suffer from two main drawbacks, namely, limited ability in detail preservation and high sensitivity to misregistration, while these two issues are of great concern in image fusion . PCNN has been introduced into medical image fusion field, PCNN and/or MST are used in several medical image fusion algorithms with encouraging results.…”

Section: Introductionmentioning

confidence: 99%

“…However, SRbased fusion methods suffer from two main drawbacks, namely, limited ability in detail preservation and high sensitivity to misregistration, while these two issues are of great concern in image fusion. 11 PCNN has been introduced into medical image fusion field, PCNN and/or MST are used in several medical image fusion algorithms 12,13 with encouraging results. However, in the literature, 12 normalized coefficient value are used to motivate the PCNN, which easily cause blur and loss of details in the fused image.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

IGM‐based perceptual multimodal medical image fusion using free energy motivated adaptive PCNN

Tang

Tian

2017

Int J Imaging Syst Tech

View full text Add to dashboard Cite

Multimodal medical image fusion merges two medical images to produce a visual enhanced fused image, to provide more accurate comprehensive pathological information to doctors for better diagnosis and treatment. In this article, we present a perceptual multimodal medical image fusion method with free energy (FE) motivated adaptive pulse coupled neural network (PCNN) by employing Internal Generative Mechanism (IGM). First, source images are divided into predicted layers and detail layers with Bayesian prediction model. Then to retain human visual system inspired features, FE is used to motivate the PCNN for processing detail layers, and large firing times are selected as coefficients. The predicted layers are fused with the averaging strategy as activity level measurement. Finally, the fused image is reconstructed by merging coefficients in both fused layers. Experimental results visually and quantitatively show that the proposed fusion strategy is superior to the state‐of‐the‐art methods.

show abstract

“…A large amount of algorithms for MS and PAN image fusion, which is also called pansharpening, has been proposed in the past decades. Concerning the categorization of existing pansharpening methods, it is widely accepted that a majority of current methods can be classified into two major categories: the component substitution (CS) methods and the methods based on multi-resolution analysis (MRA) [1][2][3][4]. Another special categorization is the methods based on PAN-modulation (PM) [5], which provide outstanding fused products with constrained spectral distortions.…”

Section: Introductionmentioning

confidence: 99%

An Image Fusion Method Based on Image Segmentation for High-Resolution Remotely-Sensed Imagery

Jing

Tang

et al. 2018

Remote Sensing

View full text Add to dashboard Cite

Fusion of high spatial resolution (HSR) multispectral (MS) and panchromatic (PAN) images has become a research focus with the development of HSR remote sensing technology. In order to reduce the spectral distortions of fused images, current image fusion methods focus on optimizing the approach used to extract spatial details from the PAN band, or on the optimization of the models employed during the injection of spatial details into the MS bands. Due to the resolution difference between the MS and PAN images, there is a large amount of mixed pixels (MPs) existing in the upsampled MS images. The fused versions of these MPs remain mixed, although they may correspond to pure PAN pixels. This is one of the reasons for spectral distortions of fusion products. However, few methods consider spectral distortions introduced by the mixed fused spectra of MPs. In this paper, an image fusion method based on image segmentation was proposed to improve the fused spectra of MPs. The MPs were identified and then fused to be as close as possible to the spectra of pure pixels, in order to reduce spectral distortions caused by fused MPs and improve the quality of fused products. A fusion experiment, using three HSR datasets recorded by WorldView-2, WorldView-3 and GeoEye-1, respectively, was implemented to compare the proposed method with several other state-of-the-art fusion methods, such as haze-and ratio-based (HR), adaptive Gram-Schmidt (GSA) and smoothing filter-based intensity modulation (SFIM). Fused products generated at the original and degraded scales were assessed using several widely-used quantitative quality indexes. Visual inspection was also employed to compare the fused images produced using the original datasets. It was demonstrated that the proposed method offers the lowest spectral distortions and more sharpened boundaries between different image objects than other methods, especially for boundaries between vegetation and non-vegetation objects.

show abstract

Image Fusion With Convolutional Sparse Representation

Cited by 773 publications

References 37 publications

Deep Convolutional Neural Network for Multi-Modal Image Restoration and Fusion

Deep Convolutional Neural Network for Multi-Modal Image Restoration and Fusion

IGM‐based perceptual multimodal medical image fusion using free energy motivated adaptive PCNN

An Image Fusion Method Based on Image Segmentation for High-Resolution Remotely-Sensed Imagery

Contact Info

Product

Resources

About