MFF-GAN: An unsupervised generative adversarial network with adaptive and gradient joint constraints for multi-focus image fusion

Zhang, Hao; Le, Zhuliang; Shao, Zhenfeng; Xu, Han; Ma, Jiayi

doi:10.1016/j.inffus.2020.08.022

Cited by 177 publications

(66 citation statements)

References 39 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Hence, we can only compare the performance of our method with the available state-of-the-art supervised methods. These include the Non-Subsampled Contournet Transform (NSCT) [ 29 ], Guided Filtering (GF) [ 35 ], Dense SIFT (DSIFT) [ 33 ], as well as the methods based on Boundary Finding (BF) [ 57 ], Convolutional Neural Network (CNN) [ 18 ], the U-net [ 41 ]; deep unsupervised algorithms FusionDN [ 43 ], MFF-GAN [ 44 ] and U2Fusion [ 42 ]. We implemented these algorithms using code acquired from their respective authors.…”

Section: Resultsmentioning

confidence: 99%

“…We used a desktop machine with 3.4 GHz Intel i7 CPU (32 RAM) and NVIDIA Titan Xp GPU (12 GB Memory) to evaluate our algorithm. For fairness, we compared our proposed algorithm with other deep unsupervised algorithms [ 42 , 43 , 44 ] since others did not perform with a GPU. The average run-time of our proposed MFNet , FusionDN, MFF-GAN and U2Fusion is 4.33, 1.55, 1.60 and 1.78 s, respectively.…”

Section: Resultsmentioning

confidence: 99%

“…This method also needs the ground truth for training the U-net fusion network model. To do this, several unsupervised multi-focus image fusion networks were proposed in [ 42 , 43 , 44 ].…”

Section: Related Workmentioning

confidence: 99%

See 2 more Smart Citations

Structural Similarity Loss for Learning to Fuse Multi-Focus Images

Gilani

Qin

Mian

2020

Sensors

View full text Add to dashboard Cite

Convolutional neural networks have recently been used for multi-focus image fusion. However, some existing methods have resorted to adding Gaussian blur to focused images, to simulate defocus, thereby generating data (with ground-truth) for supervised learning. Moreover, they classify pixels as ‘focused’ or ‘defocused’, and use the classified results to construct the fusion weight maps. This then necessitates a series of post-processing steps. In this paper, we present an end-to-end learning approach for directly predicting the fully focused output image from multi-focus input image pairs. The suggested approach uses a CNN architecture trained to perform fusion, without the need for ground truth fused images. The CNN exploits the image structural similarity (SSIM) to calculate the loss, a metric that is widely accepted for fused image quality evaluation. What is more, we also use the standard deviation of a local window of the image to automatically estimate the importance of the source images in the final fused image when designing the loss function. Our network can accept images of variable sizes and hence, we are able to utilize real benchmark datasets, instead of simulated ones, to train our network. The model is a feed-forward, fully convolutional neural network that can process images of variable sizes during test time. Extensive evaluation on benchmark datasets show that our method outperforms, or is comparable with, existing state-of-the-art techniques on both objective and subjective benchmarks.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Resultsmentioning

confidence: 99%

See 1 more Smart Citation

Structural Similarity Loss for Learning to Fuse Multi-Focus Images

Gilani

Qin

Mian

2020

Sensors

View full text Add to dashboard Cite

show abstract

“…The obtained sparse coefficients of the source images were merged, and a fused image was obtained. Generative adversarial networks (GANs) have also been exploited for increasing the depth of field, e.g., [ 60 , 61 , 62 ].…”

Section: Background and Literature Reviewmentioning

confidence: 99%

Exploiting Superpixels for Multi-Focus Image Fusion

Ilyas

Farid

Khan

et al. 2021

Entropy

View full text Add to dashboard Cite

Multi-focus image fusion is the process of combining focused regions of two or more images to obtain a single all-in-focus image. It is an important research area because a fused image is of high quality and contains more details than the source images. This makes it useful for numerous applications in image enhancement, remote sensing, object recognition, medical imaging, etc. This paper presents a novel multi-focus image fusion algorithm that proposes to group the local connected pixels with similar colors and patterns, usually referred to as superpixels, and use them to separate the focused and de-focused regions of an image. We note that these superpixels are more expressive than individual pixels, and they carry more distinctive statistical properties when compared with other superpixels. The statistical properties of superpixels are analyzed to categorize the pixels as focused or de-focused and to estimate a focus map. A spatial consistency constraint is ensured on the initial focus map to obtain a refined map, which is used in the fusion rule to obtain a single all-in-focus image. Qualitative and quantitative evaluations are performed to assess the performance of the proposed method on a benchmark multi-focus image fusion dataset. The results show that our method produces better quality fused images than existing image fusion techniques.

show abstract

“…For example, Liu [ 14 ] proposed a multi-focus image fusion method with a deep convolutional neural network; Zhong [ 15 ] proposed a remote sensing image fusion method with a convolutional neural network; Ma [ 12 ] is the first one who applied generative adversarial networks (GAN) into infrared and visible image fusion and achieved good fusion results; however, the construction of the existing FusionGAN-based method is simple, and loss fusion is imperfect, which can lead to incomplete information transfer in the fused image. Improved GAN-based methods have been proposed for image fusion, such as Zhang [ 16 ], who proposed a new generative adversarial network with adaptive and gradient joint constraints to fuse multi-focus images. Nevertheless, the methods mentioned above are effective for other kinds of source images rather than remote sensing images, and they are sensitive to the noise in an image.…”

Section: Introductionmentioning

confidence: 99%

A Noisy SAR Image Fusion Method Based on NLM and GAN

Fang

Wang

et al. 2021

Entropy

View full text Add to dashboard Cite

The unavoidable noise often present in synthetic aperture radar (SAR) images, such as speckle noise, negatively impacts the subsequent processing of SAR images. Further, it is not easy to find an appropriate application for SAR images, given that the human visual system is sensitive to color and SAR images are gray. As a result, a noisy SAR image fusion method based on nonlocal matching and generative adversarial networks is presented in this paper. A nonlocal matching method is applied to processing source images into similar block groups in the pre-processing step. Then, adversarial networks are employed to generate a final noise-free fused SAR image block, where the generator aims to generate a noise-free SAR image block with color information, and the discriminator tries to increase the spatial resolution of the generated image block. This step ensures that the fused image block contains high resolution and color information at the same time. Finally, a fused image can be obtained by aggregating all the image blocks. By extensive comparative experiments on the SEN1–2 datasets and source images, it can be found that the proposed method not only has better fusion results but is also robust to image noise, indicating the superiority of the proposed noisy SAR image fusion method over the state-of-the-art methods.

show abstract

MFF-GAN: An unsupervised generative adversarial network with adaptive and gradient joint constraints for multi-focus image fusion

Cited by 177 publications

References 39 publications

Structural Similarity Loss for Learning to Fuse Multi-Focus Images

Structural Similarity Loss for Learning to Fuse Multi-Focus Images

Exploiting Superpixels for Multi-Focus Image Fusion

A Noisy SAR Image Fusion Method Based on NLM and GAN

Contact Info

Product

Resources

About