DeepFuse: A Deep Unsupervised Approach for Exposure Fusion with Extreme Exposure Image Pairs

Abstract

We present a novel deep learning architecture for fusing static multi-exposure images. Current multi-exposure fusion (MEF) approaches use hand-crafted features to fuse input sequence. However, the weak hand-crafted representations are not robust to varying input conditions. Moreover, they perform poorly for extreme exposure image pairs. Thus, it is highly desirable to have a method that is robust to varying input conditions and capable of handling extreme exposure without artifacts. Deep representations have known to be robust to input conditions and have shown phenomenal performance in a supervised setting. However, the stumbling block in using deep learning for MEF was the lack of sufficient training data and an oracle to provide the ground-truth for supervision. To address the above issues, we have gathered a large dataset of multi-exposure image stacks for training and to circumvent the need for ground truth images, we propose an unsupervised deep learning framework for MEF utilizing a no-reference quality metric as loss function. The proposed approach uses a novel CNN architecture trained to learn the fusion operation without reference ground truth image. The model fuses a set of common low level features extracted from each image to generate artifact-free perceptually pleasing results. We perform extensive quantitative and qualitative evaluation and show that the proposed technique outperforms existing state-ofthe-art approaches for a variety of natural images.1 Exposure bias value indicates the amount of exposure offset from the auto exposure setting of an camera. For example, EV 1 is equal to doubling auto exposure time (EV 0).

Deep Convolutional Neural Network for Multi-Modal Image Restoration and Fusion

“…As we can see from this figure, the combined images generated by our method are more visually pleasant than the other approaches, with every part in focus, and with clear and sharp boundary edges. In contrast, the comparison methods, CSR [3], Deepfuse network [44] and Densefuse network [45], all lead to different levels of blurring artefacts across the boundary areas, as shown in the close-ups of the toy dog and the fence.…”

“…However, the transform domain is manually [43] proposed a simple CNN to predict the decision map for multi-focus image fusion. Prabhakar et al [44] proposed a CNN based unsupervised image fusion method to fuse one under-exposed image with an over-exposed one. Li et al [45] proposed a CNN network with the dense block structure to solve the infrared and visible image fusion problem.…”

“…The testing images are from the Lytro multi-focus image dataset [72]. We compare our method with the following three state-of-the-art approaches: CSR [3], Deepfuse network [44] and Densefuse network [45]. The visual comparison results are shown in Fig.…”

“…9 shows the fused images using different methods, including CSR [3], Deepfuse network [44], Densefuse network [45] and our CU-Net. As we can see, the method CSR [3] and Deepfuse network [44] both lose the image energy and have low contrast, which leads to low brightness in the soft-tissue regions. The method Densefuse [45] produces over-bright images, which makes many details difficult to see.…”

“…Visual comparisons of multi-focus image fusion results. (a) The near-focus image, (b) The far-focus image, (c) CSR [3], (d) Deepfuse network[44], (e) Densefuse network[45], (f) our CU-Net. Better view in electronic version.…”