End-to-End Differentiable Learning to HDR Image Synthesis for Multi-exposure Images

Abstract

Recently, high dynamic range (HDR) image reconstruction based on the multiple exposure stack from a given single exposure utilizes a deep learning framework to generate high-quality HDR images. These conventional networks focus on the exposure transfer task to reconstruct the multi-exposure stack. Therefore, they often fail to fuse the multi-exposure stack into a perceptually pleasant HDR image as the inversion artifacts occur. We tackle the problem in stack reconstruction-based methods by proposing a novel framework with a fully differentiable high dynamic range imaging (HDRI) process. By explicitly using the loss, which compares the network's output with the ground truth HDR image, our framework enables a neural network that generates the multiple exposure stack for HDRI to train stably. In other words, our differentiable HDR synthesis layer helps the deep neural network to train to create multi-exposure stacks while reflecting the precise correlations between multi-exposure images in the HDRI process. In addition, our network uses the image decomposition and the recursive process to facilitate the exposure transfer task and to adaptively respond to recursion frequency. The experimental results show that the proposed network outperforms the state-of-the-art quantitative and qualitative results in terms of both the exposure transfer tasks and the whole HDRI process.

Fast and flexible stack‐based inverse tone mapping

“…Note that the running time of stack‐based ITM methods only indicates the time of estimating the MES. We set the length of MES to 7 for the U‐Net [17], Deep Chain HDRI [10], Deep Recursive HDRI [11], and Deep Diff HDRI [22] as described in their papers. The time of the once increasing/decreasing process of these methods can be calculated by dividing the running time in Table 3 by six.…”

“…The training losses of the EAM contain the pixel‐wise loss: $$${L}_{\mathit{pix}}={\left\Vert {\widehat{O}}_{EA}-O\right\Vert }_{1},$$$ the cosine similarity loss to ensure colour correctness of the RGB vectors of each pixel: $$${L}_{\cos }=1-\frac{{\widehat{O}}_{EA}\cdot O}{{\left\Vert {\widehat{O}}_{EA}\right\Vert }_{2}{\left\Vert O\right\Vert }_{2}},$$$ and the histogram loss in [22] to ensure that the generated image has a similar global tone with the target image: $$${L}_{\mathit{hist}}=\frac{1}{L}\sum\limits _{l}^{L}{\left\Vert cn{t}_{l}\left({\widehat{O}}_{EA}\right)-cn{t}_{l}(O)\right\Vert }_{1},$$$ where O denotes the ground truth image, L denotes the intensity levels, and cnt l indicates the number of pixels, which has a rounded down intensity l in the input image. The total loss of EAM can be formulated as: $$${L}_{\mathit{EAM}}={L}_{\mathit{pix}}+{\lambda }_{\cos }{L}_{\cos }+{\lambda }_{\mathit{hist}}{L}_{\mathit{hist}},$$$ where λ cos and λ hist are set to 5 and 1 separately.…”

“…Kim et al. [22] proposed a novel framework with a fully differentiable HDR imaging process by introducing the differentiable HDR synthesis layer. Compared to the previous stack‐based ITM methods, the proposed method can generate the multi exposure stack with flexible exposure setting and achieve more accurate results.…”