Best Algorithms for HDR Image Generation. A Study of Performance Bounds

Abstract

Abstract. Since the seminal work of Mann and Picard in 1994, the standard way to build high dynamic range (hdr) images from regular cameras has been to combine a reduced number of photographs captured with different exposure times. The algorithms proposed in the literature differ in the strategy used to combine these frames. Several experimental studies comparing their performances have been reported, showing in particular that a maximum likelihood estimation yields the best results in terms of mean squared error. However, no theoretical study aiming at establishing the performance limits of the hdr estimation problem has been conducted. Another common aspect of all hdr estimation approaches is that they discard saturated values. In this paper, we address these two issues. More precisely, we derive theoretical bounds for the hdr estimation problem, and we show that, even with a small number of photographs, the maximum likelihood estimator performs extremely close to these bounds. As a second contribution, we propose a general strategy to integrate the information provided by saturated pixels in the estimation process, hence improving the estimation results. Finally, we analyze the sensitivity of the hdr estimation process to camera parameters, and we show that small errors in the camera calibration process may severely degrade the estimation results.Key words. high dynamic range imaging, irradiance estimation, exposure bracketing, multi-exposure fusion, camera acquisition model, noise modeling, censored data, exposure saturation, Cramér-Rao lower bound.1. Introduction. The human eye has the ability to capture scenes of very high dynamic range, retaining details in both dark and bright regions. This is not the case for current standard digital cameras. Indeed, the limited capacity of the sensor cells makes it impossible to record the irradiance from very bright regions for long exposures. Pixels saturate incurring in information loss under the form of censored data. On the other hand, if the exposure time is reduced in order to avoid saturation, very few photons will be captured in the dark regions and the result will be masked by the acquisition noise. Therefore the result of a single shot picture of a high dynamic range scene, taken with a regular digital camera, contains pixels which are either overexposed or too noisy.High dynamic range imaging (hdr for short) is the field of imaging that seeks to accurately capture and represent scenes with the largest possible irradiance range. The representation problem of how to display an hdr image or irradiance map in a lower range image (for computer monitors or photographic prints) while retaining localized contrast, known as tone mapping, will not be addressed here. Due to technological and physical limitations of current optical sensors, nowadays the most common way to reach high irradiance dynamic ranges is by combining multiple low dynamic range photographs, acquired with different exposure times τ 1 , τ 2 , . . . , τ T . Indeed, for a given irradiance C and expos...

A Bayesian Hyperprior Approach for Joint Image Denoising and Interpolation, With an Application to HDR Imaging

“…The main noise sources for CMOS sensors are: the Poisson photon shot noise, which can be approximated by a Gaussian distribution with equal mean and variance; the thermally generated readout noise, which is modeled as an additive Gaussian distributed noise and the spatially varying gain given by the photo response non uniformity (PRNU) [20], [27]. We thus consider the following noise model for the non saturated raw pixel value Z(p) at position p…”

“…HDR imaging aims at reproducing an extended dynamic range of luminosity compared to what can be captured using a standard digital camera, which is often not enough to produce an accurate representation of real scenes. In the case of a static scene and a static camera, the combination of multiple images with different exposure levels is a simple and efficient solution [27], [29], [30]. However, several problems arise when either the camera or the elements in the scene move [31], [32].…”

Physically Based Rendering of Synthetic Objects in Real Environments

“…The final result is then computed by a weighted average over the pixel values in the different exposures [48,143,157]. Modern HDR fusion is based on adapting the weights according to the statistical properties of the camera noise [7,73] to minimize the resulting variance of the HDR pixel estimates. The minimum number of exposures required to accurately capture a scene given an acceptable noise level has also been studied [69,84].…”

“…Dark current noise generated from thermal generation of electrons in the sensor over time is not included in the model as, especially for modern imaging sensors, it has a negligible effect at normal temperatures for exposures under 1-2 seconds [7].…”

“…which can be estimated by using the noisy estimatef i in place of the exact signal f i , as this serves as a good approximation to the true variance [7]. An in-depth overview of the radiometric camera model and procedures for calibrating the camera parameters, (g i , a i , μ R , σ 2 R ), are given in Paper E.…”