A unified framework for multi-sensor HDR video reconstruction

Abstract

One of the most successful approaches to modern high quality HDR-video capture is to use camera setups with multiple sensors imaging the scene through a common optical system. However, such systems pose several challenges for HDR reconstruction algorithms. Previous reconstruction techniques have considered debayering, denoising, resampling (alignment) and exposure fusion as separate problems. In contrast, in this paper we present a unifying approach, performing HDR assembly directly from raw sensor data. Our framework includes a camera noise model adapted to HDR video and an algorithm for spatially adaptive HDR reconstruction based on fitting of local polynomial approximations to observed sensor data. The method is easy to implement and allows reconstruction to an arbitrary resolution and output mapping. We present an implementation in CUDA and show real-time performance for an experimental 4 Mpixel multi-sensor HDR video system. We further show that our algorithm has clear advantages over existing methods, both in terms of flexibility and reconstruction quality.

Adaptive dualISO HDR reconstruction

“…In this paper, we extend the earlier framework for HDR reconstruction developed in [10,15,16] based on fitting local polynomial approximations (LPA) [5] to irregularly distributed samples around output pixels using a localized maximum likelihood estimation [30] to incorporate the heterogeneous noise of the samples. In contrast to the previous works [10,15,16], we propose a novel adaptation of the filter kernel size that allows the filter extent to adapt not only to local image structure but also the sensor noise in the region.…”

“…The method presented here extends the statistical HDR reconstruction developed by [15,16] to include reconstruction kernels which adapts to both the image content and the heterogeneous measurement noise. We assume that the input data is a raw CFA sensor image with per-pixel gain settings varying between pixel segments as described in Fig.…”

“…To model the dependence of the measured digital pixel value on the incident radiant power and the camera parameters, we use a well-established radiometric model derived from previous works [9,15]. Using this model, the non-saturated pixel values are modeled as random variables following a normal distribution:…”

“…The main contribution in this paper is an extension of the previous statistical reconstruction method for dualISO data developed in [10,15], using two novel algorithms for adapting the scale of the filtering window. In contrast to previous works [10,15], the window support is adapted both to the statistical properties of the image noise as well as the underlying signal structure contained in the image.…”

“…In contrast to previous works [10,15], the window support is adapted both to the statistical properties of the image noise as well as the underlying signal structure contained in the image. We show that the novel scale selection results in increased image quality in several examples.…”