Burst photography for high dynamic range and low-light imaging on mobile cameras

Abstract

Cell phone cameras have small apertures, which limits the number of photons they can gather, leading to noisy images in low light. They also have small sensor pixels, which limits the number of electrons each pixel can store, leading to limited dynamic range. We describe a computational photography pipeline that captures, aligns, and merges a burst of frames to reduce noise and increase dynamic range. Our system has several key features that help make it robust and efficient. First, we do not use bracketed exposures. Instead, we capture frames of constant exposure, which makes alignment more robust, and we set this exposure low enough to avoid blowing out highlights. The resulting merged image has clean shadows and high bit depth, allowing us to apply standard HDR tone mapping methods. Second, we begin from Bayer raw frames rather than the demosaicked RGB (or YUV) frames produced by hardware Image Signal Processors (ISPs) common on mobile platforms. This gives us more bits per pixel and allows us to circumvent the ISP's unwanted tone mapping and spatial denoising. Third, we use a novel FFT-based alignment algorithm and a hybrid 2D/3D Wiener filter to denoise and merge the frames in a burst. Our implementation is built atop Android's Camera2 API, which provides per-frame camera control and access to raw imagery, and is written in the Halide domain-specific language (DSL). It runs in 4 seconds on device (for a 12 Mpix image), requires no user intervention, and ships on several mass-produced cell phones.

Spatially Variant Laplacian Pyramids for Multi-frame Exposure Fusion

“…Blending of patches Once the patches for each level are reconstructed they have to be blended in a seamless manner to generate the full image. Modified raised cosine filter proposed by [9] was used to blend the overlapping patches. It is to be noted that each of the patches have an overlapping factor of 50%.…”

Stereoscopic Dark Flash for Low-light Photography

“…These methods are generally computationally expensive, and are necessarily limited in their ability to recover details in the presence of overwhelming noise. Performance can be improved by using a burst of images to denoise a single image [1], [14]- [16], though these approaches require computing a correspondence across images or some technique for being invariant to this correspondence problem, which can be problematic in the presence of significant camera or scene motion.…”

The high dynamic range imaging pipeline : Tone-mapping, distribution, and single-exposure reconstruction

“…Places and Imagenet, which are used for learning other imaging tasks. There are other datasets that potentially can be used in order to improve performance, such as from Google's HDR+ project [112], SJTU HDR Video Sequences [228], the RAISE [59] and FiveK [45] datasets of RAW images, etc. All these sources provide images at an increased bit-depth.…”