Unprocessing Images for Learned Raw Denoising

Brooks, Tim; Mildenhall, Ben; Xue, Tao; Chen, Jiawen; Sharlet, Dillon; Barron, Jonathan T.

doi:10.1109/cvpr.2019.01129

Cited by 383 publications

(348 citation statements)

References 45 publications

Supporting

Mentioning

348

Contrasting

Order By: Relevance

“…(ii) Comparison with Deblock+GAN (Nos. 3,5,7). We found that the superiority of Deblock+GAN over standard GAN tends to be achieved as the quality Table 7.…”

Section: A13 Evaluation On Compression Robust Image Generation Undementioning

confidence: 85%

Label-Noise Robust Generative Adversarial Networks

Kaneko

Ushiku

Harada

2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

Recently, generative adversarial networks (GANs), which learn data distributions through adversarial training, have gained special attention owing to their high image reproduction ability. However, one limitation of standard GANs is that they recreate training images faithfully despite image degradation characteristics such as blur, noise, and compression. To remedy this, we address the problem of blur, noise, and compression robust image generation. Our objective is to learn a non-degraded image generator directly from degraded images without prior knowledge of image degradation. The recently proposed noise robust GAN (NR-GAN) already provides a solution to the problem of noise degradation. Therefore, we first focus on blur and compression degradations. We propose blur robust GAN (BR-GAN) and compression robust GAN (CR-GAN), which learn a kernel generator and quality factor generator, respectively, with non-degraded image generators. Owing to the irreversible blur and compression characteristics, adjusting their strengths is non-trivial. Therefore, we incorporate switching architectures that can adapt the strengths in a data-driven manner. Based on BR-GAN, NR-GAN, and CR-GAN, we further propose blur, noise, and compression robust GAN (BNCR-GAN), which unifies these three models into a single model with additionally introduced adaptive consistency losses that suppress the uncertainty caused by the combination. We provide benchmark scores through large-scale comparative studies on CIFAR-10 and a generality analysis on FFHQ dataset.1 More precisely, to reduce this cost, unpaired learning methods [53,50] were devised.These methods, however, still require separate collection of non-degraded and degraded images. As an alternative, self-supervised learning methods [45, 2, 48] were also proposed; however, their application is limited to denoising.Blur, Noise, and Compression Robust Generative Adversarial Networks 3 Motivated by these existing studies, we address the following problem: "How can we learn a non-degraded image generator directly from degraded images without knowing the details of image degradation? " Particularly, in this work, we focus on three representative types of image degradation, i.e., blur, noise, and compression, and call this problem blur, noise, and compression robust image generation. As an example, we demonstrate a solution using our model in Fig. 1(c). Although the same training images ( Fig. 1(a)) are used between baseline GAN (Fig. 1(b)) and proposed blur, noise, and compression robust GAN (BNCR-GAN ) (Fig. 1(c)), BNCR-GAN succeeds in generating non-degraded images, whereas baseline GAN fails to do so.The recently proposed noise robust GAN (NR-GAN) [31] has shown promising results when its application is restricted to noise degradation. Hence, in this work, we first focus on the remaining two kinds of degradation, i.e., blur and compression, and handle all types of degradation as an advanced problem.To obtain robustness in blur and compression degradations, we first propose two variants of ...

show abstract

“…(ii) Comparison with Deblock+GAN (Nos. 3,5,7). We found that the superiority of Deblock+GAN over standard GAN tends to be achieved as the quality Table 7.…”

Section: A13 Evaluation On Compression Robust Image Generation Undementioning

confidence: 85%

Label-Noise Robust Generative Adversarial Networks

Kaneko

Ushiku

Harada

2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

show abstract

“…The means of absolute errors (MAEs) between the predicted and true motion magnitudes were generated using the MPI-Sintel optical flow dataset (Butler et al 2012). The images were inverse-tone mapped into a simulated linear space using (Brooks et al 2019), and high-magnitude noise was added, with a standard deviation of 5% of the white level. The images were then downsampled to 256×192 using a triangle kernel before being processed by the algorithms.…”

Section: E Icient Motion Magnitude Estimationmentioning

confidence: 99%

Handheld mobile photography in very low light

et al. 2019

Self Cite

View full text Add to dashboard Cite

a) Previously described result (b) Previously described result, gained (c) Our result Fig. 1. We present a system that uses a novel combination of motion-adaptive burst capture, robust temporal denoising, learning-based white balance, and tone mapping to create high quality photographs in low light on a handheld mobile device. Here we show a comparison of a photograph generated by the burst photography system described in (Hasino et al. 2016) and the system described in this paper, running on the same mobile camera. In this low-light se ing (about 0.4 lux), the previous system generates an underexposed result (a). Brightening the image (b) reveals significant noise, especially chroma noise, which results in loss of detail and an unpleasantly blotchy appearance. Additionally, the colors of the face appear too orange. Our pipeline (c) produces detailed images by selecting a longer exposure time due to the low scene motion (in this se ing, extended from 0.14 s to 0.33 s), robustly aligning and merging a larger number of frames (13 frames instead of 6), and reproducing colors reliably by training a model for predicting the white balance gains specifically in low light. Additionally, we apply local tone mapping that brightens the shadows without over-clipping highlights or sacrificing global contrast.Taking photographs in low light using a mobile phone is challenging and rarely produces pleasing results. Aside from the physical limits imposed by read noise and photon shot noise, these cameras are typically handheld, have small apertures and sensors, use mass-produced analog electronics that cannot easily be cooled, and are commonly used to photograph subjects that move, like children and pets. In this paper we describe a system for capturing clean, sharp, colorful photographs in light as low as 0.3 lux, where human vision becomes monochromatic and indistinct. To permit handheld photography without ash illumination, we capture, align, and combine multiple frames. Our system employs "motion metering", which uses an estimate of motion magnitudes (whether due to handshake or moving objects) to identify the number of frames and the per-frame exposure times that together minimize both noise and motion blur in a captured burst. We combine these frames using robust alignment and merging techniques that are specialized for high-noise imagery. To ensure accurate colors in such low light, we employ a learning-based auto white balancing algorithm. To prevent the photographs from looking like they were shot in daylight, we use tone mapping techniques inspired by illusionistic painting: increasing contrast, crushing shadows to black, and surrounding the scene with darkness. All of these processes are performed using the limited computational resources of a mobile device. Our system can be used by novice photographers to produce shareable pictures in a few seconds based on a single shu er press, even in environments so dim that humans cannot see clearly.

show abstract

“…After WB is applied to the raw-RGB image, a number of additional nonlinear photofinishing color manipulations are further applied by the ISP to render the final sRGB image [2]. These photo-finishing operations include, but are not limited to, hue/saturation manipulation, general color manipulation, and local/global tone mapping [8,27,33,44,47]. Cameras generally have multiple photo-finishing styles the user can select [2,33,34].…”

Section: Related Workmentioning

confidence: 99%

What Else Can Fool Deep Learning? Addressing Color Constancy Errors on Deep Neural Network Performance

Afifi

Brown

2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

View full text Add to dashboard Cite

There is active research targeting local image manipulations that can fool deep neural networks (DNNs) into producing incorrect results. This paper examines a type of global image manipulation that can produce similar adverse effects. Specifically, we explore how strong color casts caused by incorrectly applied computational color constancy -referred to as white balance (WB) in photography -negatively impact the performance of DNNs targeting image segmentation and classification. In addition, we discuss how existing image augmentation methods used to improve the robustness of DNNs are not well suited for modeling WB errors. To address this problem, a novel augmentation method is proposed that can emulate accurate color constancy degradation. We also explore pre-processing training and testing images with a recent WB correction algorithm to reduce the effects of incorrectly white-balanced images. We examine both augmentation and pre-processing strategies on different datasets and demonstrate notable improvements on the CIFAR-10, CIFAR-100, and ADE20K datasets.

show abstract

Unprocessing Images for Learned Raw Denoising

Cited by 383 publications

References 45 publications

Label-Noise Robust Generative Adversarial Networks

Label-Noise Robust Generative Adversarial Networks

Handheld mobile photography in very low light

What Else Can Fool Deep Learning? Addressing Color Constancy Errors on Deep Neural Network Performance

Contact Info

Product

Resources

About