Denoising with kernel prediction and asymmetric loss functions

Abstract

We present a modular convolutional architecture for denoising rendered images. We expand on the capabilities of kernel-predicting networks by combining them with a number of task-specific modules, and optimizing the assembly using an asymmetric loss. The source-aware encoder---the first module in the assembly---extracts low-level features and embeds them into a common feature space, enabling quick adaptation of a trained network to novel data. The spatial and temporal modules extract abstract, high-level features for kernel-based reconstruction, which is performed at three different spatial scales to reduce low-frequency artifacts. The complete network is trained using a class of asymmetric loss functions that are designed to preserve details and provide the user with a direct control over the variance-bias trade-off during inference. We also propose an error-predicting module for inferring reconstruction error maps that can be used to drive adaptive sampling. Finally, we present a theoretical analysis of convergence rates of kernel-predicting architectures, shedding light on why kernel prediction performs better than synthesizing the colors directly, complementing the empirical evidence presented in this and previous works. We demonstrate that our networks attain results that compare favorably to state-of-the-art methods in terms of detail preservation, low-frequency noise removal, and temporal stability on a variety of production and academic datasets.

Toward Real-World Single Image Super-Resolution: A New Benchmark and a New Model

“…This motivates us to train a kernel prediction network (KPN) for the real-world SISR task. The idea of kernel prediction is to explicitly learn a restoration kernel for each pixel, and it has been employed in applications such as denoising [2,34,48], dynamic deblurring [43,16] and video interpolation [35,36]. Though effective, the memory and computational cost of KPN is quadratically increased with the kernel size.…”

DVDNET: A Fast Network for Deep Video Denoising

“…2. The architecture of the spatial denoiser is inspired by the architectures in [8,9], while the temporal denoiser also borrows some elements from [13]. The spatial and temporal denoising blocks are composed of D spa = 12, and D temp = 6 convolutional layers, respectively.…”

Learning Explicit Smoothing Kernels for Joint Image Filtering

“…Jia et al [JDTG16] proposed to predict a dynamic convolution layer or a dynamic local filtering layer from input data and apply the layer on the input. The latter strategy, namely kernel prediction method, has been used for denoising bursts of images [MBC*18] and denoising Monte Carlo renderings [BVM*17; VRM*18]. Deformable convolutional network [DQX*17] predicts the sampling locations of the convolution operator and it was further improved by modulating the input feature amplitudes from different spatial locations [ZHLD18].…”

“…(1) and learn the kernel weights w (·,·) from a CNN model. Our strategy is similar to the kernel prediction method [JDTG16], which has been applied to denoising [MBC*18; BVM*17; VRM*18]. However, directly applying Equ.…”