A survey on deep learning-based Monte Carlo denoising

Abstract

Monte Carlo (MC) integration is used ubiquitously in realistic image synthesis because of its flexibility and generality. However, the integration has to balance estimator bias and variance, which causes visually distracting noise with low sample counts. Existing solutions fall into two categories, in-process sampling schemes and post-processing reconstruction schemes. This report summarizes recent trends in the post-processing reconstruction scheme. Recent years have seen increasing attention and significant progress in denoising MC rendering with deep learning, by training neural networks to reconstruct denoised rendering results from sparse MC samples. Many of these techniques show promising results in real-world applications, and this report aims to provide an assessment of these approaches for practitioners and researchers.

Framework for Denoising Monte Carlo Photon Transport Simulations Using Deep Learning

“…This procedure is commonly referred to as denoising. 16,17 In the past, model-based noise-adaptive filters have been proposed to address the spatially varying noise in the radiation dosage estimation context and computer graphics rendering. [18][19][20] However, improvements provided by applying these filtering-based techniques have been small to moderate, creating an equivalent speedup of only 3-to 4-fold.…”

“…16 Recent work on denoising ray-traced computer graphics, and spatially-variant noisy images in the field of computer vision, focus mainly on machine learning (ML)-based denoising methods, more specifically convolutional neural networks (CNNs). 17 Despite their promising performance compared to traditional filters, no attempt has been made, to the best of our knowledge, to adapt denoisers designed for two-dimensional (2-D) low bit-depth image domain to high dynamic range MC fluence maps. 16,21 Our motivation is therefore to develop effective CNN-based denoising techniques and compare it among state-of-the-art denoisers in the context of MC photon simulations and identify their strengths compared to traditional model-based filtering techniques.…”

Sparse Sampling and Completion for Light Transport in VPL-based Rendering

“…Rendering realistic images with global illumination evolves integrating the rendering equation, which is a recursively defined expression because the incoming lighting is to be determined by the same equation. Various efforts are attributed to high-quality rendering [Wang et al 2013;Huo et al 2015;Huo et al 2016;Huo et al 2020a;Cho et al 2021;Fan et al 2021;Huo and Yoon 2021;Huo et al 2020b;Huo 2022], or to achieve a balance between quality, performance, and flexibility [Kim et al 2020;Li et al 2021;An et al 2021;Park et al 2021;Zhang et al 2021;Li et al 2020]. Keller [1997] introduced an intermediate representation, called virtual point light (VPL), to break the recursive rendering equation, yielding a fast realistic rendering algorithm and inspiring a number of followups called VPL-based rendering.…”

Accelerating Monte Carlo simulation of light propagation in tissue mimicking turbid medium based on generative adversarial networks

“…Such problem might be solved by reconstructing precise MC simulation from the rough MC simulation modeled by small amount of photons via data-driven supervised learning methods, in which similar ideas have already be implemented to reconstruct high quality images from low quality images, for example, to reconstruct high resolution images from low resolution images, 8,9 high signal to noise ratio (SNR) images from low SNR images, 10,11 informative images from uninformative images, [12][13][14] and volumetric tomographic images from sparse projection views. 15,16 Various studies [17][18][19] about deep learning based MC denoising have also been investigated in recent years, especially for synthesizing realistic images of virtual worlds in the rendering community. Among those methods, generative adversarial networks (GAN) 20 have been proved to be one of the most promising method for image reconstruction in recent years because of its adversary network structures, consisting of a generating network and a discriminating network.…”