Path‐space Motion Estimation and Decomposition for Robust Animation Filtering

Abstract

Renderings of animation sequences with physics‐based Monte Carlo light transport simulations are exceedingly costly to generate frame‐by‐frame, yet much of this computation is highly redundant due to the strong coherence in space, time and among samples. A promising approach pursued in prior work entails subsampling the sequence in space, time, and number of samples, followed by image‐based spatio‐temporal upsampling and denoising. These methods can provide significant performance gains, though major issues remain: firstly, in a multiple scattering simulation, the final pixel color is the composite of many different light transport phenomena, and this conflicting information causes artifacts in image‐based methods. Secondly, motion vectors are needed to establish correspondence between the pixels in different frames, but it is unclear how to obtain them for most kinds of light paths (e.g. an object seen through a curved glass panel). To reduce these ambiguities, we propose a general decomposition framework, where the final pixel color is separated into components corresponding to disjoint subsets of the space of light paths. Each component is accompanied by motion vectors and other auxiliary features such as reflectance and surface normals. The motion vectors of specular paths are computed using a temporal extension of manifold exploration and the remaining components use a specialized variant of optical flow. Our experiments show that this decomposition leads to significant improvements in three image‐based applications: denoising, spatial upsampling, and temporal interpolation.

Hash-Based Hierarchical Caching and Layered Filtering for Interactive Previews in Global Illumination Rendering

“…While this method performs a range search in path space, our approach effectively brings this filtering to screen-space in a way that is simple to implement. The general decomposition method for filtering specific light paths presented by Zimmer et al [35] is closely related to our approach. However, the decomposition in our algorithm is exclusively based on recursion depth, not directly taking specific material properties into account.…”

Adaptive Temporal Sampling for Volumetric Path Tracing of Medical Data

“…[SKW∗17]. Most of these approaches also consider the temporal domain for filtering [CSK∗17], and ‐ like in our approach ‐ reuse samples in‐between frames for noise reduction [HDMS03, NSL∗07, HBGM11, MMBJ17, ZRJ∗15]. However, all of the reprojection‐based approaches consider surface rendering, where a reprojection is well defined, in contrast to the volumetric samples considered in this paper.…”

Denoising Deep Monte Carlo Renderings

“…State‐of‐the‐art denoising algorithms all rely on a combination of noisy colour and feature information to guide the denoising process. Our work builds on the joint NL‐Means filter used in previous works [RMZ13, MJL*13, KBS15, ZRJ*15], which we describe in this section. Our generalization to deep data follows in Section 4.…”

“…Multiple texture values will project into the same pixel (and bin) and since the time domain is sampled stochastically, the albedo feature will suffer from noise. Similarly to previous works [RMZ13, KBS15, ZRJ*15, MMMG16], we therefore pre‐filter the albedo and normal using our joint deep filter. For the albedo, the alpha‐weighted NL‐Means weights are computed using the flattened albedo bins with the depth values acting as an auxiliary feature.…”

“…As shown by Zimmer et al . [ZRJ*15], this type of decomposition allows tailoring the filter behaviour to the distinct noise characteristic and structure of each component, leading to higher quality denoised results. For the diffuse component (but not the specular one), we also make use of the effective irradiance proposed by Zimmer et al .…”