NeX: Real-time View Synthesis with Neural Basis Expansion

Abstract

Neural basis functionsReflectance coefficients Figure 1: (a) Each pixel in NeX multiplane image consists of an alpha transparency value, base color k 0 , and view-dependent reflectance coefficients k 1 ...k n . A linear combination of these coefficients and basis functions learned from a neural network produces the final color value. (b, c) show our synthesized images that can be rendered in real time with view-dependent effects such as the reflection on the silver spoon.

UltraSR: Spatial Encoding is a Missing Key for Implicit Image Function-based Arbitrary-Scale Super-Resolution

“…Learning 3D scene representation with a parameterized neural network has been largely explored by recent works from various angles such as implicit signed distance function [21,5,34], occupancy [31,35,14], volume rendering (i.e. radiance field) [32,26,33,37,50,44], and shapes [2,15,14]. Such implicit neural representation also started to influence traditional 2D tasks such as image representation [23,40], super-resolution [9], and medical image analysis [49].…”

“…They showed clear improvements on rendered images which were free from blurry details and structural distortion. Such technique has been used as a default setting in later 3D works such as [26,50,39,44], whose major focus were to reduce the rendering speed and improve output quality. The missing theory part on why spatial encoding tremendously boosts rendering quality has been partially analyzed in [36,8,40].…”

“…Learning how to reconstruct the high-frequency part of the image is the key in the SR task. Many recent works [32,37,50,44] have shown that a carefully designed spatial encoding can help the network recover fine details in 3D scenes. LIIF [9] overlooked the importance of these spatial encoding by directly feeding coordinates into the implicit image function represented by a vanilla MLP.…”

“…As suggested by recent neural rendering works [32,37,50,44], using spatial encoding is critical for recovering high-frequency details in 3D scenes. It is natural to ask the following questions: Does spatial encoding also matter when we render 2D images in the arbitrary-scale SR task?…”

Fast and Explicit Neural View Synthesis

“…3.3 for details). In the second category, we find recent approaches that enable efficient rendering [25,26,27] but where their continuous representation is implicit, and must be fitted via gradient descent for every new object or scene (typically taking days on commodity hardware). In this paper we present a simple yet powerful approach for novel view synthesis which explicitly encodes sources views into a volumetric representation that enables amortized rendering.…”