SNeRF: Stylized Neural Implicit Representations for 3D Scenes

Abstract

Fig. 1. Given a neural implicit scene representation trained with multiple views of a scene, SNeRF stylizes the 3D scene to match a reference style. SNeRF works with a variety of scene types (indoor, outdoor, 4D dynamic avatar) and generates novel views with cross-view consistency.This paper presents a stylized novel view synthesis method. Applying stateof-the-art stylization methods to novel views frame by frame often causes jittering artifacts due to the lack of cross-view consistency. Therefore, this paper investigates 3D scene stylization that provides a strong inductive bias for consistent novel view synthesis. Specifically, we adopt the emerging neural radiance fields (NeRF) as our choice of 3D scene representation for their capability to render high-quality novel views for a variety of scenes. However, as rendering a novel view from a NeRF requires a large number of samples, training a stylized NeRF requires a large amount of GPU memory that goes beyond an off-the-shelf GPU capacity. We introduce a new training method to address this problem by alternating the NeRF and stylization optimization steps. Such a method enables us to make full use of our hardware memory capacity to both generate images at higher resolution and adopt more expressive image style transfer methods. Our experiments show that our method produces stylized NeRFs for a wide range of content, including indoor, outdoor and dynamic scenes, and synthesizes high-quality novel views with cross-view consistency.

DR-WLC: Dimensionality Reduction cognition for object detection and pose estimation by Watching, Learning and Checking

“…In terms of navigation, NeRF is used as a map, and position estimation is performed based on RGB or depth information [49]- [52]. At the same time, researchers are also exploring NeRF-related principles and techniques in other fields, such as scene editing [53], style transfer [54].…”

Painting 3D Nature in 2D: View Synthesis of Natural Scenes from a Single Semantic Mask

“…While the above works focus on rendering realistic novel view images or reconstructing accurate 3D geometry, some works currently aim to exploit 2D pretrained models to learn a priori knowledge for the neural field. Examples include creating 3D objects driven by text [28,54,71], animating NeRF by audio signals [18,40], and stylizing scenes [26,45,81]. Unlike the previous works, in this paper, we focus on utilizing semantic image synthesis models.…”

UPST-NeRF: Universal Photorealistic Style Transfer of Neural Radiance Fields for 3D Scene

“…Stylized-NeRF [20] utilize the stylization ability of 2D stylization network and neural radiation field for 3D scene stylization, and ARF [54] proposed to stylize the more robust radiance field representation. SNeRF [36] investigated 3D scene stylization, providing a strong inductive bias for consistent novel view synthesis. INS [9] studied unifying the style transfer for 2D coordinate-based representation, neural radiance field, and signed distance function.…”

“…Many excellent works achieve this goal through texture generation [11,22,52] and semantic view synthesis [15,17]. Some recent work [5,9,16,18,20,36,54] can transfer artistic features from a single 2D image to a complete real 3D scene, thereby changing the style in the real scene.…”

“…To solve the blurry results and inconsistent appearance, Stylizing-3D-Scene [5] utilized a hyper network to transfer the style information into the scene representation. To eliminate the jittering artifacts due to the lack of cross-view consistency, SNeRF [36] investigated 3D scene stylization that provides a strong inductive bias for consistent novel view synthesis. ARF [54] proposed to stylize the more robust radiance field representation and produce consistent stylized novel views of high visual quality.…”