NeuTex: Neural Texture Mapping for Volumetric Neural Rendering

Abstract

Recent work [28, 5] has demonstrated that volumetric scene representations combined with differentiable volume rendering can enable photo-realistic rendering for challenging scenes that mesh reconstruction fails on. However, these methods entangle geometry and appearance in a "black-box" volume that cannot be edited. Instead, we present an approach that explicitly disentangles geometry-represented as a continuous 3D volume-from appearance-represented as a continuous 2D texture map. We achieve this by introducing a 3D-to-2D texture mapping (or surface parameterization) network into volumetric representations. We constrain this texture mapping network using an additional 2D-to-3D inverse mapping network and a novel cycle consistency loss to make 3D surface points map to 2D texture points that map back to the original 3D points. We demonstrate that this representation can be reconstructed using only multi-view image supervision and generates high-quality rendering results. More importantly, by separating geometry and texture, we allow users to edit appearance by simply editing 2D texture maps.

MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo

“…In particular, NeRF [29] combines MLPs with differentiable volume rendering and achieves photorealistic view synthesis. Following works have tried to advance its performance on view synthesis [27,24]; other relevant works extend it to support other neural rendering tasks like dynamic view synthesis, relighting, and editing [23,31,1,40]. However, most prior works still follow the original NeRF and require an expensive per-scene optimization process.…”

Layered neural atlases for consistent video editing

“…In 3D approaches, a geometric model of the scene is estimated, edits are then applied in 3D, and mapped back to the video frames. This approach is most commonly used in a domain where strong 3D priors exist (for example, face filters in Snapchat), or on static scenes (e.g., [Luo et al 2020;Xiang et al 2021]).…”

Stylizing 3D Scene via Implicit Representation and HyperNetwork

“…Such patch-based algorithms typically need particular design to avoid seams between patches and do not consider the texture synthesis from more holistic or global perspective. With the recent advance of deep learning techniques, different methods [Groueix et al 2018;Kanazawa et al 2018;Xiang et al 2021] are proposed to learn the correspondences between 3D shape (e.g. meshes) and the texture space for realizing the texture transfer.…”

“…meshes) and the texture space for realizing the texture transfer. For instance, Xiang et al [2021] learn a mapping from the implicit 3D representation to the 2D texture map, such that one can change the appearance of a 3D model by swapping the 2D texture map. However, such a texture mapping approach may generate unnatural results if the texture image is mapped across object edges or boundaries.…”

“…Numerous efforts have been made for controlling the appearance of the rendered 3D target. For instance, Xiang et al [2021] and Kanazawa et al [2018] formulate it as a texture synthesis problem. Specifically, they render the 3D objects with the desired texture by aligning the coordinates of the 2D UV (texture) map to those of the target object.…”