NeRFshop

Abstract

Neural Radiance Fields (NeRFs) have revolutionized novel view synthesis for captured scenes, with recent methods allowing interactive free-viewpoint navigation and fast training for scene reconstruction. However, the implicit representations used by these methods---often including neural networks and complex encodings---make them difficult to edit. Some initial methods have been proposed, but they suffer from limited editing capabilities and/or from a lack of interactivity, and are thus unsuitable for interactive editing of captured scenes. We tackle both limitations and introduce NeRFshop, a novel end-to-end method that allows users to interactively select and deform objects through cage-based transformations. NeRFshop provides fine scribble-based user control for the selection of regions or objects to edit, semi-automatic cage creation, and interactive volumetric manipulation of scene content thanks to our GPU-friendly two-level interpolation scheme. Further, we introduce a preliminary approach that reduces potential resulting artifacts of these transformations with a volumetric membrane interpolation technique inspired by Poisson image editing and provide a process that "distills" the edits into a standalone NeRF representation.

A Post Processing Technique to Automatically Remove Floater Artifacts in Neural Radiance Fields

“…Niemeyer et al [NG21] decompose a scene into multiple NeRFs, which contain one object each, allowing translation, rotation and stretching of individual objects. NeRF‐Editing [YSL*22] and [JKK*23] establish a correspondence between an explicit mesh representation and the implicit neural representation of the NeRF. Object manipulations are performed by editing this mesh representation and bending or stretching the camera rays accordingly, while they travel through these cages during a retraining process.…”

“…Jambon et al [JKK*23] present NeRFshop, which is focused on manually editing NeRFs by manually annotating and manipulating scene volumes. During a retraining step the rays path is bend in these manipulated cages, to acquire a new model that incorporates the performed edits.…”

“…Increasing the threshold for the represented scene volume in the Occupancy Clustering technique could mitigate the effects of a violated coherency assumption for transparent or thin structures. Since this increases the probability of incorporating a big floater artifact, a manual annotation of remaining artifacts comparable to the concurrent work NeRFshop [JKK*23]. This could easily be achieved by intersecting a manually annotated pixel with the occupancy grid and annotating the corresponding cluster as a floater .…”

“…Examples of these artifacts are illustrated in the rendered images on the left side of figure 1. Recent publications have addressed this issue by decomposing the scene into its view‐dependant and independent components [LTT23], scoring scene regions based on the frequency of appearance in the input views [WWT*23] or suppressing them in a retaining step after interactive annotation of the relevant regions [JKK*23]. However, these contributions require retraining of the model, adaptation of the training process or manual annotation.…”

Neural Impostor: Editing Neural Radiance Fields with Explicit Shape Manipulation

“…The employment of point clouds [XXP*22, CLW23] enables volumetric modeling but lacks geometry constraints, complicating contiguous authoring. The utilization of volumetric cages [GKE*22, YSL*22, XH22, PYL*22, JKK*23], on the other hand, offers topology constraints for stable authoring and benefits in physical simulation. Despite these advantages, their rendering speed is often hampered by complex point‐in‐tetrahedron queries.…”