Factored Neural Representation for Scene Understanding

Abstract

A long‐standing goal in scene understanding is to obtain interpretable and editable representations that can be directly constructed from a raw monocular RGB‐D video, without requiring specialized hardware setup or priors. The problem is significantly more challenging in the presence of multiple moving and/or deforming objects. Traditional methods have approached the setup with a mix of simplifications, scene priors, pretrained templates, or known deformation models. The advent of neural representations, especially neural implicit representations and radiance fields, opens the possibility of end‐to‐end optimization to collectively capture geometry, appearance, and object motion. However, current approaches produce global scene encoding, assume multiview capture with limited or no motion in the scenes, and do not facilitate easy manipulation beyond novel view synthesis. In this work, we introduce a factored neural scene representation that can directly be learned from a monocular RGB‐D video to produce object‐level neural presentations with an explicit encoding of object movement (e.g., rigid trajectory) and/or deformations (e.g., nonrigid movement). We evaluate ours against a set of neural approaches on both synthetic and real data to demonstrate that the representation is efficient, interpretable, and editable (e.g., change object trajectory). Code and data are available at: http://geometry.cs.ucl.ac.uk/projects/2023/factorednerf/.

Recent Trends in 3D Reconstruction of General Non‐Rigid Scenes

“…BANMO further decomposes the object motion into articulated and non‐rigid motion, from which both RPD and TotalRecon benefit. FactoredNeRF [WM23] assumes the root motion and segmentation masks of objects are provided at keyframes, enabling the decomposition of the scene into the background and multiple moving objects. This factored representation allows for object manipulations including removing an object or changing an object's trajectory.…”

“…3.2. Once the per‐object model is learned, individual objects can be moved, removed, duplicated, or their trajectories can be changed [WM23]. For scenes with many objects, this might be impractical as one NeRF per object needs to be learned, and controlling general properties of the scene, such as global illumination, is more challenging.…”