NeuralDiff: Segmenting 3D objects that move in egocentric videos

Abstract

Figure 1: Given a video sequence captured from an egocentric viewpoint, we segment all the objects that the actor/observer interacts with. We achieve this by means of a neural architecture, NeuralDiff, that learns to decompose each frame into a static background and a dynamic foreground, comprising the manipulated objects, which seldomly move in the sequence, and the actor's body, which moves continually and heavily occludes the scene. The neural network contains three streams that, via different inductive biases, reconstruct the background, the objects and the actor in 3D, and is thus able to render images and their segmentations also for viewpoints that do not exist in the original video sequence (see the left part of the figure, where the camera is assumed to remain at its initial position while the action unfolds).

Neural Fields in Visual Computing and Beyond

“…The application of neural fields for "high-level" semantic tasks remains an open problem. Examples of these problems include understanding 3D scene layout [WYN21], 3D scene interaction [LMW * 21a], and grouping of data into more meaningful entities [TLV21]. Furthermore, neural fields have focused on a single data modality, but exploring the fusion of multiple modalities could be a fruitful topic of research.…”

Neural Fields in Visual Computing and Beyond

“…Thus far, neural fields have primarily been used to solve "lowlevel" (e.g., image synthesis) and "mid-level" (e.g., 3D reconstruction) tasks. We believe that "high-level" tasks that involve grouping of data into more meaningful concepts is a fruitful direction of future work [TLV21]. A common limitation of many neural fields is their inability to generalize well to new data.…”