Layered neural atlases for consistent video editing

Abstract

We present a method that decomposes, and "unwraps", an input video into a set of layered 2D atlases , each providing a unified representation of the appearance of an object (or background) over the video. For each pixel in the video, our method estimates its corresponding 2D coordinate in each of the atlases, giving us a consistent parameterization of the video, along with an associated alpha (opacity) value. Importantly, we design our atlases to be interpretable and semantic, which facilitates easy and intuitive editing in the atlas domain, with minimal manual work required. Edits applied to a single 2D atlas (or input video frame) are automatically and consistently mapped back to the original video frames, while preserving occlusions, deformation, and other complex scene effects such as shadows and reflections. Our method employs a coordinate-based Multilayer Perceptron (MLP) representation for mappings, atlases, and alphas, which are jointly optimized on a per-video basis, using a combination of video reconstruction and regularization losses. By operating purely in 2D, our method does not require any prior 3D knowledge about scene geometry or camera poses, and can handle complex dynamic real world videos. We demonstrate various video editing applications, including texture mapping, video style transfer, image-to-video texture transfer, and segmentation/labeling propagation, all automatically produced by editing a single 2D atlas image.

Deformable Sprites for Unsupervised Video Decomposition

“…Although our method outputs a rich video decomposition, and not simply object masks, we evaluate it on standard video object segmentation benchmarks where we obtain competitive results. On DAVIS [26], we obtain decomposition results that are similar to recent approaches that require user mask initialization [11], while being faster to optimize (30 minutes vs. 10 hours) due to the low dimensionality of our deformation model. We further demonstrate our approach on a variety of Internet clips, where off-the-shelf segmentation methods do not generalize to discover meaningful groupings.…”

“…As such, we can easily recover Deformable Sprites for objects that do not fall into common categories. This is in contrast to layer decomposition methods such as [21], [27] or [11], which rely on input masks, either from off-the-shelf pre-trained segmentation models or from user interaction. In Figure 4, we show the masks we obtain from our representation compared to offthe-shelf Mask RCNN, and two recent motion segmentation methods [40], [38], trained on DAVIS.…”

“…In Figure 4, we show the masks we obtain from our representation compared to offthe-shelf Mask RCNN, and two recent motion segmentation methods [40], [38], trained on DAVIS. Layer decomposition methods [11], [21] that require input masks would require further user interaction, e.g., using methods such as Grab-Cut [29] to decompose these scenes.…”

“…For instance, Lao et al [16] extends the layered model to explicitly handle 3D motion. More modern methods utilize neural networks in various ways, such as augmenting object masks [21] or computing texture atlases and uv-coordinates for multiple layers using coordinate-based neural networks [11]. One shared aspect of these works is that they require reasonable input masks of an object of interest, usually provided by the user or by segmentation networks trained on pre-defined object categories.…”

NeuVV: Neural Volumetric Videos with Immersive Rendering and Editing

“…Most notably, the Neural Radiance Field (NeRF) [Mildenhall et al 2020] replaces the traditional notion of geometry and appearance with a single neural network where any new camera views can be realistically rendered by querying respective rays from the camera via neural inference. Despite its effectiveness, NeRF and its extensions have been largely focused on the static object, with a few exceptions [Kasten et al 2021;Lu et al 2020;Zhang et al 2021a] to directly tackle dynamic scenes. Further, existing solutions are still a few orders of magnitudes slower than real-time to support immersive volumography.…”