Deformable Sprites for Unsupervised Video Decomposition

Abstract

Input: Video with Flow Propagated EditsOutput: Deformable Sprites Per-frame Masks & Transforms Global SpritesLayer 2Layer 1 EditsLayer 1 Layer 2Figure 1. Video Decomposition with Deformable Sprites. Given an RGB video and its optical flow (left), we present a method that decomposes the video into layers of persistent motion groups. We represent each group as a Deformable Sprite (center), which consists of an RGB sprite image, and masks and non-rigid transforms mapping the sprite to each frame. We fit Deformable Sprites to each input video independently without any pre-training or user input. The resulting decomposition captures long-term correspondences of sprites over time, enabling effects such as propagating edits on the sprite across the entire video (right). We show full videos of our results at the project site.