Video matting of complex scenes

Abstract

This paper describes a new framework for video matting, the process of pulling a high-quality alpha matte and foreground from a video sequence. The framework builds upon techniques in natural image matting, optical flow computation, and background estimation. User interaction is comprised of garbage matte specification if background estimation is needed, and hand-drawn keyframe segmentations into "foreground," "background," and "unknown". The segmentations, called trimaps, are interpolated across the video volume using forward and backward optical flow. Competing flow estimates are combined based on information about where flow is likely to be accurate. A Bayesian matting technique uses the flowed trimaps to yield high-quality mattes of moving foreground elements with complex boundaries filmed by a moving camera. A novel technique for smoke matte extraction is also demonstrated.

Confidence-driven image co-matting

“…Fig. 6 shows the results of two videos frames from an example proposed in [33]. The two input images share very similar background appearance, except for a few local regions marked by the color rectangles.…”

Practical automatic background substitution for live video

“…Unlike interactive video matting methods [1,15,16,26], which need user interaction during video playback, automatic video matting is more appropriate for live video. The earliest kind of automatic video matting problem is constant color matting [27], which uses a constant backing color, often blue, so is usually called blue screen matting.…”

“…For images, previous methods are often samplingbased [30], affinity-based [24], or a combination of both [31], computing alpha values for the unknown region based on the known region information. For video, Chuang et al [15] use optical flow to propagate the trimap from one frame to another. Video SnapCut [1] maintains a collection of local classifiers around the object boundary.…”

“…Although border matting also uses dynamic programming to minimize an energy function that encourages alpha values varying smoothly along the boundary, the result of border matting is far from globally optimal. On the other hand, interactive approaches have been proposed to handle more complicated camera motion [1,15,16]. Since strictly real-time performance is unnecessary for such applications, they compute more precise fractional opacities along the segmentation boundary from the beginning.…”

Dynamic Color Flow: A Motion-Adaptive Color Model for Object Segmentation in Video

“…Progressive Background Completion. In case a clean plate is not available, we use a progressive background completion method similar to the one proposed in [20]. Suppose the first t frames have been segmented, the segmented backgrounds are projected onto frame t + 1 in a reverse order, from frame t to frame 1, recovering as much occluded background as possible.…”