Spatiotemporal Closure

Abstract

Abstract. Spatiotemporal segmentation is an essential task for video analysis. The strong interconnection between finding an object's spatial support and finding its motion characteristics makes the problem particularly challenging. Motivated by closure detection techniques in 2D images, this paper introduces the concept of spatiotemporal closure. Treating the spatiotemporal volume as a single entity, we extract contiguous "tubes" whose overall surface is supported by strong appearance and motion discontinuties. Formulating our closure cost over a graph of spatiotemporal superpixels, we show how it can be globally minimized using the parametric maxflow framework in an efficient manner. The resulting approach automatically recovers coherent spatiotemporal components, corresponding to objects, object parts, and object unions, providing a good set of multiscale spatiotemporal hypotheses for high-level video analysis.

Video Segmentation with Superpixels

“…A large body of literature on VS exists leveraging on appearance [11,[14][15][16], motion [3,2,12], or a combination of cues [10,[17][18][19][20][21][22][23]13]. A variety of techniques is used, e.g.…”

“…Intuitively, besides within-frame similarities used for image segmentation, VS should also use between-frame similarities to connect and thus segment corresponding regions across multiple frames. While recent work on VS proposes a variety of such between-frame similarities [2,[10][11][12][13] there is no common agreement yet on which similarities are necessary for best performance.…”

Interactive video segmentation using occlusion boundaries and temporally coherent superpixels

“…Levinshtein et al [11] generated temporally coherent superpixels, built a superpixel graph, and hypothesized multiple superpixel groupings with parametric maxflow. We adopt Levinshtein et al's method to generate superpixels, but take occlusion boundaries into account in the affinities of edges between superpixels.…”

“…In the preprocessing stage, optical flow is computed [17] forward and back and then temporally coherent superpixel segmentations for each video frame [11] (the target number of superpixels per frame is fixed, S). The next preprocessing operation is to run a simple occlusion detector that fires for forward flow inconsistent with backward flow (inequation 1) or large flow gradient magnitude (inequation 2) [18].…”

“…The difference from [11] is the 1 in the denominator, which makes the distance tend to 0 if both motions are small. We define the distance between the two superpixels of a between-frame edge as the average of an overlap distance and a color distance:…”