As-Rigid-As-Possible Stereo under Second Order Smoothness Priors

Abstract

Imposing smoothness priors is a key idea of the top-ranked global stereo models. Recent progresses demonstrated the power of second order priors which are usually defined by either explicitly considering three-pixel neighborhoods, or implicitly using a so-called 3D-label for each pixel. In contrast to the traditional first-order priors which only prefer fronto-parallel surfaces, second-order priors encourage arbitrary collinear structures. However, we still can find defective regions in matching results even under such powerful priors, e.g., large textureless regions. One reason is that most of the stereo models are non-convex, where pixelwise smoothness priors, i.e., local constraints, are too flexible to prevent the solution from trapping in bad local minimums. On the other hand, long-range spatial constraints, especially the segment-based priors, have advantages on this problem. However, segment-based priors are too rigid to handle curved surfaces. We present a mixture model to combine the benefits of these two kinds of priors, whose energy function consists of two terms 1) a Laplacian operator on the disparity map which imposes pixel-wise second-order smoothness; 2) a segment-wise matching cost as a function of quadratic surface, which encourages "as-rigid-as-possible" smoothness. To effectively solve the problem, we introduce an intermediate term to decouple the two subenergies, which enables an alternated optimization algorithm that is about an order of magnitude faster than PatchMatch [1]. Our approach is one of the top ranked models on the Middlebury benchmark at sub-pixel accuracy.

Video Generation From Single Semantic Label Map

“…Neighboring pixels usually belong to the same object, thus they tend to have similar displacement. Therefore, similar to previous work [40,30] we also add a smoothness constraint to encourage flow in a local neighborhood to be similar.…”

Depth from motion for smartphone AR

“…This fronto-parallel bias in depth estimation has been noted in the literature. [Woodford et al 2009] and [Zhang et al 2014] address this issue using specialized optimization algorithms designed to recover depth maps that minimize second-order variation, rather than first-order variation. These approaches work well, but are too expensive for real-time use and do not appear to be amenable to fast bilateral-space optimization.…”

Stereo reconstruction using top-down cues

“…For example smoothly curving surfaces can never be perfectly modelled using simple oriented planar primitives as used in many previous techniques [28,29,30], while the curved surfaces used by Zhang et al . [31] cannot model rough surfaces such as jagged stone. Despite this, reasonably chosen primitives can prove extremely effective for reconstruction within specialised domains such as urban environments.…”