Aggregation of local parametric candidates with exemplar-based occlusion handling for optical flow

Abstract

Handling all together large displacements, motion details and occlusions remains an open issue for reliable computation of optical flow in a video sequence. We propose a two-step aggregation paradigm to address this problem. The idea is to supply local motion candidates at every pixel in a first step, and then combine them to determine the global optical flow field in a second step. We exploit local parametric estimations combined with patch correspondences and we experimentally demonstrate that they are sufficient to produce highly accurate motion candidates. The aggregation step is designed as the discrete optimization of a global regularized energy. The occlusion map is estimated jointly with the flow field throughout the two steps. We propose a generic exemplar-based approach for occlusion filling with motion vectors. We achieve state-of-the-art results in computer vision benchmarks, with particularly significant improvements in the case of large displacements and occlusions.

Optical flow modeling and computation: A survey

“…For example, the 8-parameter quadratic model represents rigid motion of a planar surface in perspective projection. The small number of parameters of the affine model and its realistic local assumption make it often considered as the best trade-off between complexity and descriptiveness [27,88,175,193]. The accuracy of local affine motion estimation in appropriate estimation support was experimentally demonstrated in [88].…”

“…A smoothness constraint is assumed on n o and n m , either with a local parametric form [188,193] or a global regularization [55,89,142,159,239]. The offset formulation is also used in [10,88], but is associated to a sparsity constraint aiming at retrieving violations due to occlusions. The more general approach of [74] parametrizes intensity changes in terms of Brightness Transfer Function [106], which coefficients are learned from training data.…”

“…Patches corrupted by strong intensity edges are displaced by a mean-shift procedure to reach more homogeneous regions. The method described in [88] adapts both size and position of the square patches. This adaptation is performed thanks to the aggregation framework decoupling the estimation of candidates with predefined sizes and positions, and their selection achieved with a global model (see Section 5).…”

Spatially-Variant Kernel for Optical Flow Under Low Signal-to-Noise Ratios Application to Microscopy

“…We integrated our model in a variational framework adapted to small displacements. Other approaches to deal with large displacements and occlusions could be considered in the future [16,23,26]. …”

“…The reason is the difficulty to determine regions where motion can be 1 accurately approximated by a parametric motion model. However, it has been shown that when appropriate regions are chosen, local parametric models can yield excellent performances [27,16].…”