Multi-reference combinatorial strategy towards longer long-term dense motion estimation

Abstract

International audienceThis paper addresses the estimation of accurate long-term dense motion fields from videos of complex scenes. With computer vision applications such as video editing in mind, we exploit optical flows estimated with various inter-frame distances and combine them through multi-step integration and statistical selection (MISS). In this context, managing numerous combinations of multi-step optical flows requires a complexity reduction scheme to overcome computational and memory issues. Our contribution is two-fold. First, we provide an exhaustive analysis of available single-reference complexity reduction strategies. Second, we propose a simple and efficient alternative related to multi-reference frames multi-step integration and statistical selection (MR-MISS). Our method automatically inserts intermediate reference frames once matching failures are detected to re-generate the motion estimation process and re-correlates the resulting dense trajectories. By this way, it reaches longer accurate displacement fields while efficiently reducing the complexity. Experiments on challenging sequences reveal improved results compared to state-of-the-art methods including existing MISS schemes both in terms of complexity reduction and accuracy improvement (C) 2016 Elsevier Inc. All rights reserved

Unsupervised learning-based long-term superpixel tracking

“…Multi-step integration has been previously presented as an exhaustive candidate generation process. In practice, selecting only a subset of all possible step sequences and therefore associated multi-step paths is required to be able to build and keep in memory the multi-step integration stage outputs growing exponentially [6]. Overall, 5877241 multi-step paths can be generated for a distance of 30 frames with steps 1, 2, 5 and 10 for instance.…”

“…Establishing long-term superpixel correspondences requires to perform superpixel matching between consec-utive and distant frames and therefore to handle simultaneously small and large displacements. To address this challenge, we exploit the concept of multi-step integration introduced for long-term motion estimation using optical flow [6].…”

“…...,|F |} and S = {s j } j∈{1,...,|S|} be respectively the set of |F| and |S| connected superpixels partitioning I ref and I n .Both superpixel trajectory estimation between the reference frame and all the images of the sequence and superpixel matching to match each image to the reference frame can be considered, as in[19,6]. From-the-reference estimation isuseful for information pushing from superpixels of I ref whereas to-the-reference estimation allows information propagation over superpixels of each frame I n by pulling it from I ref .…”

“…term superpixel pairings between I ref and I n using intermediate superpixel correspondences to form a significative set of samples upon which statistical selection (Sect.3.2) is relevant. Formely introduced for optical flow integration[6], we show that this heuristic can be extended towards accurate longterm superpixel tracking. As inputs, we take a set of superpixel match fields pre-estimated from each frame of V including I ref .…”

“…As inputs, we take a set of superpixel match fields pre-estimated from each frame of V including I ref . These matches are computed between consecutive frames or with larger inter-frame distances[6] using learning-based superpixel matching (Sect.2). Let A n = {α 1 , α 2 , .…”