Multi-step flow fusion: towards accurate and dense correspondences in long video shots

Abstract

The aim of this work is to estimate dense displacement fields over long video shots. Put in sequence they are useful for representing point trajectories but also for propagating (pulling) information from a reference frame to the rest of the video. Highly elaborated optical flow estimation algorithms are at hand, and they were applied before for dense point tracking by simple accumulation, however with unavoidable position drift. On the other hand, direct long-term point matching is more robust to such deviations, but it is very sensitive to ambiguous correspondences. Why not combining the benefits of both approaches? Following this idea, we develop a multi-step flow fusion method that optimally generates dense long-term displacement fields by first merging several candidate estimated paths and then filtering the tracks in the spatio-temporal domain. Our approach permits to handle small and large displacements with improved accuracy and it is able to recover a trajectory after temporary occlusions. Especially useful for video editing applications, we attack the problem of graphic element insertion and video volume segmentation, together with a number of quantitative comparisons on ground-truth data with state-of-the-art approaches.

Dense motion estimation between distant frames: Combinatorial multi-step integration and statistical selection

“…The proposed combinatorial integration combined to SP+GO gives better performance compared to the multistep fusion (MSF) method [12] according to PSNR scores of Table 2. The MSF method itself has been shown in [12] to outperform state-of-the-art methods such as [3,6,7].…”

“…The MSF method itself has been shown in [12] to outperform state-of-the-art methods such as [3,6,7].…”

“…It links each pixel x a of I a to a corresponding position in I b . Elementary optical flow fields can be computed between consecutive frames or with different frame steps [11,12], i.e. with larger inter-frame distances.…”

“…The data term involves the matching cost C(x a , d Table 2: Registration and PSNR assessment with: the combinatorial integration followed by the global optimization (GO); by the statistical processing combined to GO (SP+GO); the multi-step fusion (MSF) method [12]. PSNR scores are computed on the whole images of Newspaper (Fig.…”

“…Following a similar approach to that presented in [10], one can select for each pixel the optimal motion vector among a set of candidate motion fields based on intrinsic motion field quality and spatial regularization. A more sophisticated processing, described in [11,12], consists in sequentially merging a set of concatenated multi-step motion fields at intermediate frames up to the target frame. However, in either case, the optimal motion vector selection strongly depends on the same optical flow assumptions that frequently fail between distant frames.…”