SceneFlowFields: Dense Interpolation of Sparse Scene Flow Correspondences

Abstract

While most scene flow methods use either variational optimization or a strong rigid motion assumption, we show for the first time that scene flow can also be estimated by dense interpolation of sparse matches. To this end, we find sparse matches across two stereo image pairs that are detected without any prior regularization and perform dense interpolation preserving geometric and motion boundaries by using edge information. A few iterations of variational energy minimization are performed to refine our results, which are thoroughly evaluated on the KITTI benchmark and additionally compared to state-of-the-art on MPI Sintel. For application in an automotive context, we further show that an optional ego-motion model helps to boost performance and blends smoothly into our approach to produce a segmentation of the scene into static and dynamic parts.

LiDAR-Flow: Dense Scene Flow Estimation from Sparse LiDAR and Stereo Images

“…That method does not consider the non-rigidly moving objects like pedestrians or bicyclists. To handle non-rigidly objects, Schuster et al [8] builds the concept of sparse matching and uses an edge-preserving interpolation method [16] to spread the sparse matches into the entire image. The reliability of generating these matches against light change, perspective deformations and occlusions is increased more by considering multiple frames of stereo [5] and using a robust interpolation method [6].…”

“…The initialization and the propagation algorithm follow [8] but with further constraints to account for the additional sparse LiDAR measurements.…”

“…The proposed consistency check algorithm generates a sparse set of matches. To speed up the interpolation, the remaining scene flow pixels, except LiDAR measurements, will be subjected to a sparsification process as in [8] (see Fig. 2).…”

“…Our LiDAR-Flow utilizes LiDAR measurements as anchor points to support each processing step across the framework. We utilize a sparse-to-dense approach for our LiDAR-Flow, which is established for image-only scene flow already [5], [6], [7], [8]. We initialize disparity with the LiDAR measurements, estimate flow based on image matching and iteratively propagate these measurements with random search into the neighborhood.…”

A Conditional Adversarial Network for Scene Flow Estimation

“…Most of these methods uses variational framework. However, Schuster et al [21] proposed scene flow estimation method based on dense interpolation of sparse matches from stereo images. The variational optimization was used at later stage for refinement.…”