Motion Cooperation: Smooth Piece-wise Rigid Scene Flow from RGB-D Images

Abstract

We propose a novel joint registration and segmentation approach to estimate scene flow from RGB-D images. Instead of assuming the scene to be composed of a number of independent rigidly-moving parts, we use non-binary labels to capture non-rigid deformations at transitions between the rigid parts of the scene. Thus, the velocity of any point can be computed as a linear combination (interpolation) of the estimated rigid motions, which provides better results than traditional sharp piecewise segmentations. Within a variational framework, the smooth segments of the scene and their corresponding rigid velocities are alternately refined until convergence. A K-means-based segmentation is employed as an initialization, and the number of regions is subsequently adapted during the optimization process to capture any arbitrary number of independently moving objects. We evaluate our approach with both synthetic and real RGB-D images that contain varied and large motions. The experiments show that our method estimates the scene flow more accurately than the most recent works in the field, and at the same time provides a meaningful segmentation of the scene based on 3D motion.

Topology-Aware Non-Rigid Point Cloud Registration

“…The above prohibit reaching definitive conclusions. However, the fact that we adopt an arguably more difficult evaluation strategy (at full resolution, which means smaller pixel size for EPE interpretation, and evaluating over whole sequences) and still obtain comparable EPE and AE absolute values to the ones reported in [11] (averages of 1.203 and 6.559, respectively), leads us to argue that FB-Warp compares favorably to MC-Flow.…”

“…PD-Flow is very accurate in estimating slow motions (e.g., in the sleeping_2 sequence), but falls behind in most cases, producing particularly large errors in sequences that contain very fast motions, such as ambush_5 and ambush_6. We also refer the reader to the Sintel-based evaluation of MC-Flow [11] ( Table 2 of that paper), another state-of-the-art scene flow algorithm with significantly better performance than PD-Flow in estimating large motions. We were unable to evaluate MC-Flow ourselves, because its implementation has not been released.…”

Deep part induction from articulated object pairs

“…To recover parts, super segments can be extracted and grouped according to their estimated rigid transformations from the motion field [Golyani et al 2017]. Alternatively, patches or points lifted from the RGBD frames can be clustered into segments based on their overall flow similarity across frames using Expectation-Maximization or coordinate descent formulations [Jaimez et al 2015;Stückler and Behnke 2015]. Parts can also be extracted from 3D point flows through direct clustering on point trajectories [Pillai et al 2014;Tzionas and Gall 2016a].…”

Multiframe Scene Flow with Piecewise Rigid Motion

“…A GPU implementation of a parallel primal-dual solver enabled real-time processing rates for RGB-D scene flow estimation. Several approaches jointly estimate segmentation and scene flow [11,19]. Motion Cooperation (MC-)Flow [11] relies on a linear subspace model, i.e., the velocity of every point is represented as a sum of estimated per-segment rigid motions.…”

“…4.2), as it is hard-coded for Kinect recordings. Unfortunately, the source code for MC-Flow [11] is not publicly available.…”