Multiframe Scene Flow with Piecewise Rigid Motion

Golyanik, Vladislav; Kim, Kihwan; Maier, Robert; NieBner, Matthias; Stricker, Didier; Kautz, Jan

doi:10.1109/3dv.2017.00039

Cited by 24 publications

(22 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Ren et al [15] predicted scene flow using iteratively semantic segmentation in stereoscopic vision with the assumption of foreground objects rigidly moving and static background. Also, in other studies, the assumption of local rigid motion and the flexible motion of each point are combined to obtain more detailed motion [18][19][20][21]. Vogel et al [22] proposed a piecewise rigid scene flow and super-pixels are used to segment the scene and constrain scene flow estimation.…”

Section: Related Workmentioning

confidence: 99%

Automatic layered RGB‐D scene flow estimation with optical flow field constraint

Liu

Jin

et al. 2020

IET image process

View full text Add to dashboard Cite

Scene flow estimation with RGB-D frames is receiving increasing attention in digital video processing and computer vision due to the widespread use of depth sensors. Existing methods based on object segmentation have shown their effectiveness for object occlusion and large displacement. However, improper segmentation often causes incomplete segmented areas or incorrect edges, which will result in inaccurate occlusion inference and scene flow estimation. To this end, an automatic layered RGB-D scene flow estimation method is proposed, which achieves more accurate layering of objects in depth image by exploring motion information. The authors exploit super-pixel segmentation for initial layering, which is beneficial for preserving edges and integrity of the objects. Furthermore, an optical flow which is highly correlated with the scene is also used for automatic layering. Finally, the utilisation of super-pixel segmentation and motion information would ensure the integrity of the object area and improve the accuracy of scene flow. They have validated their approach both qualitatively and quantitatively on several public datasets. Experimental results show that the proposed method is able to preserve the integrity of the object and achieve lower root mean square error and average angular error as compared with the current state-of-the-art algorithms. 10 IET Image Process.

show abstract

Section: Related Workmentioning

confidence: 99%

Automatic layered RGB‐D scene flow estimation with optical flow field constraint

Liu

Jin

et al. 2020

IET image process

View full text Add to dashboard Cite

show abstract

“…Its extension to a moving camera case needs to disambiguate the camera ego-motion from object scene motions in 3D. Due to the intrinsic complexity of such task, existing methods often address it with known camera parameters [1,37] or assume scene motions are piecewise rigid [21,23,10,41,42,44]. When depth is known, scene flow can be more accurately estimated.…”

Section: Related Workmentioning

confidence: 99%

“…Di↵erent from the color image, the depth map is rendered from the 3D mesh, which is less noisy and more complete than raw depth. Since the camera movement during the 3D acquisition is small between frames, we sub-sample frames at intervals of [1,2,5,10,20] to create larger motions. We employ a multi-pass rendering approach to generate depth, optical flow and rigidity mask as our ground truth.…”

Section: Refresh Datasetmentioning

confidence: 99%

Learning Rigidity in Dynamic Scenes with a Moving Camera for 3D Motion Field Estimation

Kim

Troccoli

et al. 2018

Computer Vision – ECCV 2018

Self Cite

View full text Add to dashboard Cite

Estimation of 3D motion in a dynamic scene from a temporal pair of images is a core task in many scene understanding problems. In real-world applications, a dynamic scene is commonly captured by a moving camera (i.e., panning, tilting or hand-held), increasing the task complexity because the scene is observed from di↵erent viewpoints. The primary challenge is the disambiguation of the camera motion from scene motion, which becomes more di cult as the amount of rigidity observed decreases, even with successful estimation of 2D image correspondences. Compared to other state-of-the-art 3D scene flow estimation methods, in this paper, we propose to learn the rigidity of a scene in a supervised manner from an extensive collection of dynamic scene data, and directly infer a rigidity mask from two sequential images with depths. With the learned network, we show how we can e↵ectively estimate camera motion and projected scene flow using computed 2D optical flow and the inferred rigidity mask. For training and testing the rigidity network, we also provide a new semi-synthetic dynamic scene dataset (synthetic foreground objects with a real background) and an evaluation split that accounts for the percentage of observed non-rigid pixels. Through our evaluation, we show the proposed framework outperforms current state-of-the-art scene flow estimation methods in challenging dynamic scenes.

show abstract

“…In the case of RGBD sequences, early works attempt to estimate the 3D motion field (scene flow) between consecutive frames [Christoph et al 2015;HornÃącek et al 2014;Quiroga et al 2014;Vogel et al 2014]. To recover parts, super segments can be extracted and grouped according to their estimated rigid transformations from the motion field [Golyani et al 2017]. Alternatively, patches or points lifted from the RGBD frames can be clustered into segments based on their overall flow similarity across frames using Expectation-Maximization or coordinate descent formulations [Jaimez et al 2015;Stückler and Behnke 2015].…”

Section: Prior Workmentioning

confidence: 99%

Deep part induction from articulated object pairs

Huang

Liu

et al. 2018

ACM Trans. Graph.

View full text Add to dashboard Cite

Segmentation Module Fig. 1. We present a neural network for discovering articulated parts of objects. Given a point set representing a 3D CAD model (A) and other functionally similar objects in the form of a scan, 3D mesh or RGB image (B), our network (C) can co-segment the input objects into their articulated parts and parse their underlying motion (D). The learned articulation can further be used for automatically animating the input CAD model (E).Object functionality is often expressed through part articulation -as when the two rigid parts of a scissor pivot against each other to perform the cutting function. Such articulations are often similar across objects within the same functional category. In this paper we explore how the observation of different articulation states provides evidence for part structure and motion of 3D objects. Our method takes as input a pair of unsegmented shapes representing two different articulation states of two functionally related objects, and induces their common parts along with their underlying rigid motion. This is a challenging setting, as we assume no prior shape structure, no prior shape category information, no consistent shape orientation, the articulation states may belong to objects of different geometry, plus we allow inputs to be noisy and partial scans, or point clouds lifted from RGB images. Our method learns a neural network architecture with three modules that respectively propose correspondences, estimate 3D deformation flows, and perform segmentation. To achieve optimal performance, our architecture alternates between correspondence, deformation flow, and segmentation prediction iteratively in an ICP-like fashion. Our results demonstrate that our method significantly outperforms state-of-the-art techniques in the task of discovering articulated parts of objects. In addition, our part induction is object-class agnostic and successfully generalizes to new and unseen objects.

show abstract

Multiframe Scene Flow with Piecewise Rigid Motion

Cited by 24 publications

References 29 publications

Automatic layered RGB‐D scene flow estimation with optical flow field constraint

Automatic layered RGB‐D scene flow estimation with optical flow field constraint

Learning Rigidity in Dynamic Scenes with a Moving Camera for 3D Motion Field Estimation

Deep part induction from articulated object pairs

Contact Info

Product

Resources

About