DynamicFusion: Reconstruction and tracking of non-rigid scenes in real-time

Newcombe, Richard; Fox, Dieter; Seitz, Steven M.

doi:10.1109/cvpr.2015.7298631

Cited by 796 publications

(676 citation statements)

References 37 publications

Supporting

Mentioning

651

Contrasting

Unclassified

Order By: Relevance

“…However, these methods cannot capture dynamic scenes. Newcombe et al [40], Dou et al [16] and Innmann et al [27] support changes to the scene by deforming the reconstructed volume. These methods expect accurate object tracking, which can fail under complex or fast movement.…”

Section: Volumetric Methodsmentioning

confidence: 99%

Real-time scene reconstruction and triangle mesh generation using multiple RGB-D cameras

Meerits

Nozick

Saitō

2017

J Real-Time Image Proc

View full text Add to dashboard Cite

We present a novel 3D reconstruction system that can generate a stable triangle mesh using data from multiple RGB-D sensors in real time for dynamic scenes. The first part of the system uses moving least squares (MLS) point set surfaces to smooth and filter point clouds acquired from RGB-D sensors. The second part of the system generates triangle meshes from point clouds. The whole pipeline is executed on the GPU and is tailored to scale linearly with the size of the input data. Our contributions include changes to the MLS method for improving meshing, a fast triangle mesh generation method and GPU implementations of all parts of the pipeline.

show abstract

Section: Volumetric Methodsmentioning

confidence: 99%

Real-time scene reconstruction and triangle mesh generation using multiple RGB-D cameras

Meerits

Nozick

Saitō

2017

J Real-Time Image Proc

View full text Add to dashboard Cite

show abstract

“…The most notable in this area is the work of Newcombe et al, in particular Kinect Fusion [23] and Dynamic Fusion [22]. However, since this work focuses on tracking and modelling from strictly monocular RGB video, a more detailed review is omitted here.…”

Section: Related Workmentioning

confidence: 99%

TMAGIC: A Model-Free 3D Tracker

Lebeda

Hadfield

Bowden

2017

IEEE Trans. on Image Process.

View full text Add to dashboard Cite

Abstract-Significant effort has been devoted within the visual tracking community to rapid learning of object properties on the fly. However, state-of-the-art approaches still often fail in cases such as rapid out-of-plane rotation, when the appearance changes suddenly. One of the major contributions of this work is a radical rethinking of the traditional wisdom of modelling 3D motion as appearance change during tracking. Instead, 3D motion is modelled as 3D motion. This intuitive but previously unexplored approach provides new possibilities in visual tracking research.Firstly, 3D tracking is more general, as large out-of-plane motion is often fatal for 2D trackers, but helps 3D trackers to build better models. Secondly, the tracker's internal model of the object can be used in many different applications and it could even become the main motivation, with tracking supporting reconstruction rather than vice versa. This effectively bridges the gap between visual tracking and Structure from Motion.A new benchmark dataset of sequences with extreme out-ofplane rotation is presented and an online leader-board offered to stimulate new research in the relatively underdeveloped area of 3D tracking. The proposed method, provided as a baseline, is capable of successfully tracking these sequences, all of which pose a considerable challenge to 2D trackers (error reduced by 46 %).

show abstract

“…However, these approaches assume a reconstruction of the full non-rigid object surface at each time frame and do not easily extend to 4D alignment of partial surface reconstructions or depth maps. The wide-spread availability of low-cost depth sensors has motivated the development of methods for temporal correspondence or alignment and 4D modelling from partial dynamic surface observations [8,9,10,11]. Scene flow techniques [12,13] typically estimate the pairwise surface or volume correspondence between reconstructions at successive frames but do not extend to 4D alignment or correspondence across complete sequences due to drift and failure for rapid and complex motion.…”

Section: Introductionmentioning

confidence: 99%

“…However these methods fail in the case of occlusion, large motions, background clutter, deformation, moving cameras and appearance of new parts of objects. Recent work has introduced approaches, such as DynamicFusion [8], for 4D modelling from depth image sequences integrating temporal observations of non-rigid shape to resolve fine detail. Approaches to 4D modelling from partial surface observations are currently limited to relatively simple isolated objects such as the human face or upper-body and do not handle large non-rigid deformations such as loose clothing.…”

Section: Introductionmentioning

confidence: 99%

4D Match Trees for Non-rigid Surface Alignment

Mustafa

Kim

Hilton

2016

Computer Vision – ECCV 2016

View full text Add to dashboard Cite

Abstract. This paper presents a method for dense 4D temporal alignment of partial reconstructions of non-rigid surfaces observed from single or multiple moving cameras of complex scenes. 4D Match Trees are introduced for robust global alignment of non-rigid shape based on the similarity between images across sequences and views. Wide-timeframe sparse correspondence between arbitrary pairs of images is established using a segmentation-based feature detector (SFD) which is demonstrated to give improved matching of non-rigid shape. Sparse SFD correspondence allows the similarity between any pair of image frames to be estimated for moving cameras and multiple views. This enables the 4D Match Tree to be constructed which minimises the observed change in non-rigid shape for global alignment across all images. Dense 4D temporal correspondence across all frames is then estimated by traversing the 4D Match tree using optical flow initialised from the sparse feature matches. The approach is evaluated on single and multiple view images sequences for alignment of partial surface reconstructions of dynamic objects in complex indoor and outdoor scenes to obtain a temporally consistent 4D representation. Comparison to previous 2D and 3D scene flow demonstrates that 4D Match Trees achieve reduced errors due to drift and improved robustness to large non-rigid deformations.

show abstract

DynamicFusion: Reconstruction and tracking of non-rigid scenes in real-time

Abstract: Real-time reconstructions of a moving scene with DynamicFusion; both the person and the camera are moving. The initially noisy and incomplete model is progressively denoised and completed over time (left to right).

Cited by 796 publications

References 37 publications

Real-time scene reconstruction and triangle mesh generation using multiple RGB-D cameras

Real-time scene reconstruction and triangle mesh generation using multiple RGB-D cameras

TMAGIC: A Model-Free 3D Tracker

4D Match Trees for Non-rigid Surface Alignment

Contact Info

Product

Resources

About