MultiBodySync: Multi-Body Segmentation and Motion Estimation via 3D Scan Synchronization

Abstract

We present MultiBodySync, a novel, end-to-end trainable multi-body motion segmentation and rigid registration framework for multiple input 3D point clouds. The two non-trivial challenges posed by this multi-scan multibody setting that we investigate are: (i) guaranteeing correspondence and segmentation consistency across multiple input point clouds capturing different spatial arrangements of bodies or body parts; and (ii) obtaining robust motionbased rigid body segmentation applicable to novel object categories.We propose an approach to address these issues that incorporates spectral synchronization into an iterative deep declarative network, so as to simultaneously recover consistent correspondences as well as motion segmentation. At the same time, by explicitly disentangling the correspondence and motion segmentation estimation modules, we achieve strong generalizability across different object categories. Our extensive evaluations demonstrate that our method is effective on various datasets ranging from rigid parts in articulated objects to individually moving objects in a 3D scene, be it single-view or full point clouds. Code at https://github.com/ huangjh-pub/multibody-sync.

Lepard: Learning partial point cloud matching in rigid and deformable scenes

“…2) In deformable cases, our method does not explicitly handle topological changes. A potential solution is to jointly learn matching with motion segmentation as in [23,27]. 3) Finally, matching and registration for deformable point clouds in the low overlapping cases is particularly challenging due to data incompleteness and the high complexity of non-rigid motion.…”

OPD: Single-view 3D Openable Part Detection

“…Input CC Seg OP PS #cat #obj #part Snapshot [8] 3D mesh 368 368 Shape2Motion [34] PC 45 2440 6762 RPMNet [41] PC 969 1420 DeepPartInduction [43] Pair of PCs 16 16881 MultiBodySync [10] Multiple PCs 16 ScrewNet [11] Depth video 9 4496 4496 Liu et al [16] RGB video 3 ANCSH [13] Single-view PC 5 237 343 Abbatematteo et al [ This structure-agnostic approach allows us to more easily achieve cross-category generality. That is, a single model can tackle instances from a variety of object categories (e.g.…”

“…More recent work has started to explore training of single models for motion prediction across categories and structures [10,11,11,16]. Zeng et al [44] proposed an optical flow-based approach on RGB-D images given segmentation masks of the moving part and fixed part.…”

“…They evaluate only on ground truth segmentation and do not investigate how part segmentation and detection influences the accuracy of motion prediction. Others have proposed to predict articulated part pose from depth sequences [11], image video sequences [16], or synchronizing multiple point clouds [10].…”