RPM-Net

Abstract

We introduce RPM-Net, a deep learning-based approach which simultaneously infers movable parts and hallucinates their motions from a single, un-segmented, and possibly partial, 3D point cloud shape. RPM-Net is a novel Recurrent Neural Network (RNN), composed of an encoder-decoder pair with interleaved Long Short-Term Memory (LSTM) components, which together predict a temporal sequence of pointwise displacements for the input point cloud. At the same time, the displacements allow the network to learn movable parts, resulting in a motion-based shape segmentation. Recursive applications of RPM-Net on the obtained parts can predict finer-level part motions, resulting in a hierarchical object segmentation. Furthermore, we develop a separate network to estimate part mobilities, e.g., per-part motion parameters, from the segmented motion sequence. Both networks learn deep predictive models from a training set that exemplifies a variety of mobilities for diverse objects. We show results of simultaneous motion and part predictions from synthetic and real scans of 3D objects exhibiting a variety of part mobilities, possibly involving multiple movable parts. 1 The subtle difference between part mobility and part motion is that the term mobility refers to the extent that a part can move; it emphasizes the geometric or transformation characteristics of part motions, e.g., the types of the motions and the reference point or axis of a rotation, etc. On the other hand, motion is a more general term which also encompasses measures reflecting physical properties, such as speed and acceleration. In our work, RPM-Net predicts movable parts and their motions simultaneously, while the second network, Mobility-Net, further predicts part mobilities.