Semantic object reconstruction via casual handheld scanning

Abstract

We introduce a learning-based method to reconstruct objects acquired in a casual handheld scanning setting with a depth camera. Our method is based on two core components. First, a deep network that provides a semantic segmentation and labeling of the frames of an input RGBD sequence. Second, an alignment and reconstruction method that employs the semantic labeling to reconstruct the acquired object from the frames. We demonstrate that the use of a semantic labeling improves the reconstructions of the objects, when compared to methods that use only the depth information of the frames. Moreover, since training a deep network requires a large amount of labeled data, a key contribution of our work is an active self-learning framework to simplify the creation of the training data. Specifically, we iteratively predict the labeling of frames with the neural network, reconstruct the object from the labeled frames, and evaluate the confidence of the labeling, to incrementally train the neural network while requiring only a small amount of user-provided annotations. We show that this method enables the creation of data for training a neural network with high accuracy, while requiring only little manual effort.

InstanceFusion: Real‐time Instance‐level 3D Reconstruction Using a Single RGBD Camera

“…SSCNet [SYZ*17] uses a dilation‐based 3D context module to efficiently expand the receptive field and enable 3D context learning. Hu et al [HWVK*18] design a deep network to produce semantic labels of the RGBD frames, and then use these labels to improve the quality of semantic reconstruction. Hou et al [HDN19] present 3D‐SIS network, which takes the 2D/3D features into training and achieves instance segmentation result from the entire 3D scene.…”

“…According to the data processing type, state‐of‐the‐art (SOTA) methods of 3D semantic reconstruction can be roughly divided into two main classes. One class of methods ( e.g ., [KMYG12], [CLW*14], [HZCP16], [HWVK*18], [HSXF19]) rely on the geometry post‐processing techniques: they stop the sensor scanning, then segment the whole reconstructed 3D scene model into individual objects. Another class of methods (e.g., [TTN16], [RA17], [MHDL17], [RBA18], [LZZZ18]) adopt online processing pipeline, which directly project 2D segmentation results to 3D space, yielding individual reconstructed 3D models.…”

Automatic Pose Generation for Robotic 3-D Scanning of Mechanical Parts

“…None of the previous approaches for scan automation was designed to utilize geometric priors of mechanical parts or CAD models; they assume a continuous [5], [6] smooth surface [4], [7]. Hu et al [33] suggested a method to incorporate semantic labeling instead of primitive labels, to assist registration and reconstruction during the scanning stage. However, in contrast to our work, this work is focused on registration and reconstruction, and the problem of exhaustive scan path is still in the control of the user without any systematic suggestions.…”