Database‐Assisted Object Retrieval for Real‐Time 3D Reconstruction

Abstract

Figure 1: As a cluttered 3D scene is scanned and reconstructed in real time (top), we continuously query a large shape database, retrieving and registering similar objects to the scan (bottom). Noisy, partial scanned geometry is then replaced with the models, resulting in a complete, high-quality semantic reconstruction (middle). AbstractIn recent years, real-time 3D scanning technology has developed significantly and is now able to capture large environments with considerable accuracy. Unfortunately, the reconstructed geometry still suffers from incompleteness, due to occlusions and lack of view coverage, resulting in unsatisfactory reconstructions. In order to overcome these fundamental physical limitations, we present a novel reconstruction approach based on retrieving objects from a 3D shape database while scanning an environment in real-time. With this approach, we are able to replace scanned RGB-D data with complete, hand-modeled objects from shape databases. We align and scale retrieved models to the input data to obtain a high-quality virtual representation of the real-world environment that is quite faithful to the original geometry. In contrast to previous methods, we are able to retrieve objects in cluttered and noisy scenes even when the database contains only similar models, but no exact matches. In addition, we put a strong focus on object retrieval in an interactive scanning context -our algorithm runs directly on 3D scanning data structures, and is able to query databases of thousands of models in an online fashion during scanning.

Simultaneous Object Segmentation and Recognition by Merging CNN Outputs from Uniformly Distributed Multiple Viewpoints

“…Li et al [20] proposed a multi-view-based object recognition method that works in real-time by extracting keypoints near 3D corners. While it achieves high efficiency with respect to the computational cost, it cannot manage smooth-surfaced objects whose keypoints are hard to extract (e.g., spherical objects).…”

Bayesian Eigenobjects: A Unified Framework for 3D Robot Perception

“…One approach [37,36,16] is to construct a large database of complete and high quality object scans. When a novel object is encountered, a query is performed to find the most similar known object.…”

“…These approaches suffer however, if an exact match is not found. While some approaches still attempt to find a nearest match in such a case [16,3], the results will be poor if the query object is sufficiently different from any in the database. Looked at another way, instancebased database models are necessarily discrete, containing only a finite number of exemplars, and will yield poor results if coverage of the space is insufficient.…”

“…Similar to Li et al [16], we assume that the partial query objects consist of filled, empty, and unknown pixels. Let d denote the number of known (filled and empty) elements of o q .…”

Unsupervised learning of spatial-temporal models of objects in a long-term autonomy scenario

“…The SLAM++ system of Moreno et al [7] uses an a-priori constructed object database both for object recognition and for camera localization. Similarly, Li et al [8] use a database of object models which replace segmented objects in the scene, with an emphasis on maintaining and matching against large databases of objects. In our work we focus on the individual dynamic elements and we make no assumptions about them being known ahead of time.…”