What and Where: 3D Object Recognition with Accurate Pose

Abstract

Abstract. Many applications of 3D object recognition, such as augmented reality or robotic manipulation, require an accurate solution for the 3D pose of the recognized objects. This is best accomplished by building a metrically accurate 3D model of the object and all its feature locations, and then fitting this model to features detected in new images. In this chapter, we describe a system for constructing 3D metric models from multiple images taken with an uncalibrated handheld camera, recognizing these models in new images, and precisely solving for object pose. This is demonstrated in an augmented reality application where objects must be recognized, tracked, and superimposed on new images taken from arbitrary viewpoints without perceptible jitter. This approach not only provides for accurate pose, but also allows for integration of features from multiple training images into a single model that provides for more reliable recognition 1 .

Street View Motion-from-Structure-from-Motion

“…Such points are employed for for image localization relative to an SfM model by Se et al [23], Gordon & Lowe [11], Irschara et al [16], and Li et al [17]. In those works, each 3D point is augmented with SIFT descriptors (perhaps averaged, perhaps quantized).…”

Model-Based Pose Estimation for Rigid Objects

“…The approach assumes an affine projection model and incurs high computational cost. Gordon and Lowe [3] describe a system based on SIFT features for recognizing learnt models in new images and solving for their pose. Intended for use in an augmented reality application, this system estimates the pose of the camera with respect to a set of mostly stationary objects in its environment rather than the other way round.…”

3D Object Recognition and Pose Estimation for Multiple Objects Using Multi-Prioritized RANSAC and Model Updating

“…Four systems are compared in their performance: the paradigm system proposed in [10] (G&L), where only image-to-model matching and RANSAC is employed, and our system by sequentially adding spatial feature clustering (S.C.), Multi-Prioritized RANSAC (MP-R), and model updating.…”

“…In the feature-based paradigm, as pioneered in [10,18], a 3D sparse point cloud representing the target object is reconstructed by applying Structure from Motion to features tracked over a set of training images. Once the model is obtained offline, on-line recognition and pose estimation is performed by matching the image features against the model features and solving the Perspective-n-Point problem for the 2D-3D correspondences.…”

“…Feature-based methods can be grouped on the basis of the feature used, e.g., edges [7], shape [5], patches [17], interest points [10,18]. We share the same approach of the latter, as we use SIFT features as interest points [15].…”