Semantic structure from motion

Abstract

Conventional rigid structure from motion (SFM) addresses the problem of recovering the camera parameters (motion) and the 3D locations (structure) of scene points, given observed 2D image feature points. In this paper, we propose a new formulation called Semantic Structure From Motion (SSFM). In addition to the geometrical constraints provided by SFM, SSFM takes advantage of both semantic and geometrical properties associated with objects in the scene (Fig. 1). These properties allow us to recover not only the structure and motion but also the 3D locations, poses, and categories of objects in the scene. We cast this problem as a max-likelihood problem where geometry (cameras, points, objects) and semantic information (object classes) are simultaneously estimated. The key intuition is that, in addition to image features, the measurements of objects across views provide additional geometrical constraints that relate cameras and scene parameters. These constraints make the geometry estimation process more robust and, in turn, make object detection more accurate. Our framework has the unique ability to: i) estimate camera poses only from object detections, ii) enhance camera pose estimation, compared to feature-point-based SFM algorithms, iii) improve object detections given multiple uncalibrated images, compared to independently detecting objects in single images. Extensive quantitative results on three datasets -LiDAR cars, street-view pedestrians, and Kinect office desktop -verify our theoretical claims.

Semantic Structure from Motion: A Novel Framework for Joint Object Recognition and 3D Reconstruction

“…[5] attempts joint estimation using a "cognitive loop" but requires a dedicated stereo-camera architecture and makes assumptions about camera motion. Having our preliminary result published as [1], we are the first to make these two steps coherent within a setting that requires only images with uncalibrated cameras (up to internal parameters) and arbitrary scene-camera configurations.…”

Fine-Grained Categorization for 3D Scene Understanding

“…4.2 evaluates our fine-grained categorization in isolation, we now move on to the more challenging task of applying it in the context of a 3D scene understanding task, on a recently proposed street scene data set [1,21]. To that end, we design an idealized experiment, (1) (2) (3) (4) Figure 2: Depth estimation results.…”

“…It offers the prospect of removing false positive predictions by imposing additional constraints on the layout of objects in the scene, either in the form of scene priors (such as ground plane affinity [14,31], mechanics [12], or humancentric functions [13]) or the likelihood of observations, given the current hypothesis (such as the agreement between predicted and observed object poses [1]). In this paper, we aim to expand upon the latter aspect, by providing fine-grained category predictions as additional input cues to scene-level reasoning.…”

“…As baselines (i), we consider a HOG [5] template with a linear SVM, and locality constrained linear coding (LLC [29]), which is one of the most powerful image-level classifiers to date (among the state-of-the-art on Caltech-101 [8] and -256 [11] classification benchmarks). For ii), we compare our partbank (PB) computed on response maps of the DPM [10] car detector as provided by the authors [9] (PB(DPM)), and part-bank computed on response maps of the bank of 8 viewpointdependent DPMs proposed by [1] (PB(mvDPM)). Since the latter explicitly distinguishes between different viewpoints, we expect the corresponding part response maps to be more informative than the ones of PB(DPM).…”

“…Data set. We use the Ford campus vision and lidar data set [1,21] for testing, as it provides calibrated camera images as well as registered point cloud data that can serve as the basis for metric 3D evaluation. Applying fine-grained categorization on this data set is challenging, as its statistics deviate largely from our car-types data set used for training, both w.r.t.…”