3D Object Localisation from Multi-view Image Detections

Abstract

In this work we present a novel approach to recover objects 3D position and occupancy in a generic scene using only 2D object detections from multiple view images. The method reformulates the problem as the estimation of a quadric (ellipsoid) in 3D given a set of 2D ellipses fitted to the object detection bounding boxes in multiple views. We show that a closed-form solution exists in the dual-space using a minimum of three views while a solution with two views is possible through the use of non-linear optimisation and object constraints on the size of the object shape. In order to make the solution robust toward inaccurate bounding boxes, a likely occurrence in object detection methods, we introduce a data preconditioning technique and a non-linear refinement of the closed form solution based on implicit subspace constraints. Results on synthetic tests and on different real datasets, involving challenging scenarios, demonstrate the applicability and potential of our method in several realistic scenarios.

Visual Graphs from Motion (VGfM): Scene Understanding with Object Geometry Reasoning

“…(b, c) is the point cloud, 3D quadrics (using same color labelling), camera poses for two camera, the ground truth is shown in green. It can be seen that the proposed solution overcomes the limitation of [27].…”

“…Recently, new techniques have emerged to estimate the 3D spatial layout of the objects as well as their occupancy [27,11,2]. These techniques rely on the quality of deep learning object detectors [27,11] or the use of additional range data [2]. Similarly volumetric approaches have been used to construct the layout of objects in rooms, or construct objects and regress their positioning [33].…”

“…In the next section, we present an extension named LfD with Constraints (LfDC) which is based on linear constraints on the quadric centers. It has the advantage of being a fast closed-form solution while being more robust than LfD [27].…”

“…the camera is calibrated). The LfD method described in [27] solves the problem in the dual space where it can be linearized as:…”

“…where 0 x denotes a column vector of zeros of length x, and the matrix M i ∈ R 6F ×(10+F ) and the vector w i ∈ R 10+F are defined as follow: [27] in red and LfDC in blue with cross and triangle respectively for the centers. (b, c) is the point cloud, 3D quadrics (using same color labelling), camera poses for two camera, the ground truth is shown in green.…”