Exploiting Symmetry and/or Manhattan Properties for 3D Object Structure Estimation from Single and Multiple Images

Abstract

Many man-made objects have intrinsic symmetries and Manhattan structure. By assuming an orthographic projection model, this paper addresses the estimation of 3D structures and camera projection using symmetry and/or Manhattan structure cues, which occur when the input is singleor multiple-image from the same category, e.g., multiple different cars. Specifically, analysis on the single image case implies that Manhattan alone is sufficient to recover the camera projection, and then the 3D structure can be reconstructed uniquely exploiting symmetry. However, Manhattan structure can be difficult to observe from a single image due to occlusion. To this end, we extend to the multipleimage case which can also exploit symmetry but does not require Manhattan axes. We propose a novel rigid structure from motion method, exploiting symmetry and using multiple images from the same category as input. Experimental results on the Pascal3D+ dataset show that our method significantly outperforms baseline methods. 1 We experimented with using Manhattan for the multiple-image case, but found it gave negligible improvement. Please see the derivations of using Manhattan for the multiple-image case in the supplementary material.

Inferring 3D Shapes of Unknown Rigid Objects in Clutter Through Inverse Physics Reasoning

“…Inferring hidden facets in the space of all possible 3D surfaces is computationally challenging for robotic manipulation tasks that require real-time inference. Therefore, we limit the space of hypotheses by exploiting the Manhattan properties that are commonly made in the literature [33]. The Manhattan structure assumption states that the occluded facets have curvatures similar to the observed ones.…”

Reconstructing vehicles from a single image: Shape priors for road scene understanding

“…We differ from [11] in that we capture inherent geometric constraints that vehicles exhibit (symmetry and planarity, for instance) which would result in more meaningful shape estimates. This was partially addressed in [7], which tries to use symmetry and Manhattan properties of objects to recover more meaningful shapes. However, the reconstructions assume a weak-orthographic projection model, whereas we use a projective camera model and a globally optimal pose estimation pipeline.…”

“…Simultaneous pose and shape estimation of vehicles is illposed when only a single image is available [7]. Guided by the motivation that humans make use of prior information about the vehicle to reason about 3D shape and pose, we decouple the pose and shape estimation problems.…”

“…However, we find this fact rarely exploited for monocular reconstruction. Symmetry constraints are imposed for reconstruction in [7], but they propose a factorization-based solution for the weak-orthographic projection case. On the other hand, we impose symmetry constraints for the projective case and solve it using nonlinear least squares.…”

“…Central Idea: Estimating shape and pose simultaneously from a single image is an ill-posed problem, and suffers from several ambiguities as demonstrated in [7]. Most notably, when the object pose is unconstrained, optimizing on shape reprojection error results in arbitrary deformations in the estimated 3D shape.…”