Omnidirectional 3D reconstruction in augmented Manhattan worlds

Abstract

Abstract-This paper proposes a method for high-quality omnidirectional 3D reconstruction of augmented Manhattan worlds from catadioptric stereo video sequences. In contrast to existing works we do not rely on constructing virtual perspective views, but instead propose to optimize depth jointly in a unified omnidirectional space. Furthermore, we show that plane-based prior models can be applied even though planes in 3D do not project to planes in the omnidirectional domain. Towards this goal, we propose an omnidirectional slanted-plane Markov random field model which relies on plane hypotheses extracted using a novel voting scheme for 3D planes in omnidirectional space. To quantitatively evaluate our method we introduce a dataset which we have captured using our autonomous driving platform AnnieWAY which we equipped with two horizontally aligned catadioptric cameras and a Velodyne HDL-64E laser scanner for precise ground truth depth measurements. As evidenced by our experiments, the proposed method clearly benefits from the unified view and significantly outperforms existing stereo matching techniques both quantitatively and qualitatively. Furthermore, our method is able to reduce noise and the obtained depth maps can be represented very compactly by a small number of image segments and plane parameters.

Towards visual ego-motion learning in robots

“…Our approach differentiates itself from existing solutions on various fronts as shown in Table I. We evaluate the performance of our proposed approach on various publicly-available datasets including the KITTI dataset [21], the Multi-FOV synthetic dataset [27] (pinhole, fisheye, and catadioptric lenses), an omnidirectionalcamera dataset [28], and on the Oxford Robotcar 1000km Dataset [29].…”

“…In our experiments, we found that while our proposed solution was sufficiently powerful to model different camera optics, it was significantly better at modeling pinhole lenses as compared to fisheye and Sensor fusion with learned ego-motion: On fusing our proposed VO method with intermittent GPS updates (every 150 frames, black circles), the pose-graph optimized ego-motion solution (in green) achieves sufficiently high accuracy relative to ground truth. We test on a variety of publicly-available datasets including (a) Multi-FOV synthetic dataset [27] (pinhole shown above), (b) an omnidirectional-camera dataset [28], (c) Oxford Robotcar 1000km Dataset [29] (2015-11-13-10-28-08) (d-h) KITTI dataset [21]. Weak supervision such as GPS measurements can be especially advantageous in recovering improved estimates for localization, while simultaneously minimizing uncertainties associated with pure VO-based approaches.…”

Stereo reconstruction using top-down cues

“…This has gained significant popularity, due to it's robustness to minor calibration inaccuracies. Recent extensions of this idea include the addition of weighting terms [17], an iterative variant [18] and the augmented Manhattan world assumption [19,20].…”

“…In Table 5 we expand on this evaluation, by testing the contribution of every individual energy term from Equation (20). In all cases, removing an energy term causes an increase in the error, meaning that each term encodes useful information for 3D reconstruction, and none of the terms are redundant.…”

Room Layout Estimation with Object and Material Attributes Information Using a Spherical Camera

“…Schoenbein et al [22] proposed a high-quality omnidirectional 3D reconstruction of Manhattan worlds from catadioptric stereo video cameras. However, these catadioptric omnidirectional cameras have a large number of systematic parameters including the camera and mirror calibration.…”