Eliminating the Blind Spot: Adapting 3D Object Detection and Monocular Depth Estimation to 360$$^\circ $$ Panoramic Imagery

Garanderie, Grégoire Payen de La; Atapour-Abarghouei, Amir; Breckon, Toby P.

doi:10.1007/978-3-030-01261-8_48

Cited by 70 publications

(46 citation statements)

References 51 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These models can be trained with one projection model and tested with another, allowing them to utilize the large body of pinhole datasets. There were several other approaches aside from full rotational invariance and distortion-aware filters, including graph CNNs [22], style transfer [23], and increased filter sizes in the polar regions [24].…”

Section: Beyond Pinhole Projectionmentioning

confidence: 99%

Unsupervised Learning of Depth and Ego-Motion From Cylindrical Panoramic Video

Sharma

Ventura

2019

2019 IEEE International Conference on Artificial Intelligence and Virtual Reality (AIVR)

View full text Add to dashboard Cite

We introduce a convolutional neural network model for unsupervised learning of depth and ego-motion from cylindrical panoramic video. Panoramic depth estimation is an important technology for applications such as virtual reality, 3D modeling, and autonomous robotic navigation. In contrast to previous approaches for applying convolutional neural networks to panoramic imagery, we use the cylindrical panoramic projection which allows for the use of the traditional CNN layers such as convolutional filters and max pooling without modification. Our evaluation of synthetic and real data shows that unsupervised learning of depth and ego-motion on cylindrical panoramic images can produce high-quality depth maps and that an increased field-of-view improves ego-motion estimation accuracy. We also introduce Headcam, a novel dataset of panoramic video collected from a helmet-mounted camera while biking in an urban setting.

show abstract

Section: Beyond Pinhole Projectionmentioning

confidence: 99%

Unsupervised Learning of Depth and Ego-Motion From Cylindrical Panoramic Video

Sharma

Ventura

2019

2019 IEEE International Conference on Artificial Intelligence and Virtual Reality (AIVR)

View full text Add to dashboard Cite

show abstract

“…On the other hand, a single camera, although cannot provide explicit depth information, is several orders of magnitude cheaper than the LiDAR and can capture the scene clearly up to approximately 100 meters. Although people have explored the possibility of monocular 3D object detection for a decade [77,6,75,33,76,43,12,60,31,32,21], state-of-the-art monocular methods can only yield drastically low performance in contrast to the high performance achieved by the LiDAR-based methods (e.g., 13.6% average precision (AP) [60] vs. 86.5% AP [20] on the moderate set of cars of KITTI [14] dataset).…”

Section: Introductionmentioning

confidence: 99%

Monocular 3D Object Detection with Pseudo-LiDAR Point Cloud

Weng

Kitani

2019

2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW)

238

142

View full text Add to dashboard Cite

Monocular 3D scene understanding tasks, such as object size estimation, heading angle estimation and 3D localization, is challenging. Successful modern day methods for 3D scene understanding require the use of a 3D sensor. On the other hand, single image based methods have significantly worse performance. In this work, we aim at bridging the performance gap between 3D sensing and 2D sensing for 3D object detection by enhancing LiDAR-based algorithms to work with single image input. Specifically, we perform monocular depth estimation and lift the input image to a point cloud representation, which we call pseudo-LiDAR point cloud. Then we can train a LiDAR-based 3D detection network with our pseudo-LiDAR end-to-end. Following the pipeline of two-stage 3D detection algorithms, we detect 2D object proposals in the input image and extract a point cloud frustum from the pseudo-LiDAR for each proposal. Then an oriented 3D bounding box is detected for each frustum. To handle the large amount of noise in the pseudo-LiDAR, we propose two innovations: (1) use a 2D-3D bounding box consistency constraint, adjusting the predicted 3D bounding box to have a high overlap with its corresponding 2D proposal after projecting onto the image; (2) use the instance mask instead of the bounding box as the representation of 2D proposals, in order to reduce the number of points not belonging to the object in the point cloud frustum. Through our evaluation on the KITTI benchmark, we achieve the top-ranked performance on both bird's eye view and 3D object detection among all monocular methods, effectively quadrupling the performance over previous state-of-the-art. Our code is available at https: //github.com/xinshuoweng/Mono3D_PLiDAR.

show abstract

“…All previous monocular methods can only detect objects from the front-facing camera, ignoring objects on the sides and rear of the vehicle. While lidar methods can be used effectively for 360 degrees detection, [46] proposes the first 360 degrees panoramic image based method for 3D object detection. They estimate dense depth maps of panoramic images and adapt standard object detection methods for the equirectangular representation.…”

Section: A Monocular Image Based Methodsmentioning

confidence: 99%

“…The main drawbacks of monocular based methods is the lack of depth cues, which limits detection and localization accuracy specially for far and occluded objects, and sensitivity to lighting and weather conditions, limiting the use of these methods for day time. Also, since most methods rely on a front facing camera (except for [46]), it is only possible to detect objects in front of the vehicle, contrasting to point clouds methods that, in principle, have a coverage all around the vehicle. We summarize the methodology/contributions and limitations of monocular methods in Table IV.…”

Section: A Monocular Image Based Methodsmentioning

confidence: 99%