Towards Autonomous Driving: a Multi-Modal 360° Perception Proposal

Beltrán, Jorge Ramírez; Guindel, Carlos; Cortes, Irene; Barrera, Alejandro; Astudillo, Armando; Urdiales, Jesús; Alvarez, Mario; Bekka, Farid; Milanés, Vicente; García, Fernando

doi:10.1109/itsc45102.2020.9294494

Cited by 14 publications

(13 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…5 shows the output of the perception pipeline over time in a situation with three agents being tracked in consecutive frames. Experimental results in [10] prove the adequacy of the solution, which can populate the World Model with stable detections that are accurately localized.…”

Section: A Perceptionmentioning

confidence: 71%

See 1 more Smart Citation

The Tornado Project: An Automated Driving Demonstration in Peri-Urban and Rural Areas

Milanés

González

Navas

et al. 2022

IEEE Intell. Transport. Syst. Mag.

Self Cite

View full text Add to dashboard Cite

This paper presents the results of a two-week robottaxi service demonstration in peri-urban and rural areas. A fully robotized Renault ZOE was available for general public use in the Rambouillet Territory in France. The driving zone included several complex scenarios as two-way narrow road driving, a tunnel crossing with lane reduction from two-way road up to a single narrow lane or roundabouts, allowing to evaluate the maturity of the vehicle for such application. This paper describes the scientific and technical development especially from perception, navigation and control point of view to carry out such demonstration. Results indicate that even if the vehicle was able to autonomously navigate through peri-urban and rural areas, there are still some technical challenges that limit its integration with the transport system.

show abstract

Section: A Perceptionmentioning

confidence: 71%

“…The perception solution stems from the one presented in [10], where an analysis of the performance of the complete pipeline is provided. In brief, the information provided by the sensors is fused through a low-level fusion approach for obstacle detection, classification, 3D box estimation, and tracking.…”

Section: A Perceptionmentioning

confidence: 99%

The Tornado Project: An Automated Driving Demonstration in Peri-Urban and Rural Areas

Milanés

González

Navas

et al. 2022

IEEE Intell. Transport. Syst. Mag.

Self Cite

View full text Add to dashboard Cite

show abstract

“…It requires a full understanding of the surrounding environment, which omnidirectional vision excels at. Some works focus on setting up 360 ˝platform for autonomous driving [213], [214]. Specifically, [213] utilized a stereo camera, a polarization camera and a panoramic camera to form a multimodal visual system to capture omnidirectional landscape.…”

Section: Applicationsmentioning

confidence: 99%

“…Specifically, [213] utilized a stereo camera, a polarization camera and a panoramic camera to form a multimodal visual system to capture omnidirectional landscape. [214] introduced a multi-modal 360 ˝perception proposal based on visual and LiDAR scanners for 3D object detection and tracking. In addition to the platform, the emergence of public omnidirectional datasets for autonomous driving are crucial for the application of DL methods.…”

Section: Applicationsmentioning

confidence: 99%

Deep Learning for Omnidirectional Vision: A Survey and New Perspectives

Ai¹,

Cao²,

Zhu³

et al. 2022

Preprint

View full text Add to dashboard Cite

To whom it may concern,In this paper, we provide an in-depth and comprehensive survey on recent technical advancements in deep learning for omnidirectional vision, and also provide new perspectives for the future direction. Deep learning (DL) has recently been applied to omnidirectional vision. DL-based omnidirectional vision methods often achieve the state-of-the-art (SoTA) performance on various benchmark datasets. Diverse deep neural network (DNN) models have been developed, ranging from the convolutions neural network (CNN)-based models to vision transformers (ViT)-based ones. Generally, the SoTA DNN-based methods differ from each other in four major aspects: convolutional filters used to extract features from the omnidirectional image (ODI) data, network design considering the input numbers and projection types, omnidirectional vision with novel learning strategies, and practical applications. In this paper, we provide a comprehensive and systematic review and analysis of the recent progress in DL methods for omnidirectional vision. There are some previous surveys in the literature. However, some of them are focused on the specific vision tasks, especially room layout reconstruction, 3D scene geometry recovery. Moreover, some provide limited reviews of omnidirectional video streaming methods. Unlike existing surveys, this paper highlights the importance of deep learning and probe the recent advances for omnidirectional vision, both methodically and comprehensively. To the best of our knowledge, this is the first survey to comprehensively review and analyze the DL methods for omnidirectional vision. The main contributions of this paper to the community are five folds: (I) We comprehensively review and analyze the DL methods for omnidirectional vision, including the omnidirectional imaging principle, representation learning, datasets, a taxonomy, and applications, to highlight the differences and difficulties with the 2D planner image data. (II) We conduct an analytical study of recent trends of DL for omnidirectional vision, both hierarchically and structurally. Moreover, we offer insights into the discussion and challenge of each category. (III) We summarize the latest novel learning strategies and potential applications for omnidirectional vision.(I) We provide insightful discussions of the challenges and open problems yet to be solved and propose the potential future directions to spur more in-depth research by the community.(II) We create an open-source repository that provides a taxonomy of all the mentioned works and code links, and hope it can shed light on future research. The organization of this paper is structured as follows. In Sec.2, we introduce the imaging principle of ODI, convolution methods for omnidirectional vision, and some representative datasets. In Sec.3, we review and analyze the existing DL approaches for various tasks and provide taxonomies to categorize the relevant papers. Sec.4 covers novel learning paradigms for the tasks in omnidirectional vision, e.g., unsupervised learning, transfer learning, and reinforcement learning. Sec.5 then scrutinizes the applications, followed by Sec. 6, where we discuss open problems and future directions. Finally, we conclude this paper in Sec. 7. Furthermore, we summarize most, if not all but representative, works (over 200 papers) in the last five years, which were published in the top-tier conferences and journals in computer vision/graphics and machine learning. Due to the lack of space, we show the experimental results in Sec. 2 of the suppl. material. We believe that this paper will bring significant interest in such crucial topics in the community and provide fundamental technical references for further research.

show abstract

“…Due to the particular features of these sensory technologies, they are suitable to be part of the same perception system, providing complementary information. In that kind of design, data from the different sensors must be appropriately combined before inference making use of fusion techniques [2], [3]. In the most usual setup, sensors have overlapping fields of view (as in Fig.…”

Section: Introductionmentioning

confidence: 99%

Automatic Extrinsic Calibration Method for LiDAR and Camera Sensor Setups

Beltrán,

Guindel,

de la Escalera

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Most sensor setups for onboard autonomous perception are composed of LiDARs and vision systems, as they provide complementary information that improves the reliability of the different algorithms necessary to obtain a robust scene understanding. However, the effective use of information from different sources requires an accurate calibration between the sensors involved, which usually implies a tedious and burdensome process. We present a method to calibrate the extrinsic parameters of any pair of sensors involving LiDARs, monocular or stereo cameras, of the same or different modalities. The procedure is composed of two stages: first, reference points belonging to a custom calibration target are extracted from the data provided by the sensors to be calibrated, and second, the optimal rigid transformation is found through the registration of both point sets. The proposed approach can handle devices with very different resolutions and poses, as usually found in vehicle setups. In order to assess the performance of the proposed method, a novel evaluation suite built on top of a popular simulation framework is introduced. Experiments on the synthetic environment show that our calibration algorithm significantly outperforms existing methods, whereas real data tests corroborate the results obtained in the evaluation suite. Open-source code is available at https://github.com/beltransen/velo2cam calibration.

show abstract

Towards Autonomous Driving: a Multi-Modal 360° Perception Proposal

Cited by 14 publications

References 27 publications

The Tornado Project: An Automated Driving Demonstration in Peri-Urban and Rural Areas

The Tornado Project: An Automated Driving Demonstration in Peri-Urban and Rural Areas

Deep Learning for Omnidirectional Vision: A Survey and New Perspectives

Automatic Extrinsic Calibration Method for LiDAR and Camera Sensor Setups

Contact Info

Product

Resources

About