Why did the robot cross the road? — Learning from multi-modal sensor data for autonomous road crossing

Radwan, Noha; Winterhalter, Wera; Dornhege, Christian; Burgard, Wolfram

doi:10.1109/iros.2017.8206347

Cited by 10 publications

(17 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In our previous work (Radwan et al, 2017), we proposed a classification approach to predict the safety of an intersection for crossing by training a random forest classifier on tracked detections from both radar and LiDAR scanners, which enables fast and reliable detections of oncoming traffic. Although this method has the advantage of being independent to the intersection type, it lacks the ability to generalize to new unseen scenarios as it learns a discriminative model of the problem.…”

Section: Related Workmentioning

confidence: 99%

“…In order to evaluate the overall performance of our model for the joint tasks of motion prediction, TLR, and intersection safety prediction, we extend our previously proposed dataset (Radwan et al, 2017) with additional sequences and labels for each of the aforementioned tasks. The FSC dataset consists of tracked detections of cars, cyclists, and pedestrians captured at different intersections in Freiburg, Germany using a 3D LiDAR scanner and Delphi electronically scanning radars (ESRs) mounted on our robotic platform shown in Figure 6 (Radwan et al, 2017). Note that both the data capturing procedure and all experiments on this dataset were conducted using this robotic platform.…”

Section: Experimental Evaluationmentioning

confidence: 99%

“…We benchmark our IA-TCNN architecture on several publicly available datasets, namely ETH (Pellegrini et al, 2009), UCY (Lerner et al, 2007), and L-CAS (Yan et al, 2017), in addition to our own Freiburg Street Crossing (FSC) dataset (Radwan et al, 2017). For the TLR task, we benchmark on Nexar (Nexar, 2016) and Bosch (Behrendt and Novak, 2017) datasets as well as the FSC dataset.…”

Section: Introductionmentioning

confidence: 99%

“…We extend the previously introduced FSC dataset (Radwan et al, 2017) consisting of images, LiDAR as well as radar data by eight sequences captured at various intersections along with annotations for the traffic light state, trajectory annotations for the tracked dynamic objects and the corresponding crossing decision, and make the dataset publicly available.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Multimodal interaction-aware motion prediction for autonomous street crossing

Radwan

Burgard

Valada

2020

The International Journal of Robotics Research

Self Cite

View full text Add to dashboard Cite

For mobile robots navigating on sidewalks, the ability to safely cross street intersections is essential. Most existing approaches rely on the recognition of the traffic light signal to make an informed crossing decision. Although these approaches have been crucial enablers for urban navigation, the capabilities of robots employing such approaches are still limited to navigating only on streets that contain signalized intersections. In this article, we address this challenge and propose a multimodal convolutional neural network framework to predict the safety of a street intersection for crossing. Our architecture consists of two subnetworks: an interaction-aware trajectory estimation stream ( interaction-aware temporal convolutional neural network (IA-TCNN)), that predicts the future states of all observed traffic participants in the scene; and a traffic light recognition stream AtteNet. Our IA-TCNN utilizes dilated causal convolutions to model the behavior of all the observable dynamic agents in the scene without explicitly assigning priorities to the interactions among them, whereas AtteNet utilizes squeeze-excitation blocks to learn a content-aware mechanism for selecting the relevant features from the data, thereby improving the noise robustness. Learned representations from the traffic light recognition stream are fused with the estimated trajectories from the motion prediction stream to learn the crossing decision. Incorporating the uncertainty information from both modules enables our architecture to learn a likelihood function that is robust to noise and mispredictions from either subnetworks. Simultaneously, by learning to estimate motion trajectories of the surrounding traffic participants and incorporating knowledge of the traffic light signal, our network learns a robust crossing procedure that is invariant to the type of street intersection. Furthermore, we extend our previously introduced Freiburg Street Crossing dataset with sequences captured at multiple intersections of varying types, demonstrating complex interactions among the traffic participants as well as various lighting and weather conditions. We perform comprehensive experimental evaluations on public datasets as well as our Freiburg Street Crossing dataset, which demonstrate that our network achieves state-of-the-art performance for each of the subtasks, as well as for the crossing safety prediction. Moreover, we deploy the proposed architectural framework on a robotic platform and conduct real-world experiments that demonstrate the suitability of the approach for real-time deployment and robustness to various environments.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Experimental Evaluationmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Multimodal interaction-aware motion prediction for autonomous street crossing

Radwan

Burgard

Valada

2020

The International Journal of Robotics Research

Self Cite

View full text Add to dashboard Cite

show abstract

“…In addition to the technical implementation, the navigation of mobile robot systems outdoors has been the subject of interdisciplinary research for several years [30]. Challenges here are the recognition and classification of passable paths [31], changing lighting situations during the day [32], dynamic obstacles such as people, cyclists or pets [33], intersections and road crossings [34].…”

Section: Mobile Robot Systems Based On Quadrupedsmentioning

confidence: 99%

Living and Interacting with Robots: Engaging Users in the Development of a Mobile Robot

Varney¹,

Henke²,

Janßen³

2020

Becoming Human With Humanoid - From Physical Interaction to Social Intelligence

View full text Add to dashboard Cite

Mobile robots such as Aldebaran's humanoid Pepper currently find their way into society. Many research projects already try to match humanoid robots with humans by letting them assist, e.g., in geriatric care or simply for purposes of keeping company or entertainment. However, many of these projects deal with acceptance issues that come with a new type of interaction between humans and robots. These issues partly originate from different types of robot locomotion, limited human-like behaviour as well as limited functionalities in general. At the same time, animal-type robots-quadrupeds such as Boston Dynamic's WildCat-and underactuated robots are on the rise and present social scientists with new challenges such as the concept of uncanny valley. The possible positive aspects of the unusual cooperations and interactions, however, are mostly pushed into the background. This paper describes an approach of a project at a research institution in Germany that aims at developing a setting of human-robot-interaction and collaboration that engages the designated users in the whole process.

show abstract

Pre-robotic Navigation Identification of Pedestrian Crossings and Their Orientations

Ahmed

Matsumaru

2021

Field and Service Robotics

View full text Add to dashboard Cite

Why did the robot cross the road? — Learning from multi-modal sensor data for autonomous road crossing

Cited by 10 publications

References 19 publications

Multimodal interaction-aware motion prediction for autonomous street crossing

Multimodal interaction-aware motion prediction for autonomous street crossing

Living and Interacting with Robots: Engaging Users in the Development of a Mobile Robot

Pre-robotic Navigation Identification of Pedestrian Crossings and Their Orientations

Contact Info

Product

Resources

About