Multimodal interaction-aware motion prediction for autonomous street crossing

Abstract

For mobile robots navigating on sidewalks, the ability to safely cross street intersections is essential. Most existing approaches rely on the recognition of the traffic light signal to make an informed crossing decision. Although these approaches have been crucial enablers for urban navigation, the capabilities of robots employing such approaches are still limited to navigating only on streets that contain signalized intersections. In this article, we address this challenge and propose a multimodal convolutional neural network framework to predict the safety of a street intersection for crossing. Our architecture consists of two subnetworks: an interaction-aware trajectory estimation stream ( interaction-aware temporal convolutional neural network (IA-TCNN)), that predicts the future states of all observed traffic participants in the scene; and a traffic light recognition stream AtteNet. Our IA-TCNN utilizes dilated causal convolutions to model the behavior of all the observable dynamic agents in the scene without explicitly assigning priorities to the interactions among them, whereas AtteNet utilizes squeeze-excitation blocks to learn a content-aware mechanism for selecting the relevant features from the data, thereby improving the noise robustness. Learned representations from the traffic light recognition stream are fused with the estimated trajectories from the motion prediction stream to learn the crossing decision. Incorporating the uncertainty information from both modules enables our architecture to learn a likelihood function that is robust to noise and mispredictions from either subnetworks. Simultaneously, by learning to estimate motion trajectories of the surrounding traffic participants and incorporating knowledge of the traffic light signal, our network learns a robust crossing procedure that is invariant to the type of street intersection. Furthermore, we extend our previously introduced Freiburg Street Crossing dataset with sequences captured at multiple intersections of varying types, demonstrating complex interactions among the traffic participants as well as various lighting and weather conditions. We perform comprehensive experimental evaluations on public datasets as well as our Freiburg Street Crossing dataset, which demonstrate that our network achieves state-of-the-art performance for each of the subtasks, as well as for the crossing safety prediction. Moreover, we deploy the proposed architectural framework on a robotic platform and conduct real-world experiments that demonstrate the suitability of the approach for real-time deployment and robustness to various environments.

Understanding Socially Aware Robot Navigation

“…Whereas, when robots and human goals can be explicitly defined in an environment, then planning based approaches are extremely suitable. Few real world experiments (Bartoli et al, 2018;Bera et al, 2016;Radwan et al, 2018;Vasquez, 2016) conducted by different institutes in the domain of human motion prediction are illustrated in Table 4. It reports the study design depicting the number of participants and quantities measured during the experiment.…”

“…Bera et al (Bera et al, 2016), proposed path prediction based on global and local movement patterns on different videos captured by overhead cameras but their experiment lacked testing their model with the robots. Bartoli et al (Bartoli et al, 2018) and Radwan et al (Radwan et al, 2018) models performed well by combining human motion approaches with context aware mapping by utilizing LSTM sequential model to achieve energy optimization as well as offered better system tolerance to new unseen scenarios.…”

Gated Recurrent Fusion to Learn Driving Behavior from Temporal Multimodal Data

“…However, temporal fusion has received much lesser attention even though they generate a lot of multimodal temporal data coming from a range of sensors like Camera, LiDAR (Light Detection and Ranging), wheel odometry, etc. Typical driving automation tasks of interest are learning driver behavior [3] and intent [12], motion forecasting [28], [29], object detection [16], learning affordances [30], action regression [15], [31], semantic segmentation [32], [33] among others.…”

Self-Supervised Moving Vehicle Detection from Audio-Visual Cues

“…Also, robots operating in areas reserved for pedestrians such as sidewalks or pedestrian zones, e.g., delivery robots, benefit substantially from precise detections. Such detections, for example, enable delivery robot applications to safely cross a street [11], [14], [15].…”