Towards trustworthy multi‐modal motion prediction: Holistic evaluation and interpretability of outputs

Abstract

Predicting the motion of other road agents enables autonomous vehicles to perform safe and efficient path planning. This task is very complex, as the behaviour of road agents depends on many factors and the number of possible future trajectories can be considerable (multi‐modal). Most prior approaches proposed to address multi‐modal motion prediction are based on complex machine learning systems that have limited interpretability. Moreover, the metrics used in current benchmarks do not evaluate all aspects of the problem, such as the diversity and admissibility of the output. The authors aim to advance towards the design of trustworthy motion prediction systems, based on some of the requirements for the design of Trustworthy Artificial Intelligence. The focus is on evaluation criteria, robustness, and interpretability of outputs. First, the evaluation metrics are comprehensively analysed, the main gaps of current benchmarks are identified, and a new holistic evaluation framework is proposed. Then, a method for the assessment of spatial and temporal robustness is introduced by simulating noise in the perception system. To enhance the interpretability of the outputs and generate more balanced results in the proposed evaluation framework, an intent prediction layer that can be attached to multi‐modal motion prediction models is proposed. The effectiveness of this approach is assessed through a survey that explores different elements in the visualisation of the multi‐modal trajectories and intentions. The proposed approach and findings make a significant contribution to the development of trustworthy motion prediction systems for autonomous vehicles, advancing the field towards greater safety and reliability.

Self-Evaluation of Trajectory Predictors for Autonomous Driving

“…The results show that the monitor achieves the highest area under the receiver operating characteristic curve compared to the baseline methods, indicating its strong performance in detecting prediction failures. Carrasco et al [28] propose a multi-modal motion prediction system that integrates evaluation criteria, robustness analysis, and interpretability of outputs. They analyze the limitations of current benchmarks and propose a new holistic evaluation framework that considers accuracy, diversity, and compliance with traffic rules.…”

Static and Dynamic Performance Analysis of Cable-Stayed Bridges with Cables Damaged Fire

“…On With the rapid development of technology, we have seen the potential to use machine learning (ML), artificial intelligence (AI), and artificial neural network (ANN) algorithms to solve practical engineering problems [36][37][38][39]. In terms of convergence speed in simulation calculations, multi-modal motion prediction models for vehicles, and solving numerical models with disturbance suppression, ML and AI have significant advantages [40][41][42]. The numerical simulations or AI methods will also have more applications in the field of geotechnical engineering, such as construction process monitoring, multi-physical field coupling, and reliability analysis, which can effectively promote the development of geotechnical engineering [43][44][45].…”

Q‐EANet: Implicit social modeling for trajectory prediction via experience‐anchored queries

“…For interpretability of DLbased models, we believe it can be reflected in several aspects, including feature importance analysis, visualization-based explanations, comparative experiments, and explanatory rules. [14] serves as a pertinent example in this regard. The research specifically delved into the visualization of multi-modal predictions, including their corresponding probabilities, and examines the influence of different modalities and prediction ranges on the interpretability of the prediction system.…”