Explaining a Deep Reinforcement Learning (DRL)-Based Automated Driving Agent in Highway Simulations

Abstract

As deep learning models have become increasingly complex, it is critical to understand their decision-making, particularly in safety-relevant applications. In order to support a quantitative interpretation of an autonomous agent trained through Deep Reinforcement Learning (DRL) in the highway-env simulation environment, we propose a framework featuring three types of views for analyzing data: (i) episode timeline, (ii) frame by frame, and (iii) aggregated statistical analysis, also including heatmaps for a better spatial understanding. Our methodology allowed a novel, consistent description of the behavior of the agent. The main motivator for the taken action is typically the longitudinal distance from the second-closest and, to a lower extent, third-closest vehicle. In the overtakes, also the agent's position in lanes becomes relevant. The analysis identified interesting patterns and an issue in the last frames of an episode, when the agent is unable to overtake the last two vehicles, arguably because of the lack of reference vehicles ahead. We observed a clear differentiation between attention and SHAP values (estimating the importance of each feature for each decision), reflecting the architecture of the neural network, where the first layer implements the attention mechanism, while the deeper ones make the actual decision. Attention focuses on the proximity of the ego, while the decision is taken on a wider horizon, denoting a valuable anticipation capability. To support research, the proposed framework is released as open source.

Alessandro De Gloria, a Pioneer in Electronic Engineering Applications

“…In the years, the collaboration involved several research fields and projects, such as Human-Computer Interaction (Comunicar and Aide), machine learning (Edel), vehicle-to-X cooperation (Safespot), and collaborative mobility (Team). In the last period, Alessandro and his group have been deeply involved in piloting automated driving functions (ADFs) for SAE level 3 and 4 vehicle automation, particularly by developing the big data management architecture [22], based on the Measurify open-source framework [23], and developing machine learning models for perception (e.g., [24]) and decision making (e.g., [25]). Also, in this area, a key appreciated peculiarity was the multidisciplinary approach, which meant designing applications and systems able to effectively exploit the underlying hardware in a holistic, application-requirement-centred view.…”

Implementing Deep Reinforcement Learning (DRL)-based Driving Styles for Non-Player Vehicles

“…RL is a branch of machine learning (ML) in which an agent (i.e., the model) learns, through experience, to take actions in an environment in order to maximize the cumulative reward in an episode [8], [9]. Based on literature about real-world automated driving (e.g., [10]- [14]), we argue that DRL could be useful to develop realistic NPVs for SGs, also faithfully implementing different driving styles (e.g., standard, comfort, aggressive), for a more realistic and difficulty-level-adaptable representation of the traffic surrounding the ego vehicle.…”