Human-like autonomous car-following model with deep reinforcement learning

Abstract

This study proposes a framework for human-like autonomous car-following planning based on deep reinforcement learning (deep RL). Historical driving data are fed into a simulation environment where an RL agent learns from trial and error interactions based on a reward function that signals how much the agent deviates from the empirical data. Through these interactions, an optimal policy, or car-following model that maps in a human-like way from speed, relative speed between a lead and following vehicle, and inter-vehicle spacing to acceleration of a following vehicle is finally obtained. The model can be continuously updated when more data are fed in. Two thousand car-following periods extracted from the 2015 Shanghai Naturalistic Driving Study were used to train the model and compare its performance with that of traditional and recent datadriven car-following models. As shown by this study's results, a deep deterministic policy gradient car-following model that uses disparity between simulated and observed speed as the reward function and considers a reaction delay of 1s, denoted as DDPGvRT, can reproduce human-like car-following behavior with higher accuracy than traditional and recent data-driven car-following models. Specifically, the DDPGvRT model has a spacing validation error of 18% and speed validation error of 5%, which are less than those of other models, including the intelligent driver model, models based on locally weighted regression, and conventional neural network-based models. Moreover, the DDPGvRT demonstrates good capability of generalization to various driving situations and can adapt to different drivers by continuously learning. This study demonstrates that reinforcement learning methodology can offer insight into driver behavior and can contribute to the development of human-like autonomous driving algorithms and traffic-flow models.

An Autonomous Path Planning Model for Unmanned Ships Based on Deep Reinforcement Learning

“…In the above formula, U a (s) is the potential energy value of the gravitational field at the point s, U r (s) is the potential energy value of the repulsive field at the point s, and U(s) is the potential energy of the point s. U a (s) and U r (s) are obtained by Formulas (10) and (11), respectively.…”

Implementing Autonomous Driving Behaviors Using a Message Driven Petri Net Framework

“…Building the POMDP model is not an easy task because it is necessary to define not only the states, actions, and transitions but also all the transition probabilities and rewards. Some of those parameters can be learned from experimental data using some reinforcement learning or deep learning techniques [24,25]. A question remaining is to evaluate the performance of systems of this kind against other solutions, such as simpler finite state machines, where it is easier to model the traffic rules and change the model as the traffic rules are updated.…”

Car-Following Modeling Incorporating Driving Memory Based on Autoencoder and Long Short-Term Memory Neural Networks

“…Wang et al [2] developed a car-following model that can better describe freeway driving behaviors based on an adaptive neuro-fuzzy inference system and wavelet analysis for denoising. Zhu et al [14] proposed a framework for a human-like autonomous car-following model based on the deep deterministic policy gradient algorithm of RL, which showed good capability of generalization under various driving situations. It can be driver-adapted by successive learning and is outperforming most of existing data-driven models.…”