Self-Optimizing Path Tracking Controller for Intelligent Vehicles Based on Reinforcement Learning

Abstract

The path tracking control system is a crucial component for autonomous vehicles; it is challenging to realize accurate tracking control when approaching a wide range of uncertain situations and dynamic environments, particularly when such control must perform as well as, or better than, human drivers. While many methods provide state-of-the-art tracking performance, they tend to emphasize constant PID control parameters, calibrated by human experience, to improve tracking accuracy. A detailed analysis shows that PID controllers inefficiently reduce the lateral error under various conditions, such as complex trajectories and variable speed. In addition, intelligent driving vehicles are highly non-linear objects, and high-fidelity models are unavailable in most autonomous systems. As for the model-based controller (MPC or LQR), the complex modeling process may increase the computational burden. With that in mind, a self-optimizing, path tracking controller structure, based on reinforcement learning, is proposed. For the lateral control of the vehicle, a steering method based on the fusion of the reinforcement learning and traditional PID controllers is designed to adapt to various tracking scenarios. According to the pre-defined path geometry and the real-time status of the vehicle, the interactive learning mechanism, based on an RL framework (actor–critic—a symmetric network structure), can realize the online optimization of PID control parameters in order to better deal with the tracking error under complex trajectories and dynamic changes of vehicle model parameters. The adaptive performance of velocity changes was also considered in the tracking process. The proposed controlling approach was tested in different path tracking scenarios, both the driving simulator platforms and on-site vehicle experiments have verified the effects of our proposed self-optimizing controller. The results show that the approach can adaptively change the weights of PID to maintain a tracking error (simulation: within ±0.071 m; realistic vehicle: within ±0.272 m) and steering wheel vibration standard deviations (simulation: within ±0.04°; realistic vehicle: within ±80.69°); additionally, it can adapt to high-speed simulation scenarios (the maximum speed is above 100 km/h and the average speed through curves is 63–76 km/h).

A Review of Autonomous Vehicle Path Tracking Algorithm Research

“…To solve the problem of feature representation and online learning capability in learning control of uncertain dynamic systems, a multicore online RL method for path tracking control was proposed in the literature [24], where a multicore feature learning framework was designed based on pairwise heuristic planning, and simulations under S-curve and urban road conditions verified that the controller has better tracking accuracy and stability performance than LQR controller and PP controller.Ma et al combined RL and PID were combined to propose a self-seeking optimal path tracking control based on the interactive learning mechanism of the RL framework to achieve online optimization of the PID control parameters, and the simulation and real vehicle tests proved that the control method has better tracking performance in high-speed conditions (maximum speed above 100 km/h) [104].…”

Research on intelligent vehicle path tracking control based on preview distance optimization

“…Control algorithms, as the core component of path tracking control systems, have been the focus of most researchers, who aim to enhance the precision of path tracking control and its robustness. Currently, widely used control algorithms include PID control [7][8][9], fuzzy control [10][11][12], model predictive control [13][14][15], and sliding mode control [16][17][18]. In [8], the authors proposed a steering method that integrates reinforcement learning with traditional PID controllers.…”

“…Currently, widely used control algorithms include PID control [7][8][9], fuzzy control [10][11][12], model predictive control [13][14][15], and sliding mode control [16][17][18]. In [8], the authors proposed a steering method that integrates reinforcement learning with traditional PID controllers. This approach employs an RL framework with interactive learning mechanisms, enabling adaptive adjustment of the PID control parameters and maintaining excellent tracking accuracy even on complex trajectories.…”