Weights-varying MPC for Autonomous Vehicle Guidance: a Deep Reinforcement Learning Approach

Abstract

Determining the optimal cost function parameters of Model Predictive Control (MPC) to optimize multiple control objectives is a challenging and time-consuming task. Multiobjective Bayesian Optimization (BO) techniques solve this problem by determining a Pareto optimal parameter set for an MPC with static weights. However, a single parameter set may not deliver the most optimal closed-loop control performance when the context of the MPC operating conditions changes during its operation, urging the need to adapt the cost function weights at runtime. Deep Reinforcement Learning (RL) algorithms can automatically learn context-dependent optimal parameter sets and dynamically adapt for a Weightsvarying MPC (WMPC). However, learning cost function weights from scratch in a continuous action space may lead to unsafe operating states. To solve this, we propose a novel approach limiting the RL actions within a safe learning space representing a catalog of pre-optimized BO Pareto-optimal weight sets. We conceive a RL agent not to learn in a continuous space but to proactively anticipate upcoming control tasks and to choose the most optimal discrete actions, each corresponding to a single set of Pareto optimal weights, context-dependent. Hence, even an untrained RL agent guarantees a safe and optimal performance. Experimental results demonstrate that an untrained RL-WMPC shows Pareto-optimal closed-loop behavior and training the RL-WMPC helps exhibit a performance beyond the Pareto-front.

Incorporating Recurrent Reinforcement Learning into Model Predictive Control for Adaptive Control in Autonomous Driving

“…Comparatively, Reinforcement Learning (RL) can modify MPC parameters at each time step to quickly adapt to dynamic environments. Zarrouki et al (2021) and Song and Scaramuzza (2020) both learn a policy that can improve parameters of MPC's cost function, while Gros and Zanon (2020); Amos et al (2018) aim to modify both transition and cost function's parameters. Nevertheless, none of these methods consider the non-stationary scenario in an autonomous driving system, which violates the essential Markov property in RL.…”

Prediction Horizon-varying Model Predictive Control (MPC)for Autonomous Vehicle Control

“…In [16], a neural network model predictive control(NNMPC) is d proposed, which used neural network (NN) to learn and predict vehicle dynamics based on measured states and input variables. The concept of using Reinforcement Learning (RL) to learn MPC cost function parameters is introduced in [17], and [18] proposes a weights-varying MPC based on deep reinforcement learning to adjust cost function weights in different situations. Prediction horizon refers to the time range used to predict the system's future behavior during the control process, and is the key parameter affecting performance and computational burden of the control system in MPC [19].…”

Prediction Horizon-Varying Model Predictive Control (MPC) for Autonomous Vehicle Control

“…Ref. [20] proposes a weights-varying MPC using a deep reinforcement learning (DRL) algorithm to adjust cost function weights in different situations. Ref.…”

“…Ref. [21] proposed a novel approach limiting DRL actions within a safe learning space, and the proposed DRL algorithm can automatically learn context-dependent optimal parameter sets and dynamically adapt for a weights-varying MPC. Ref.…”