Model-Reference Reinforcement Learning Control of Autonomous Surface Vehicles with Uncertainties

Abstract

This paper presents a novel model-reference reinforcement learning control method for uncertain autonomous surface vehicles. The proposed control combines a conventional control method with deep reinforcement learning. With the conventional control, we can ensure the learning-based control law provides closed-loop stability for the overall system, and potentially increase the sample efficiency of the deep reinforcement learning. With the reinforcement learning, we can directly learn a control law to compensate for modeling uncertainties. In the proposed control, a nominal system is employed for the design of a baseline control law using a conventional control approach. The nominal system also defines the desired performance for uncertain autonomous vehicles to follow. In comparison with traditional deep reinforcement learning methods, our proposed learning-based control can provide stability guarantees and better sample efficiency. We demonstrate the performance of the new algorithm via extensive simulation results.

Model-Reference Reinforcement Learning Control of Autonomous Surface Vehicles

“…Note that both θ and φ are a set of parameters whose dimensions are determined by the deep neural network setup. Details on the design of Q θ (s t , u l,t ) and π φ (u l,t |s t ) can be found in [27]. The deep neural network for Q θ is called critic, while the one for π φ is called actor [27].…”

Lyapunov-Based Reinforcement Learning State Estimator

“…Lemma 2. (Policy improvement) Considering the last updated policy π old and the new policy π new to be obtained from (27), L πnew(k) ≤ L π old (k) holds for ∀x k ∈ S and ∀a k ∈ A.…”

“…Only until recently, the asymptotic stability in model-free RL is given for robotic control tasks [26]. In [27], [28], the stability of a system with a combination of a classic baseline controller and a RL controller is proved for autonomous surface vehicles with collisions.…”

Model-Reference Reinforcement Learning for Collision-Free Tracking Control of Autonomous Surface Vehicles

“…Some of the work in this paper has been accepted to be presented in the 59th IEEE Conference on Decision and Control (CDC) that will be hosted at December, 2020. The online version of our CDC paper can be found in [48]. In our CDC paper, the collision avoidance problem is not addressed.…”