Model-Reference Reinforcement Learning for Collision-Free Tracking Control of Autonomous Surface Vehicles

Zhang, Qingrui; Pan, Wei; Reppa, Vasso

doi:10.1109/tits.2021.3086033

Cited by 43 publications

(23 citation statements)

References 47 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Reinforcement Learning for uncertainty: Compared with the existing GP-based and neuralnetwork-based approaches, RL, an adaptive and interactive learning approach, is introduced to model highly dynamic uncertainties in recent work [15]. Distributional RL constructs the entire distributions of the action-value function instead of the traditional expectation, where, to some extent, it addresses the key challenge of traditional RL, i.e., biasing the actions with high variance values in policy optimization [9].…”

Section: Related Workmentioning

confidence: 99%

QuaDUE-CCM: Interpretable Distributional Reinforcement Learning using Uncertain Contraction Metrics for Precise Quadrotor Trajectory Tracking

Wang¹,

O’Keeffe²,

Qian³

et al. 2022

Preprint

View full text Add to dashboard Cite

Accuracy and stability are common requirements for Quadrotor trajectory tracking systems. Designing an accurate and stable tracking controller remains challenging, particularly in unknown and dynamic environments with complex aerodynamic disturbances. We propose a Quantile-approximation-based Distributional-reinforced Uncertainty Estimator (QuaDUE) to accurately identify the effects of aerodynamic disturbances, i.e., the uncertainties between the true and estimated Control Contraction Metrics (CCMs). Taking inspiration from contraction theory and integrating the QuaDUE for uncertainties, our novel CCMbased trajectory tracking framework tracks any feasible reference trajectory precisely whilst guaranteeing exponential convergence. More importantly, the convergence and training acceleration of the distributional RL are guaranteed and analyzed, respectively, from theoretical perspectives. We also demonstrate our system under unknown and diverse aerodynamic forces. Under large aerodynamic forces (>2 m/s 2 ), compared with the classic data-driven approach, our QuaDUE-CCM achieves at least a 56.6% improvement in tracking error. Compared with QuaDRED-MPC, a distributional RL-based approach, QuaDUE-CCM achieves at least a 3 times improvement in contraction rate.

show abstract

Section: Related Workmentioning

confidence: 99%

QuaDUE-CCM: Interpretable Distributional Reinforcement Learning using Uncertain Contraction Metrics for Precise Quadrotor Trajectory Tracking

Wang¹,

O’Keeffe²,

Qian³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Model Uncertainties: They are usually caused by the unknown dynamics [134]- [136], underactuated ASVs [137], high-speed maneuvering situation [138], sensor errors [135], or environmental disturbances [139], which may introduce unknown parameters, terms or functions into an ASV control system.…”

Section: A Definition and Key Problemsmentioning

confidence: 99%

“…In addition, based on DRL, the control law can be learned directly to compensate for uncertainties and disturbances [139]. The reward functions of DRL have a very large impact on the learned desired behavior.…”

Section: The Limitation Of DLmentioning

confidence: 99%

Survey of Deep Learning for Autonomous Surface Vehicles in the Marine Environment

Ye¹,

Yin²,

Wang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Within the next several years, there will be a high level of autonomous technology that will be available for widespread use, which will reduce labor costs, increase safety, save energy, enable difficult unmanned tasks in harsh environments, and eliminate human error. Compared to software development for other autonomous vehicles, maritime software development, especially on aging but still functional fleets, is described as being in a very early and emerging phase. This introduces very large challenges and opportunities for researchers and engineers to develop maritime autonomous systems. Recent progress in sensor and communication technology has introduced the use of autonomous surface vehicles (ASVs) in applications such as coastline surveillance, oceanographic observation, multivehicle cooperation, and search and rescue missions. Advanced artificial intelligence technology, especially deep learning (DL) methods that conduct nonlinear mapping with self-learning representations, has brought the concept of full autonomy one step closer to reality. This paper surveys the existing work regarding the implementation of DL methods in ASV-related fields. First, the scope of this work is described after reviewing surveys on ASV developments and technologies, which draws attention to the research gap between DL and maritime operations. Then, DL-based navigation, guidance, control (NGC) systems and cooperative operations, are presented. Finally, this survey is completed by highlighting the current challenges and future research directions.

show abstract

“…In comparison to existing data-driven approaches, Reinforcement Learning (RL), an interactive learning process, is able to learn complex and changeable disturbances -i.e., the errors between the true and estimated values -using much less model information [14]. The key challenge of most existing RL approaches [15] is that policy optimization biases toward actions with high variance value estimates, since some of these values will be overestimated by random chance [16].…”

Section: Introduction Accurate Trajectory Tracking For Autonomous Unm...mentioning

confidence: 99%

Interpretable Stochastic Model Predictive Control using Distributional Reinforced Estimation for Quadrotor Tracking Systems

Wang¹,

O’Keeffe²,

Qian³

et al. 2022

Preprint

View full text Add to dashboard Cite

This paper presents a novel trajectory tracker for autonomous quadrotor navigation in dynamic and complex environments. The proposed framework integrates a distributional Reinforcement Learning (RL) estimator for unknown aerodynamic effects into a Stochastic Model Predictive Controller (SMPC) for trajectory tracking. Aerodynamic effects derived from drag forces and moment variations are difficult to model directly and accurately. Most current quadrotor tracking systems therefore treat them as simple 'disturbances' in conventional control approaches. We propose Quantile-approximationbased Distributional Reinforced-disturbance-estimator, an aerodynamic disturbance estimator, to accurately identify disturbances, i.e., uncertainties between the true and estimated values of aerodynamic effects. Simplified Affine Disturbance Feedback is employed for control parameterization to guarantee convexity, which we then integrate with a SMPC to achieve sufficient and non-conservative control signals. We demonstrate our system to improve the cumulative tracking errors by at least 66% with unknown and diverse aerodynamic forces compared with recent state-of-the-art. Concerning traditional Reinforcement Learning's non-interpretability, we provide convergence and stability guarantees of Distributional RL and SMPC, respectively, with non-zero mean disturbances.

show abstract

Model-Reference Reinforcement Learning for Collision-Free Tracking Control of Autonomous Surface Vehicles

Abstract: Qingrui Zhang (Member, IEEE) received the B.S. degree in automatic control from the Harbin Institute of Technology, Harbin, China, in 2013, and the Ph.D. degree in aerospace science and engineering from the

Cited by 43 publications

References 47 publications

QuaDUE-CCM: Interpretable Distributional Reinforcement Learning using Uncertain Contraction Metrics for Precise Quadrotor Trajectory Tracking

QuaDUE-CCM: Interpretable Distributional Reinforcement Learning using Uncertain Contraction Metrics for Precise Quadrotor Trajectory Tracking

Survey of Deep Learning for Autonomous Surface Vehicles in the Marine Environment

Interpretable Stochastic Model Predictive Control using Distributional Reinforced Estimation for Quadrotor Tracking Systems

Contact Info

Product

Resources

About