Application of reinforcement learning on self-tuning PID controller for soccer robot multi-agent system

Hakim, Aulia El; Hindersah, Hilwadi; Rijanto, Estiko

doi:10.1109/rict-icevt.2013.6741546

Cited by 13 publications

(10 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Online tuning of PID controllers has been thoroughly studied and major approaches include fuzzy inference [6], stochastic approximation from data [7], heuristic optimization [8], and their combinations [9]. More recently, machine learning techniques such as reinforcement learning [10] and supervised learning [11] have been applied to tune PID controllers.…”

Section: State Of the Artmentioning

confidence: 99%

General Dynamic Neural Networks for the Adaptive Tuning of an Omni-Directional Drive System for Reactive Swarm Robotics

Zhao

Dorigo

Allwright

2021

2021 25th International Conference on Methods and Models in Automation and Robotics (MMAR)

View full text Add to dashboard Cite

We demonstrate the use of general dynamic neural networks (GDNNs) for the online tuning of an omni-directional drive system for reactive swarm robots. The drive system used in this work consists of four motor-encoder-microcontroller modules each constituting a single-input single-output (SISO) proportional, integral, and differential (PID) control system. For a given target velocity, a neural network generates the parameters for each PID control system. In this paper, we evaluate and compare two different network structures for generating the PID parameters for the control systems using a hardware platform that we also presented in this paper. We analyze the performance of the system with respect to ISO performance indicators, our results show that both network structures are able to learn and tune the parameters for each PID control system to increase the accuracy of the drive system in comparison to fixed untuned PID parameters that are close to the output of a randomly initialized network.

show abstract

Section: State Of the Artmentioning

confidence: 99%

General Dynamic Neural Networks for the Adaptive Tuning of an Omni-Directional Drive System for Reactive Swarm Robotics

Zhao

Dorigo

Allwright

2021

2021 25th International Conference on Methods and Models in Automation and Robotics (MMAR)

View full text Add to dashboard Cite

show abstract

“…It has the advantages of not relying on models and having good learning effects for complex systems. In order to improve the control performance, some scholars combined Q-learning with PID control and proposed many excellent control methods [33,34]. In the autonomous underwater vehicle system, reference [35] proposed a Q-learning PID controller based on RBFNN to improve control performance, in which Q-learning neural network was used to adaptively optimize control parameters.…”

Section: Introductionmentioning

confidence: 99%

A Q‐Learning‐Based Parameters Adaptive Algorithm for Formation Tracking Control of Multi‐Mobile Robot Systems

et al. 2022

View full text Add to dashboard Cite

This paper proposes an adaptive formation tracking control algorithm optimized by Q-learning scheme for multiple mobile robots. In order to handle the model uncertainties and external disturbances, a desired linear extended state observer is designed to develop an adaptive formation tracking control strategy. Then an adaptive method of sliding mode control parameters optimized by Q-learning scheme is employed, which can avoid the complex parameter tuning process. Furthermore, the stability of the closed-loop control system is rigorously proved by means of matrix properties of graph theory and Lyapunov theory, and the formation tracking errors can be guaranteed to be uniformly ultimately bounded. Finally, simulations are presented to show the proposed algorithm has the advantages of faster convergence rate, higher tracking accuracy, and better steady-state performance.

show abstract

“…The combination of fuzzy PID and RL is reported in Boubertakh and Glorennec (2006); Boubertakh et al (2010). Applications of RL-based self-tuning PID include soccer robot El Hakim et al (2013), multicopter Park et al (2019), and human-in-theloop physical assistive control Zhong and Li (2019). For process control, reports in Lawrence et al (2020b,a) consider the PID tuning in sample-by-sample and episodic modes, respectively.…”

Section: Introductionmentioning

confidence: 99%

Stability-Preserving Automatic Tuning of PID Control with Reinforcement Learning

Lakhani¹,

Chowdhury²,

Lu³

2021

Preprint

View full text Add to dashboard Cite

PID control has been the dominant control strategy in the process industry due to its simplicity in design and effectiveness in controlling a wide range of processes. However, traditional methods on PID tuning often require extensive domain knowledge and field experience. To address the issue, this work proposes an automatic PID tuning framework based on reinforcement learning (RL), particularly the deterministic policy gradient (DPG) method. Different from existing studies on using RL for PID tuning, in this work, we consider the closed-loop stability throughout the RL-based tuning process. In particular, we propose a novel episodic tuning framework that allows for an episodic closed-loop operation under selected PID parameters where the actor and critic networks are updated once at the end of each episode. To ensure the closed-loop stability during the tuning, we initialize the training with a conservative but stable baseline PID controller and the resultant reward is used as a benchmark score. A supervisor mechanism is used to monitor the running reward (e.g., tracking error) at each step in the episode. As soon as the running reward exceeds the benchmark score, the underlying controller is replaced by the baseline controller as an early correction to prevent instability. Moreover, we use layer normalization to standardize the input to each layer in actor and critic networks to overcome the issue of policy saturation at action bounds, to ensure the convergence to the optimum. The developed methods are validated through setpoint tracking experiments on a second-order plus dead-time system. Simulation results show that with our scheme, the closed-loop stability can be maintained throughout RL explorations and the explored PID parameters by the RL agent converge quickly to the optimum.

show abstract

Application of reinforcement learning on self-tuning PID controller for soccer robot multi-agent system

Cited by 13 publications

References 4 publications

General Dynamic Neural Networks for the Adaptive Tuning of an Omni-Directional Drive System for Reactive Swarm Robotics

General Dynamic Neural Networks for the Adaptive Tuning of an Omni-Directional Drive System for Reactive Swarm Robotics

A Q‐Learning‐Based Parameters Adaptive Algorithm for Formation Tracking Control of Multi‐Mobile Robot Systems

Stability-Preserving Automatic Tuning of PID Control with Reinforcement Learning

Contact Info

Product

Resources

About