Stability-Preserving Automatic Tuning of PID Control with Reinforcement Learning

Abstract

PID control has been the dominant control strategy in the process industry due to its simplicity in design and effectiveness in controlling a wide range of processes. However, traditional methods on PID tuning often require extensive domain knowledge and field experience. To address the issue, this work proposes an automatic PID tuning framework based on reinforcement learning (RL), particularly the deterministic policy gradient (DPG) method. Different from existing studies on using RL for PID tuning, in this work, we consider the closed-loop stability throughout the RL-based tuning process. In particular, we propose a novel episodic tuning framework that allows for an episodic closed-loop operation under selected PID parameters where the actor and critic networks are updated once at the end of each episode. To ensure the closed-loop stability during the tuning, we initialize the training with a conservative but stable baseline PID controller and the resultant reward is used as a benchmark score. A supervisor mechanism is used to monitor the running reward (e.g., tracking error) at each step in the episode. As soon as the running reward exceeds the benchmark score, the underlying controller is replaced by the baseline controller as an early correction to prevent instability. Moreover, we use layer normalization to standardize the input to each layer in actor and critic networks to overcome the issue of policy saturation at action bounds, to ensure the convergence to the optimum. The developed methods are validated through setpoint tracking experiments on a second-order plus dead-time system. Simulation results show that with our scheme, the closed-loop stability can be maintained throughout RL explorations and the explored PID parameters by the RL agent converge quickly to the optimum.

Multi-Phase Focused PID Adaptive Tuning with Reinforcement Learning

“…PID parameters can be optimized using two main methods: classical tuning methods and optimization methods [8,9]. Classical methods often rely on heuristic tuning or model construction to address specific engineering challenges [10]. However, these methods, such as the Ziegler-Nichols method [11] and the Cohen-Coon method [12], make assumptions about the underlying model or attempt to estimate the best approximation of the PID parameters [13].…”

“…Ref. [10] proposed an RL-based stabilitypreserving PID adaptive tuning framework that guarantees the controller's stability in the entire closed-loop process and found that this approach reduced the MSE tracking error by more than 10% compared with other methods. However, this method requires a known set of PID parameters as reference values to guide the agent's exploration.…”

“…Yet this will introduce the problem of overly high-dimensional inputs to the RL networks. In another study [10], episodic data are described as a state customized to equal the time step, reducing the input dimension, but still potentially resulting in information redundancy. Our observations show that the early stages of the control process exhibit significant output fluctuations, while the output value stabilizes progressively towards the setpoint in the later stages, known as the tuning characteristic of PID control.…”

A Data-Driven Approach for PIDs Tuning in Energy-intensive Industry: Application to Evaporator in Kraft Pulp Mill

“…Due to simplicity and ease of implementation, proportional-integral-derivative (PID) controllers have been extensively used in industrial processes over the past few decades, which consist of almost 90% of controllers in industrial control loops s (Al-Bargothi et al 2019). They capture the system historical behaviors through the integration part and forecast the future behavior of the system via the differentiation part (Lakhani et al 2021). At the same time, the advancement of modern science and technology improves the complexity of industrial processes, leading to potential problems such as unstable control loops (Shamsuzzoha 2018), which may contribute to the low effectiveness of PID tuning.…”

“…This method does not need too much advanced technical support, but it depends on the personal professional ability, which cannot ensure the accuracy and adaptability of tuning. Apart from manual tuning, PID tuning approaches are categorized into heuristic tuning, rule-based tuning, and model-based tuning (Lakhani et al 2021). Heuristic tuning is based on trial and error according to the prior knowledge on the control processes and its corresponding PID parameters, which is easy to implement but time-consuming.…”