Meta-reinforcement learning for the tuning of PI controllers: An offline approach

McClement, Daniel G.; Lawrence, Nathan P.; Backström, Johan U.; Loewen, Philip D.; Forbes, Michael G.; Gopaluni, R. Bhushan

doi:10.1016/j.jprocont.2022.08.002

Cited by 17 publications

(13 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This paper extends our previous work which focused only on systems which could be modeled as first order plus time delay and PI controllers [1]. In particular, in Section III we extend our framework to second order systems and PID controllers.…”

Section: Introductionmentioning

confidence: 63%

“…We use PPO because of its sample efficiency and robustness to the choice of model hyperparameters. Full implementation and technical details can be found in [1], [11].…”

Section: Background a Reinforcement Learningmentioning

confidence: 99%

“…systems within the task distribution p(T )) without any online training. Additionally, nearly any system's data can be modified to be "in-distribution" [1]. • The meta-RL agent is a single model that is trained once, offline, meaning one does not need to specify hyperparameters on a task-by-task basis.…”

Section: A Tasks States Actions Costsmentioning

confidence: 99%

“…In addition to the RL state, the critic conditions its estimates of the value function on the true process parameters (K, τ 1 , τ 2 , and θ), as well as the deep hidden state -the hidden state of the second recurrent layer -of the actor. Knowledge of a task's process dynamics, as well as knowledge of the actor's internal representation of the process dynamics through its hidden state, allows the critic to more accurately estimate the value function, which improves the quality of the surrogate objective function used to train the actor and the training efficiency of the algorithm [1]. Equipping the critic with this information also allows it to operate as a simpler feedforward neural network rather than a recurrent network like the actor.…”

Section: B Rl Agent Structurementioning

confidence: 99%

“…We consider the following sets of SOPTD model parameters in our experiments: K and τ 1 are sampled from the interval [0.25, 1.0]; The ratios τ2 τ1 and θ τ1 are sampled from the interval [0.0, 1.0]. In [1] we demonstrate how training across this range of parameters can be quite versatile in practice using data augmentation to adjust the dynamics of most second order processes into this distribution from the meta-RL agent's perspective. The PPO algorithm is adapted from Open AI's "Spinning Up" implementation and modified to accommodate the inclusion of a recurrent neural network and distribution of control tasks [22].…”

Section: Training Proceduresmentioning

confidence: 99%

See 4 more Smart Citations

Meta-Reinforcement Learning for Adaptive Control of Second Order Systems

McClement

Lawrence

Forbes

et al. 2022

2022 IEEE International Symposium on Advanced Control of Industrial Processes (AdCONIP)

Self Cite

View full text Add to dashboard Cite

Meta-learning is a branch of machine learning which aims to synthesize data from a distribution of related tasks to efficiently solve new ones. In process control, many systems have similar and well-understood dynamics, which suggests it is feasible to create a generalizable controller through meta-learning. In this work, we formulate a meta reinforcement learning (meta-RL) control strategy that takes advantage of known, offline information for training, such as a model structure. The meta-RL agent is trained over a distribution of model parameters, rather than a single model, enabling the agent to automatically adapt to changes in the process dynamics while maintaining performance. A key design element is the ability to leverage model-based information offline during training, while maintaining a model-free policy structure for interacting with new environments. Our previous work has demonstrated how this approach can be applied to the industrially-relevant problem of tuning proportional-integral controllers to control first order processes. In this work, we briefly reintroduce our methodology and demonstrate how it can be extended to proportional-integral-derivative controllers and second order systems.

show abstract

Section: Introductionmentioning

confidence: 63%

“…We use PPO because of its sample efficiency and robustness to the choice of model hyperparameters. Full implementation and technical details can be found in [1], [11].…”

Section: Background a Reinforcement Learningmentioning

confidence: 99%

Section: A Tasks States Actions Costsmentioning

confidence: 99%

Section: B Rl Agent Structurementioning

confidence: 99%

Section: Training Proceduresmentioning

confidence: 99%

See 3 more Smart Citations

Meta-Reinforcement Learning for Adaptive Control of Second Order Systems

McClement

Lawrence

Forbes

et al. 2022

2022 IEEE International Symposium on Advanced Control of Industrial Processes (AdCONIP)

Self Cite

View full text Add to dashboard Cite

show abstract

Data-Driven Automotive Development: Federated Reinforcement Learning for Calibration and Control

Rudolf

Schürmann

Skull³

et al. 2022

Proceedings

View full text Add to dashboard Cite

Robust and performant controllers are essential for industrial applications. However, deriving controller parameters for complex and nonlinear systems is challenging and time-consuming. To facilitate automatic controller parametrization, this work presents a novel approach using deep reinforcement learning (DRL) with N-dimensional Bspline geometries (BSGs). We focus on the control of parametervariant systems, a class of systems with complex behavior which depends on the operating conditions. For this system class, gain-scheduling control structures are widely used in applications across industries due to well-known design principles. Facilitating the expensive controller parametrization task regarding these control structures, we deploy an DRL agent. Based on control system observations, the agent autonomously decides how to adapt the controller parameters. We make the adaptation process more efficient by introducing BSGs to map the controller parameters which may depend on numerous operating conditions. To preprocess time-series data and extract a fixed-length feature vector, we use a long shortterm memory (LSTM) neural networks. Furthermore, this work contributes actor regularizations that are relevant to real-world environments which differ from training. Accordingly, we apply dropout layer normalization to the actor and critic networks of the truncated quantile critic (TQC) algorithm. To show our approach's working principle and effectiveness, we train and evaluate the DRL agent on the parametrization task of an industrial control structure with parameter lookup tables.

show abstract

A reinforcement learning-based transformed inverse model strategy for nonlinear process control

Dutta,

Upreti

2023

Computers & Chemical Engineering

View full text Add to dashboard Cite

Meta-reinforcement learning for the tuning of PI controllers: An offline approach

Cited by 17 publications

References 27 publications

Meta-Reinforcement Learning for Adaptive Control of Second Order Systems

Meta-Reinforcement Learning for Adaptive Control of Second Order Systems

Data-Driven Automotive Development: Federated Reinforcement Learning for Calibration and Control

A reinforcement learning-based transformed inverse model strategy for nonlinear process control

Contact Info

Product

Resources

About