Integration of reinforcement learning and model predictive control to optimize semi‐batch bioreactor

Oh, Tae Hoon; Park, Hyun Min; Kim, Jong Woo; Lee, Jong Min

doi:10.1002/aic.17658

Cited by 30 publications

(8 citation statements)

References 52 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The actor networks in SPER-SAC are parameterized DNNs consisting of 4 hidden layers with 256 hidden nodes, therefore the number of nodes in each layer as [12,256,256,2], whose inputs are the vector of process variables listed in table 1 and outputs the set value of the substrate feed flowrate in the form of a Gaussian distribution. In contrast, the number of network nodes in each layer of the critic network is [13,256,256,1], and its input consists of the process variables and the output of the actor network.…”

Section: Training Environmentmentioning

confidence: 99%

“…To address these challenges, reinforcement learning (RL) has been shown to be a potential alternative to the traditional control methods [10][11][12]. First, as a data-driven approach, the agent (analogous to the controller) in RL learns the control action (analogous to the control output) by interacting with the environment directly (analogous to the system/process) at each time step [13].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Batch process control based on reinforcement learning with segmented prioritized experience replay

Xu,

Ma,

Tao

2024

Meas. Sci. Technol.

View full text Add to dashboard Cite

Batch process is difficult to control accurately due to their complex nonlinear dynamics and unstable operating conditions. The traditional methods such as model predictive control, will seriously affect control performance when process model is inaccurate. In contrast, reinforcement learning (RL) provides an viable alternative by interacting directly with the environment to learn optimal strategy. This paper proposes a batch process controller based on the segmented prioritized experience replay soft actor-critic. Soft actor-critic(SAC) combines off-policy updates and maximum entropy reinforcement learning with an actor-critic formulation, which can obtain a more robust control strategy than other RL methods. To improve the efficiency of the experience replay mechanism in tasks with long episodes and multiple phases, a new method of sampling experience called segmented prioritized experience replay (SPER) is designed in SAC. In addition, a novel reward function is set for the SPER-SAC based controller to deal with the sparse reward. Finally, the effectiveness of the SPER-SAC based controller for batch process examples is demonstrated by comparing with the conventional RL-based control methods.

show abstract

Section: Training Environmentmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Batch process control based on reinforcement learning with segmented prioritized experience replay

Xu,

Ma,

Tao

2024

Meas. Sci. Technol.

View full text Add to dashboard Cite

show abstract

“…Model-free reinforcement learning, on the other hand, obtains optimal strategies through real-time interaction between the agent and the environment without requiring precision. For example, Lee et al integrated MPC with the double-deep Q-network algorithm to obtain the optimal substrate feeding strategy in industrial-scale penicillin production, effectively reducing the operating cost of semi-intermittent bioreactors [ 104 ]. Benton et al optimized the feeding process for cyanobacterial-phycocyanin (C-PC) production by the Asynchronous Advantage Actor-Critic (A3C) algorithm with asynchronous learning control and finally increased the product yield by 52.1% [ 105 ].…”

Section: Development Of Modelingmentioning

confidence: 99%

From Spatial-Temporal Multiscale Modeling to Application: Bridging the Valley of Death in Industrial Biotechnology

et al. 2023

View full text Add to dashboard Cite

The Valley of Death confronts industrial biotechnology with a significant challenge to the commercialization of products. Fortunately, with the integration of computation, automation and artificial intelligence (AI) technology, the industrial biotechnology accelerates to cross the Valley of Death. The Fourth Industrial Revolution (Industry 4.0) has spurred advanced development of intelligent biomanufacturing, which has evolved the industrial structures in line with the worldwide trend. To achieve this, intelligent biomanufacturing can be structured into three main parts that comprise digitalization, modeling and intellectualization, with modeling forming a crucial link between the other two components. This paper provides an overview of mechanistic models, data-driven models and their applications in bioprocess development. We provide a detailed elaboration of the hybrid model and its applications in bioprocess engineering, including strain design, process control and optimization, as well as bioreactor scale-up. Finally, the challenges and opportunities of biomanufacturing towards Industry 4.0 are also discussed.

show abstract

“…Deep neural networks (DNNs) have been immensely successful for building autonomous systems for forecasting [1], monitoring [2], fault detection [3] and control [4] to name a few. The performance of DNNs is heavily reliant on the fidelity and quantity of data used for training.…”

Section: Introductionmentioning

confidence: 99%

Uncertainty-aware deep learning for digital twin-driven monitoring: Application to fault detection in power lines

Das¹,

Gjorgiev²,

Sansavini³

2023

Preprint

View full text Add to dashboard Cite

Deep neural networks (DNNs) are often coupled with physics-based models or data-driven surrogate models to perform fault detection and health monitoring of systems in the low data regime. These models serve as digital twins to generate large quantities of data to train DNNs which would otherwise be difficult to obtain from the real-life system. However, such models can exhibit parametric uncertainty that propagates to the generated data. In addition, DNNs exhibit uncertainty in the parameters learnt during training. In such a scenario, the performance of the DNN model will be influenced by the uncertainty in the physics-based model as well as the parameters of the DNN. In this article, we quantify the impact of both these sources of uncertainty on the performance of the DNN. We perform explicit propagation of uncertainty in input data through all layers of the DNN, as well as implicit prediction of output uncertainty to capture the former. Furthermore, we adopt Monte Carlo dropout to capture uncertainty in DNN parameters. We demonstrate the approach for fault detection of power lines with a physics-based model, two types of input data and three different neural network architectures. We compare the performance of such uncertainty-aware probabilistic models with their deterministic counterparts. The results show that the probabilistic models provide important information regarding the confidence of predictions, while also delivering an improvement in performance over deterministic models.

show abstract

Integration of reinforcement learning and model predictive control to optimize semi‐batch bioreactor

Cited by 30 publications

References 52 publications

Batch process control based on reinforcement learning with segmented prioritized experience replay

Batch process control based on reinforcement learning with segmented prioritized experience replay

From Spatial-Temporal Multiscale Modeling to Application: Bridging the Valley of Death in Industrial Biotechnology

Uncertainty-aware deep learning for digital twin-driven monitoring: Application to fault detection in power lines

Contact Info

Product

Resources

About