Model-free control based on reinforcement learning for a wastewater treatment problem

Syafiie, S.; Tadeo, Fernando; Martínez, Ernesto; Álvarez, Teresa

doi:10.1016/j.asoc.2009.10.018

Cited by 52 publications

(17 citation statements)

References 25 publications

(48 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A real-world implementation of the RL control method is made possible by applying data-mining algorithms to construct models from data and generate synthetic data for pre-training the RL algorithm, without the need to physically interact with the system. This also represents an original contribution compared to other control problems like [25,24]. In order to address this problem and in a different context for building heating systems, an interesting and different approach is proposed by Costanzo et al [35], and inspired by the work in [36], which consists in using a neural network to create virtual data that are used together with experimental data to approximate the Q-function (state-action value function) of fitted Q-learning and reduce the amount of system interaction required to learn a "good" policy.…”

Section: Related Work and Contributionsmentioning

confidence: 94%

“…Syafiie et al proposed a model-free control approach for advanced oxidation processes (or Fenton process), since according to the authors it is extremely difficult do develop a precise mathematical model and the system is subject to several uncertainties and time-evolving characteristics [24]. As an alternative to proportional-integral-derivative (PID) controllers, Hernández-del-Olmo et al explored RL for oxygen control in the N-ammonia removal process, whose main objective was to minimize WTTP operational costs (including energy costs) [25].…”

Section: Related Work and Contributionsmentioning

confidence: 99%

See 1 more Smart Citation

Data-driven predictive energy optimization in a wastewater pumping station

et al. 2019

View full text Add to dashboard Cite

Urban wastewater sector is being pushed to optimize processes in order to reduce energy consumption without compromising its quality standards. Energy costs can represent a significant share of the global operational costs (between 50% and 60%) in an intensive energy consumer. Pumping is the largest consumer of electrical energy in a wastewater treatment plant. Thus, the optimal control of pump units can help the utilities to decrease operational costs. This work describes an innovative predictive control policy for wastewater variable-frequency pumps that minimize electrical energy consumption, considering uncertainty forecasts for wastewater intake rate and information collected by sensors accessible through the Supervisory Control and Data Acquisition system. The proposed control method combines statistical learning (regression and predictive models) and deep reinforcement learning (Proximal Policy Optimization). The following main original contributions are produced: i) model-free and data-driven predictive control;ii) control philosophy focused on operating the tank with a variable wastewater setpoint level; iii) use of supervised learning to generate synthetic data for pre-training the reinforcement learning policy, without the need to physically interact with the system.The results for a real case-study during 90 days show a 16.7% decrease in electrical energy consumption while still achieving a 97% reduction in the number of alarms (tank level above 7.2 meters) when compared with the current operating scenario (operating with a fixed set-point level). The numerical analysis showed that the proposed data-driven method is able to explore the trade-off between number of alarms and consumption minimization, offering different options to decision-makers.

show abstract

Section: Related Work and Contributionsmentioning

confidence: 94%

Section: Related Work and Contributionsmentioning

confidence: 99%

Data-driven predictive energy optimization in a wastewater pumping station

et al. 2019

View full text Add to dashboard Cite

show abstract

“…A biosensor based on gas phase monitoring was designed to measure the hydrogen peroxide concentrations in Modrzejewska, Guwy, Dinsdale, and Hawkes (2007). It is worth to mention also a specific continuous monitoring of hydrogen peroxide formed in situ was proposed by Oh, Kim, Kang, Oh, and Kang (2005), or a model-free learning control to deal with the chemical characteristic variations with time is given by Syafiie, Tadeo, Martinez, and Alvarez (2011).…”

Section: Introductionmentioning

confidence: 99%

AH∞control for optimizing the advanced oxidation processes—Case of a catalytic ozonation reactor

Abouzlam

Ouvrard

Mehdi

et al. 2015

Control Engineering Practice

View full text Add to dashboard Cite

“…For a class of unknown general nonlinear systems such as WWTP, adaptive dynamic programming (ADP)-also known as reinforcement learning, has attracted more and more attention of experts recently [8][9][10][11][12][13][14][15][16][17][18][19][20][21][22][23][24]. The core of ADP is the approximation of Bellman's equation or the Hamilton-Jacobi-Bellman (HJB) equation by learning [10,12,20].…”

Section: Introductionmentioning

confidence: 99%

Heuristic Dynamic Programming Using Echo State Network For Multivariable Tracking Control Of Wastewater Treatment Process

Bo¹,

Qiao²

2014

Asian Journal of Control

View full text Add to dashboard Cite

This paper proposes a heuristic dynamic programming (HDP) scheme to simultaneously control the dissolved oxygen concentration and the nitrate level in wastewater treatment processes (WWTP). Unlike traditional HDP schemes, the optimal control values are calculated in an analytical way by the proposed HDP controller. It can reduce the learning burden of the HDP controller to a great extent. The system model and the evaluation index J are approximated by two echo state networks (ESNs). Gradient-based learning algorithms are employed to train both ESNs online, and the convergence of the training algorithm is investigated based on Lyapunov theory. The performance of the proposed ESN-based HDP (E-HDP) controller is tested and evaluated on a WWTP benchmark. Experimental results demonstrate that the proposed approach can achieve effective performance.

show abstract

Model-free control based on reinforcement learning for a wastewater treatment problem

Cited by 52 publications

References 25 publications

Data-driven predictive energy optimization in a wastewater pumping station

Data-driven predictive energy optimization in a wastewater pumping station

AH∞control for optimizing the advanced oxidation processes—Case of a catalytic ozonation reactor

Heuristic Dynamic Programming Using Echo State Network For Multivariable Tracking Control Of Wastewater Treatment Process

Contact Info

Product

Resources

About