Deep Reinforcement Learning-Based Smart Joint Control Scheme for On/Off Pumping Systems in Wastewater Treatment Plants

Abstract

In this paper, we propose a deep reinforcement learning (DRL) based predictive control scheme for reducing the energy consumption and energy cost of pumping systems in wastewater treatment plants (WWTP), in which the pumps are operated in a binary mode, using on/off signals. As global energy consumption increases, the efficient operation of energy-intensive facilities has also become important. A WWTP in Busan, Republic of Korea is used as the target of this study. This WWTP is a large energyconsuming facility, and the pumping station accounts for a significant portion of the energy consumption of the WWTP. The framework of the proposed scheme consists of a deep neural network (DNN) model for forecasting wastewater inflow and a DRL agent for controlling the on/off signals of the pumping system, where proximal policy optimization (PPO) and deep Q-neural network (DQN) are employed as the DRL agents. To implement smart control with DRL, a reward function is designed to consider the energy consumption amount and electricity price information. In particular, new features and penalty factors for pump switching, which are essential for preventing pump wear, are also considered. The performance of our designed DRL agents is compared with those of WWTP experts and conventional approaches such as scheduling method and model predictive control (MPC), in which integer linear programming (ILP) optimization is employed. Results show that the designed agents outperform the other approaches in terms of compliance with operating rules and reducing energy costs.

Novel Intelligent Control Framework for WWTP Optimization To Achieve Stable and Sustainable Operation

“…According to Table S2, the variation ranges of AR and Q ER were 10,000–25,000 and 1500–2000 m 3 /h, respectively. Considering the actual equipment precision and model training cost, their intervals were set as 1000 and 100 m 3 /h, respectively. , In this case, we obtained an action space of 96(16 × 6) AR- Q ER pairs for the agent selection.…”

“…In terms of reward setting, the optimization goal is to maximize energy efficiency while ensuring effluent safety of the WWTP. ,, First, we chose TN to represent effluent quality for simplicity and interpretability of the results, which is the key indicator to measure the effect of the NRP. When effluent TN is lower than pre-set limit 9 mg/L (60% of the discharge limit 15 mg/L), the risk of nonstandard discharge is relatively low and TN is not included in the penalty item.…”

“…When effluent TN is lower than pre-set limit 9 mg/L (60% of the discharge limit 15 mg/L), the risk of nonstandard discharge is relatively low and TN is not included in the penalty item. Otherwise, TN was added to the penalty items and adequate attention would be given to TN when choosing strategies . Regarding the energy consumption, we added the value of the control parameters (AR and Q ER ) to the penalty items.…”

“…By interacting with the “environment” (i.e., WWTPs in this study), RL is reported to be able to intelligently find the optimal control parameters to achieve certain goals. For instance, RL models have been used to adjust parameters such as pH and ammonia (NH 4 + -N) concentration in biochemical tank, and the results indicate that they can effectively achieve presetting goals such as effluent safety or cost reduction. , These works are instructive for process optimization at WWTPs, but the results may be relatively simplified without considering the complexity and multi-disturbance of the actual process because most of studies were conducted with synthetic data from activated sludge models . In addition, complex processes such as NRP have not yet been studied from the perspective of full-scale and multi-objective optimization.…”

The Harm and Countermeasures of Automobile Exhaust Pollution Based on Deep Reinforcement Learning

“…(2) Increase the market share of clean energy vehicles From the perspective of the car itself, the fundamental way to reduce exhaust emissions is to develop clean energy, develop clean energy vehicles, increase the market share of clean energy vehicles in the Beijing-Tianjin-Hebei region, and reduce the use of high-pollution and high-emission vehicles in the region ratio [18].…”