Deep Reinforcement Learning with Uncertain Data for Real-Time Stormwater System Control and Flood Mitigation

Abstract

Flooding in many areas is becoming more prevalent due to factors such as urbanization and climate change, requiring modernization of stormwater infrastructure. Retrofitting standard passive systems with controllable valves/pumps is promising, but requires real-time control (RTC). One method of automating RTC is reinforcement learning (RL), a general technique for sequential optimization and control in uncertain environments. The notion is that an RL algorithm can use inputs of real-time flood data and rainfall forecasts to learn a policy for controlling the stormwater infrastructure to minimize measures of flooding. In real-world conditions, rainfall forecasts and other state information are subject to noise and uncertainty. To account for these characteristics of the problem data, we implemented Deep Deterministic Policy Gradient (DDPG), an RL algorithm that is distinguished by its capability to handle noise in the input data. DDPG implementations were trained and tested against a passive flood control policy. Three primary cases were studied: (i) perfect data, (ii) imperfect rainfall forecasts, and (iii) imperfect water level and forecast data. Rainfall episodes (100) that caused flooding in the passive system were selected from 10 years of observations in Norfolk, Virginia, USA; 85 randomly selected episodes were used for training and the remaining 15 unseen episodes served as test cases. Compared to the passive system, all RL implementations reduced flooding volume by 70.5% on average, and performed within a range of 5%. This suggests that DDPG is robust to noisy input data, which is essential knowledge to advance the real-world applicability of RL for stormwater RTC.

Towards the long term implementation of real time control of combined sewer systems: a review of performance and influencing factors

“…Darsono & Labadie 2007) can be particularly vulnerable to these temporary changes if not used in the training data, with decreased efficacy as a result (Saliba et al 2020). The black-box nature of reinforcement agents can make it harder for posterior analysis of why decision were made and addition of safety control can be harder (Bowes et al, 2020). Methods should be developed to consider these factors in data-driven techniques as well.…”

A systematic review of municipal smart water for climate adaptation and mitigation

“…Similarly, Mullapudi et al (2020) develop an algorithm trains a reinforcement learning agent to control RTC valves in a distributed stormwater system. Saliba et al (2020) examine retrofitting passive stormwater systems with RTCs using a reinforcement learning algorithm, Deep Deterministic Policy Gradient, which is capable of handling noisy input data. Edmondson et al (2018) prototype a smart sewer asset management model to monitor and evaluate real-time performance of the system and predict flooding.…”

“…There are 17 studies that focus on smart, municipal stormwater management. The bulk of this research addresses dynamic control of stormwater flows through RTCs from the watershed to the city (Bowes et al, 2020;Ibrahim, 2020;Joseph-Duran et al, 2015;Maiolo et al, 2020;Mullapudi et al, 2018Mullapudi et al, , 2020Sadler et al, 2020;Saliba et al, 2020;Shishegar et al, 2021). Related to this work, there is also some research around monitoring and optimizing wastewater systems to prevent combined sewer overflows (Edmondson et al, 2018;Lund et al, 2019;Zhang et al, 2018).…”

Flooding and Overflow Mitigation Using Deep Reinforcement Learning Based on Koopman Operator of Urban Drainage Systems

“…In previous studies, the data of the experimental trials and feedback used for the RL training were provided by multiple interactions between RL agent and storm water management model (SWMM) (Mullapudi et al., 2020; Saliba et al., 2020). This comes with a heavy computational burden associated with model simulations (Mullapudi et al., 2020).…”

“…This comes with a heavy computational burden associated with model simulations (Mullapudi et al., 2020). In addition, the inputs of the RL agent were variables relevant to control, which were far less than the data provided by SWMM (Mullapudi et al., 2020; Saliba et al., 2020). This leads to low efficiency of data usage and wastage of computer memory (Mullapudi et al., 2020; Saliba et al., 2020).…”