Aggregation–Decomposition-Based Multi-Agent Reinforcement Learning for Multi-Reservoir Operations Optimization

Abstract

Stochastic dynamic programming (SDP) is a widely-used method for reservoir operations optimization under uncertainty but suffers from the dual curses of dimensionality and modeling. Reinforcement learning (RL), a simulation-based stochastic optimization approach, can nullify the curse of modeling that arises from the need for calculating a very large transition probability matrix. RL mitigates the curse of the dimensionality problem, but cannot solve it completely as it remains computationally intensive in complex multi-reservoir systems. This paper presents a multi-agent RL approach combined with an aggregation/decomposition (AD-RL) method for reducing the curse of dimensionality in multi-reservoir operation optimization problems. In this model, each reservoir is individually managed by a specific operator (agent) while co-operating with other agents systematically on finding a near-optimal operating policy for the whole system. Each agent makes a decision (release) based on its current state and the feedback it receives from the states of all upstream and downstream reservoirs. The method, along with an efficient artificial neural network-based robust procedure for the task of tuning Q-learning parameters, has been applied to a real-world five-reservoir problem, i.e., the Parambikulam–Aliyar Project (PAP) in India. We demonstrate that the proposed AD-RL approach helps to derive operating policies that are better than or comparable with the policies obtained by other stochastic optimization methods with less computational burden.

Application of the Systems Approach to the Management of Complex Water Systems

“…Stojkovic and Simonovic [16], Hooshyar et al [17], and Aivazidou and Tsolakis [18] address various issues in managing complex water problems. Stojkovic and Simonovic study the impact of climate change on the management of a complex multipurpose water system and present a set of steps of the climate change impact analysis process.…”

Predictive MPC-Based Operation of Urban Drainage Systems Using Input Data-Clustered Artificial Neural Networks Rainfall Forecasting Models

“…Urban drainage systems (UDSs) that convey stormwater away from cities can also be managed with nonstructural measures, such as optimizing operation policies with intelligent control approaches [6][7][8][9][10][11]. Computationally effective stochastic optimization models have been successfully implemented in optimizing reservoir system operations, e.g., [12,13], which are yet to be applied in UDSs. In this area of work, pre-emptive prediction of floods using enhanced rainfall forecasting algorithms can afford us additional possibilities to optimally employ flood control devices.…”

Evolutionary algorithm-based multiobjective reservoir operation policy optimisation under uncertainty

“…However, SDP has been referred to as having three curses (Giuliani et al 2016, Giuliani et al 2021: (I) multiple objectives-an inability to explicitly account for multiobjective tradeoffs, so a weighted sum method is often used (Soleimani et al 2016, Ortiz-Partida et al 2019, Celeste et al 2021; (ii) dimensionality-discretisation of the system state significantly increases computational cost when a large system is considered (Sahu, Mclaughlin 2018, Dobson et al 2019, Hooshyar et al 2020; and (iii) modelling-all variables need to be described in the simulation model, thus specific model and problem formulations are needed (Mortazavi et al 2012, Soleimani et al 2016, Sahu, Mclaughlin 2018, Dobson et al 2019, Ortiz-Partida et al 2019, Hooshyar et al 2020, Celeste et al 2021, which restrict its real-world applications.…”