DRL-EPANET: Deep reinforcement learning for optimal control at scale in Water Distribution Systems

Anas Belfadil, David Modesto, Jose Antonio Martin H.

Keywords: Reinforcement Learning, Epanet, Water Distribution Systems, Optimal Control, Large Action Spaces

TL;DR: Using reinforcement learning for optimal control at scale in Water Distribution Systems

Abstract: Deep Reinforcement Learning has undergone a revolution in recent years, enabling researchers to tackle a variety of previously inaccessible sequential decision problems. Nevertheless, this method is not widely employed in Water Distribution Systems. In this paper, we demonstrate that DRL can be coupled with the popular hydraulic simulator Epanet and that DRL-Epanet can be applied to a variety of difficult WDS problems. As an example, we use it for pressure control in WDS. We show that DRL-Epanet is scalable to massive action spaces and demonstrate its effectiveness on a problem with more than one million possible actions at each time step. We also demonstrate that it can deal with uncertainties such as stochastic demands, contamination, and other risks; for instance, we address the problem of pressure control in the presence of random pipe breaks. We show that the BDQ algorithm is capable of learning in this context, and we enhance it with an algorithmic modification, BDQF (BDQ with Fixed actions), that achieves better rewards, especially when non-fixed actions are sparse. Finally, we argue that DRL-Epanet can be used for real-time control in smart WDS, which is an advantage over existing methods.