RL2Grid: Benchmarking Reinforcement Learning in Power Grid Operations

Enrico Marchesini, Benjamin Donnot, Constance Crozier, Ian Dytham, Christian Merz, Lars Schewe, Nico Westerbeck, Cathy Wu, Antoine Marot, Priya L. Donti

Keywords: Reinforcement Learning, Power Grids, Benchmark

TL;DR: We present RL2Grid, a benchmark representing realistic power grid operations aimed at fostering the maturity of RL methods, and progress in power grid operations.

Abstract: Reinforcement learning (RL) has the potential to transform power grid operations by providing adaptive, scalable controllers essential for grid decarbonization and resilience. However, despite their promise, today's RL methods struggle to deal with complex dynamics, aleatoric uncertainty, long-horizon goals, and hard physical constraints, limiting their application in challenging real-world problems. This paper presents RL2Grid, a benchmark developed in collaboration with European power system operators to accelerate progress in grid control and foster the maturity of RL. Our work builds upon Grid2Op, a power grid simulation framework developed by RTE France, to provide standardized tasks, state and action spaces, and rewards within a common interface, presenting a common basis for monitoring and promoting progress. Additionally, we formalize heuristic-guided transitions and safety constraints derived from human operator knowledge and safe practices to reflect grid requirements. We evaluate and compare popular RL baselines across the increasingly complex settings represented within RL2Grid, establishing reference performance metrics and offering insights into the effectiveness of different approaches. Our findings indicate that power grids present substantial challenges for modern RL, underscoring the need for novel methods capable of dealing with complex real-world physical systems.

Journal Edition Interest: No

Submission Number: 29