Benchmarking Quantum Reinforcement Learning

Nico Meyer, Christian Ufrecht, George Yammine, Georgios Kontes, Christopher Mutschler, Daniel Scherer

TL;DR: We introduce a statistical estimator for benchmarking sample complexity in quantum reinforcement learning.

Abstract: Benchmarking and establishing proper statistical validation metrics for reinforcement learning (RL) remain ongoing challenges, where no consensus has been established yet. The emergence of quantum computing and its potential applications in quantum reinforcement learning (QRL) further complicate benchmarking efforts. To enable valid performance comparisons and to streamline current research in this area, we propose a novel benchmarking methodology, which is based on a statistical estimator for sample complexity and a definition of statistical outperformance. Furthermore, considering QRL, our methodology casts doubt on some previous claims regarding its superiority. We conducted experiments on a novel benchmarking environment with flexible levels of complexity. While we still identify possible advantages, our findings are more nuanced overall. We discuss the potential limitations of these results and explore their implications for empirical research on quantum advantage in QRL.

Lay Summary:

Quantum computing is an exciting new technology that could revolutionize reinforcement learning (RL) and machine learning in general. At least, this statement is frequently made in the literature. Upon closer inspection, however, claims of quantum reinforcement learning (QRL) superiority are often based on insufficient statistical evaluation. This reflects the ongoing challenges of establishing proper statistical validation metrics for RL.

In our work, we introduce a statistical estimator for benchmarking and establish a statistically robust comparison between RL and corresponding quantum versions, along with a suitable environment. In line with previous work, we find indications of QRL outperforming RL, but our findings are more nuanced. Finally, we contextualize these results in terms of their implications for empirical quantum advantage in QRL.

With our results, we emphasize the importance of robust benchmarking and statistical backing as part of performance evaluation. We hope our suggestions will guide researchers to identify trends in small toy problems amenable to current quantum computers. As quantum hardware evolves and problem sizes grow, these trends might or might not persist, potentially guiding us toward quantum advantage in machine learning.

Link To Code: https://github.com/nicomeyer96/qrl-benchmark

Primary Area: Reinforcement Learning->Everything Else

Keywords: Reinforcement Learning, Quantum Computing, Benchmarking, Sample Complexity, Quantum Machine Learning

Submission Number: 5834