A Large-Scale Analysis on Methodological Choices in Deep Reinforcement Learning

Ezgi Korkmaz

Keywords: scientific analysis, methodological choices

Abstract: Deep reinforcement learning research has been the center of remarkable scientific progress for the past decade. From winning one of the most challenging games to algorithmic advancements that allowed solving problems without even explicitly knowing the rules of the task at hand reinforcement learning research progress has been the epicenter of many breakthrough ideas. In this paper, we analyze the methodological issues in deep reinforcement learning. We introduce the theoretical foundations of the underlying causes outlining that the asymptotic performance of deep reinforcement learning algorithms does not have a monotone relationship to the performance in the regimes where data becomes scarce. The extensive large-scale empirical analysis provided in our paper discovers that a major line of deep reinforcement learning research under the canonical methodological choices resulted in suboptimal conclusions.

Primary Area: reinforcement learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 10145