Intriguing Properties of Deep Neural Policy Manifold: Intrinsic Correlation and Deep Neural Policy Curvature

ICLR 2026 Conference Submission19769 Authors

Keywords: deep neural policy curvature, properties of deep neural policy manifold

Abstract: The progress in deep reinforcement learning research has allowed us to construct policies that can solve complex problems in high-dimensional MDPs leading to the creation of AI agents that can strategize and reason solely by interacting with an environment without any supervision. Yet, the knowledge we have on the underlying structure of the deep neural policy manifold is limited. In this paper, we discover that there is a strong correlation between the advantage function and the gradient of the loss targeting directions of instability. By leveraging this intrinsic correlation, we propose a novel algorithm that can diagnose deep neural policy decision volatilities when their environment contains instabilities. We provide theoretical foundations for this intrinsic correlation, and we conduct extensive empirical analysis in the Arcade Learning Environment with high-dimensional observations. From algorithmic and architectural changes to natural distributional shifts and worst-case perturbations, our proposed method can identify and diagnose the differences by leveraging the intrinsic correlation. Our analysis reveals foundational properties of the deep neural policies trained in high-dimensional MDPs, and our work, while laying the groundwork for reliability, is further a fundamental step towards constructing stable and generalizable policies.

Primary Area: reinforcement learning

Submission Number: 19769