A Differential and Pointwise Control Approach to Reinforcement Learning

Minh Phuong Nguyen, Chandrajit L. Bajaj

Keywords: Hamiltonian Dynamics, Policy Optimization, Pointwise Convergence, Continuous-Time Control, Physics-Informed Learning, Scientific Computing

TL;DR: We propose Differential RL, a physics-informed framework that reformulates RL as a differential control problem. Its algorithm, dfPO, achieves pointwise convergence and outperforms standard RL in low-data scientific computing tasks.

Abstract: Reinforcement learning (RL) in continuous state-action spaces remains challenging in scientific computing due to poor sample efficiency and lack of pathwise physical consistency. We introduce Differential Reinforcement Learning (Differential RL), a novel framework that reformulates RL from a continuous-time control perspective via a differential dual formulation. This induces a Hamiltonian structure that embeds physics priors and ensures consistent trajectories without requiring explicit constraints. To implement Differential RL, we develop Differential Policy Optimization (dfPO), a pointwise, stage-wise algorithm that refines local movement operators along the trajectory for improved sample efficiency and dynamic alignment. We establish pointwise convergence guarantees, a property not available in standard RL, and derive a competitive theoretical regret bound of $\mathcal{O}(K^{5/6})$. Empirically, dfPO outperforms standard RL baselines on representative scientific computing tasks, including surface modeling, grid control, and molecular dynamics, under low-data and physics-constrained conditions.

Primary Area: Theory (e.g., control theory, learning theory, algorithmic game theory)

Submission Number: 28010