Dynamics of learning when learning dynamics using neural networks

ICLR 2026 Conference Submission22061 Authors

Keywords: neural network, dynamical systems, learning dynamics

Abstract: When neural networks are trained from data to model the dynamics of physical systems, they encounter a persistent challenge: the long-time dynamics they produce are often unphysical or unstable. We analyze the origin of such unphysical instabilities when learning linear dynamical systems, focusing on the learning dynamics of gradient descent. We make several analytical findings, which empirical observations suggest extend to nonlinear dynamical systems. First, the rate of convergence of the learning dynamics of gradient descent is uneven and depends on the distribution of energy in the data. As a special case, in directions in which the data have no energy, the true dynamics of the physical system cannot be learned. High dimensionality also inhibits learning. Second, in the unlearnable directions, the model dynamics produced by the neural network depend on the weight initialization, and common weight initialization schemes can produce unstable model dynamics. Third, injecting synthetic noise into the data adds damping to the learning dynamics and can stabilize the learned model dynamics, though doing so undesirably biases the learned model dynamics. For each contributor to unphysical instability, we suggest mitigative strategies. We also highlight important differences between learning discrete-time and continuous-time dynamics, and discuss extensions to nonlinear systems.

Primary Area: learning on time series and dynamical systems

Submission Number: 22061