Can transformers truly understand dynamical systems?

Zheng-Meng Zhai, Ying-Cheng Lai

Keywords: Dynamical system; Critical transition; Transformer; Reservoir computing

Abstract: Transformer architectures have recently surged as promising solutions for nonlinear dynamical systems, often proposed as foundation models capable of zero-shot dynamics reconstruction and forecasting. Despite this success, it remains unclear whether they can truly serve as reliable digital twins of dynamical systems, i.e., whether they capture the underlying physics in distinct parameter regimes. In nonlinear dynamics, reservoir computing (RC) has already demonstrated broad success, as it is intrinsically a dynamical system capable of capturing not only the dynamical climate of the target system but more importantly, how the climate changes with parameter. Transformers, in contrast, rely on permutation-invariant attention mechanisms, which can limit their ability to capture how temporal structure changes with parameter. To address this issue, we take predicting catastrophic collapse, which occurs when bifurcation parameters cross critical thresholds, as a benchmark task. Models are trained on trajectories in normal parameter regimes and then tested on parameters in an unseen regime with system collapse. Our results show that Transformers, across configurations, consistently fail to capture collapse, while RC reliably predicts the transitions. This surprising finding raises questions about the generalization ability of Transformers to dynamical systems, a topic warranting future research.

Primary Area: applications to physical sciences (physics, chemistry, biology, etc.)

Submission Number: 9664