Generalizing Dynamics Modeling Easier from Representation Perspective
Keywords: Dynamics Modeling, Ordinary Differential Equations, Pre-trained Language Models
Abstract: Learning system dynamics from observations is a critical problem in many applications over various real-world complex systems, e.g., climate, ecology, and fluid systems. Recently, the neural-based dynamics modeling method has become the prevalent solution, where its basic idea is to embed the original states of objects into a latent space before learning the dynamics using neural-based methods such as neural Ordinary Differential Equations (ODE). Given observations from different complex systems, the existing dynamics modeling methods offer a specific model for each observation, resulting in poor generalization. Inspired by the great success of pre-trained models, we raise a question: whether we can conduct a generalized Pre-trained Dynamic EncoDER (PDEDER), which, for various complex systems, can embed their original states into a latent space, where the dynamics can be easier captured. To conduct this generalized PDEDER, we collect 153 sets of real-world and synthetic observations from 24 complex systems. Inspired by the success of time series forecasting using Pre-trained Language Models (PLM), we can employ any PLM and further update it over these dynamic observations by tokenization techniques to achieve the generalized PDEDER. Given any future dynamic observation, we can fine-tune PDEDERwith any specific dynamics modeling method. We evaluate PDEDER on 18 dynamic systems by long/short-term forecasting under both in-domain and cross-domain settings and the empirical results indicate the effectiveness of PDEDER.
Primary Area: learning on time series and dynamical systems
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 375
Loading