Transformers for Modeling Physical Systems

Nicholas Geneva, Nicholas Zabaras

Keywords: Scientific Machine Learning, Deep Learning, Physics, Surrogate Modeling, Koopman, Transformers, Attention

Abstract: Transformers are widely used in neural language processing due to their ability to model longer-term dependencies in text. Although these models achieve state-of-the-art performance for many language related tasks, their applicability outside of the neural language processing field has been minimal. In this work, we propose the use of transformer models for the prediction of dynamical systems representative of physical phenomena. The use of Koopman based embeddings provide a unique and powerful method for projecting any dynamical system into a vector representation which can then be predicted by a transformer model. The proposed model is able to accurately predict various dynamical systems and outperform classical methods that are commonly used in the scientific machine learning literature.

One-sentence Summary: Using Koopman embeddings, transformer models can accurately predict physical dynamics.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2010.03957/code)

Reviewed Version (pdf): https://openreview.net/references/pdf?id=ooaUVEysN