Decoder Only Transformer for Physics Informed Neural Networks

ICLR 2026 Conference Submission19675 Authors

Keywords: Physics Informed Neural Network, PINN, Neural Operators

Abstract:

Physics-Informed Neural Networks (PINNs) approximate PDE solutions by embedding physical constraints into training, yet MLP-based backbones often suffer from instability and loss of fidelity on long horizons. Recent sequence models (e.g., Transformers) alleviate some of these issues, but their encoder–decoder design adds parameters and memory pressure with limited benefit for autoregressive pseudo-sequences.
We introduce \textbf{DoPformer}, a \emph{decoder-only} Transformer tailored to physics-informed learning. DoPformer consumes short spatio–temporal pseudo-sequences, uses multi-head self-attention with WaveAct activations, and applies a sequential physics loss across the window. Removing the encoder and cross-attention yields a lighter model while preserving long-range temporal coupling through self-attention.
To further boost spectral accuracy, we explore two optional modules: (i) a Fourier \emph{neural-operator} branch (\textit{DoPformer+NO}) that improves oscillatory regimes and long-horizon rollouts; and (ii) a compact \emph{KAN}-based feed-forward replacement (\textit{DoPformer+KAN}) that drastically reduces parameters while maintaining strong accuracy.
Across convection, reaction, wave, and 2D Navier–Stokes equations, DoPformer consistently improves PINN accuracy and stability; the NO and KAN variants deliver additional gains depending on stiffness and spectral content. Our numerical results show that on these benchmarks DoPformer attains state-of-the-art accuracy among physics-informed models while using substantially fewer parameters.

Primary Area: neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)

Submission Number: 19675