Physics-Informed Neural Networks with Fourier Features and Attention-Driven Decoding

Rohan Arni, Carlos Blanco

Additional Submission Instructions: For the camera-ready version, please include the author names and affiliations, funding disclosures, and acknowledgements.

Track: Track 1: Original Research/Position/Education/Attention Track

Keywords: pinns, physics, differential equations, fourier features, neural solvers, transformers, self-attention

TL;DR: The Spectral PINNsformer is a compact, decoder-only Transformer PINN that uses Fourier features to mitigate spectral bias and self-attention to capture spatiotemporal correlations, outperforming PINNs and encoder-decoder PINNsFormers on benchmarks.

Abstract: Physics-Informed Neural Networks (PINNs) are a useful framework for approximating partial differential equation solutions using deep learning methods. In this paper, we propose a principled redesign of the PINNsformer, a Transformer-based PINN architecture. We present the Spectral PINNSformer (S-Pformer), a refinement of encoder-decoder PINNSformers that addresses two key issues; 1. the redundancy (i.e. increased parameter count) of the encoder, and 2. the mitigation of spectral bias. We find that the encoder is unnecessary for capturing spatiotemporal correlations when relying solely on self-attention, thereby reducing parameter count. Further, we integrate Fourier feature embeddings to explicitly mitigate spectral bias, enabling adaptive encoding of multiscale behaviors in the frequency domain. Our model outperforms encoder-decoder PINNSformer architectures across all benchmarks, achieving or outperforming MLP performance while reducing parameter count significantly.

Submission Number: 88