Relative Position Biases for Transformer PINNs

Fedor Buzaev, Andrei Ermakov, Mariia Ivanova, Fedor Ratnikov, Denis Derkach, Ilya Makarov

Keywords: PINN, ALiBi, RoPE

TL;DR: We add positional bias into transformer PINNs

Abstract:

Transformer-based physics-informed neural networks (PINNs) have recently improved
PDE solving by modeling spatiotemporal interactions with self-attention,
yet most variants still rely on absolute coordinate embeddings. We ask whether
relative positional structure – a key inductive bias in modern Transformers can
be made coordinate-aware and yield an accuracy boost with minimal overhead
for PDE PINNs. Building on a PINNsFormer-style decoder-only baseline (SPformer),
we introduce two drop-in modifications: (i) a spatial-distance ALiBi
attention bias and (ii) spatial RoPE applied to attention queries/keys using normalized
physical coordinates. Across multiple PDE benchmarks and random seeds,
both relative schemes consistently improve over the baseline, with spatial RoPE
providing the strongest gains in our experiments. Our results suggest that injecting
coordinate-relative structure into attention is a simple and effective upgrade
for Transformer PINNs.

Journal Opt In: Yes, I want to participate in the IOP focus collection submission

Submission Number: 137