GRLA: Bridging Softmax and Linear Attention via Gaussian RBF Kernel for Lightweight Image Super-Resolution

ICLR 2026 Conference Submission16242 Authors

Keywords: Lightweight image super-resolution, Softmax Attention, Linear Attention, Gaussian RBF

Abstract: Lightweight image super-resolution (SR) requires effective modeling of long-range dependencies under stringent computational constraints. Although self-attention mechanisms are highly effective for this task, their quadratic computational complexity imposes a prohibitive constraint in lightweight SR applications. Existing linear attention methods reduce complexity to linear but significantly underperform compared to Softmax attention due to their inability to explicitly model the Euclidean distance between query and key vectors. Through mathematical derivation, we demonstrate that the core operation of standard Softmax attention, $\exp({Q}_i^T {K}_j)$, is equivalent to an unnormalized Gaussian Radial Basis Function (GRBF) kernel. Building on this insight, we propose a GRBF-based linear attention mechanism (GRBFLA), which reformulates a distance-aware GRBF kernel that is amenable to Taylor series expansion, enabling linear approximation. This kernel progressively approximates the behavior of standard Softmax attention while maintaining linear complexity. Based on GRBFLA, we develop a lightweight image SR architecture termed GRLA. Experimental results show that for ×4 SR on the Manga109 dataset, GRLA outperforms the representative self-attention model SwinIR-light by 0.57 dB in PSNR while reducing computational cost FLOPs by 11\%. Compared to the state-of-the-art Mamba-based lightweight model MambaIRv2-light, GRLA achieves a 0.25 dB higher PSNR with a 25\% reduction in FLOPs.

Primary Area: applications to computer vision, audio, language, and other modalities

Submission Number: 16242