Spectraformer: A Unified Random Feature Framework for Transformer

Duke Nguyen, Aditya Joshi, Flora D. Salim

Keywords: linearized attention, transformer, efficient transformer, kernel, random features

TL;DR: Spectraformer is a generic random feature framework for kernelized attention, that systematically compares past works with new alternatives. Hence, it identifies a novel combination of algorithms outperforming existing random feature transformers.

Abstract: Linearization of attention using various kernel approximation and kernel learning techniques has shown promise. Past methods use a subset of combinations of component functions and weight matrices within the random features paradigm. We identify the need for a systematic comparison of different combinations of weight matrices and component functions for attention learning in Transformer. In this work, we introduce $\textit{Spectraformer}$, a unified framework for approximating and learning the kernel function in linearized attention of the Transformer. We experiment with broad classes of component functions and weight matrices for three textual tasks in the LRA benchmark. Our findings indicate that different kernels are good at different tasks and that kernel choice is fundamental to performant models. Our code is available at: https://anonymous.4open.science/r/spectraformer-8A97.

Primary Area: learning theory

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 8555