Beyond Position: the emergence of wavelet-like properties in Transformers
Abstract: This paper studies how transformer models develop robust wavelet-like properties that effectively compensate for the theoretical limitations of Rotary Position Embeddings (RoPE), providing insights into how these networks process sequential information across different scales. Through theoretical analysis and empirical validation across models ranging from 1B to 12B parameters, we show that attention heads naturally evolve to implement multi-resolution processing analogous to wavelet transforms. Our analysis establishes that attention heads consistently organize into complementary frequency bands with systematic power distribution patterns, and these wavelet-like characteristics become more pronounced in larger models. We provide mathematical analysis showing how these properties align with optimal solutions to the fundamental uncertainty principle between positional precision and frequency resolution. Our findings suggest that the effectiveness of modern transformer architectures stems significantly from their development of optimal multi-resolution decompositions that naturally address the theoretical constraints of position encoding.
Paper Type: Long
Research Area: Interpretability and Analysis of Models for NLP
Research Area Keywords: Positional encoding, attention, wavelet, LLMs, Transformer
Contribution Types: Model analysis & interpretability, Theory
Languages Studied: English
Submission Number: 469
Loading