DAPE V2: Process Attention Score as Feature Map for Length Extrapolation

ACL ARR 2025 February Submission178 Authors

Abstract:

The attention mechanism is a fundamental component of the Transformer model, contributing to interactions among distinct tokens. In general, the attention scores are determined simply by the key-query products. However, this work's occasional trial (combining DAPE and NoPE) of including additional MLPs on attention scores without position encoding indicates that the classical key-query multiplication may limit the performance of Transformers.
In this work, we conceptualize attention as a feature map and apply the convolution operator (for neighboring attention scores across different heads) to mimic the processing methods in computer vision. Specifically, \textbf{the main contribution of this paper is identifying and interpreting the Transformer length extrapolation problem as a result of the limited expressiveness of the naive query and key dot product, and we successfully translate the length extrapolation issue into a well-understood feature map processing problem,} which is called Convolutional Data-Adaptive Position Encoding (CDAPE).
The novel insight, which can be adapted to various attention-related models, reveals that the current Transformer architecture has the potential for further evolution. Extensive experiments demonstrate that treating attention as a feature map and applying convolution as a processing method significantly enhances Transformer performance.

Paper Type: Long

Research Area: Language Modeling

Research Area Keywords: Transformers, data-adaptive positional encoding, long context, length generalization

Contribution Types: Model analysis & interpretability

Languages Studied: English

Submission Number: 178