Layer-wise Positional Bias in Short-Context Language Modeling

ACL ARR 2026 January Submission9359 Authors

Keywords: language models, positional bias, model interpretability, layer analysis, attribution methods, architectural properties, transformer models

Abstract: Language models often show a preference for using information from specific positions in the input regardless of semantic relevance. While positional bias has been studied in various contexts—from attention sinks to task performance degradation in long-context settings—prior work has not established how these biases evolve across individual layers and input positions, or how they vary independent of task complexity. We introduce an attribution-based framework to analyze positional effects in short-context language modeling. Using layer conductance with a sliding-window approach, we quantify how each layer distributes importance across input positions, yielding layer-wise positional importance profiles. We find that these profiles are architecture-specific, stable across inputs, and invariant to lexical scrambling. Characterizing these profiles, we find prominent recency bias that increases with depth and subtle primacy bias that diminishes through model depth. Beyond positional structure, we also show that early layers preferentially weight content words over function words across all positions, while later layers lose this word-type differentiation.

Paper Type: Long

Research Area: Interpretability and Analysis of Models for NLP

Research Area Keywords: feature attribution;

Contribution Types: Model analysis & interpretability

Languages Studied: English

Submission Number: 9359