Do Deeper Layers Always Abstract? A Task-Dependent Geometric Analysis of LLM

ACL ARR 2026 January Submission8334 Authors

Keywords: latent geometry, task-dependent geometric evolution, abstract representations

Abstract: The prevailing view holds that deeper layers of large language models (LLMs) naturally yield more abstract representations. We revisit this assumption by analyzing the latent geometry of LLaMA-2-7B across three semantic tasks: lexical modeling (ML), sentence inference (MI), and abstractive summarization (MG). Using two probe-free diagnostics—effective dimensionality ($d_{\text{eff}}$) and a discrete curvature proxy ($\kappa$)—we trace how representations evolve with depth. Our results show that abstraction is not universal but task-driven: MG exhibits an expand–compress trajectory in $d_{\text{eff}}$ with sharp curvature bursts in upper layers, while MI remains nearly flat and ML shows gradual refinement. Quantitatively, MG’s peak curvature exceeds MI’s by over $30\times$, highlighting nonlinear folding as a hallmark of generative abstraction. These findings challenge depth-centric assumptions, introduce reproducible geometric tools for analyzing LLM representations, and suggest new geometry-aware objectives for controlling abstraction.

Paper Type: Short

Research Area: Interpretability and Analysis of Models for NLP

Research Area Keywords: latent geometry, task-dependent geometric evolution,abstract representations

Contribution Types: Model analysis & interpretability

Languages Studied: English

Submission Number: 8334