Sink or Spike: Representational Geometry Mediates Generation Quality in LLMs

Keywords: Attention Sinks, Large Language Models, Singular Value Decomposition, Representational Geometry

TL;DR: Perturbing LLM representations using their SVD yields a consistent effect on the generated text as well as the internal attention maps.

Abstract: Large Language Models (LLMs) rely on contextual representations for their input and output tokens in order to operate. Having an understanding of how these representations are organized and how this organization relates to the text the model generates is integral to the ongoing development of safer, more interpretable LLMs. However, even the representations created for a single prompt define a complicated, high-dimensional geometry, making this a difficult task. We take a step towards understanding this complicated relationship by using the singular value decomposition (SVD) of the representations for a single prompt to guide a perturbation procedure. We find two perturbation strategies that regularly cause two different LLMs to produce coherent responses that are irrelevant to the prompt or text that is relevant but repetitive, respectively. Analyzing the attention maps for these generations reveals a statistically significant trend where these two types of generations are associated with a negative or positive spike in the maximum attention value on the attention sink for each of the output tokens.

Anonymization: This submission has been anonymized for double-blind review via the removal of identifying information such as names, affiliations, and identifying URLs.

Style Files: I have used the style files.

Submission Number: 91