Correlation Dimension of Autoregressive Large Language Models

Xin Du, Kumiko Tanaka-Ishii

Keywords: correlation dimension, fractal dimension, large language models, self-similarity, complexity, degeneration, hallucination, LLM evaluation

TL;DR: We propose correlation dimension as a practical, model-agnostic metric that captures structural complexity and detects degeneration in large language model outputs beyond what perplexity reveals.

Abstract:

Large language models (LLMs) have achieved remarkable progress in natural
language generation, yet they continue to display puzzling behaviors—such as
repetition and incoherence—even when exhibiting low perplexity. This
highlights a key limitation of conventional evaluation metrics, which
emphasize local prediction accuracy while overlooking long-range structural
complexity. We introduce correlation dimension, a fractal-geometric measure
of self-similarity, to quantify the epistemological complexity of text as
perceived by a language model. This measure captures the hierarchical
recurrence structure of language, bridging local and global properties in a
unified framework. Through extensive experiments, we show that correlation
dimension (1) reveals three distinct phases during pretraining, (2) reflects
context-dependent complexity, (3) indicates a model's tendency toward
hallucination, and (4) reliably detects multiple forms of degeneration in
generated text. The method is computationally efficient, robust to model
quantization (down to 4-bit precision), broadly applicable across
autoregressive architectures (e.g., Transformer and Mamba), and provides
fresh insight into the generative dynamics of LLMs.

Primary Area: Deep learning (e.g., architectures, generative models, optimization for deep networks, foundation models, LLMs)

Submission Number: 24688